8ec9ff4a12
* admin: add plugin runtime UI page and route wiring * pb: add plugin gRPC contract and generated bindings * admin/plugin: implement worker registry, runtime, monitoring, and config store * admin/dash: wire plugin runtime and expose plugin workflow APIs * command: add flags to enable plugin runtime * admin: rename remaining plugin v2 wording to plugin * admin/plugin: add detectable job type registry helper * admin/plugin: add scheduled detection and dispatch orchestration * admin/plugin: prefetch job type descriptors when workers connect * admin/plugin: add known job type discovery API and UI * admin/plugin: refresh design doc to match current implementation * admin/plugin: enforce per-worker scheduler concurrency limits * admin/plugin: use descriptor runtime defaults for scheduler policy * admin/ui: auto-load first known plugin job type on page open * admin/plugin: bootstrap persisted config from descriptor defaults * admin/plugin: dedupe scheduled proposals by dedupe key * admin/ui: add job type and state filters for plugin monitoring * admin/ui: add per-job-type plugin activity summary * admin/plugin: split descriptor read API from schema refresh * admin/ui: keep plugin summary metrics global while tables are filtered * admin/plugin: retry executor reservation before timing out * admin/plugin: expose scheduler states for monitoring * admin/ui: show per-job-type scheduler states in plugin monitor * pb/plugin: rename protobuf package to plugin * admin/plugin: rename pluginRuntime wiring to plugin * admin/plugin: remove runtime naming from plugin APIs and UI * admin/plugin: rename runtime files to plugin naming * admin/plugin: persist jobs and activities for monitor recovery * admin/plugin: lease one detector worker per job type * admin/ui: show worker load from plugin heartbeats * admin/plugin: skip stale workers for detector and executor picks * plugin/worker: add plugin worker command and stream runtime scaffold * plugin/worker: implement vacuum detect and execute handlers * admin/plugin: document external vacuum plugin worker starter * command: update plugin.worker help to reflect implemented flow * command/admin: drop legacy Plugin V2 label * plugin/worker: validate vacuum job type and respect min interval * plugin/worker: test no-op detect when min interval not elapsed * command/admin: document plugin.worker external process * plugin/worker: advertise configured concurrency in hello * command/plugin.worker: add jobType handler selection * command/plugin.worker: test handler selection by job type * command/plugin.worker: persist worker id in workingDir * admin/plugin: document plugin.worker jobType and workingDir flags * plugin/worker: support cancel request for in-flight work * plugin/worker: test cancel request acknowledgements * command/plugin.worker: document workingDir and jobType behavior * plugin/worker: emit executor activity events for monitor * plugin/worker: test executor activity builder * admin/plugin: send last successful run in detection request * admin/plugin: send cancel request when detect or execute context ends * admin/plugin: document worker cancel request responsibility * admin/handlers: expose plugin scheduler states API in no-auth mode * admin/handlers: test plugin scheduler states route registration * admin/plugin: keep worker id on worker-generated activity records * admin/plugin: test worker id propagation in monitor activities * admin/dash: always initialize plugin service * command/admin: remove plugin enable flags and default to enabled * admin/dash: drop pluginEnabled constructor parameter * admin/plugin UI: stop checking plugin enabled state * admin/plugin: remove docs for plugin enable flags * admin/dash: remove unused plugin enabled check method * admin/dash: fallback to in-memory plugin init when dataDir fails * admin/plugin API: expose worker gRPC port in status * command/plugin.worker: resolve admin gRPC port via plugin status * split plugin UI into overview/configuration/monitoring pages * Update layout_templ.go * add volume_balance plugin worker handler * wire plugin.worker CLI for volume_balance job type * add erasure_coding plugin worker handler * wire plugin.worker CLI for erasure_coding job type * support multi-job handlers in plugin worker runtime * allow plugin.worker jobType as comma-separated list * admin/plugin UI: rename to Workers and simplify config view * plugin worker: queue detection requests instead of capacity reject * Update plugin_worker.go * plugin volume_balance: remove force_move/timeout from worker config UI * plugin erasure_coding: enforce local working dir and cleanup * admin/plugin UI: rename admin settings to job scheduling * admin/plugin UI: persist and robustly render detection results * admin/plugin: record and return detection trace metadata * admin/plugin UI: show detection process and decision trace * plugin: surface detector decision trace as activities * mini: start a plugin worker by default * admin/plugin UI: split monitoring into detection and execution tabs * plugin worker: emit detection decision trace for EC and balance * admin workers UI: split monitoring into detection and execution pages * plugin scheduler: skip proposals for active assigned/running jobs * admin workers UI: add job queue tab * plugin worker: add dummy stress detector and executor job type * admin workers UI: reorder tabs to detection queue execution * admin workers UI: regenerate plugin template * plugin defaults: include dummy stress and add stress tests * plugin dummy stress: rotate detection selections across runs * plugin scheduler: remove cross-run proposal dedupe * plugin queue: track pending scheduled jobs * plugin scheduler: wait for executor capacity before dispatch * plugin scheduler: skip detection when waiting backlog is high * plugin: add disk-backed job detail API and persistence * admin ui: show plugin job detail modal from job id links * plugin: generate unique job ids instead of reusing proposal ids * plugin worker: emit heartbeats on work state changes * plugin registry: round-robin tied executor and detector picks * add temporary EC overnight stress runner * plugin job details: persist and render EC execution plans * ec volume details: color data and parity shard badges * shard labels: keep parity ids numeric and color-only distinction * admin: remove legacy maintenance UI routes and templates * admin: remove dead maintenance endpoint helpers * Update layout_templ.go * remove dummy_stress worker and command support * refactor plugin UI to job-type top tabs and sub-tabs * migrate weed worker command to plugin runtime * remove plugin.worker command and keep worker runtime with metrics * update helm worker args for jobType and execution flags * set plugin scheduling defaults to global 16 and per-worker 4 * stress: fix RPC context reuse and remove redundant variables in ec_stress_runner * admin/plugin: fix lifecycle races, safe channel operations, and terminal state constants * admin/dash: randomize job IDs and fix priority zero-value overwrite in plugin API * admin/handlers: implement buffered rendering to prevent response corruption * admin/plugin: implement debounced persistence flusher and optimize BuildJobDetail memory lookups * admin/plugin: fix priority overwrite and implement bounded wait in scheduler reserve * admin/plugin: implement atomic file writes and fix run record side effects * admin/plugin: use P prefix for parity shard labels in execution plans * admin/plugin: enable parallel execution for cancellation tests * admin: refactor time.Time fields to pointers for better JSON omitempty support * admin/plugin: implement pointer-safe time assignments and comparisons in plugin core * admin/plugin: fix time assignment and sorting logic in plugin monitor after pointer refactor * admin/plugin: update scheduler activity tracking to use time pointers * admin/plugin: fix time-based run history trimming after pointer refactor * admin/dash: fix JobSpec struct literal in plugin API after pointer refactor * admin/view: add D/P prefixes to EC shard badges for UI consistency * admin/plugin: use lifecycle-aware context for schema prefetching * Update ec_volume_details_templ.go * admin/stress: fix proposal sorting and log volume cleanup errors * stress: refine ec stress runner with math/rand and collection name - Added Collection field to VolumeEcShardsDeleteRequest for correct filename construction. - Replaced crypto/rand with seeded math/rand PRNG for bulk payloads. - Added documentation for EcMinAge zero-value behavior. - Added logging for ignored errors in volume/shard deletion. * admin: return internal server error for plugin store failures Changed error status code from 400 Bad Request to 500 Internal Server Error for failures in GetPluginJobDetail to correctly reflect server-side errors. * admin: implement safe channel sends and graceful shutdown sync - Added sync.WaitGroup to Plugin struct to manage background goroutines. - Implemented safeSendCh helper using recover() to prevent panics on closed channels. - Ensured Shutdown() waits for all background operations to complete. * admin: robustify plugin monitor with nil-safe time and record init - Standardized nil-safe assignment for *time.Time pointers (CreatedAt, UpdatedAt, CompletedAt). - Ensured persistJobDetailSnapshot initializes new records correctly if they don't exist on disk. - Fixed debounced persistence to trigger immediate write on job completion. * admin: improve scheduler shutdown behavior and logic guards - Replaced brittle error string matching with explicit r.shutdownCh selection for shutdown detection. - Removed redundant nil guard in buildScheduledJobSpec. - Standardized WaitGroup usage for schedulerLoop. * admin: implement deep copy for job parameters and atomic write fixes - Implemented deepCopyGenericValue and used it in cloneTrackedJob to prevent shared state. - Ensured atomicWriteFile creates parent directories before writing. * admin: remove unreachable branch in shard classification Removed an unreachable 'totalShards <= 0' check in classifyShardID as dataShards and parityShards are already guarded. * admin: secure UI links and use canonical shard constants - Added rel="noopener noreferrer" to external links for security. - Replaced magic number 14 with erasure_coding.TotalShardsCount. - Used renderEcShardBadge for missing shard list consistency. * admin: stabilize plugin tests and fix regressions - Composed a robust plugin_monitor_test.go to handle asynchronous persistence. - Updated all time.Time literals to use timeToPtr helper. - Added explicit Shutdown() calls in tests to synchronize with debounced writes. - Fixed syntax errors and orphaned struct literals in tests. * Potential fix for code scanning alert no. 278: Slice memory allocation with excessive size value Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Potential fix for code scanning alert no. 283: Uncontrolled data used in path expression Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * admin: finalize refinements for error handling, scheduler, and race fixes - Standardized HTTP 500 status codes for store failures in plugin_api.go. - Tracked scheduled detection goroutines with sync.WaitGroup for safe shutdown. - Fixed race condition in safeSendDetectionComplete by extracting channel under lock. - Implemented deep copy for JobActivity details. - Used defaultDirPerm constant in atomicWriteFile. * test(ec): migrate admin dockertest to plugin APIs * admin/plugin_api: fix RunPluginJobTypeAPI to return 500 for server-side detection/filter errors * admin/plugin_api: fix ExecutePluginJobAPI to return 500 for job execution failures * admin/plugin_api: limit parseProtoJSONBody request body to 1MB to prevent unbounded memory usage * admin/plugin: consolidate regex to package-level validJobTypePattern; add char validation to sanitizeJobID * admin/plugin: fix racy Shutdown channel close with sync.Once * admin/plugin: track sendLoop and recv goroutines in WorkerStream with r.wg * admin/plugin: document writeProtoFiles atomicity — .pb is source of truth, .json is human-readable only * admin/plugin: extract activityLess helper to deduplicate nil-safe OccurredAt sort comparators * test/ec: check http.NewRequest errors to prevent nil req panics * test/ec: replace deprecated ioutil/math/rand, fix stale step comment 5.1→3.1 * plugin(ec): raise default detection and scheduling throughput limits * topology: include empty disks in volume list and EC capacity fallback * topology: remove hard 10-task cap for detection planning * Update ec_volume_details_templ.go * adjust default * fix tests --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
206 lines
7.0 KiB
Markdown
206 lines
7.0 KiB
Markdown
# Admin Worker Plugin System (Design)
|
|
|
|
This document describes the plugin system for admin-managed workers, implemented in parallel with the current maintenance/worker mechanism.
|
|
|
|
## Scope
|
|
|
|
- Add a new plugin protocol and runtime model for multi-language workers.
|
|
- Keep all current admin + worker code paths untouched.
|
|
- Use gRPC for all admin-worker communication.
|
|
- Let workers describe job configuration UI declaratively via protobuf.
|
|
- Persist all job type configuration under admin server data directory.
|
|
- Support detector workers and executor workers per job type.
|
|
- Add end-to-end workflow observability (activities, active jobs, progress).
|
|
|
|
## New Contract
|
|
|
|
- Proto file: `weed/pb/plugin.proto`
|
|
- gRPC service: `PluginControlService.WorkerStream`
|
|
- Connection model: worker-initiated long-lived bidirectional stream.
|
|
|
|
Why this model:
|
|
|
|
- Works for workers in any language with gRPC support.
|
|
- Avoids admin dialing constraints in NAT/private networks.
|
|
- Allows command/response, progress streaming, and heartbeat over one channel.
|
|
|
|
## Core Runtime Components (Admin Side)
|
|
|
|
1. `PluginRegistry`
|
|
- Tracks connected workers and their per-job-type capabilities.
|
|
- Maintains liveness via heartbeat timeout.
|
|
|
|
2. `SchemaCoordinator`
|
|
- For each job type, asks one capable worker for `JobTypeDescriptor`.
|
|
- Caches descriptor version and refresh timestamp.
|
|
|
|
3. `ConfigStore`
|
|
- Persists descriptor + saved config values in `dataDir`.
|
|
- Stores both:
|
|
- Admin-owned runtime config (detection interval, dispatch concurrency, retry).
|
|
- Worker-owned config values (plugin-specific detection/execution knobs).
|
|
|
|
4. `DetectorScheduler`
|
|
- Per job type, chooses one detector worker (`can_detect=true`).
|
|
- Sends `RunDetectionRequest` with saved configs + cluster context.
|
|
- Accepts `DetectionProposals`, dedupes by `dedupe_key`, inserts jobs.
|
|
|
|
5. `JobDispatcher`
|
|
- Chooses executor worker (`can_execute=true`) for each pending job.
|
|
- Sends `ExecuteJobRequest`.
|
|
- Consumes `JobProgressUpdate` and `JobCompleted`.
|
|
|
|
6. `WorkflowMonitor`
|
|
- Builds live counters and timeline from events:
|
|
- activities per job type,
|
|
- active jobs,
|
|
- per-job progress/state,
|
|
- worker health/load.
|
|
|
|
## Worker Responsibilities
|
|
|
|
1. Register capabilities on connect (`WorkerHello`).
|
|
2. Expose job type descriptor (`ConfigSchemaResponse`) including UI schemas:
|
|
- admin config form,
|
|
- worker config form,
|
|
- defaults.
|
|
3. Run detection on demand (`RunDetectionRequest`) and return proposals.
|
|
4. Execute assigned jobs (`ExecuteJobRequest`) and stream progress.
|
|
5. Heartbeat regularly with slot usage and running work.
|
|
6. Handle cancellation requests (`CancelRequest`) for in-flight detection/execution.
|
|
|
|
## Declarative UI Model
|
|
|
|
UI is fully derived from protobuf schema:
|
|
|
|
- `ConfigForm`
|
|
- `ConfigSection`
|
|
- `ConfigField`
|
|
- `ConfigOption`
|
|
- `ValidationRule`
|
|
- `ConfigValue` (typed scalar/list/map/object value container)
|
|
|
|
Result:
|
|
|
|
- Admin can render forms without hardcoded task structs.
|
|
- New job types can ship UI schema from worker binary alone.
|
|
- Worker language is irrelevant as long as it can emit protobuf messages.
|
|
|
|
## Detection and Dispatch Flow
|
|
|
|
1. Worker connects and registers capabilities.
|
|
2. Admin requests descriptor per job type.
|
|
3. Admin persists descriptor and editable config values.
|
|
4. On detection interval (admin-owned setting):
|
|
- Admin chooses one detector worker for that job type.
|
|
- Sends `RunDetectionRequest` with:
|
|
- `AdminRuntimeConfig`,
|
|
- `admin_config_values`,
|
|
- `worker_config_values`,
|
|
- `ClusterContext` (master/filer/volume grpc locations, metadata).
|
|
5. Detector emits `DetectionProposals` and `DetectionComplete`.
|
|
6. Admin dedupes and enqueues jobs.
|
|
7. Dispatcher assigns jobs to any eligible executor worker.
|
|
8. Executor emits `JobProgressUpdate` and `JobCompleted`.
|
|
9. Monitor updates workflow UI in near-real-time.
|
|
|
|
## Persistence Layout (Admin Data Dir)
|
|
|
|
Current layout under `<admin-data-dir>/plugin/`:
|
|
|
|
- `job_types/<job_type>/descriptor.pb`
|
|
- `job_types/<job_type>/descriptor.json`
|
|
- `job_types/<job_type>/config.pb`
|
|
- `job_types/<job_type>/config.json`
|
|
- `job_types/<job_type>/runs.json`
|
|
- `jobs/tracked_jobs.json`
|
|
- `activities/activities.json`
|
|
|
|
`config.pb` should use `PersistedJobTypeConfig` from `plugin.proto`.
|
|
|
|
## Admin UI
|
|
|
|
- Route: `/plugin`
|
|
- Includes:
|
|
- runtime status,
|
|
- workers/capabilities,
|
|
- declarative descriptor-driven config forms,
|
|
- run history (last 10 success + last 10 errors),
|
|
- tracked jobs and activity stream,
|
|
- manual actions for schema refresh, detection, and detect+execute workflow.
|
|
|
|
## Scheduling Policy (Initial)
|
|
|
|
Detector selection per job type:
|
|
- only workers with `can_detect=true`.
|
|
- prefer healthy worker with highest free detection slots.
|
|
- lease ends when heartbeat timeout or stream drop.
|
|
|
|
Execution dispatch:
|
|
- only workers with `can_execute=true`.
|
|
- select by available execution slots and least active jobs.
|
|
- retry on failure using admin runtime retry config.
|
|
|
|
## Safety and Reliability
|
|
|
|
- Idempotency: dedupe proposals by (`job_type`, `dedupe_key`).
|
|
- Backpressure: enforce max jobs per detection run.
|
|
- Timeouts: detection and execution timeout from admin runtime config.
|
|
- Replay-safe persistence: write job state changes before emitting UI events.
|
|
- Heartbeat-based failover for detector/executor reassignment.
|
|
|
|
## Backward Compatibility
|
|
|
|
- Legacy `worker.proto` runtime remains internally available where still referenced.
|
|
- External CLI worker path is moved to plugin runtime behavior.
|
|
- Runtime is enabled by default on admin worker gRPC server.
|
|
|
|
## Incremental Rollout Plan
|
|
|
|
Phase 1
|
|
- Introduce protocol and storage models only.
|
|
|
|
Phase 2
|
|
- Build admin registry/scheduler/dispatcher behind feature flag.
|
|
|
|
Phase 3
|
|
- Add dedicated plugin UI pages and metrics.
|
|
|
|
Phase 4
|
|
- Port one existing job type (e.g. vacuum) as external worker plugin.
|
|
|
|
Phase 4 status (starter)
|
|
- Added `weed worker` command as an external `plugin.proto` worker process.
|
|
- Initial handler implements `vacuum` job type with:
|
|
- declarative descriptor/config form response (`ConfigSchemaResponse`),
|
|
- detection via master topology scan (`RunDetectionRequest`),
|
|
- execution via existing vacuum task logic (`ExecuteJobRequest`),
|
|
- heartbeat/load reporting for monitor UI.
|
|
- Legacy maintenance-worker-specific CLI path is removed.
|
|
|
|
Run example:
|
|
- Start admin: `weed admin -master=localhost:9333`
|
|
- Start worker: `weed worker -admin=localhost:23646`
|
|
- Optional explicit job type: `weed worker -admin=localhost:23646 -jobType=vacuum`
|
|
- Optional stable worker ID persistence: `weed worker -admin=localhost:23646 -workingDir=/var/lib/seaweedfs-plugin`
|
|
|
|
Phase 5
|
|
- Migrate remaining job types and deprecate old mechanism.
|
|
|
|
## Agreed Defaults
|
|
|
|
1. Detector multiplicity
|
|
- Exactly one detector worker per job type at a time. Admin selects one worker and runs detection there.
|
|
|
|
2. Secret handling
|
|
- No encryption at rest required for plugin config in this phase.
|
|
|
|
3. Schema compatibility
|
|
- No migration policy required yet; this is a new system.
|
|
|
|
4. Execution ownership
|
|
- Same worker is allowed to do both detection and execution.
|
|
|
|
5. Retention
|
|
- Keep last 10 successful runs and last 10 error runs per job type.
|