Files

Chris Lu 8ec9ff4a12 Refactor plugin system and migrate worker runtime (#8369 )

* admin: add plugin runtime UI page and route wiring

* pb: add plugin gRPC contract and generated bindings

* admin/plugin: implement worker registry, runtime, monitoring, and config store

* admin/dash: wire plugin runtime and expose plugin workflow APIs

* command: add flags to enable plugin runtime

* admin: rename remaining plugin v2 wording to plugin

* admin/plugin: add detectable job type registry helper

* admin/plugin: add scheduled detection and dispatch orchestration

* admin/plugin: prefetch job type descriptors when workers connect

* admin/plugin: add known job type discovery API and UI

* admin/plugin: refresh design doc to match current implementation

* admin/plugin: enforce per-worker scheduler concurrency limits

* admin/plugin: use descriptor runtime defaults for scheduler policy

* admin/ui: auto-load first known plugin job type on page open

* admin/plugin: bootstrap persisted config from descriptor defaults

* admin/plugin: dedupe scheduled proposals by dedupe key

* admin/ui: add job type and state filters for plugin monitoring

* admin/ui: add per-job-type plugin activity summary

* admin/plugin: split descriptor read API from schema refresh

* admin/ui: keep plugin summary metrics global while tables are filtered

* admin/plugin: retry executor reservation before timing out

* admin/plugin: expose scheduler states for monitoring

* admin/ui: show per-job-type scheduler states in plugin monitor

* pb/plugin: rename protobuf package to plugin

* admin/plugin: rename pluginRuntime wiring to plugin

* admin/plugin: remove runtime naming from plugin APIs and UI

* admin/plugin: rename runtime files to plugin naming

* admin/plugin: persist jobs and activities for monitor recovery

* admin/plugin: lease one detector worker per job type

* admin/ui: show worker load from plugin heartbeats

* admin/plugin: skip stale workers for detector and executor picks

* plugin/worker: add plugin worker command and stream runtime scaffold

* plugin/worker: implement vacuum detect and execute handlers

* admin/plugin: document external vacuum plugin worker starter

* command: update plugin.worker help to reflect implemented flow

* command/admin: drop legacy Plugin V2 label

* plugin/worker: validate vacuum job type and respect min interval

* plugin/worker: test no-op detect when min interval not elapsed

* command/admin: document plugin.worker external process

* plugin/worker: advertise configured concurrency in hello

* command/plugin.worker: add jobType handler selection

* command/plugin.worker: test handler selection by job type

* command/plugin.worker: persist worker id in workingDir

* admin/plugin: document plugin.worker jobType and workingDir flags

* plugin/worker: support cancel request for in-flight work

* plugin/worker: test cancel request acknowledgements

* command/plugin.worker: document workingDir and jobType behavior

* plugin/worker: emit executor activity events for monitor

* plugin/worker: test executor activity builder

* admin/plugin: send last successful run in detection request

* admin/plugin: send cancel request when detect or execute context ends

* admin/plugin: document worker cancel request responsibility

* admin/handlers: expose plugin scheduler states API in no-auth mode

* admin/handlers: test plugin scheduler states route registration

* admin/plugin: keep worker id on worker-generated activity records

* admin/plugin: test worker id propagation in monitor activities

* admin/dash: always initialize plugin service

* command/admin: remove plugin enable flags and default to enabled

* admin/dash: drop pluginEnabled constructor parameter

* admin/plugin UI: stop checking plugin enabled state

* admin/plugin: remove docs for plugin enable flags

* admin/dash: remove unused plugin enabled check method

* admin/dash: fallback to in-memory plugin init when dataDir fails

* admin/plugin API: expose worker gRPC port in status

* command/plugin.worker: resolve admin gRPC port via plugin status

* split plugin UI into overview/configuration/monitoring pages

* Update layout_templ.go

* add volume_balance plugin worker handler

* wire plugin.worker CLI for volume_balance job type

* add erasure_coding plugin worker handler

* wire plugin.worker CLI for erasure_coding job type

* support multi-job handlers in plugin worker runtime

* allow plugin.worker jobType as comma-separated list

* admin/plugin UI: rename to Workers and simplify config view

* plugin worker: queue detection requests instead of capacity reject

* Update plugin_worker.go

* plugin volume_balance: remove force_move/timeout from worker config UI

* plugin erasure_coding: enforce local working dir and cleanup

* admin/plugin UI: rename admin settings to job scheduling

* admin/plugin UI: persist and robustly render detection results

* admin/plugin: record and return detection trace metadata

* admin/plugin UI: show detection process and decision trace

* plugin: surface detector decision trace as activities

* mini: start a plugin worker by default

* admin/plugin UI: split monitoring into detection and execution tabs

* plugin worker: emit detection decision trace for EC and balance

* admin workers UI: split monitoring into detection and execution pages

* plugin scheduler: skip proposals for active assigned/running jobs

* admin workers UI: add job queue tab

* plugin worker: add dummy stress detector and executor job type

* admin workers UI: reorder tabs to detection queue execution

* admin workers UI: regenerate plugin template

* plugin defaults: include dummy stress and add stress tests

* plugin dummy stress: rotate detection selections across runs

* plugin scheduler: remove cross-run proposal dedupe

* plugin queue: track pending scheduled jobs

* plugin scheduler: wait for executor capacity before dispatch

* plugin scheduler: skip detection when waiting backlog is high

* plugin: add disk-backed job detail API and persistence

* admin ui: show plugin job detail modal from job id links

* plugin: generate unique job ids instead of reusing proposal ids

* plugin worker: emit heartbeats on work state changes

* plugin registry: round-robin tied executor and detector picks

* add temporary EC overnight stress runner

* plugin job details: persist and render EC execution plans

* ec volume details: color data and parity shard badges

* shard labels: keep parity ids numeric and color-only distinction

* admin: remove legacy maintenance UI routes and templates

* admin: remove dead maintenance endpoint helpers

* Update layout_templ.go

* remove dummy_stress worker and command support

* refactor plugin UI to job-type top tabs and sub-tabs

* migrate weed worker command to plugin runtime

* remove plugin.worker command and keep worker runtime with metrics

* update helm worker args for jobType and execution flags

* set plugin scheduling defaults to global 16 and per-worker 4

* stress: fix RPC context reuse and remove redundant variables in ec_stress_runner

* admin/plugin: fix lifecycle races, safe channel operations, and terminal state constants

* admin/dash: randomize job IDs and fix priority zero-value overwrite in plugin API

* admin/handlers: implement buffered rendering to prevent response corruption

* admin/plugin: implement debounced persistence flusher and optimize BuildJobDetail memory lookups

* admin/plugin: fix priority overwrite and implement bounded wait in scheduler reserve

* admin/plugin: implement atomic file writes and fix run record side effects

* admin/plugin: use P prefix for parity shard labels in execution plans

* admin/plugin: enable parallel execution for cancellation tests

* admin: refactor time.Time fields to pointers for better JSON omitempty support

* admin/plugin: implement pointer-safe time assignments and comparisons in plugin core

* admin/plugin: fix time assignment and sorting logic in plugin monitor after pointer refactor

* admin/plugin: update scheduler activity tracking to use time pointers

* admin/plugin: fix time-based run history trimming after pointer refactor

* admin/dash: fix JobSpec struct literal in plugin API after pointer refactor

* admin/view: add D/P prefixes to EC shard badges for UI consistency

* admin/plugin: use lifecycle-aware context for schema prefetching

* Update ec_volume_details_templ.go

* admin/stress: fix proposal sorting and log volume cleanup errors

* stress: refine ec stress runner with math/rand and collection name

- Added Collection field to VolumeEcShardsDeleteRequest for correct filename construction.
- Replaced crypto/rand with seeded math/rand PRNG for bulk payloads.
- Added documentation for EcMinAge zero-value behavior.
- Added logging for ignored errors in volume/shard deletion.

* admin: return internal server error for plugin store failures

Changed error status code from 400 Bad Request to 500 Internal Server Error for failures in GetPluginJobDetail to correctly reflect server-side errors.

* admin: implement safe channel sends and graceful shutdown sync

- Added sync.WaitGroup to Plugin struct to manage background goroutines.
- Implemented safeSendCh helper using recover() to prevent panics on closed channels.
- Ensured Shutdown() waits for all background operations to complete.

* admin: robustify plugin monitor with nil-safe time and record init

- Standardized nil-safe assignment for *time.Time pointers (CreatedAt, UpdatedAt, CompletedAt).
- Ensured persistJobDetailSnapshot initializes new records correctly if they don't exist on disk.
- Fixed debounced persistence to trigger immediate write on job completion.

* admin: improve scheduler shutdown behavior and logic guards

- Replaced brittle error string matching with explicit r.shutdownCh selection for shutdown detection.
- Removed redundant nil guard in buildScheduledJobSpec.
- Standardized WaitGroup usage for schedulerLoop.

* admin: implement deep copy for job parameters and atomic write fixes

- Implemented deepCopyGenericValue and used it in cloneTrackedJob to prevent shared state.
- Ensured atomicWriteFile creates parent directories before writing.

* admin: remove unreachable branch in shard classification

Removed an unreachable 'totalShards <= 0' check in classifyShardID as dataShards and parityShards are already guarded.

* admin: secure UI links and use canonical shard constants

- Added rel="noopener noreferrer" to external links for security.
- Replaced magic number 14 with erasure_coding.TotalShardsCount.
- Used renderEcShardBadge for missing shard list consistency.

* admin: stabilize plugin tests and fix regressions

- Composed a robust plugin_monitor_test.go to handle asynchronous persistence.
- Updated all time.Time literals to use timeToPtr helper.
- Added explicit Shutdown() calls in tests to synchronize with debounced writes.
- Fixed syntax errors and orphaned struct literals in tests.

* Potential fix for code scanning alert no. 278: Slice memory allocation with excessive size value

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for code scanning alert no. 283: Uncontrolled data used in path expression

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* admin: finalize refinements for error handling, scheduler, and race fixes

- Standardized HTTP 500 status codes for store failures in plugin_api.go.
- Tracked scheduled detection goroutines with sync.WaitGroup for safe shutdown.
- Fixed race condition in safeSendDetectionComplete by extracting channel under lock.
- Implemented deep copy for JobActivity details.
- Used defaultDirPerm constant in atomicWriteFile.

* test(ec): migrate admin dockertest to plugin APIs

* admin/plugin_api: fix RunPluginJobTypeAPI to return 500 for server-side detection/filter errors

* admin/plugin_api: fix ExecutePluginJobAPI to return 500 for job execution failures

* admin/plugin_api: limit parseProtoJSONBody request body to 1MB to prevent unbounded memory usage

* admin/plugin: consolidate regex to package-level validJobTypePattern; add char validation to sanitizeJobID

* admin/plugin: fix racy Shutdown channel close with sync.Once

* admin/plugin: track sendLoop and recv goroutines in WorkerStream with r.wg

* admin/plugin: document writeProtoFiles atomicity — .pb is source of truth, .json is human-readable only

* admin/plugin: extract activityLess helper to deduplicate nil-safe OccurredAt sort comparators

* test/ec: check http.NewRequest errors to prevent nil req panics

* test/ec: replace deprecated ioutil/math/rand, fix stale step comment 5.1→3.1

* plugin(ec): raise default detection and scheduling throughput limits

* topology: include empty disks in volume list and EC capacity fallback

* topology: remove hard 10-task cap for detection planning

* Update ec_volume_details_templ.go

* adjust default

* fix tests

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

2026-02-18 13:42:41 -08:00

7.0 KiB

Raw Blame History

Admin Worker Plugin System (Design)

This document describes the plugin system for admin-managed workers, implemented in parallel with the current maintenance/worker mechanism.

Scope

Add a new plugin protocol and runtime model for multi-language workers.
Keep all current admin + worker code paths untouched.
Use gRPC for all admin-worker communication.
Let workers describe job configuration UI declaratively via protobuf.
Persist all job type configuration under admin server data directory.
Support detector workers and executor workers per job type.
Add end-to-end workflow observability (activities, active jobs, progress).

New Contract

Proto file: weed/pb/plugin.proto
gRPC service: PluginControlService.WorkerStream
Connection model: worker-initiated long-lived bidirectional stream.

Why this model:

Works for workers in any language with gRPC support.
Avoids admin dialing constraints in NAT/private networks.
Allows command/response, progress streaming, and heartbeat over one channel.

Core Runtime Components (Admin Side)

PluginRegistry

Tracks connected workers and their per-job-type capabilities.
Maintains liveness via heartbeat timeout.

SchemaCoordinator

For each job type, asks one capable worker for JobTypeDescriptor.
Caches descriptor version and refresh timestamp.

ConfigStore

Persists descriptor + saved config values in dataDir.
Stores both:
- Admin-owned runtime config (detection interval, dispatch concurrency, retry).
- Worker-owned config values (plugin-specific detection/execution knobs).

DetectorScheduler

Per job type, chooses one detector worker (can_detect=true).
Sends RunDetectionRequest with saved configs + cluster context.
Accepts DetectionProposals, dedupes by dedupe_key, inserts jobs.

JobDispatcher

Chooses executor worker (can_execute=true) for each pending job.
Sends ExecuteJobRequest.
Consumes JobProgressUpdate and JobCompleted.

WorkflowMonitor

Builds live counters and timeline from events:
- activities per job type,
- active jobs,
- per-job progress/state,
- worker health/load.

Worker Responsibilities

Register capabilities on connect (WorkerHello).
Expose job type descriptor (ConfigSchemaResponse) including UI schemas:

admin config form,
worker config form,
defaults.

Run detection on demand (RunDetectionRequest) and return proposals.
Execute assigned jobs (ExecuteJobRequest) and stream progress.
Heartbeat regularly with slot usage and running work.
Handle cancellation requests (CancelRequest) for in-flight detection/execution.

Declarative UI Model

UI is fully derived from protobuf schema:

ConfigForm
ConfigSection
ConfigField
ConfigOption
ValidationRule
ConfigValue (typed scalar/list/map/object value container)

Result:

Admin can render forms without hardcoded task structs.
New job types can ship UI schema from worker binary alone.
Worker language is irrelevant as long as it can emit protobuf messages.

Detection and Dispatch Flow

Worker connects and registers capabilities.
Admin requests descriptor per job type.
Admin persists descriptor and editable config values.
On detection interval (admin-owned setting):

Admin chooses one detector worker for that job type.
Sends RunDetectionRequest with:
- AdminRuntimeConfig,
- admin_config_values,
- worker_config_values,
- ClusterContext (master/filer/volume grpc locations, metadata).

Detector emits DetectionProposals and DetectionComplete.
Admin dedupes and enqueues jobs.
Dispatcher assigns jobs to any eligible executor worker.
Executor emits JobProgressUpdate and JobCompleted.
Monitor updates workflow UI in near-real-time.

Persistence Layout (Admin Data Dir)

Current layout under <admin-data-dir>/plugin/:

job_types/<job_type>/descriptor.pb
job_types/<job_type>/descriptor.json
job_types/<job_type>/config.pb
job_types/<job_type>/config.json
job_types/<job_type>/runs.json
jobs/tracked_jobs.json
activities/activities.json

config.pb should use PersistedJobTypeConfig from plugin.proto.

Admin UI

Route: /plugin
Includes:
- runtime status,
- workers/capabilities,
- declarative descriptor-driven config forms,
- run history (last 10 success + last 10 errors),
- tracked jobs and activity stream,
- manual actions for schema refresh, detection, and detect+execute workflow.

Scheduling Policy (Initial)

Detector selection per job type:

only workers with can_detect=true.
prefer healthy worker with highest free detection slots.
lease ends when heartbeat timeout or stream drop.

Execution dispatch:

only workers with can_execute=true.
select by available execution slots and least active jobs.
retry on failure using admin runtime retry config.

Safety and Reliability

Idempotency: dedupe proposals by (job_type, dedupe_key).
Backpressure: enforce max jobs per detection run.
Timeouts: detection and execution timeout from admin runtime config.
Replay-safe persistence: write job state changes before emitting UI events.
Heartbeat-based failover for detector/executor reassignment.

Backward Compatibility

Legacy worker.proto runtime remains internally available where still referenced.
External CLI worker path is moved to plugin runtime behavior.
Runtime is enabled by default on admin worker gRPC server.

Incremental Rollout Plan

Phase 1

Introduce protocol and storage models only.

Phase 2

Build admin registry/scheduler/dispatcher behind feature flag.

Phase 3

Add dedicated plugin UI pages and metrics.

Phase 4

Port one existing job type (e.g. vacuum) as external worker plugin.

Phase 4 status (starter)

Added weed worker command as an external plugin.proto worker process.
Initial handler implements vacuum job type with:
- declarative descriptor/config form response (ConfigSchemaResponse),
- detection via master topology scan (RunDetectionRequest),
- execution via existing vacuum task logic (ExecuteJobRequest),
- heartbeat/load reporting for monitor UI.
Legacy maintenance-worker-specific CLI path is removed.

Run example:

Start admin: weed admin -master=localhost:9333
Start worker: weed worker -admin=localhost:23646
Optional explicit job type: weed worker -admin=localhost:23646 -jobType=vacuum
Optional stable worker ID persistence: weed worker -admin=localhost:23646 -workingDir=/var/lib/seaweedfs-plugin

Phase 5

Migrate remaining job types and deprecate old mechanism.

Agreed Defaults

Detector multiplicity

Exactly one detector worker per job type at a time. Admin selects one worker and runs detection there.

Secret handling

No encryption at rest required for plugin config in this phase.

Schema compatibility

No migration policy required yet; this is a new system.

Execution ownership

Same worker is allowed to do both detection and execution.

Retention

Keep last 10 successful runs and last 10 error runs per job type.

7.0 KiB Raw Blame History