* fix(admin): reduce memory usage and verbose logging for large clusters (#8919)
The admin server used excessive memory and produced thousands of log lines
on clusters with many volumes (e.g., 33k volumes). Three root causes:
1. Scanner duplicated all volume metrics: getVolumeHealthMetrics() created
VolumeHealthMetrics objects, then convertToTaskMetrics() copied them all
into identical types.VolumeHealthMetrics. Now uses the task-system type
directly, eliminating the duplicate allocation and removing convertToTaskMetrics.
2. All previous task states loaded at startup: LoadTasksFromPersistence read
and deserialized every .pb file from disk, logging each one. With thousands
of balance tasks persisted, this caused massive startup I/O, memory usage,
and log noise (including unguarded DEBUG glog.Infof per task). Now starts
with an empty queue — the scanner re-detects current needs from live cluster
state. Terminal tasks are purged from memory and disk when new scan results
arrive.
3. Verbose per-volume/per-node logging: V(2) and V(3) logs produced thousands
of lines per scan. Per-volume logs bumped to V(4), per-node/rack/disk logs
bumped to V(3). Topology summary now logs counts instead of full node ID arrays.
Also removes lastTopologyInfo field from MaintenanceScanner — the raw protobuf
topology is returned as a local value and not retained between 30-minute scans.
* fix(admin): delete stale task files at startup, add DeleteAllTaskStates
Old task .pb files from previous runs were left on disk. The periodic
CleanupCompletedTasks still loads all files to find completed ones —
the same expensive 4GB path from the pprof profile.
Now at startup, DeleteAllTaskStates removes all .pb files by scanning
the directory without reading or deserializing them. The scanner will
re-detect any tasks still needed from live cluster state.
* fix(admin): don't persist terminal tasks to disk
CompleteTask was saving failed/completed tasks to disk where they'd
accumulate. The periodic cleanup only triggered for completed tasks,
not failed ones. Now terminal tasks are deleted from disk immediately
and only kept in memory for the current session's UI.
* fix(admin): cap in-memory tasks to 100 per job type
Without a limit, the task map grows unbounded — balance could create
thousands of pending tasks for a cluster with many imbalanced volumes.
Now AddTask rejects new tasks when a job type already has 100 in the
queue. The scanner will re-detect skipped volumes on the next scan.
* fix(admin): address PR review - memory-only purge, active-only capacity
- purgeTerminalTasks now only cleans in-memory map (terminal tasks are
already deleted from disk by CompleteTask)
- Per-type capacity limit counts only active tasks (pending/assigned/
in_progress), not terminal ones
- When at capacity, purge terminal tasks first before rejecting
* fix(admin): fix orphaned comment, add TaskStatusCancelled to terminal switch
- Move hasQueuedOrActiveTaskForVolume comment to its function definition
- Add TaskStatusCancelled to the terminal state switch in CompleteTask
so cancelled task files are deleted from disk
* chore: remove unreachable dead code across the codebase
Remove ~50,000 lines of unreachable code identified by static analysis.
Major removals:
- weed/filer/redis_lua: entire unused Redis Lua filer store implementation
- weed/wdclient/net2, resource_pool: unused connection/resource pool packages
- weed/plugin/worker/lifecycle: unused lifecycle plugin worker
- weed/s3api: unused S3 policy templates, presigned URL IAM, streaming copy,
multipart IAM, key rotation, and various SSE helper functions
- weed/mq/kafka: unused partition mapping, compression, schema, and protocol functions
- weed/mq/offset: unused SQL storage and migration code
- weed/worker: unused registry, task, and monitoring functions
- weed/query: unused SQL engine, parquet scanner, and type functions
- weed/shell: unused EC proportional rebalance functions
- weed/storage/erasure_coding/distribution: unused distribution analysis functions
- Individual unreachable functions removed from 150+ files across admin,
credential, filer, iam, kms, mount, mq, operation, pb, s3api, server,
shell, storage, topology, and util packages
* fix(s3): reset shared memory store in IAM test to prevent flaky failure
TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey was flaky because
the MemoryStore credential backend is a singleton registered via init().
Earlier tests that create anonymous identities pollute the shared store,
causing LookupAnonymous() to unexpectedly return true.
Fix by calling Reset() on the memory store before the test runs.
* style: run gofmt on changed files
* fix: restore KMS functions used by integration tests
* fix(plugin): prevent panic on send to closed worker session channel
The Plugin.sendToWorker method could panic with "send on closed channel"
when a worker disconnected while a message was being sent. The race was
between streamSession.close() closing the outgoing channel and sendToWorker
writing to it concurrently.
Add a done channel to streamSession that is closed before the outgoing
channel, and check it in sendToWorker's select to safely detect closed
sessions without panicking.
remove min_interval_seconds from plugin workers and default vacuum interval to 17m
The worker-level min_interval_seconds was redundant with the admin-side
DetectionIntervalSeconds, complicating scheduling logic. Remove it from
vacuum, volume_balance, erasure_coding, and ec_balance handlers.
Also change the vacuum default DetectionIntervalSeconds from 2 hours to
17 minutes to match the previous default behavior.
* fix: maintenance task topology lookup, retry, and stale task cleanup
1. Strip gRPC port from ServerAddress in SyncTask using ToHttpAddress()
so task targets match topology disk keys (NodeId format).
2. Skip capacity check when topology has no disks yet (startup race
where tasks are loaded from persistence before first topology update).
3. Don't retry permanent errors like "volume not found" - these will
never succeed on retry.
4. Cancel all pending tasks for each task type before re-detection,
ensuring stale proposals from previous cycles are cleaned up.
This prevents stale tasks from blocking new detection and from
repeatedly failing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* logs
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
* less lock scope
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(admin): release mutex before disk I/O in maintenance queue
saveTaskState performs synchronous BoltDB writes. Calling it while
holding mq.mutex.Lock() in AddTask, GetNextTask, and CompleteTask
blocks all readers (GetTasks via RLock) for the full disk write
duration on every task state change.
During a maintenance scan AddTasksFromResults calls AddTask for every
volume — potentially hundreds of times — meaning the write lock is
held almost continuously. The HTTP handler for /maintenance calls
GetTasks which blocks on RLock, exceeding the 30s timeout and
returning 408 to the browser.
Fix: update in-memory state (mq.tasks, mq.pendingTasks) under the
lock as before, then unlock before calling saveTaskState. In-memory
state is the authoritative source; persistence is crash-recovery only
and does not require lock protection during the write.
* fix(admin): add mutex to ConfigPersistence to synchronize tasks/ filesystem ops
saveTaskState is now called outside mq.mutex, meaning SaveTaskState,
LoadAllTaskStates, DeleteTaskState, and CleanupCompletedTasks can be
invoked concurrently from multiple goroutines. ConfigPersistence had no
internal synchronization, creating races on the tasks/ directory:
- concurrent os.WriteFile + os.ReadFile on the same .pb file could
yield a partial read and unmarshal error
- LoadAllTaskStates (ReadDir + per-file ReadFile) could see a
directory entry for a file being written or deleted concurrently
- CleanupCompletedTasks (LoadAllTaskStates + DeleteTaskState) could
race with SaveTaskState on the same file
Fix: add tasksMu sync.Mutex to ConfigPersistence, acquired at the top
of SaveTaskState, LoadTaskState, LoadAllTaskStates, DeleteTaskState,
and CleanupCompletedTasks. Extract private Locked helpers so that
CleanupCompletedTasks (which holds tasksMu) can call them internally
without deadlocking.
---------
Co-authored-by: Anton Ustyugov <anton@devops>
* admin: fix capacity leak in maintenance system by preserving Task IDs
Preserve the original TaskID generated during detection and sync task
states (Assign/Complete/Retry) with ActiveTopology. This ensures that
capacity reserved during task assignment is properly released when a
task completes or fails, preventing 'need 9, have 0' capacity exhaustion.
Fixes https://github.com/seaweedfs/seaweedfs/issues/8202
* Update weed/admin/maintenance/maintenance_queue.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update weed/admin/maintenance/maintenance_queue.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* test: rename ActiveTopologySync to TaskIDPreservation
Rename the test case to more accurately reflect its scope, as suggested
by the code review bot.
* Add TestMaintenanceQueue_ActiveTopologySync to verify task state synchronization and capacity management
* Implement task assignment rollback and add verification test
* Enhance ActiveTopology.CompleteTask to support pending tasks
* Populate storage impact in MaintenanceIntegration.SyncTask
* Release capacity in RemoveStaleWorkers when worker becomes unavailable
* Release capacity in MaintenanceManager.CancelTask when pending task is cancelled
* Sync reloaded tasks with ActiveTopology in LoadTasksFromPersistence
* Add verification tests for consistent capacity management lifecycle
* Add TestMaintenanceQueue_RetryCapacitySync to verify capacity tracking during retries
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix float stepping
* do not auto refresh
* only logs when non 200 status
* fix maintenance task sorting and cleanup redundant handler logic
* Refactor log retrieval to persist to disk and fix slowness
- Move log retrieval to disk-based persistence in GetMaintenanceTaskDetail
- Implement background log fetching on task completion in worker_grpc_server.go
- Implement async background refresh for in-progress tasks
- Completely remove blocking gRPC calls from the UI path to fix 10s timeouts
- Cleanup debug logs and performance profiling code
* Ensure consistent deterministic sorting in config_persistence cleanup
* Replace magic numbers with constants and remove debug logs
- Added descriptive constants for truncation limits and timeouts in admin_server.go and worker_grpc_server.go
- Replaced magic numbers with these constants throughout the codebase
- Verified removal of stdout debug printing
- Ensured consistent truncation logic during log persistence
* Address code review feedback on history truncation and logging logic
- Fix AssignmentHistory double-serialization by copying task in GetMaintenanceTaskDetail
- Fix handleTaskCompletion logging logic (mutually exclusive success/failure logs)
- Remove unused Timeout field from LogRequestContext and sync select timeouts with constants
- Ensure AssignmentHistory is only provided in the top-level field for better JSON structure
* Implement goroutine leak protection and request deduplication
- Add request deduplication in RequestTaskLogs to prevent multiple concurrent fetches for the same task
- Implement safe cleanup in timeout handlers to avoid race conditions in pendingLogRequests map
- Add a 10s cooldown for background log refreshes in GetMaintenanceTaskDetail to prevent spamming
- Ensure all persistent log-fetching goroutines are bounded and efficiently managed
* Fix potential nil pointer panics in maintenance handlers
- Add nil checks for adminServer in ShowTaskDetail, ShowMaintenanceWorkers, and UpdateTaskConfig
- Update getMaintenanceQueueData to return a descriptive error instead of nil when adminServer is uninitialized
- Ensure internal helper methods consistently check for adminServer initialization before use
* Strictly enforce disk-only log reading
- Remove background log fetching from GetMaintenanceTaskDetail to prevent timeouts and network calls during page view
- Remove unused lastLogFetch tracking fields to clean up dead code
- Ensure logs are only updated upon task completion via handleTaskCompletion
* Refactor GetWorkerLogs to read from disk
- Update /api/maintenance/workers/:id/logs endpoint to use configPersistence.LoadTaskExecutionLogs
- Remove synchronous gRPC call RequestTaskLogs to prevent timeouts and bad gateway errors
- Ensure consistent log retrieval behavior across the application (disk-only)
* Fix timestamp parsing in log viewer
- Update task_detail.templ JS to handle both ISO 8601 strings and Unix timestamps
- Fix "Invalid time value" error when displaying logs fetched from disk
- Regenerate templates
* master: fallback to HDD if SSD volumes are full in Assign
* worker: improve EC detection logging and fix skip counters
* worker: add Sync method to TaskLogger interface
* worker: implement Sync and ensure logs are flushed before task completion
* admin: improve task log retrieval with retries and better timeouts
* admin: robust timestamp parsing in task detail view
* Fix nil pointer panic in maintenance worker when receiving empty task assignment
When a worker requests a task and none are available, the admin server
sends an empty TaskAssignment message. The worker was attempting to log
the task details without checking if the TaskId was empty, causing a
nil pointer dereference when accessing taskAssign.Params.VolumeId.
This fix adds a check for empty TaskId before processing the assignment,
preventing worker crashes and improving stability in production environments.
* Add EC integration test for admin-worker maintenance system
Adds comprehensive integration test that verifies the end-to-end flow
of erasure coding maintenance tasks:
- Admin server detects volumes needing EC encoding
- Workers register and receive task assignments
- EC encoding is executed and verified in master topology
- File read-back validation confirms data integrity
The test uses unique absolute working directories for each worker to
prevent ID conflicts and ensure stable worker registration. Includes
proper cleanup and process management for reliable test execution.
* Improve maintenance system stability and task deduplication
- Add cross-type task deduplication to prevent concurrent maintenance
operations on the same volume (EC, balance, vacuum)
- Implement HasAnyTask check in ActiveTopology for better coordination
- Increase RequestTask timeout from 5s to 30s to prevent unnecessary
worker reconnections
- Add TaskTypeNone sentinel for generic task checks
- Update all task detectors to use HasAnyTask for conflict prevention
- Improve config persistence and schema handling
* Add GitHub Actions workflow for EC integration tests
Adds CI workflow that runs EC integration tests on push and pull requests
to master branch. The workflow:
- Triggers on changes to admin, worker, or test files
- Builds the weed binary
- Runs the EC integration test suite
- Uploads test logs as artifacts on failure for debugging
This ensures the maintenance system remains stable and worker-admin
integration is validated in CI.
* go version 1.24
* address comments
* Update maintenance_integration.go
* support seconds
* ec prioritize over balancing in tests
* Fix issue #7880: Tasks use Volume IDs instead of ip:port
When volume servers are registered with custom IDs, tasks were attempting
to connect using the ID instead of the actual ip:port address, causing
connection failures.
Modified task detection logic in balance, erasure coding, and vacuum tasks
to resolve volume server IDs to their actual ip:port addresses using
ActiveTopology information.
* Use server addresses directly instead of translating from IDs
Modified VolumeHealthMetrics to include ServerAddress field populated
directly from topology DataNodeInfo.Address. Updated task detection
logic to use addresses directly without runtime lookups.
Changes:
- Added ServerAddress field to VolumeHealthMetrics
- Updated maintenance scanner to populate ServerAddress
- Modified task detection to use ServerAddress for Node fields
- Updated DestinationPlan to include TargetAddress
- Removed runtime address lookups in favor of direct address usage
* Address PR comments: add ServerAddress field, improve error handling
- Add missing ServerAddress field to VolumeHealthMetrics struct
- Add warning in vacuum detection when server not found in topology
- Improve error handling in erasure coding to abort task if sources missing
- Make vacuum task stricter by skipping if server not found in topology
* Refactor: Extract common address resolution logic into shared utility
- Created weed/worker/tasks/util/address.go with ResolveServerAddress function
- Updated balance, erasure_coding, and vacuum detection to use the shared utility
- Removed code duplication and improved maintainability
- Consistent error handling across all task types
* Fix critical issues in task address resolution
- Vacuum: Require topology availability and fail if server not found (no fallback to ID)
- Ensure all task types consistently fail early when topology is incomplete
- Prevent creation of tasks that would fail due to missing server addresses
* Address additional PR feedback
- Add validation for empty addresses in ResolveServerAddress
- Remove redundant serverAddress variable in vacuum detection
- Improve robustness of address resolution
* Improve error logging in vacuum detection
- Include actual error details in log message for better diagnostics
- Make error messages consistent with other task types
* Fix worker reconnection race condition causing context canceled errors
Fixes#7824
This commit fixes critical connection stability issues between admin server
and workers that manifested as rapid reconnection cycles with 'context canceled'
errors, particularly after 24+ hours of operation in containerized environments.
Root Cause:
-----------
Race condition where TWO goroutines were calling stream.Recv() on the same
gRPC bidirectional stream concurrently:
1. sendRegistrationSync() started a goroutine that calls stream.Recv()
2. handleIncoming() also calls stream.Recv() in a loop
Per gRPC specification, only ONE goroutine can call Recv() on a stream at a
time. Concurrent Recv() calls cause undefined behavior, manifesting as
'context canceled' errors and stream corruption.
The race occurred during worker reconnection:
- Sometimes sendRegistrationSync goroutine read the registration response first (success)
- Sometimes handleIncoming read it first, causing sendRegistrationSync to timeout
- This left the stream in an inconsistent state, triggering 'context canceled' error
- The error triggered rapid reconnection attempts, creating a reconnection storm
Why it happened after 24 hours:
Container orchestration systems (Docker Swarm/Kubernetes) periodically restart
pods. Over time, workers reconnect multiple times. Each reconnection had a chance
of hitting the race condition. Eventually the race manifested and caused the
connection storm.
Changes:
--------
weed/worker/client.go:
- Start handleIncoming and handleOutgoing goroutines BEFORE sending registration
- Use sendRegistration() instead of sendRegistrationSync()
- Ensures only ONE goroutine (handleIncoming) calls stream.Recv()
- Eliminates race condition entirely
weed/admin/dash/worker_grpc_server.go:
- Clean up old connection when worker reconnects with same ID
- Cancel old connection context to stop its goroutines
- Prevents resource leaks and stale connection accumulation
Impact:
-------
Before: Random 'context canceled' errors during reconnection, rapid reconnection
cycles, resource leaks, requires manual restart to recover
After: Reliable reconnection, single Recv() goroutine, proper cleanup,
stable operation over 24+ hours
Testing:
--------
Build verified successful with no compilation errors.
How to reproduce the bug:
1. Start admin server and worker
2. Restart admin server (simulates container recreation)
3. Worker reconnects
4. Race condition may manifest, causing 'context canceled' error
5. Observe rapid reconnection cycles in logs
The fix is backward compatible and requires no configuration changes.
* Add MaxConnectionAge to gRPC server for Docker Swarm DNS handling
- Configure MaxConnectionAge and MaxConnectionAgeGrace for gRPC server
- Expand error detection in shouldInvalidateConnection for better cache invalidation
- Add connection lifecycle logging for debugging
* Add topology validation and nil-safety checks
- Add validation guards in UpdateTopology to prevent invalid updates
- Add nil-safety checks in rebuildIndexes
- Add GetDiskCount method for diagnostic purposes
* Fix worker registration race condition
- Reorder goroutine startup in WorkerStream to prevent race conditions
- Add defensive cleanup in unregisterWorker with panic-safe channel closing
* Add comprehensive topology update logging
- Enhance UpdateTopologyInfo with detailed logging of datacenter/node/disk counts
- Add metrics logging for topology changes
* Add periodic diagnostic status logging
- Implement topologyStatusLoop running every 5 minutes
- Add logTopologyStatus function reporting system metrics
- Run as background goroutine in maintenance manager
* Enhance master client connection logging
- Add connection timing logs in tryConnectToMaster
- Add reconnection attempt counting in KeepConnectedToMaster
- Improve diagnostic visibility for connection issues
* Remove unused sendRegistrationSync function
- Function is no longer called after switching to asynchronous sendRegistration
- Contains the problematic concurrent stream.Recv() pattern that caused race conditions
- Cleanup as suggested in PR review
* Clarify comment for channel closing during disconnection
- Improve comment to explain why channels are closed and their effect
- Make the code more self-documenting as suggested in PR review
* Address code review feedback: refactor and improvements
- Extract topology counting logic to shared helper function
CountTopologyResources() to eliminate duplication between
topology_management.go and maintenance_integration.go
- Use gRPC status codes for more robust error detection in
shouldInvalidateConnection(), falling back to string matching
for transport-level errors
- Add recover wrapper for channel close consistency in
cleanupStaleConnections() to match unregisterWorker() pattern
* Update grpc_client_server.go
* Fix data race on lastSeen field access
- Add mutex protection around conn.lastSeen = time.Now() in WorkerStream method
- Ensures thread-safe access consistent with cleanupStaleConnections
* Fix goroutine leaks in worker reconnection logic
- Close streamExit in reconnect() before creating new connection
- Close streamExit in attemptConnection() when sendRegistration fails
- Prevents orphaned handleOutgoing/handleIncoming goroutines from previous connections
- Ensures proper cleanup of goroutines competing for shared outgoing channel
* Minor cleanup improvements for consistency and clarity
- Remove redundant string checks in shouldInvalidateConnection that overlap with gRPC status codes
- Add recover block to Stop() method for consistency with other channel close operations
- Maintains valuable DNS and transport-specific error detection while eliminating redundancy
* Improve topology update error handling
- Return descriptive errors instead of silently preserving topology for invalid updates
- Change nil topologyInfo case to return 'rejected invalid topology update: nil topologyInfo'
- Change empty DataCenterInfos case to return 'rejected invalid topology update: empty DataCenterInfos (had X nodes, Y disks)'
- Keep existing glog.Warningf calls but append error details to logs before returning errors
- Allows callers to distinguish rejected updates and handle them appropriately
* Refactor safe channel closing into helper method
- Add safeCloseOutgoingChannel helper method to eliminate code duplication
- Replace repeated recover blocks in Stop, unregisterWorker, and cleanupStaleConnections
- Improves maintainability and ensures consistent error handling across all channel close operations
- Maintains same panic recovery behavior with contextual source identification
* Make connection invalidation string matching case-insensitive
- Convert error string to lowercase once for all string.Contains checks
- Improves robustness by catching error message variations from different sources
- Eliminates need for separate 'DNS resolution' and 'dns' checks
- Maintains same error detection coverage with better reliability
* Clean up warning logs in UpdateTopology to avoid duplicating error text
- Remove duplicated error phrases from glog.Warningf messages
- Keep concise contextual warnings that don't repeat the fmt.Errorf content
- Maintain same error returns for backward compatibility
* Add robust validation to prevent topology wipeout during master restart
- Reject topology updates with 0 nodes when current topology has nodes
- Prevents transient empty topology from overwriting valid state
- Improves resilience during master restart scenarios
- Maintains backward compatibility for legitimate empty topology updates
* refactor planning into task detection
* refactoring worker tasks
* refactor
* compiles, but only balance task is registered
* compiles, but has nil exception
* avoid nil logger
* add back ec task
* setting ec log directory
* implement balance and vacuum tasks
* EC tasks will no longer fail with "file not found" errors
* Use ReceiveFile API to send locally generated shards
* distributing shard files and ecx,ecj,vif files
* generate .ecx files correctly
* do not mount all possible EC shards (0-13) on every destination
* use constants
* delete all replicas
* rename files
* pass in volume size to tasks
* initial design
* added simulation as tests
* reorganized the codebase to move the simulation framework and tests into their own dedicated package
* integration test. ec worker task
* remove "enhanced" reference
* start master, volume servers, filer
Current Status
✅ Master: Healthy and running (port 9333)
✅ Filer: Healthy and running (port 8888)
✅ Volume Servers: All 6 servers running (ports 8080-8085)
🔄 Admin/Workers: Will start when dependencies are ready
* generate write load
* tasks are assigned
* admin start wtih grpc port. worker has its own working directory
* Update .gitignore
* working worker and admin. Task detection is not working yet.
* compiles, detection uses volumeSizeLimitMB from master
* compiles
* worker retries connecting to admin
* build and restart
* rendering pending tasks
* skip task ID column
* sticky worker id
* test canScheduleTaskNow
* worker reconnect to admin
* clean up logs
* worker register itself first
* worker can run ec work and report status
but:
1. one volume should not be repeatedly worked on.
2. ec shards needs to be distributed and source data should be deleted.
* move ec task logic
* listing ec shards
* local copy, ec. Need to distribute.
* ec is mostly working now
* distribution of ec shards needs improvement
* need configuration to enable ec
* show ec volumes
* interval field UI component
* rename
* integration test with vauuming
* garbage percentage threshold
* fix warning
* display ec shard sizes
* fix ec volumes list
* Update ui.go
* show default values
* ensure correct default value
* MaintenanceConfig use ConfigField
* use schema defined defaults
* config
* reduce duplication
* refactor to use BaseUIProvider
* each task register its schema
* checkECEncodingCandidate use ecDetector
* use vacuumDetector
* use volumeSizeLimitMB
* remove
remove
* remove unused
* refactor
* use new framework
* remove v2 reference
* refactor
* left menu can scroll now
* The maintenance manager was not being initialized when no data directory was configured for persistent storage.
* saving config
* Update task_config_schema_templ.go
* enable/disable tasks
* protobuf encoded task configurations
* fix system settings
* use ui component
* remove logs
* interface{} Reduction
* reduce interface{}
* reduce interface{}
* avoid from/to map
* reduce interface{}
* refactor
* keep it DRY
* added logging
* debug messages
* debug level
* debug
* show the log caller line
* use configured task policy
* log level
* handle admin heartbeat response
* Update worker.go
* fix EC rack and dc count
* Report task status to admin server
* fix task logging, simplify interface checking, use erasure_coding constants
* factor in empty volume server during task planning
* volume.list adds disk id
* track disk id also
* fix locking scheduled and manual scanning
* add active topology
* simplify task detector
* ec task completed, but shards are not showing up
* implement ec in ec_typed.go
* adjust log level
* dedup
* implementing ec copying shards and only ecx files
* use disk id when distributing ec shards
🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk
📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId
🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest
💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId])
📂 File System: EC shards and metadata land in the exact disk directory planned
* Delete original volume from all locations
* clean up existing shard locations
* local encoding and distributing
* Update docker/admin_integration/EC-TESTING-README.md
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* check volume id range
* simplify
* fix tests
* fix types
* clean up logs and tests
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* add ui for maintenance
* valid config loading. fix workers page.
* refactor
* grpc between admin and workers
* add a long-running bidirectional grpc call between admin and worker
* use the grpc call to heartbeat
* use the grpc call to communicate
* worker can remove the http client
* admin uses http port + 10000 as its default grpc port
* one task one package
* handles connection failures gracefully with exponential backoff
* grpc with insecure tls
* grpc with optional tls
* fix detecting tls
* change time config from nano seconds to seconds
* add tasks with 3 interfaces
* compiles reducing hard coded
* remove a couple of tasks
* remove hard coded references
* reduce hard coded values
* remove hard coded values
* remove hard coded from templ
* refactor maintenance package
* fix import cycle
* simplify
* simplify
* auto register
* auto register factory
* auto register task types
* self register types
* refactor
* simplify
* remove one task
* register ui
* lazy init executor factories
* use registered task types
* DefaultWorkerConfig remove hard coded task types
* remove more hard coded
* implement get maintenance task
* dynamic task configuration
* "System Settings" should only have system level settings
* adjust menu for tasks
* ensure menu not collapsed
* render job configuration well
* use templ for ui of task configuration
* fix ordering
* fix bugs
* saving duration in seconds
* use value and unit for duration
* Delete WORKER_REFACTORING_PLAN.md
* Delete maintenance.json
* Delete custom_worker_example.go
* remove address from workers
* remove old code from ec task
* remove creating collection button
* reconnect with exponential backoff
* worker use security.toml
* start admin server with tls info from security.toml
* fix "weed admin" cli description
