* fix: decrypt SSE-encrypted objects in S3 replication sink
* fix: add SSE decryption support to GCS, Azure, B2, Local sinks
* fix: return error instead of warning for SSE-C objects during replication
* fix: close readers after upload to prevent resource leaks
* fix: return error for unknown SSE types instead of passing through ciphertext
* refactor(repl_util): extract CloseReader/CloseMaybeDecryptedReader helpers
The io.Closer close-on-error and defer-close pattern was duplicated in
copyWithDecryption and the S3 sink. Extract exported helpers to keep a
single implementation and prevent future divergence.
* fix(repl_util): warn on mixed SSE types across chunks in detectSSEType
detectSSEType previously returned the SSE type of the first encrypted
chunk without inspecting the rest. If an entry somehow has chunks with
different SSE types, only the first type's decryption would be applied.
Now scans all chunks and logs a warning on mismatch.
* fix(repl_util): decrypt inline SSE objects during replication
Small SSE-encrypted objects stored in entry.Content were being copied
as ciphertext because:
1. detectSSEType only checked chunk metadata, but inline objects have
no chunks — now falls back to checking entry.Extended for SSE keys
2. Non-S3 sinks short-circuited on len(entry.Content)>0, bypassing
the decryption path — now call MaybeDecryptContent before writing
Adds MaybeDecryptContent helper for decrypting inline byte content.
* fix(repl_util): add KMS initialization for replication SSE decryption
SSE-KMS decryption was not wired up for filer.backup — the only
initialization was for SSE-S3 key manager. CreateSSEKMSDecryptedReader
requires a global KMS provider which is only loaded by the S3 API
auth-config path.
Add InitializeSSEForReplication helper that initializes both SSE-S3
(from filer KEK) and SSE-KMS (from Viper config [kms] section /
WEED_KMS_* env vars). Replace the SSE-S3-only init in filer_backup.go.
* fix(replicator): initialize SSE decryption for filer.replicate
The SSE decryption setup was only added to filer_backup.go, but the
notification-based replicator (filer.replicate) uses the same sinks
and was missing the required initialization. Add SSE init in
NewReplicator so filer.replicate can decrypt SSE objects.
* refactor(repl_util): fold entry param into CopyFromChunkViews
Remove the CopyFromChunkViewsWithEntry wrapper and add the entry
parameter directly to CopyFromChunkViews, since all callers already
pass it.
* fix(repl_util): guard SSE init with sync.Once, error on mixed SSE types
InitializeWithFiler overwrites the global superKey on every call.
Wrap InitializeSSEForReplication with sync.Once so repeated calls
(e.g. from NewReplicator) are safe.
detectSSEType now returns an error instead of logging a warning when
chunks have inconsistent SSE types, so replication aborts rather than
silently applying the wrong decryption to some chunks.
* fix(repl_util): allow SSE init retry, detect conflicting metadata, add tests
- Replace sync.Once with mutex+bool so transient failures (e.g. filer
unreachable) don't permanently prevent initialization. Only successful
init flips the flag; failed attempts allow retries.
- Remove v.IsSet("kms") guard that prevented env-only KMS configs
(WEED_KMS_*) from being detected. Always attempt KMS loading and let
LoadConfigurations handle "no config found".
- detectSSEType now checks for conflicting extended metadata keys
(e.g. both SeaweedFSSSES3Key and SeaweedFSSSEKMSKey present) and
returns an error instead of silently picking the first match.
- Add table-driven tests for detectSSEType, MaybeDecryptReader, and
MaybeDecryptContent covering plaintext, uniform SSE, mixed chunks,
inline SSE via extended metadata, conflicting metadata, and SSE-C.
* test(repl_util): add SSE-S3 and SSE-KMS integration tests
Add round-trip encryption/decryption tests:
- SSE-S3: encrypt with CreateSSES3EncryptedReader, decrypt with
CreateSSES3DecryptedReader, verify plaintext matches
- SSE-KMS: encrypt with AES-CTR, wire a mock KMSProvider via
SetGlobalKMSProvider, build serialized KMS metadata, verify
MaybeDecryptReader and MaybeDecryptContent produce correct plaintext
Fix existing tests to check io.ReadAll errors.
* test(repl_util): exercise full SSE-S3 path through MaybeDecryptReader
Replace direct CreateSSES3DecryptedReader calls with end-to-end tests
that go through MaybeDecryptReader → decryptSSES3 →
DeserializeSSES3Metadata → GetSSES3IV → CreateSSES3DecryptedReader.
Uses WEED_S3_SSE_KEK env var + a mock filer client to initialize the
global key manager with a test KEK, then SerializeSSES3Metadata to
build proper envelope-encrypted metadata. Cleanup restores the key
manager state.
* fix(localsink): write to temp file to prevent truncated replicas
The local sink truncated the destination file before writing content.
If decryption or chunk copy failed, the file was left empty/truncated,
destroying the previous replica.
Write to a temp file in the same directory and atomically rename on
success. On any error the temp file is cleaned up and the existing
replica is untouched.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(kafka): resolve consumer group resumption timeout in e2e tests
Three issues caused ConsumerGroupResumption to time out when the second
consumer tried to resume from committed offsets:
1. ForceCompleteRebalance deadlock: performCleanup() held group.Mu.Lock
then called ForceCompleteRebalance() which tried to acquire the same
lock — a guaranteed deadlock on Go's non-reentrant sync.Mutex. Fixed
by requiring callers to hold the lock (matching actual call sites).
2. Unbounded fallback fetch: when the multi-batch fetch timed out, the
fallback GetStoredRecords call used the connection context (no
deadline). A slow broker gRPC call could block the data-plane
goroutine indefinitely, causing head-of-line blocking for all
responses on that connection. Fixed with a 10-second timeout.
3. HWM lookup failure caused empty responses: after a consumer leaves
and the partition is deactivated, GetLatestOffset can fail. The
fetch handler treated this as "no data" and entered the long-poll
loop (up to 10s × 4 retries = 40s timeout). Fixed by assuming data
may exist when HWM lookup fails, so the actual fetch determines
availability.
* fix(kafka): address review feedback on HWM sentinel and fallback timeout
- Don't expose synthetic HWM (requestedOffset+1) to clients; keep
result.highWaterMark at 0 when the real HWM lookup fails.
- Tie fallback timeout to client's MaxWaitTime instead of a fixed 10s,
so one slow partition doesn't hold the reader beyond the request budget.
* fix(kafka): use large HWM sentinel and clamp fallback timeout
- Use requestedOffset+10000 as sentinel HWM instead of +1, so
FetchMultipleBatches doesn't artificially limit to 1 record.
- Add 2s floor to fallback timeout so disk reads via gRPC have
a reasonable chance even when maxWaitMs is small or zero.
* fix(kafka): use MaxInt64 sentinel and derive HWM from fetch result
- Use math.MaxInt64 as HWM sentinel to avoid integer overflow risk
(previously requestedOffset+10000 could wrap on large offsets).
- After the fetch, derive a meaningful HWM from newOffset so the
client never sees MaxInt64 or 0 in the response.
* fix(kafka): use remaining time budget for fallback fetch
The fallback was restarting the full maxWaitMs budget even though the
multi-batch fetch already consumed part of it. Now compute remaining
time from either the parent context deadline or maxWaitMs minus
elapsed, skip the fallback if budget is exhausted, and clamp to
[2s, 10s] bounds.
* fix(gcssink): prevent empty object finalization on write failure
The GCS writer was created unconditionally with defer wc.Close(),
which finalizes the upload even when content decryption or copy
fails. This silently overwrites valid objects with empty data.
Remove the unconditional defer, explicitly close on success to
propagate errors, and delete the object on write failure.
* fix(gcssink): use context cancellation instead of obj.Delete on failure
obj.Delete() after a failed write would delete the existing object at
that key, causing data loss on updates. Use a cancelable context
instead — cancelling before Close() aborts the GCS upload without
touching any pre-existing object.
* fix(azuresink): delete freshly created blob on write failure
appendBlobClient.Create() runs before content decryption and copy.
If MaybeDecryptContent or CopyFromChunkViews fails, an empty blob
is left behind, silently replacing any previous valid data. Add
cleanup that deletes the blob on content write errors when we were
the ones who created it.
* fix(azuresink): track recreated blobs for cleanup on write failure
handleExistingBlob deletes and recreates the blob when overwrite is
needed, but freshlyCreated was only set on the initial Create success
path. Set freshlyCreated = needsWrite after handleExistingBlob so
recreated blobs are also cleaned up on content write failure.
* fix(s3): apply PutObject multipart expansion to STS session policy evaluation (#8929)
PR #8445 added logic to implicitly grant multipart upload actions when
s3:PutObject is authorized, but only in the S3 API policy engine's
CompiledStatement.MatchesAction(). STS session policies are evaluated
through the IAM policy engine's matchesActions() -> awsIAMMatch() path,
which did plain pattern matching without the multipart expansion.
Add the same multipart expansion logic to the IAM policy engine's
matchesActions() so that session policies containing s3:PutObject
correctly allow multipart upload operations.
* fix: make multipart action set lookup case-insensitive and optimize
Address PR review feedback:
- Lowercase multipartActionSet keys and use strings.ToLower for lookup,
since AWS IAM actions are case-insensitive
- Only check for s3:PutObject permission when the requested action is
actually a multipart action, avoiding unnecessary awsIAMMatch calls
- Add test case for case-insensitive multipart action matching
* fix(admin): reduce memory usage and verbose logging for large clusters (#8919)
The admin server used excessive memory and produced thousands of log lines
on clusters with many volumes (e.g., 33k volumes). Three root causes:
1. Scanner duplicated all volume metrics: getVolumeHealthMetrics() created
VolumeHealthMetrics objects, then convertToTaskMetrics() copied them all
into identical types.VolumeHealthMetrics. Now uses the task-system type
directly, eliminating the duplicate allocation and removing convertToTaskMetrics.
2. All previous task states loaded at startup: LoadTasksFromPersistence read
and deserialized every .pb file from disk, logging each one. With thousands
of balance tasks persisted, this caused massive startup I/O, memory usage,
and log noise (including unguarded DEBUG glog.Infof per task). Now starts
with an empty queue — the scanner re-detects current needs from live cluster
state. Terminal tasks are purged from memory and disk when new scan results
arrive.
3. Verbose per-volume/per-node logging: V(2) and V(3) logs produced thousands
of lines per scan. Per-volume logs bumped to V(4), per-node/rack/disk logs
bumped to V(3). Topology summary now logs counts instead of full node ID arrays.
Also removes lastTopologyInfo field from MaintenanceScanner — the raw protobuf
topology is returned as a local value and not retained between 30-minute scans.
* fix(admin): delete stale task files at startup, add DeleteAllTaskStates
Old task .pb files from previous runs were left on disk. The periodic
CleanupCompletedTasks still loads all files to find completed ones —
the same expensive 4GB path from the pprof profile.
Now at startup, DeleteAllTaskStates removes all .pb files by scanning
the directory without reading or deserializing them. The scanner will
re-detect any tasks still needed from live cluster state.
* fix(admin): don't persist terminal tasks to disk
CompleteTask was saving failed/completed tasks to disk where they'd
accumulate. The periodic cleanup only triggered for completed tasks,
not failed ones. Now terminal tasks are deleted from disk immediately
and only kept in memory for the current session's UI.
* fix(admin): cap in-memory tasks to 100 per job type
Without a limit, the task map grows unbounded — balance could create
thousands of pending tasks for a cluster with many imbalanced volumes.
Now AddTask rejects new tasks when a job type already has 100 in the
queue. The scanner will re-detect skipped volumes on the next scan.
* fix(admin): address PR review - memory-only purge, active-only capacity
- purgeTerminalTasks now only cleans in-memory map (terminal tasks are
already deleted from disk by CompleteTask)
- Per-type capacity limit counts only active tasks (pending/assigned/
in_progress), not terminal ones
- When at capacity, purge terminal tasks first before rejecting
* fix(admin): fix orphaned comment, add TaskStatusCancelled to terminal switch
- Move hasQueuedOrActiveTaskForVolume comment to its function definition
- Add TaskStatusCancelled to the terminal state switch in CompleteTask
so cancelled task files are deleted from disk
* fix(master): fast resume state and default resumeState to true
When resumeState is enabled in single-master mode, the raft server had
existing log entries so the self-join path couldn't promote to leader.
The server waited the full election timeout (10-20s) before self-electing.
Fix by temporarily setting election timeout to 1ms before Start() when
in single-master + resumeState mode with existing log, then restoring
the original timeout after leader election. This makes resume near-instant.
Also change the default for resumeState from false to true across all
CLI commands (master, mini, server) so state is preserved by default.
* fix(master): prevent fastResume goroutine from hanging forever
Use defer to guarantee election timeout is always restored, and bound
the polling loop with a timeout so it cannot spin indefinitely if
leader election never succeeds.
* fix(master): use ticker instead of time.After in fastResume polling loop
fix(admin): use gRPC address for current server in RaftListClusterServers
The old Raft implementation was returning the HTTP address
(ms.option.Master) for the current server, while peers used gRPC
addresses (peer.ConnectionString). The Admin UI's GetClusterMasters()
converts all addresses from gRPC to HTTP via GrpcAddressToServerAddress
(port - 10000), which produced a negative port (-667) for the current
server since its address was already in HTTP format (port 9333).
Use ToGrpcAddress() for consistency with both HashicorpRaft (which
stores gRPC addresses) and old Raft peers.
Fixes#8921
* feat(admin): add profiling options for debugging high memory/CPU usage
Add -debug, -debug.port, -cpuprofile, and -memprofile flags to the admin
command, matching the profiling support already available in master, volume,
and other server commands. This enables investigation of resource usage
issues like #8919.
* refactor(admin): move profiling flags into AdminOptions struct
Move cpuprofile and memprofile flags from global variables into the
AdminOptions struct and init() function for consistency with other flags.
* fix(debug): bind pprof server to localhost only and document profiling flags
StartDebugServer was binding to all interfaces (0.0.0.0), exposing
runtime profiling data to the network. Restrict to 127.0.0.1 since
this is a development/debugging tool.
Also add a "Debugging and Profiling" section to the admin command's
help text documenting the new flags.
* fix(s3): include static identities in listing operations
Static identities loaded from -s3.config file were only stored in the
S3 API server's in-memory state. Listing operations (s3.configure shell
command, aws iam list-users) queried the credential manager which only
returned dynamic identities from the backend store.
Register static identities with the credential manager after loading
so they are included in LoadConfiguration and ListUsers results, and
filtered out before SaveConfiguration to avoid persisting them to the
dynamic store.
Fixes https://github.com/seaweedfs/seaweedfs/discussions/8896
* fix: avoid mutating caller's config and defensive copies
- SaveConfiguration: use shallow struct copy instead of mutating the
caller's config.Identities field
- SetStaticIdentities: skip nil entries to avoid panics
- GetStaticIdentities: defensively copy PolicyNames slice to avoid
aliasing the original
* fix: filter nil static identities and sync on config reload
- SetStaticIdentities: filter nil entries from the stored slice (not
just from staticNames) to prevent panics in LoadConfiguration/ListUsers
- Extract updateCredentialManagerStaticIdentities helper and call it
from both startup and the grace.OnReload handler so the credential
manager's static snapshot stays current after config file reloads
* fix: add mutex for static identity fields and fix ListUsers for store callers
- Add sync.RWMutex to protect staticIdentities/staticNames against
concurrent reads during config reload
- Revert CredentialManager.ListUsers to return only store users, since
internal callers (e.g. DeletePolicy) look up each user in the store
and fail on non-existent static entries
- Merge static usernames in the filer gRPC ListUsers handler instead,
via the new GetStaticUsernames method
- Fix CI: TestIAMPolicyManagement/managed_policy_crud_lifecycle was
failing because DeletePolicy iterated static users that don't exist
in the store
* fix: show static identities in admin UI and weed shell
The admin UI and weed shell s3.configure command query the filer's
credential manager via gRPC, which is a separate instance from the S3
server's credential manager. Static identities were only registered
on the S3 server's credential manager, so they never appeared in the
filer's responses.
- Add CredentialManager.LoadS3ConfigFile to parse a static S3 config
file and register its identities
- Add FilerOptions.s3ConfigFile so the filer can load the same static
config that the S3 server uses
- Wire s3ConfigFile through in weed mini and weed server modes
- Merge static usernames in filer gRPC ListUsers handler
- Add CredentialManager.GetStaticUsernames helper
- Add sync.RWMutex to protect concurrent access to static identity
fields
- Avoid importing weed/filer from weed/credential (which pulled in
filer store init() registrations and broke test isolation)
- Add docker/compose/s3_static_users_example.json
* fix(admin): make static users read-only in admin UI
Static users loaded from the -s3.config file should not be editable
or deletable through the admin UI since they are managed via the
config file.
- Add IsStatic field to ObjectStoreUser, set from credential manager
- Hide edit, delete, and access key buttons for static users in the
users table template
- Show a "static" badge next to static user names
- Return 403 Forbidden from UpdateUser and DeleteUser API handlers
when the target user is a static identity
* fix(admin): show details for static users
GetObjectStoreUserDetails called credentialManager.GetUser which only
queries the dynamic store. For static users this returned
ErrUserNotFound. Fall back to GetStaticIdentity when the store lookup
fails.
* fix(admin): load static S3 identities in admin server
The admin server has its own credential manager (gRPC store) which is
a separate instance from the S3 server's and filer's. It had no static
identity data, so IsStaticIdentity returned false (edit/delete buttons
shown) and GetStaticIdentity returned nil (details page failed).
Pass the -s3.config file path through to the admin server and call
LoadS3ConfigFile on its credential manager, matching the approach
used for the filer.
* fix: use protobuf is_static field instead of passing config file path
The previous approach passed -s3.config file path to every component
(filer, admin). This is wrong because the admin server should not need
to know about S3 config files.
Instead, add an is_static field to the Identity protobuf message.
The field is set when static identities are serialized (in
GetStaticIdentities and LoadS3ConfigFile). Any gRPC client that loads
configuration via GetConfiguration automatically sees which identities
are static, without needing the config file.
- Add is_static field (tag 8) to iam_pb.Identity proto message
- Set IsStatic=true in GetStaticIdentities and LoadS3ConfigFile
- Admin GetObjectStoreUsers reads identity.IsStatic from proto
- Admin IsStaticUser helper loads config via gRPC to check the flag
- Filer GetUser gRPC handler falls back to GetStaticIdentity
- Remove s3ConfigFile from AdminOptions and NewAdminServer signature
* feat(s3): add concurrent chunk prefetch for large file downloads
Add a pipe-based prefetch pipeline that overlaps chunk fetching with
response writing during S3 GetObject, SSE downloads, and filer proxy.
While chunk N streams to the HTTP response, fetch goroutines for the
next K chunks establish HTTP connections to volume servers ahead of
time, eliminating the RTT gap between sequential chunk fetches.
Uses io.Pipe for minimal memory overhead (~1MB per download regardless
of chunk size, vs buffering entire chunks). Also increases the
streaming read buffer from 64KB to 256KB to reduce syscall overhead.
Benchmark results (64KB chunks, prefetch=4):
- 0ms latency: 1058 → 2362 MB/s (2.2× faster)
- 5ms latency: 11.0 → 41.7 MB/s (3.8× faster)
- 10ms latency: 5.9 → 23.3 MB/s (4.0× faster)
- 20ms latency: 3.1 → 12.1 MB/s (3.9× faster)
* fix: address review feedback for prefetch pipeline
- Fix data race: use *chunkPipeResult (pointer) on channel to avoid
copying struct while fetch goroutines write to it. Confirmed clean
with -race detector.
- Remove concurrent map write: retryWithCacheInvalidation no longer
updates fileId2Url map. Producer only reads it; consumer never writes.
- Use mem.Allocate/mem.Free for copy buffer to reduce GC pressure.
- Add local cancellable context so consumer errors (client disconnect)
immediately stop the producer and all in-flight fetch goroutines.
* fix(test): remove dead code and add Range header support in test server
- Remove unused allData variable in makeChunksAndServer
- Add Range header handling to createTestServer for partial chunk
read coverage (206 Partial Content, 416 Range Not Satisfiable)
* fix: correct retry condition and goroutine leak in prefetch pipeline
- Fix retry condition: use result.fetchErr/result.written instead of
copied to decide cache-invalidation retry. The old condition wrongly
triggered retry when the fetch succeeded but the response writer
failed on the first write (copied==0 despite fetcher having data).
Now matches the sequential path (stream.go:197) which checks whether
the fetcher itself wrote zero bytes.
- Fix goroutine leak: when the producer's send to the results channel
is interrupted by context cancellation, the fetch goroutine was
already launched but the result was never sent to the channel. The
drain loop couldn't handle it. Now waits on result.done before
returning so every fetch goroutine is properly awaited.
* feat(s3): store and return checksum headers for additional checksum algorithms
When clients upload with --checksum-algorithm (SHA256, CRC32, etc.),
SeaweedFS validated the checksum but discarded it. The checksum was
never stored in metadata or returned in PUT/HEAD/GET responses.
Now the checksum is computed alongside MD5 during upload, stored in
entry extended attributes, and returned as the appropriate
x-amz-checksum-* header in all responses.
Fixes#8911
* fix(s3): address review feedback and CI failures for checksum support
- Gate GET/HEAD checksum response headers on x-amz-checksum-mode: ENABLED
per AWS S3 spec, fixing FlexibleChecksumError on ranged GETs and
multipart copies
- Verify computed checksum against client-provided header value for
non-chunked uploads, returning BadDigest on mismatch
- Add nil check for getCheckSumWriter to prevent panic
- Handle comma-separated values in X-Amz-Trailer header
- Use ordered slice instead of map for deterministic checksum header
selection; extract shared mappings into package-level vars
* fix(s3): skip checksum header for ranged GET responses
The stored checksum covers the full object. Returning it for ranged
(partial) responses causes SDK checksum validation failures because the
SDK validates the header value against the partial content received.
Skip emitting x-amz-checksum-* headers when a Range request header is
present, fixing PyArrow large file read failures.
* fix(s3): reject unsupported checksum algorithm with 400
detectRequestedChecksumAlgorithm now returns an error code when
x-amz-sdk-checksum-algorithm or x-amz-checksum-algorithm contains
an unsupported value, instead of silently ignoring it.
* feat(s3): compute composite checksum for multipart uploads
Store the checksum algorithm during CreateMultipartUpload, then during
CompleteMultipartUpload compute a composite checksum from per-part
checksums following the AWS S3 spec: concatenate raw per-part checksums,
hash with the same algorithm, format as "base64-N" where N is part count.
The composite checksum is persisted on the final object entry and
returned in HEAD/GET responses (gated on x-amz-checksum-mode: ENABLED).
Reuses existing per-part checksum storage from putToFiler and the
getCheckSumWriter/checksumHeaders infrastructure.
* fix(s3): validate checksum algorithm in CreateMultipartUpload, error on missing part checksums
- Move detectRequestedChecksumAlgorithm call before mkdir callback so
an unsupported algorithm returns 400 before the upload is created
- Change computeCompositeChecksum to return an error when a part is
missing its checksum (the upload was initiated with a checksum
algorithm, so all parts must have checksums)
- Propagate the error as ErrInvalidPart in CompleteMultipartUpload
* fix(s3): return checksum header in CompleteMultipartUpload response, validate per-part algorithm
- Add ChecksumHeaderName/ChecksumValue fields to CompleteMultipartUploadResult
and set the x-amz-checksum-* HTTP response header in the handler, matching
the AWS S3 CompleteMultipartUpload response spec
- Validate that each part's stored checksum algorithm matches the upload's
expected algorithm before assembling the composite checksum; return an
error if a part was uploaded with a different algorithm
* fix(filer): remove cancellation guard from RollbackTransaction and clean up #8909
RollbackTransaction is a cleanup operation that must succeed even when
the context is cancelled — guarding it causes the exact orphaned state
that #8909 was trying to prevent.
Also:
- Use single-evaluation `if err := ctx.Err(); err != nil` pattern
instead of double-calling ctx.Err()
- Remove spurious blank lines before guards
- Add context.DeadlineExceeded test coverage
- Simplify tests from ~230 lines to ~130 lines
* fix(filer): call cancel() in expiredCtx and test rollback with expired context
- Call cancel() instead of suppressing it to avoid leaking timer resources
- Test RollbackTransaction with both cancelled and expired contexts
* chore: remove unreachable dead code across the codebase
Remove ~50,000 lines of unreachable code identified by static analysis.
Major removals:
- weed/filer/redis_lua: entire unused Redis Lua filer store implementation
- weed/wdclient/net2, resource_pool: unused connection/resource pool packages
- weed/plugin/worker/lifecycle: unused lifecycle plugin worker
- weed/s3api: unused S3 policy templates, presigned URL IAM, streaming copy,
multipart IAM, key rotation, and various SSE helper functions
- weed/mq/kafka: unused partition mapping, compression, schema, and protocol functions
- weed/mq/offset: unused SQL storage and migration code
- weed/worker: unused registry, task, and monitoring functions
- weed/query: unused SQL engine, parquet scanner, and type functions
- weed/shell: unused EC proportional rebalance functions
- weed/storage/erasure_coding/distribution: unused distribution analysis functions
- Individual unreachable functions removed from 150+ files across admin,
credential, filer, iam, kms, mount, mq, operation, pb, s3api, server,
shell, storage, topology, and util packages
* fix(s3): reset shared memory store in IAM test to prevent flaky failure
TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey was flaky because
the MemoryStore credential backend is a singleton registered via init().
Earlier tests that create anonymous identities pollute the shared store,
causing LookupAnonymous() to unexpectedly return true.
Fix by calling Reset() on the memory store before the test runs.
* style: run gofmt on changed files
* fix: restore KMS functions used by integration tests
* fix(plugin): prevent panic on send to closed worker session channel
The Plugin.sendToWorker method could panic with "send on closed channel"
when a worker disconnected while a message was being sent. The race was
between streamSession.close() closing the outgoing channel and sendToWorker
writing to it concurrently.
Add a done channel to streamSession that is closed before the outgoing
channel, and check it in sendToWorker's select to safely detect closed
sessions without panicking.
* feat(s3): support WEED_S3_SSE_KEY env var for SSE-S3 KEK
Add support for providing the SSE-S3 Key Encryption Key (KEK) via the
WEED_S3_SSE_KEY environment variable (hex-encoded 256-bit key). This
avoids storing the master key in plaintext on the filer at /etc/s3/sse_kek.
Key source priority:
1. WEED_S3_SSE_KEY environment variable (recommended)
2. Existing filer KEK at /etc/s3/sse_kek (backward compatible)
3. Auto-generate and save to filer (deprecated for new deployments)
Existing deployments with a filer-stored KEK continue to work unchanged.
A deprecation warning is logged when auto-generating a new filer KEK.
* refactor(s3): derive KEK from any string via HKDF instead of requiring hex
Accept any secret string in WEED_S3_SSE_KEY and derive a 256-bit key
using HKDF-SHA256 instead of requiring a hex-encoded key. This is
simpler for users — no need to generate hex, just set a passphrase.
* feat(s3): add WEED_S3_SSE_KEK and WEED_S3_SSE_KEY env vars for KEK
Two env vars for providing the SSE-S3 Key Encryption Key:
- WEED_S3_SSE_KEK: hex-encoded, same format as /etc/s3/sse_kek.
If the filer file also exists, they must match.
- WEED_S3_SSE_KEY: any string, 256-bit key derived via HKDF-SHA256.
Refuses to start if /etc/s3/sse_kek exists (must delete first).
Only one may be set. Existing filer-stored KEKs continue to work.
Auto-generating and storing new KEKs on filer is deprecated.
* fix(s3): stop auto-generating KEK, fail only when SSE-S3 is used
Instead of auto-generating a KEK and storing it on the filer when no
key source is configured, simply leave SSE-S3 disabled. Encrypt and
decrypt operations return a clear error directing the user to set
WEED_S3_SSE_KEK or WEED_S3_SSE_KEY.
* refactor(s3): move SSE-S3 KEK config to security.toml
Move KEK configuration from standalone env vars to security.toml's new
[sse_s3] section, following the same pattern as JWT keys and TLS certs.
[sse_s3]
kek = "" # hex-encoded 256-bit key (same format as /etc/s3/sse_kek)
key = "" # any string, HKDF-derived
Viper's WEED_ prefix auto-mapping provides env var support:
WEED_SSE_S3_KEK and WEED_SSE_S3_KEY.
All existing behavior is preserved: filer KEK fallback, mismatch
detection, and HKDF derivation.
* refactor(s3): rename SSE-S3 config keys to s3.sse.kek / s3.sse.key
Use [s3.sse] section in security.toml, matching the existing naming
convention (e.g. [s3.*]). Env vars: WEED_S3_SSE_KEK, WEED_S3_SSE_KEY.
* fix(s3): address code review findings for SSE-S3 KEK
- Don't hold mutex during filer retry loop (up to 20s of sleep).
Lock only to write filerClient and superKey.
- Remove dead generateAndSaveSuperKeyToFiler and unused constants.
- Return error from deriveKeyFromSecret instead of ignoring it.
- Fix outdated doc comment on InitializeWithFiler.
- Use t.Setenv in tests instead of manual os.Setenv/Unsetenv.
* fix(s3): don't block startup on filer errors when KEK is configured
- When s3.sse.kek is set, a temporarily unreachable filer no longer
prevents startup. The filer consistency check becomes best-effort
with a warning.
- Same treatment for s3.sse.key: filer unreachable logs a warning
instead of failing.
- Rewrite error messages to suggest migration instead of file deletion,
avoiding the risk of orphaning encrypted data.
Finding 3 (restore auto-generation) intentionally skipped — auto-gen
was removed by design to avoid storing plaintext KEK on filer.
* fix(test): set WEED_S3_SSE_KEY in SSE integration test server startup
SSE-S3 no longer auto-generates a KEK, so integration tests must
provide one. Set WEED_S3_SSE_KEY=test-sse-s3-key in all weed mini
invocations in the test Makefile.
* fix(s3): use URL-safe secret keys for admin dashboard users and service accounts
The dashboard's generateSecretKey() used base64.StdEncoding which produces
+, /, and = characters that break S3 signature authentication. Reuse the
IAM package's GenerateSecretAccessKey() which was already fixed in #7990.
Fixes#8898
* fix: handle error from GenerateSecretAccessKey instead of ignoring it
* feat(s3): add STS GetFederationToken support
Implement the AWS STS GetFederationToken API, which allows long-term IAM
users to obtain temporary credentials scoped down by an optional inline
session policy. This is useful for server-side applications that mint
per-user temporary credentials.
Key behaviors:
- Requires SigV4 authentication from a long-term IAM user
- Rejects calls from temporary credentials (session tokens)
- Name parameter (2-64 chars) identifies the federated user
- DurationSeconds supports 900-129600 (15 min to 36 hours, default 12h)
- Optional inline session policy for permission scoping
- Caller's attached policies are embedded in the JWT token
- Returns federated user ARN: arn:aws:sts::<account>:federated-user/<Name>
No performance impact on the S3 hot path — credential vending is a
separate control-plane operation, and all policy data is embedded in
the stateless JWT token.
* fix(s3): address GetFederationToken PR review feedback
- Fix Name validation: max 32 chars (not 64) per AWS spec, add regex
validation for [\w+=,.@-]+ character whitelist
- Refactor parseDurationSeconds into parseDurationSecondsWithBounds to
eliminate duplicated duration parsing logic
- Add sts:GetFederationToken permission check via VerifyActionPermission
mirroring the AssumeRole authorization pattern
- Change GetPoliciesForUser to return ([]string, error) so callers fail
closed on policy-resolution failures instead of silently returning nil
- Move temporary-credentials rejection before SigV4 verification for
early rejection and proper test coverage
- Update tests: verify specific error message for temp cred rejection,
add regex validation test cases (spaces, slashes rejected)
* refactor(s3): use sts.Action* constants instead of hard-coded strings
Replace hard-coded "sts:AssumeRole" and "sts:GetFederationToken" strings
in VerifyActionPermission calls with sts.ActionAssumeRole and
sts.ActionGetFederationToken package constants.
* fix(s3): pass through sts: prefix in action resolver and merge policies
Two fixes:
1. mapBaseActionToS3Format now passes through "sts:" prefix alongside
"s3:" and "iam:", preventing sts:GetFederationToken from being
rewritten to s3:sts:GetFederationToken in VerifyActionPermission.
This also fixes the existing sts:AssumeRole permission checks.
2. GetFederationToken policy embedding now merges identity.PolicyNames
(from SigV4 identity) with policies from the IAM manager (which may
include group-attached policies), deduplicated via a map. Previously
the IAM manager lookup was skipped when identity.PolicyNames was
non-empty, causing group policies to be omitted from the token.
* test(s3): add integration tests for sts: action passthrough and policy merge
Action resolver tests:
- TestMapBaseActionToS3Format_ServicePrefixPassthrough: verifies s3:, iam:,
and sts: prefixed actions pass through unchanged while coarse actions
(Read, Write) are mapped to S3 format
- TestResolveS3Action_STSActionsPassthrough: verifies sts:AssumeRole,
sts:GetFederationToken, sts:GetCallerIdentity pass through ResolveS3Action
unchanged with both nil and real HTTP requests
Policy merge tests:
- TestGetFederationToken_GetPoliciesForUser: tests IAMManager.GetPoliciesForUser
with no user store (error), missing user, user with policies, user without
- TestGetFederationToken_PolicyMergeAndDedup: tests that identity.PolicyNames
and IAM-manager-resolved policies are merged and deduplicated (SharedPolicy
appears in both sources, result has 3 unique policies)
- TestGetFederationToken_PolicyMergeNoManager: tests that when IAM manager is
unavailable, identity.PolicyNames alone are embedded
* test(s3): add end-to-end integration tests for GetFederationToken
Add integration tests that call GetFederationToken using real AWS SigV4
signed HTTP requests against a running SeaweedFS instance, following the
existing pattern in test/s3/iam/s3_sts_assume_role_test.go.
Tests:
- TestSTSGetFederationTokenValidation: missing name, name too short/long,
invalid characters, duration too short/long, malformed policy, anonymous
rejection (7 subtests)
- TestSTSGetFederationTokenRejectTemporaryCredentials: obtains temp creds
via AssumeRole then verifies GetFederationToken rejects them
- TestSTSGetFederationTokenSuccess: basic success, custom 1h duration,
36h max duration with expiration time verification
- TestSTSGetFederationTokenWithSessionPolicy: creates a bucket, obtains
federated creds with GetObject-only session policy, verifies GetObject
succeeds and PutObject is denied using the AWS SDK S3 client
* fix(s3): skip directories before marker in ListObjectVersions pagination
ListObjectVersions was re-traversing the entire directory tree from the
beginning on every paginated request, only skipping entries at the leaf
level. For buckets with millions of objects in deep hierarchies, this
caused exponentially slower responses as pagination progressed.
Two optimizations:
1. Use keyMarker to compute a startFrom position at each directory level,
skipping directly to the relevant entry instead of scanning from the
beginning (mirroring how ListObjects uses marker descent).
2. Skip recursing into subdirectories whose keys are entirely before the
keyMarker.
Changes per-page cost from O(entries_before_marker) to O(tree_depth).
* test(s3): add integration test for deep-hierarchy version listing pagination
Adds TestVersioningPaginationDeepDirectoryHierarchy which creates objects
across 20 subdirectories at depth 6 (mimicking Veeam 365 backup layout)
and paginates through them with small maxKeys. Verifies correctness
(no duplicates, sorted order, all objects found) and checks that later
pages don't take dramatically longer than earlier ones — the symptom
of the pre-fix re-traversal bug. Also tests delimiter+pagination
interaction across subdirectories.
* test(s3): strengthen deep-hierarchy pagination assertions
- Replace timing warning (t.Logf) with a failing assertion (t.Errorf)
so pagination regressions actually fail the test.
- Replace generic count/uniqueness/sort checks on CommonPrefixes with
exact equality against the expected prefix slice, catching wrong-but-
sorted results.
* test(s3): use allKeys for exact assertion in deep-hierarchy pagination test
Wire the allKeys slice (previously unused dead code) into the version
listing assertion, replacing generic count/uniqueness/sort checks with
an exact equality comparison against the keys that were created.
* STS: add GetCallerIdentity support
Implement the AWS STS GetCallerIdentity action, which returns the
ARN, account ID, and user ID of the caller based on SigV4 authentication.
This is commonly used by AWS SDKs and CLI tools (e.g. `aws sts get-caller-identity`)
to verify credentials and determine the authenticated identity.
* test: remove trivial GetCallerIdentity tests
Remove the XML unmarshal test (we don't consume this response as input)
and the routing constant test (just asserts a literal equals itself).
* fix: route GetCallerIdentity through STS in UnifiedPostHandler and use stable UserId
- UnifiedPostHandler only dispatched actions starting with "AssumeRole" to STS,
so GetCallerIdentity in a POST body would fall through to the IAM path and
get AccessDenied for non-admin users. Add explicit check for GetCallerIdentity.
- Use identity.Name as UserId instead of credential.AccessKey, which is a
transient value and incorrect for STS assumed-role callers.
* fix(weed/worker/tasks/ec_balance): non-recursive reportProgress
* fix(ec_balance): call ReportProgressWithStage and include volumeID in log
The original fix replaced infinite recursion with a glog.Infof, but
skipped the framework progress callback. This adds the missing
ReportProgressWithStage call so the admin server receives EC balance
progress, and includes volumeID in the log for disambiguation.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer.sync: show active chunk transfers when sync progress stalls
When the sync watermark is not advancing, print each in-progress chunk
transfer with its file path, bytes received so far, and current status
(downloading, uploading, or waiting with backoff duration). This helps
diagnose which files are blocking progress during replication.
Closes#8542
* filer.sync: include last error in stall diagnostics
* filer.sync: fix data races in ChunkTransferStatus
Add sync.RWMutex to ChunkTransferStatus and lock around all field
mutations in fetchAndWrite. ActiveTransfers now returns value copies
under RLock so callers get immutable snapshots.
* fix(s3): use recursive delete for .versions directory cleanup
When only delete markers remain in a .versions directory,
updateLatestVersionAfterDeletion tried to delete it non-recursively,
which failed with "fail to delete non-empty folder" because the delete
marker entries were still present. Use recursive deletion so the
directory and its remaining delete marker entries are cleaned up together.
* fix(s3): guard .versions directory deletion against truncated listings
When the version listing is truncated (>1000 entries), content versions
may exist beyond the first page. Skip the recursive directory deletion
in this case to prevent data loss.
* fix(s3): preserve delete markers in .versions directory
Delete markers must be preserved per S3 semantics — they are only
removed by an explicit DELETE with versionId. The previous fix would
recursively delete the entire .versions directory (including delete
markers) when no content versions were found.
Now the logic distinguishes three cases:
1. Content versions exist → update latest version metadata
2. Only delete markers remain (or listing truncated) → keep directory
3. Truly empty → safe to delete directory (non-recursive)
* fix(admin): respect urlPrefix in S3 bucket and S3Tables navigation links (#8884)
Several admin UI templates used hardcoded URLs (templ.SafeURL) instead of
dash.PUrl(ctx, ...) for navigation links, causing 404 errors when the
admin is deployed with --urlPrefix.
Fixed in: s3_buckets.templ, s3tables_buckets.templ, s3tables_tables.templ
* fix(admin): URL-escape bucketName in S3Tables navigation links
Add url.PathEscape(bucketName) for consistency and correctness in
s3tables_tables.templ (back-to-namespaces link) and s3tables_buckets.templ
(namespace link), matching the escaping already used in the table details link.
* S3: map canned ACL to file permissions and add configurable default file mode
S3 uploads were hardcoded to 0660 regardless of ACL headers. Now the
X-Amz-Acl header maps to Unix file permissions per-object:
- public-read, authenticated-read, bucket-owner-read → 0644
- public-read-write → 0666
- private, bucket-owner-full-control → 0660
Also adds -defaultFileMode / -s3.defaultFileMode flag to set a
server-wide default when no ACL header is present.
Closes#8874
* Address review feedback for S3 file mode feature
- Extract hardcoded 0660 to defaultFileMode constant
- Change parseDefaultFileMode to return error instead of calling Fatalf
- Add -s3.defaultFileMode flag to filer.go and mini.go (was missing)
- Add doc comment to S3Options about updating all four flag sites
- Add TestResolveFileMode with 10 test cases covering ACL mapping,
server default, and priority ordering
* fix(s3): remove customer encryption key from SSE-C debug log
The debug log in validateAndParseSSECHeaders was logging the raw
customer-provided encryption key bytes in hex format (keyBytes=%x),
leaking sensitive key material to log output. Remove the key bytes
from the log statement while keeping the MD5 hash comparison info.
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* filer.sync: support per-cluster mTLS with -a.security and -b.security flags
When syncing between two clusters that use different certificate authorities,
a single security.toml cannot authenticate to both. Add -a.security and
-b.security flags so each filer can use its own security.toml for TLS.
Closes#8481
* security: fatal on failure to read explicitly provided security config
When -a.security or -b.security is specified, falling back to insecure
credentials on read error would silently bypass mTLS. Fatal instead.
* fix(filer.sync): use source filer's fromTsMs flag in initOffsetFromTsMs
A→B was using bFromTsMs and B→A was using aFromTsMs — these were
swapped. Each path should seed the target's offset with the source
filer's starting timestamp.
* security: return error from LoadClientTLSFromFile, resolve relative PEM paths
Change LoadClientTLSFromFile to return (grpc.DialOption, error) so
callers can handle failures explicitly instead of a silent insecure
fallback. Resolve relative PEM paths (grpc.ca, grpc.client.cert,
grpc.client.key) against the config file's directory.
* Fix Admin UI master list showing gRPC port instead of HTTP port for followers (#8867)
Raft stores server addresses as gRPC addresses. The Admin UI was using
these addresses directly via ToHttpAddress(), which cannot extract the
HTTP port from a plain gRPC address. Use GrpcAddressToServerAddress()
to properly convert gRPC addresses back to HTTP addresses.
* Use httpAddress consistently as masterMap key
Address review feedback: masterInfo.Address (HTTP form) was already
computed but the raw address was used as the map key, causing
potential key mismatches between topology and raft data.
* fix(s3api): fix AWS Signature V2 format and validation
* fix(s3api): Skip space after "AWS" prefix (+1 offset)
* test(s3api): add unit tests for Signature V2 authentication fix
* fix(s3api): simply comparing signatures
* validation for the colon extraction in expectedAuth
* fix(shell): avoid marking skipped or unplaced volumes as fixed
---------
Co-authored-by: chrislu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Process .ecj deletions during EC decode and vacuum decoded volume (#8798)
When decoding EC volumes back to normal volumes, deletions recorded in
the .ecj journal were not being applied before computing the dat file
size or checking for live needles. This caused the decoded volume to
include data for deleted files and could produce false positives in the
all-deleted check.
- Call RebuildEcxFile before HasLiveNeedles/FindDatFileSize in
VolumeEcShardsToVolume so .ecj deletions are merged into .ecx first
- Vacuum the decoded volume after mounting in ec.decode to compact out
deleted needle data from the .dat file
- Add integration tests for decoding with non-empty .ecj files
* storage: add offline volume compaction helper
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ec: compact decoded volumes before deleting shards
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ec: address PR review comments
- Fall back to data directory for .ecx when idx directory lacks it
- Make compaction failure non-fatal during EC decode
- Remove misleading "buffer: 10%" from space check error message
* ec: collect .ecj from all shard locations during decode
Each server's .ecj only contains deletions for needles whose data
resides in shards held by that server. Previously, sources with no
new data shards to contribute were skipped entirely, losing their
.ecj deletion entries. Now .ecj is always appended from every shard
location so RebuildEcxFile sees the full set of deletions.
* ec: add integration tests for .ecj collection during decode
TestEcDecodePreservesDeletedNeedles: verifies that needles deleted
via VolumeEcBlobDelete are excluded from the decoded volume.
TestEcDecodeCollectsEcjFromPeer: regression test for the fix in
collectEcShards. Deletes a needle only on a peer server that holds
no new data shards, then verifies the deletion survives decode via
.ecj collection.
* ec: address review nits in decode and tests
- Remove double error wrapping in mountDecodedVolume
- Check VolumeUnmount error in peer ecj test
- Assert 404 specifically for deleted needles, fail on 5xx
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: replace modulo hashing with consistent hash ring
Introduce HashRing with virtual nodes (CRC32-based consistent hashing)
to replace the modulo-based hashKeyToServer. When a filer node is
removed, only keys that hashed to that node are remapped to the next
server on the ring, leaving all other mappings stable. This is the
foundation for backup replication — the successor on the ring is
always the natural takeover node.
* dlm: add Generation and IsBackup fields to Lock
Lock now carries IsBackup (whether this node holds the lock as a backup
replica) and Generation (a monotonic fencing token that increments on
each fresh acquisition, stays the same on renewal). Add helper methods:
AllLocks, PromoteLock, DemoteLock, InsertBackupLock, RemoveLock, GetLock.
* dlm: add ReplicateLock RPC and generation/is_backup proto fields
Add generation field to LockResponse for fencing tokens.
Add generation and is_backup fields to Lock message.
Add ReplicateLock RPC for primary-to-backup lock replication.
Add ReplicateLockRequest/ReplicateLockResponse messages.
* dlm: add async backup replication to DistributedLockManager
Route lock/unlock via consistent hash ring's GetPrimaryAndBackup().
After a successful lock or unlock on the primary, asynchronously
replicate the operation to the backup server via ReplicateFunc
callback. Single-server deployments skip replication.
* dlm: add ReplicateLock handler and backup-aware topology changes
Add ReplicateLock gRPC handler for primary-to-backup replication.
Revise OnDlmChangeSnapshot to handle three cases on topology change:
- Promote backup locks when this node becomes primary
- Demote primary locks when this node becomes backup
- Transfer locks when this node is neither primary nor backup
Wire up SetupDlmReplication during filer server initialization.
* dlm: expose generation fencing token in lock client
LiveLock now captures the generation from LockResponse and exposes it
via Generation() method. Consumers can use this as a fencing token to
detect stale lock holders.
* dlm: update empty folder cleaner to use consistent hash ring
Replace local modulo-based hashKeyToServer with LockRing.GetPrimary()
which uses the shared consistent hash ring for folder ownership.
* dlm: add unit tests for consistent hash ring
Test basic operations, consistency on server removal (only keys from
removed server move), backup-is-successor property (backup becomes
new primary when primary is removed), and key distribution balance.
* dlm: add integration tests for lock replication failure scenarios
Test cases:
- Primary crash with backup promotion (backup has valid token)
- Backup crash with primary continuing
- Both primary and backup crash (lock lost, re-acquirable)
- Rolling restart across all nodes
- Generation fencing token increments on new acquisition
- Replication failure (primary still works independently)
- Unlock replicates deletion to backup
- Lock survives server addition (topology change)
- Consistent hashing minimal disruption (only removed server's keys move)
* dlm: address PR review findings
1. Causal replication ordering: Add per-lock sequence number (Seq) that
increments on every mutation. Backup rejects incoming mutations with
seq <= current seq, preventing stale async replications from
overwriting newer state. Unlock replication also carries seq and is
rejected if stale.
2. Demote-after-handoff: OnDlmChangeSnapshot now transfers the lock to
the new primary first and only demotes to backup after a successful
TransferLocks RPC. If the transfer fails, the lock stays as primary
on this node.
3. SetSnapshot candidateServers leak: Replace the candidateServers map
entirely instead of appending, so removed servers don't linger.
4. TransferLocks preserves Generation and Seq: InsertLock now accepts
generation and seq parameters. After accepting a transferred lock,
the receiving node re-replicates to its backup.
5. Rolling restart test: Add re-replication step after promotion and
assert survivedCount > 0. Add TestDLM_StaleReplicationRejected.
6. Mixed-version upgrade note: Add comment on HashRing documenting that
all filer nodes must be upgraded together.
* dlm: serve renewals locally during transfer window on node join
When a new node joins and steals hash ranges from surviving nodes,
there's a window between ring update and lock transfer where the
client gets redirected to a node that doesn't have the lock yet.
Fix: if the ring says primary != self but we still hold the lock
locally (non-backup, matching token), serve the renewal/unlock here
rather than redirecting. The lock will be transferred by
OnDlmChangeSnapshot, and subsequent requests will go to the new
primary once the transfer completes.
Add tests:
- TestDLM_NodeDropAndJoin_OwnershipDisruption: measures disruption
when a node drops and a new one joins (14/100 surviving-node locks
disrupted, all handled by transfer logic)
- TestDLM_RenewalDuringTransferWindow: verifies renewal succeeds on
old primary during the transfer window
* dlm: master-managed lock ring with stabilization batching
The master now owns the lock ring membership. Instead of filers
independently reacting to individual ClusterNodeUpdate add/remove
events, the master:
1. Tracks filer membership in LockRingManager
2. Batches rapid changes with a 1-second stabilization timer
(e.g., a node drop + join within 1 second → single ring update)
3. Broadcasts the complete ring snapshot atomically via the new
LockRingUpdate message in KeepConnectedResponse
Filers receive the ring as a complete snapshot and apply it via
SetSnapshot, ensuring all filers converge to the same ring state
without intermediate churn.
This eliminates the double-churn problem where a rapid drop+join
would fire two separate ring mutations, each triggering lock
transfers and disrupting ownership on surviving nodes.
* dlm: track ring version, reject stale updates, remove dead code
SetSnapshot now takes a version parameter from the master. Stale
updates (version < current) are rejected, preventing reordered
messages from overwriting a newer ring state. Version 0 is always
accepted for bootstrap.
Remove AddServer/RemoveServer from LockRing — the ring is now
exclusively managed by the master via SetSnapshot. Remove the
candidateServers map that was only used by those methods.
* dlm: fix SelectLocks data race, advance generation on backup insert
- SelectLocks: change RLock to Lock since the function deletes map
entries, which is a write operation and causes a data race under RLock.
- InsertBackupLock: advance nextGeneration to at least the incoming
generation so that after failover promotion, new lock acquisitions
get a generation strictly greater than any replicated lock.
- Bump replication failure log from V(1) to Warningf for production
visibility.
* dlm: fix SetSnapshot race, test reliability, timer edge cases
- SetSnapshot: hold LockRing lock through both version update and
Ring.SetServers() so they're atomic. Prevents a concurrent caller
from seeing the new version but applying stale servers.
- Transfer window test: search for a key that actually moves primary
when filer4 joins, instead of relying on a fixed key that may not.
- renewLock redirect: pass the existing token to the new primary
instead of empty string, so redirected renewals work correctly.
- scheduleBroadcast: check timer.Stop() return value. If the timer
already fired, the callback picks up latest state.
- FlushPending: only broadcast if timer.Stop() returns true (timer
was still pending). If false, the callback is already running.
- Fix test comment: "idempotent" → "accepted, state-changing".
* dlm: use wall-clock nanoseconds for lock ring version
The lock ring version was an in-memory counter that reset to 0 on
master restart. A filer that had seen version 5 would reject version 1
from the restarted master.
Fix: use time.Now().UnixNano() as the version. This survives master
restarts without persistence — the restarted master produces a
version greater than any pre-restart value.
* dlm: treat expired lock owners as missing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: reject stale lock transfers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: order replication by generation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: bootstrap lock ring on reconnect
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* chore(weed/wdclient): prune unused functions
* chore(weed/wdclient): prune test-only functions and associated tests
* chore(weed/wdclient): remove dead cursor field
The cursor field and its initialization are no longer used after
the removal of getLocationIndex.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* Use Unix sockets for gRPC between co-located services in weed server
Extends the Unix socket gRPC optimization (added for mini mode in #8856)
to `weed server`. Registers Unix socket paths for each service's gRPC
port before startup, so co-located services (master, volume, filer, S3)
communicate via Unix sockets instead of TCP loopback.
Only services actually started in this process get registered. The gRPC
port is resolved early (port + 10000 if unset) so the socket path is
known before any service dials another.
* Refactor gRPC Unix socket registration into a data-driven loop
* Fix stale admin lock metric when lock expires and is reacquired (#8857)
When a lock expired without an explicit unlock and a different client
acquired it, the old client's metric was never cleared, causing
multiple clients to appear as simultaneously holding the lock.
* Use DeleteLabelValues instead of Set(0) to remove stale metric series
Avoids cardinality explosion from accumulated stale series when
client names are dynamic.
* rename metadata events
* fix subscription filter to use NewEntry.Name for rename path matching
The server-side subscription filter constructed the new path using
OldEntry.Name instead of NewEntry.Name when checking if a rename
event's destination matches the subscriber's path prefix. This could
cause events to be incorrectly filtered when a rename changes the
file name.
* fix bucket events to handle rename of bucket directories
onBucketEvents only checked IsCreate and IsDelete. A bucket directory
rename via AtomicRenameEntry now emits a single rename event (both
OldEntry and NewEntry non-nil), which matched neither check. Handle
IsRename by deleting the old bucket and creating the new one.
* fix replicator to handle rename events across directory boundaries
Two issues fixed:
1. The replicator filtered events by checking if the key (old path)
was under the source directory. Rename events now use the old path
as key, so renames from outside into the watched directory were
silently dropped. Now both old and new paths are checked, and
cross-boundary renames are converted to create or delete.
2. NewParentPath was passed to the sink without remapping to the
sink's target directory structure, causing the sink to write
entries at the wrong location. Now NewParentPath is remapped
alongside the key.
* fix filer sync to handle rename events crossing directory boundaries
The early directory-prefix filter only checked resp.Directory (old
parent). Rename events now carry the old parent as Directory, so
renames from outside the source path into it were dropped before
reaching the existing cross-boundary handling logic. Check both old
and new directories against sourcePath and excludePaths so the
downstream old-key/new-key logic can properly convert these to
create or delete operations.
* fix metadata event path matching
* fix metadata event consumers for rename targets
* Fix replication rename target keys
Logical rename events now reach replication sinks with distinct source and target paths.\n\nHandle non-filer sinks as delete-plus-create on the translated target key, and make the rename fallback path create at the translated target key too.\n\nAdd focused tests covering non-filer renames, filer rename updates, and the fallback path.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix filer sync rename path scoping
Use directory-boundary matching instead of raw prefix checks when classifying source and target paths during filer sync.\n\nAlso apply excludePaths per side so renames across excluded boundaries downgrade cleanly to create/delete instead of being misclassified as in-scope updates.\n\nAdd focused tests for boundary matching and rename classification.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix replicator directory boundary checks
Use directory-boundary matching instead of raw prefix checks when deciding whether a source or target path is inside the watched tree or an excluded subtree.\n\nThis prevents sibling paths such as /foo and /foobar from being misclassified during rename handling, and preserves the earlier rename-target-key fix.\n\nAdd focused tests for boundary matching and rename classification across sibling/excluded directories.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix etc-remote rename-out handling
Use boundary-safe source/target directory membership when classifying metadata events under DirectoryEtcRemote.\n\nThis prevents rename-out events from being processed as config updates, while still treating them as removals where appropriate for the remote sync and remote gateway command paths.\n\nAdd focused tests for update/removal classification and sibling-prefix handling.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Defer rename events until commit
Queue logical rename metadata events during atomic and streaming renames and publish them only after the transaction commits successfully.\n\nThis prevents subscribers from seeing delete or logical rename events for operations that later fail during delete or commit.\n\nAlso serialize notification.Queue swaps in rename tests and add failure-path coverage.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip descendant rename target lookups
Avoid redundant target lookups during recursive directory renames once the destination subtree is known absent.\n\nThe recursive move path now inserts known-absent descendants directly, and the test harness exercises prefixed directory listing so the optimization is covered by a directory rename regression test.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten rename review tests
Return filer_pb.ErrNotFound from the bucket tracking store test stub so it follows the FilerStore contract, and add a webhook filter case for same-name renames across parent directories.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix HardLinkId format verb in InsertEntryKnownAbsent error
HardLinkId is a byte slice. %d prints each byte as a decimal number
which is not useful for an identifier. Use %x to match the log line
two lines above.
* only skip descendant target lookup when source and dest use same store
moveFolderSubEntries unconditionally passed skipTargetLookup=true for
every descendant. This is safe when all paths resolve to the same
underlying store, but with path-specific store configuration a child's
destination may map to a different backend that already holds an entry
at that path. Use FilerStoreWrapper.SameActualStore to check per-child
and fall back to the full CreateEntry path when stores differ.
* add nil and create edge-case tests for metadata event scope helpers
* extract pathIsEqualOrUnder into util.IsEqualOrUnder
Identical implementations existed in both replication/replicator.go and
command/filer_sync.go. Move to util.IsEqualOrUnder (alongside the
existing FullPath.IsUnder) and remove the duplicates.
* use MetadataEventTargetDirectory for new-side directory in filer sync
The new-side directory checks and sourceNewKey computation used
message.NewParentPath directly. If NewParentPath were empty (legacy
events, older filer versions during rolling upgrades), sourceNewKey
would be wrong (/filename instead of /dir/filename) and the
UpdateEntry parent path rewrite would panic on slice bounds.
Derive targetDir once from MetadataEventTargetDirectory, which falls
back to resp.Directory when NewParentPath is empty, and use it
consistently for all new-side checks and the sink parent path.
* Use Unix sockets for gRPC between co-located services in mini mode
In `weed mini`, all services run in one process. Previously, inter-service
gRPC traffic (volume↔master, filer↔master, S3↔filer, worker↔admin, etc.)
went through TCP loopback. This adds a gRPC Unix socket registry in the pb
package: mini mode registers a socket path per gRPC port at startup, each
gRPC server additionally listens on its socket, and GrpcDial transparently
routes to the socket via WithContextDialer when a match is found.
Standalone commands (weed master, weed filer, etc.) are unaffected since
no sockets are registered. TCP listeners are kept for external clients.
* Handle Serve error and clean up socket file in ServeGrpcOnLocalSocket
Log non-expected errors from grpcServer.Serve (ignoring
grpc.ErrServerStopped) and always remove the Unix socket file
when Serve returns, ensuring cleanup on Stop/GracefulStop.
* fix(worker): pass compaction revision and file sizes in EC volume copy
The worker EC task was sending CopyFile requests without the current
compaction revision (defaulting to 0) and with StopOffset set to
math.MaxInt64. After a vacuum compaction this caused the volume server
to reject the copy or return stale data.
Read the volume file status first and forward the compaction revision
and actual file sizes so the copy is consistent with the compacted
volume.
* propagate erasure coding task context
* fix(worker): validate volume file status and detect short copies
Reject zero dat file size from ReadVolumeFileStatus — a zero-sized
snapshot would produce 0-byte copies and broken EC shards.
After streaming, verify totalBytes matches the expected stopOffset
and return an error on short copies instead of logging success.
* fix(worker): reject zero idx file size in volume status validation
A non-empty dat with zero idx indicates an empty or corrupt volume.
Without this guard, copyFileFromSource gets stopOffset=0, produces a
0-byte .idx, passes the short-copy check, and generateEcShardsLocally
runs against a volume with no index.
* fix fake plugin volume file status
* fix plugin volume balance test fixtures
* fix(filer): apply default disk type after location-prefix resolution in gRPC AssignVolume
The gRPC AssignVolume path was applying the filer's default DiskType to
the request before calling detectStorageOption. This caused the default
to shadow any disk type configured via a filer location-prefix rule,
diverging from the HTTP write path which applies the default only when
no rule matches.
Extract resolveAssignStorageOption to apply the filer default disk type
after detectStorageOption, so location-prefix rules take precedence.
* fix(filer): apply default disk type after location-prefix resolution in TUS upload path
Same class of bug as the gRPC AssignVolume fix: the TUS tusWriteData
handler called detectStorageOption0 but never applied the filer's
default DiskType when no location-prefix rule matched. This made TUS
uploads ignore the -disk flag entirely.
* fix(s3): preserve explicit directory markers during empty folder cleanup
PR #8292 switched empty-folder cleanup from per-folder implicit checks
to bucket-level policy, inadvertently dropping the check that preserved
explicitly created directories (e.g., PUT /bucket/folder/). This caused
user-created folders to be deleted when their last file was removed.
Add IsDirectoryKeyObject check in executeCleanup to skip folders that
have a MIME type set, matching the canonical pattern used throughout the
S3 listing and delete handlers.
* fix: handle ErrNotFound in IsDirectoryKeyObject for race safety
Entry may be deleted between the emptiness check and the directory
marker lookup. Treat not-found as false rather than propagating
the error, avoiding unnecessary error logging in the cleanup path.
* refactor: consolidate directory marker tests and tidy error handling
- Combine two separate test functions into a table-driven test
- Nest ErrNotFound check inside the err != nil block
* notification.kafka: add SASL authentication and TLS support (#8827)
Wire sarama SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512) and TLS
configuration into the Kafka notification producer and consumer,
enabling connections to secured Kafka clusters.
* notification.kafka: validate mTLS config
* kafka notification: validate partial mTLS config, replace panics with errors
- Reject when only one of tls_client_cert/tls_client_key is provided
- Replace three panic() calls in KafkaInput.initialize with returned errors
* kafka notification: enforce minimum TLS 1.2 for Kafka connections
