* Add -insecureSkipVerify flag and config option for filer.sync HTTPS connections
When using filer.sync between clusters with different CAs (e.g., separate
OpenShift clusters), TLS certificate verification fails with "x509:
certificate signed by unknown authority". This adds two ways to skip TLS
certificate verification:
1. CLI flag: `weed filer.sync -insecureSkipVerify ...`
2. Config option: `insecure_skip_verify = true` under [https.client] in
security.toml
Closes#8778
* Add insecure_skip_verify option for HTTPS client in security.toml
When using filer.sync between clusters with different CAs (e.g., separate
OpenShift clusters), TLS certificate verification fails. Adding
insecure_skip_verify = true under [https.client] in security.toml allows
skipping TLS certificate verification.
The option is read during global HTTP client initialization so it applies
to all HTTPS connections including filer.sync proxy reads and writes.
Closes#8778
---------
Co-authored-by: Copilot <copilot@github.com>
* test: restore maintenance mode coverage in TestVolumeMarkReadonlyWritableErrorPaths
PR #8360 removed the maintenance mode assertions because the refactored
check ordering (volume lookup before maintenance check) caused the
original test to hit "not found" instead of "maintenance mode" — the
test used a non-existent volume ID.
Restore coverage by allocating a real volume, then verifying:
- existing volume in maintenance mode returns "maintenance mode"
- non-existent volume in maintenance mode still returns "not found"
(validating the new check ordering)
* test: add coverage for ScrubVolume MarkBrokenVolumesReadonly flag
PR #8360 added the mark_broken_volumes_readonly field to ScrubVolumeRequest
but no tests exercised the new logic paths. Add three integration tests:
- HealthyVolume: flag is a no-op when scrub finds no broken volumes
- CorruptVolume: corrupted .idx triggers broken detection; without the
flag the volume stays writable, with the flag it becomes read-only
- MaintenanceMode: makeVolumeReadonly fails under maintenance and
ScrubVolume propagates the error via errors.Join
* refactor: extract CorruptIndexFile and EnableMaintenanceMode test helpers
Move duplicated idx corruption and maintenance mode setup into
framework.CorruptIndexFile() and framework.EnableMaintenanceMode()
helpers. Use defer for file close in the corruption helper.
* Give the `ScrubVolume()` RPC an option to flag found broken volumes as read-only.
Also exposes this option in the shell `volume.scrub` command.
* Remove redundant test in `TestVolumeMarkReadonlyWritableErrorPaths`.
417051bb slightly rearranges the logic for `VolumeMarkReadonly()` and `VolumeMarkWritable()`,
so calling them for invalid volume IDs will actually yield that error, instead of checking
maintnenance mode first.
* Fix filerExcludeFileName to support directory names and path components
The original implementation only matched excludeFileName against
message.NewEntry.Name, which caused two issues:
1. Nil pointer panic on delete events (NewEntry is nil)
2. Files inside excluded directories were still backed up because
the parent directory name was not checked
This patch:
- Checks all path components in resp.Directory against the regexp
- Adds nil guard for message.NewEntry before accessing .Name
- Also checks message.OldEntry.Name for rename/delete events
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add -filerExcludePathPattern flag and fix nil panic in filerExcludeFileName
Separate concerns between two exclude mechanisms:
- filerExcludeFileName: matches entry name only (leaf node)
- filerExcludePathPattern (NEW): matches any path component via regexp,
so files inside matched directories are also excluded
Also fixes nil pointer panic when filerExcludeFileName encounters
delete events where NewEntry is nil.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Refactor exclude logic: per-side exclusion for rename events, reduce duplication
- Extract isEntryExcluded() to compute exclusion per old/new side,
so rename events crossing an exclude boundary are handled as
delete + create instead of being entirely skipped
- Extract compileExcludePattern() to deduplicate regexp compilation
- Replace strings.Split with allocation-free pathContainsMatch()
- Check message.NewParentPath (not just resp.Directory) for new side
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Move regexp compilation out of retry loop to fail fast on config errors
compileExcludePattern for -filerExcludeFileName and -filerExcludePathPattern
are configuration-time validations that will never succeed on retry.
Move them to runFilerBackup before the reconnect loop and use glog.Fatalf
on failure, so invalid patterns are caught immediately at startup instead
of being retried every 1.7 seconds indefinitely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add wildcard matching helpers for path and filename exclusion
* Replace regexp exclude patterns with wildcard-based flags, deprecate -filerExcludeFileName
Add -filerExcludeFileNames and -filerExcludePathPatterns flags that accept
comma-separated wildcard patterns (*, ?) using the existing wildcard library.
Mark -filerExcludeFileName as deprecated but keep its regexp behavior.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* shell: add s3.bucket.access command for anonymous access policy (#7738)
Add a new weed shell command to view or change the anonymous access
policy of an S3 bucket without external tools.
Usage:
s3.bucket.access -name <bucket> -access read,list
s3.bucket.access -name <bucket> -access none
Supported permissions: read, write, list. The command writes a standard
bucket policy with Principal "*" and warns if no anonymous IAM identity
exists.
* shell: fix anonymous identity hint in s3.bucket.access warning
The anonymous identity doesn't need IAM actions — the bucket policy
controls what anonymous users can do.
* shell: only warn about anonymous identity when write access is set
Read and list operations use AuthWithPublicRead which evaluates bucket
policies directly without requiring the anonymous identity. Only write
operations go through the normal auth flow that needs it.
* shell: rewrite s3.bucket.access to use IAM actions instead of bucket policies
Replace the bucket policy approach with direct IAM identity actions,
matching the s3.configure pattern. The user is auto-created if it does
not exist.
Usage:
s3.bucket.access -name <bucket> -user anonymous -access Read,List
s3.bucket.access -name <bucket> -user anonymous -access none
s3.bucket.access -name <bucket> -user anonymous
Actions are stored as "Action:bucket" on the identity, same as
s3.configure -actions=Read -buckets=my-bucket.
* shell: return flag parse errors instead of swallowing them
* shell: normalize action names case-insensitively in s3.bucket.access
Accept actions in any case (read, READ, Read) and normalize to canonical
form (Read, Write, List, etc.) before storing. This matches the
case-insensitive handling of "none" and avoids confusing rejections.
Add an "Anonymous" checkbox next to the username field in the Create User
modal. When checked, the username is set to "anonymous" and the credential
generation checkbox is disabled since anonymous users do not need keys.
The checkbox is only shown when no anonymous user exists yet. The
manage-access-keys button in the users table is hidden for the anonymous
user.
* filer: add StreamMutateEntry bidi streaming RPC
Add a bidirectional streaming RPC that carries all filer mutation
types (create, update, delete, rename) over a single ordered stream.
This eliminates per-request connection overhead for pipelined
operations and guarantees mutation ordering within a stream.
The server handler delegates each request to the existing unary
handlers (CreateEntry, UpdateEntry, DeleteEntry) and uses a proxy
stream adapter for rename operations to reuse StreamRenameEntry logic.
The is_last field signals completion for multi-response operations
(rename sends multiple events per request; create/update/delete
always send exactly one response with is_last=true).
* mount: add streaming mutation multiplexer (streamMutateMux)
Implement a client-side multiplexer that routes all filer mutation RPCs
(create, update, delete, rename) over a single bidirectional gRPC
stream. Multiple goroutines submit requests through a send channel;
a dedicated sendLoop serializes them on the stream; a recvLoop
dispatches responses to waiting callers via per-request channels.
Key features:
- Lazy stream opening on first use
- Automatic reconnection on stream failure
- Permanent fallback to unary RPCs if filer returns Unimplemented
- Monotonic request_id for response correlation
- Multi-response support for rename operations (is_last signaling)
The mux is initialized on WFS and closed during unmount cleanup.
No call sites use it yet — wiring comes in subsequent commits.
* mount: route CreateEntry and UpdateEntry through streaming mux
Wire all CreateEntry call sites to use wfs.streamCreateEntry() which
routes through the StreamMutateEntry stream when available, falling
back to unary RPCs otherwise. Also wire Link's UpdateEntry calls
through wfs.streamUpdateEntry().
Updated call sites:
- flushMetadataToFiler (file flush after write)
- Mkdir (directory creation)
- Symlink (symbolic link creation)
- createRegularFile non-deferred path (Mknod)
- flushFileMetadata (periodic metadata flush)
- Link (hard link: update source + create link + rollback)
* mount: route UpdateEntry and DeleteEntry through streaming mux
Wire remaining mutation call sites through the streaming mux:
- saveEntry (Setattr/chmod/chown/utimes) → streamUpdateEntry
- Unlink → streamDeleteEntry (replaces RemoveWithResponse)
- Rmdir → streamDeleteEntry (replaces RemoveWithResponse)
All filer mutations except Rename now go through StreamMutateEntry
when the filer supports it, with automatic unary RPC fallback.
* mount: route Rename through streaming mux
Wire Rename to use streamMutate.Rename() when available, with
fallback to the existing StreamRenameEntry unary stream.
The streaming mux sends rename as a StreamRenameEntryRequest oneof
variant. The server processes it through the existing rename logic
and sends multiple StreamRenameEntryResponse events (one per moved
entry), with is_last=true on the final response.
All filer mutations now go through a single ordered stream.
* mount: fix stream mux connection ownership
WithGrpcClient(streamingMode=true) closes the gRPC connection when
the callback returns, destroying the stream. Own the connection
directly via pb.GrpcDial so it stays alive for the stream's lifetime.
Close it explicitly in recvLoop on stream failure and in Close on
shutdown.
* mount: fix rename failure for deferred-create files
Three fixes for rename operations over the streaming mux:
1. lookupEntry: fall back to local metadata store when filer returns
"not found" for entries in uncached directories. Files created with
deferFilerCreate=true exist only in the local leveldb store until
flushed; lookupEntry skipped the local store when the parent
directory had never been readdir'd, causing rename to fail with
ENOENT.
2. Rename: wait for pending async flushes and force synchronous flush
of dirty metadata before sending rename to the filer. Covers the
writebackCache case where close() defers the flush to a background
worker that may not complete before rename fires.
3. StreamMutateEntry: propagate rename errors from server to client.
Add error/errno fields to StreamMutateEntryResponse so the mount
can map filer errors to correct FUSE status codes instead of
silently returning OK. Also fix the existing Rename error handler
which could return fuse.OK on unrecognized errors.
* mount: fix streaming mux error handling, sendLoop lifecycle, and fallback
Address PR review comments:
1. Server: populate top-level Error/Errno on StreamMutateEntryResponse for
create/update/delete errors, not just rename. Previously update errors
were silently dropped and create/delete errors were only in nested
response fields that the client didn't check.
2. Client: check nested error fields in CreateEntry (ErrorCode, Error)
and DeleteEntry (Error) responses, matching CreateEntryWithResponse
behavior.
3. Fix sendLoop lifecycle: give each stream generation a stopSend channel.
recvLoop closes it on error to stop the paired sendLoop. Previously a
reconnect left the old sendLoop draining sendCh, breaking ordering.
4. Transparent fallback: stream helpers and doRename fall back to unary
RPCs on transport errors (ErrStreamTransport), including the first
Unimplemented from ensureStream. Previously the first call failed
instead of degrading.
5. Filer rotation in openStream: try all filer addresses on dial failure,
matching WithFilerClient behavior. Stop early on Unimplemented.
6. Pass metadata-bearing context to StreamMutateEntry RPC call so
sw-client-id header is actually sent.
7. Gate lookupEntry local-cache fallback on open dirty handle or pending
async flush to avoid resurrecting deleted/renamed entries.
8. Remove dead code in flushFileMetadata (err=nil followed by if err!=nil).
9. Use string matching for rename error-to-errno mapping in the mount to
stay portable across Linux/macOS (numeric errno values differ).
* mount: make failAllPending idempotent with delete-before-close
Change failAllPending to collect pending entries into a local slice
(deleting from the sync.Map first) before closing channels. This
prevents double-close panics if called concurrently. Also remove the
unused err parameter.
* mount: add stream generation tracking and teardownStream
Introduce a generation counter on streamMutateMux that increments each
time a new stream is created. Requests carry the generation they were
enqueued for so sendLoop can reject stale requests after reconnect.
Add teardownStream(gen) which is idempotent (only acts when gen matches
current generation and stream is non-nil). Both sendLoop and recvLoop
call it on error, replacing the inline cleanup in recvLoop. sendLoop now
actively triggers teardown on send errors instead of silently exiting.
ensureStream waits for the prior generation's recvDone before creating a
new stream, ensuring all old pending waiters are failed before reconnect.
recvLoop now takes the stream, generation, and recvDone channel as
parameters to avoid accessing shared fields without the lock.
* mount: harden Close to prevent races with teardownStream
Nil out stream, cancel, and grpcConn under the lock so that any
concurrent teardownStream call from recvLoop/sendLoop becomes a no-op.
Call failAllPending before closing sendCh to unblock waiters promptly.
Guard recvDone with a nil check for the case where Close is called
before any stream was ever opened.
* mount: make errCh receive ctx-aware in doUnary and Rename
Replace the blocking <-sendReq.errCh with a select that also observes
ctx.Done(). If sendLoop exits via stopSend without consuming a buffered
request, the caller now returns ctx.Err() instead of blocking forever.
The buffered errCh (capacity 1) ensures late acknowledgements from
sendLoop don't block the sender.
* mount: fix sendLoop/Close race and recvLoop/teardown pending channel race
Three related fixes:
1. Stop closing sendCh in Close(). Closing the shared producer channel
races with callers who passed ensureStream() but haven't sent yet,
causing send-on-closed-channel panics. sendCh is now left open;
ensureStream checks m.closed to reject new callers.
2. Drain buffered sendCh items on shutdown. sendLoop defers drainSendCh()
on exit so buffered requests get an ErrStreamTransport on their errCh
instead of blocking forever. Close() drains again for any stragglers
enqueued between sendLoop's drain and the final shutdown.
3. Move failAllPending from teardownStream into recvLoop's defer.
teardownStream (called from sendLoop on send error) was closing
pending response channels while recvLoop could be between
pending.Load and the channel send — a send-on-closed-channel panic.
recvLoop is now the sole closer of pending channels, eliminating
the race. Close() waits on recvDone (with cancel() to guarantee
Recv unblocks) so pending cleanup always completes.
* filer/mount: add debug logging for hardlink lifecycle
Add V(0) logging at every point where a HardLinkId is created, stored,
read, or deleted to trace orphaned hardlink references. Logging covers:
- gRPC server: CreateEntry/UpdateEntry when request carries HardLinkId
- FilerStoreWrapper: InsertEntry/UpdateEntry when entry has HardLinkId
- handleUpdateToHardLinks: entry path, HardLinkId, counter, chunk count
- setHardLink: KvPut with blob size
- maybeReadHardLink: V(1) on read attempt and successful decode
- DeleteHardLink: counter decrement/deletion events
- Mount Link(): when NewHardLinkId is generated and link is created
This helps diagnose how a git pack .rev file ended up with a
HardLinkId during a clone (no hard links should be involved).
* test: add git clone/pull integration test for FUSE mount
Shell script that exercises git operations on a SeaweedFS mount:
1. Creates a bare repo on the mount
2. Clones locally, makes 3 commits, pushes to mount
3. Clones from mount bare repo into an on-mount working dir
4. Verifies clone integrity (files, content, commit hashes)
5. Pushes 2 more commits with renames and deletes
6. Checks out an older revision on the mount clone
7. Returns to branch and pulls with real changes
8. Verifies file content, renames, deletes after pull
9. Checks git log integrity and clean status
27 assertions covering file existence, content, commit hashes,
file counts, renames, deletes, and git status. Run against any
existing mount: bash test-git-on-mount.sh /path/to/mount
* test: add git clone/pull FUSE integration test to CI suite
Add TestGitOperations to the existing fuse_integration test framework.
The test exercises git's full file operation surface on the mount:
1. Creates a bare repo on the mount (acts as remote)
2. Clones locally, makes 3 commits (files, bulk data, renames), pushes
3. Clones from mount bare repo into an on-mount working dir
4. Verifies clone integrity (content, commit hash, file count)
5. Pushes 2 more commits with new files, renames, and deletes
6. Checks out an older revision on the mount clone
7. Returns to branch and pulls with real fast-forward changes
8. Verifies post-pull state: content, renames, deletes, file counts
9. Checks git log integrity (5 commits) and clean status
Runs automatically in the existing fuse-integration.yml CI workflow.
* mount: fix permission check with uid/gid mapping
The permission checks in createRegularFile() and Access() compared
the caller's local uid/gid against the entry's filer-side uid/gid
without applying the uid/gid mapper. With -map.uid 501:0, a directory
created as uid 0 on the filer would not match the local caller uid
501, causing hasAccess() to fall through to "other" permission bits
and reject write access (0755 → other has r-x, no w).
Fix: map entry uid/gid from filer-space to local-space before the
hasAccess() call so both sides are in the same namespace.
This fixes rsync -a failing with "Permission denied" on mkstempat
when using uid/gid mapping.
* mount: fix Mkdir/Symlink returning filer-side uid/gid to kernel
Mkdir and Symlink used `defer wfs.mapPbIdFromFilerToLocal(entry)` to
restore local uid/gid, but `outputPbEntry` writes the kernel response
before the function returns — so the kernel received filer-side
uid/gid (e.g., 0:0). macFUSE then caches these and rejects subsequent
child operations (mkdir, create) because the caller uid (501) doesn't
match the directory owner (0), and "other" bits (0755 → r-x) lack
write permission.
Fix: replace the defer with an explicit call to mapPbIdFromFilerToLocal
before outputPbEntry, so the kernel gets local uid/gid.
Also add nil guards for UidGidMapper in Access and createRegularFile
to prevent panics in tests that don't configure a mapper.
This fixes rsync -a "Permission denied" on mkpathat for nested
directories when using uid/gid mapping.
* mount: fix Link outputting filer-side uid/gid to kernel, add nil guards
Link had the same defer-before-outputPbEntry bug as Mkdir and Symlink:
the kernel received filer-side uid/gid because the defer hadn't run
yet when outputPbEntry wrote the response.
Also add nil guards for UidGidMapper in Access and createRegularFile
so tests without a mapper don't panic.
Audit of all outputPbEntry/outputFilerEntry call sites:
- Mkdir: fixed in prior commit (explicit map before output)
- Symlink: fixed in prior commit (explicit map before output)
- Link: fixed here (explicit map before output)
- Create (existing file): entry from maybeLoadEntry (already mapped)
- Create (deferred): entry has local uid/gid (never mapped to filer)
- Create (non-deferred): createRegularFile defer runs before return
- Mknod: createRegularFile defer runs before return
- Lookup: entry from lookupEntry (already mapped)
- GetAttr: entry from maybeReadEntry/maybeLoadEntry (already mapped)
- readdir: entry from cache (mapIdFromFilerToLocal) or filer (mapped)
- saveEntry: no kernel output
- flushMetadataToFiler: no kernel output
- flushFileMetadata: no kernel output
* test: fix git test for same-filesystem FUSE clone
When both the bare repo and working clone live on the same FUSE mount,
git's local transport uses hardlinks and cross-repo stat calls that
fail on FUSE. Fix:
- Use --no-local on clone to disable local transport optimizations
- Use reset --hard instead of checkout to stay on branch
- Use fetch + reset --hard origin/<branch> instead of git pull to
avoid local transport stat failures during fetch
* adjust logging
* test: use plain git clone/pull to exercise real FUSE behavior
Remove --no-local and fetch+reset workarounds. The test should use
the same git commands users run (clone, reset --hard, pull) so it
reveals real FUSE issues rather than hiding them.
* test: enable V(1) logging for filer/mount and collect logs on failure
- Run filer and mount with -v=1 so hardlink lifecycle logs (V(0):
create/delete/insert, V(1): read attempts) are captured
- On test failure, automatically dump last 16KB of all process logs
(master, volume, filer, mount) to test output
- Copy process logs to /tmp/seaweedfs-fuse-logs/ for CI artifact upload
- Update CI workflow to upload SeaweedFS process logs alongside test output
* mount: clone entry for filer flush to prevent uid/gid race
flushMetadataToFiler and flushFileMetadata used entry.GetEntry() which
returns the file handle's live proto entry pointer, then mutated it
in-place via mapPbIdFromLocalToFiler. During the gRPC call window, a
concurrent Lookup (which takes entryLock.RLock but NOT fhLockTable)
could observe filer-side uid/gid (e.g., 0:0) on the file handle entry
and return it to the kernel. The kernel caches these attributes, so
subsequent opens by the local user (uid 501) fail with EACCES.
Fix: proto.Clone the entry before mapping uid/gid for the filer request.
The file handle's live entry is never mutated, so concurrent Lookup
always sees local uid/gid.
This fixes the intermittent "Permission denied" on .git/FETCH_HEAD
after the first git pull on a mount with uid/gid mapping.
* mount: add debug logging for stale lock file investigation
Add V(0) logging to trace the HEAD.lock recreation issue:
- Create: log when O_EXCL fails (file already exists) with uid/gid/mode
- completeAsyncFlush: log resolved path, saved path, dirtyMetadata,
isDeleted at entry to trace whether async flush fires after rename
- flushMetadataToFiler: log the dir/name/fullpath being flushed
This will show whether the async flush is recreating the lock file
after git renames HEAD.lock → HEAD.
* mount: prevent async flush from recreating renamed .lock files
When git renames HEAD.lock → HEAD, the async flush from the prior
close() can run AFTER the rename and re-insert HEAD.lock into the
meta cache via its CreateEntryRequest response event. The next git
pull then sees HEAD.lock and fails with "File exists".
Fix: add isRenamed flag on FileHandle, set by Rename before waiting
for the pending async flush. The async flush checks this flag and
skips the metadata flush for renamed files (same pattern as isDeleted
for unlinked files). The data pages still flush normally.
The Rename handler flushes deferred metadata synchronously (Case 1)
before setting isRenamed, ensuring the entry exists on the filer for
the rename to proceed. For already-released handles (Case 2), the
entry was created by a prior flush.
* mount: also mark renamed inodes via entry.Attributes.Inode fallback
When GetInode fails (Forget already removed the inode mapping), the
Rename handler couldn't find the pending async flush to set isRenamed.
The async flush then recreated the .lock file on the filer.
Fix: fall back to oldEntry.Attributes.Inode to find the pending async
flush when the inode-to-path mapping is gone. Also extract
MarkInodeRenamed into a method on FileHandleToInode for clarity.
* mount: skip async metadata flush when saved path no longer maps to inode
The isRenamed flag approach failed for refs/remotes/origin/HEAD.lock
because neither GetInode nor oldEntry.Attributes.Inode could find the
inode (Forget already evicted the mapping, and the entry's stored
inode was 0).
Add a direct check in completeAsyncFlush: before flushing metadata,
verify that the saved path still maps to this inode in the inode-to-path
table. If the path was renamed or removed (inode mismatch or not found),
skip the metadata flush to avoid recreating a stale entry.
This catches all rename cases regardless of whether the Rename handler
could set the isRenamed flag.
* mount: wait for pending async flush in Unlink before filer delete
Unlink was deleting the filer entry first, then marking the draining
async-flush handle as deleted. The async flush worker could race between
these two operations and recreate the just-unlinked entry on the filer.
This caused git's .lock files (e.g. refs/remotes/origin/HEAD.lock) to
persist after git pull, breaking subsequent git operations.
Move the isDeleted marking and add waitForPendingAsyncFlush() before
the filer delete so any in-flight flush completes first. Even if the
worker raced past the isDeleted check, the wait ensures it finishes
before the filer delete cleans up any recreated entry.
* mount: reduce async flush and metadata flush log verbosity
Raise completeAsyncFlush entry log, saved-path-mismatch skip log, and
flushMetadataToFiler entry log from V(0) to V(3)/V(4). These fire for
every file close with writebackCache and are too noisy for normal use.
* filer: reduce hardlink debug log verbosity from V(0) to V(4)
HardLinkId logs in filerstore_wrapper, filerstore_hardlink, and
filer_grpc_server fire on every hardlinked file operation (git pack
files use hardlinks extensively) and produce excessive noise.
* mount/filer: reduce noisy V(0) logs for link, rmdir, and empty folder check
- weedfs_link.go: hardlink creation logs V(0) → V(4)
- weedfs_dir_mkrm.go: non-empty folder rmdir error V(0) → V(1)
- empty_folder_cleaner.go: "not empty" check log V(0) → V(4)
* filer: handle missing hardlink KV as expected, not error
A "kv: not found" on hardlink read is normal when the link blob was
already cleaned up but a stale entry still references it. Log at V(1)
for not-found; keep Error level for actual KV failures.
* test: add waitForDir before git pull in FUSE git operations test
After git reset --hard, the FUSE mount's metadata cache may need a
moment to settle on slow CI. The git pull subprocess (unpack-objects)
could fail to stat the working directory. Poll for up to 5s.
* Update git_operations_test.go
* wait
* test: simplify FUSE test framework to use weed mini
Replace the 4-process setup (master + volume + filer + mount) with
2 processes: "weed mini" (all-in-one) + "weed mount". This simplifies
startup, reduces port allocation, and is faster on CI.
* test: fix mini flag -admin → -admin.ui
* filer.sync: replace O(n) conflict check with O(depth) index lookups
The MetadataProcessor.conflictsWith() scanned all active jobs linearly
for every new event dispatch. At high concurrency (256-1024), this O(n)
scan under the activeJobsLock became a bottleneck that throttled the
event dispatch pipeline, negating the benefit of higher -concurrency
values.
Replace the linear scan with three index maps:
- activeFilePaths: O(1) exact file path lookup
- activeDirPaths: O(1) directory path lookup per ancestor
- descendantCount: O(1) check for active jobs under a directory
Conflict check is now O(depth) where depth is the path depth (typically
3-6 levels), constant regardless of active job count. Benchmark confirms
~81ns per check whether there are 32 or 1024 active jobs.
Also replace the O(n) watermark scan with minActiveTs tracking so
non-oldest job completions are O(1).
Ref: #8771
* filer.sync: replace O(n) watermark rescan with min-heap lazy deletion
Address review feedback:
- Replace minActiveTs O(n) rescan with a tsMinHeap using lazy deletion.
Each TsNs is pushed once and popped once, giving O(log n) amortized
watermark tracking regardless of completion order.
- Fix benchmark to consume conflictsWith result via package-level sink
variable to prevent compiler elision.
The watermark advancement semantics (conservative, sets to completing
job's TsNs) are unchanged from the original code. This is intentionally
safe for idempotent replay on restart.
* feat(k8s): added possibility to specify service.type for multiple services in helm chart
* fix(k8s): removed headless (clusterIP: None) from services
* fix(k8s): keep master and filer services headless for StatefulSet compatibility
Master and filer services must remain headless (clusterIP: None) because
their StatefulSets reference them via serviceName for stable pod DNS.
Revert the service.type change for these two services and remove their
unused service config from values.yaml. S3 and SFTP remain configurable.
---------
Co-authored-by: Andreas Røste <andreas2101@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mount: defer file creation gRPC to flush time for faster small file writes
When creating a file via FUSE Create(), skip the synchronous gRPC
CreateEntry call to the filer. Instead, allocate the inode and build
the entry locally, deferring the filer create to the Flush/Release path
where flushMetadataToFiler already sends a CreateEntry with chunk data.
This eliminates one synchronous gRPC round-trip per file during creation.
For workloads with many small files (e.g. 30K files), this reduces the
per-file overhead from ~2 gRPC calls to ~1.
Mknod retains synchronous filer creation since it has no file handle
and thus no flush path.
* mount: use bounded worker pool for async flush operations
Replace unbounded goroutine spawning in writebackCache async flush
with a fixed-size worker pool backed by a channel. When many files
are closed rapidly (e.g., cp -r of 30K files), the previous approach
spawned one goroutine per file, leading to resource contention on
gRPC/HTTP connections and high goroutine overhead.
The worker pool size matches ConcurrentWriters (default 128), which
provides good parallelism while bounding resource usage. Work items
are queued into a buffered channel and processed by persistent worker
goroutines.
* mount: fix deferred create cache visibility and async flush race
Three fixes for the deferred create and async flush changes:
1. Insert a local placeholder entry into the metadata cache during
deferred file creation so that maybeLoadEntry() can find the file
for duplicate-create checks, stat, and readdir. Uses InsertEntry
directly (not applyLocalMetadataEvent) to avoid triggering the
directory hot-threshold eviction that would wipe the entry.
2. Fix race in ReleaseHandle where asyncFlushWg.Add(1) and the
channel send happened after pendingAsyncFlushMu was unlocked.
A concurrent WaitForAsyncFlush could observe a zero counter,
close the channel, and cause a send-on-closed panic. Move Add(1)
before the unlock; keep the send after unlock to avoid deadlock
with workers that acquire the same mutex during cleanup.
3. Update TestCreateCreatesAndOpensFile to flush the file handle
before verifying the CreateEntry gRPC call, since file creation
is now deferred to flush time.
* S3: reject part uploads after AbortMultipartUpload
PutObjectPartHandler did not verify that the multipart upload session
still exists before accepting parts. After AbortMultipartUpload deleted
the upload directory, the ErrNotFound from getEntry was silently ignored
(treated as "may be non-SSE upload"), allowing parts to be stored as
orphaned files.
Now return ErrNoSuchUpload when the upload directory is not found,
matching AWS S3 behavior.
Fixes#8766
* S3: check upload existence unconditionally in PutObjectPartHandler
Move the getEntry call out of the SSE-type conditional so the upload
existence check runs for all part uploads, including SSE-C. Previously
the SSE-C path skipped the check entirely, allowing parts to be uploaded
after abort when SSE-C headers were present.
Also flattens the nested SSE branching by one level now that getEntry
is called once upfront.
* S3: address PR review feedback for PutObjectPartHandler
- Log at error level when getEntry fails with an unexpected error,
since we return ErrInternalError to the client
- Distinguish base IV decode errors from length validation failures
with separate, clearer error messages
---------
Co-authored-by: Copilot <copilot@github.com>
* filer: add FilerError enum and error_code field to CreateEntryResponse
Add a machine-readable error code alongside the existing string error
field. This follows the precedent set by PublishMessageResponse in the
MQ broker proto. The string field is kept for human readability and
backward compatibility.
Defined codes: OK, ENTRY_NAME_TOO_LONG, PARENT_IS_FILE,
EXISTING_IS_DIRECTORY, EXISTING_IS_FILE, ENTRY_ALREADY_EXISTS.
* filer: add sentinel errors and error code mapping in filer_pb
Define sentinel errors (ErrEntryNameTooLong, ErrParentIsFile, etc.) in
the filer_pb package so both the filer and consumers can reference them
without circular imports.
Add FilerErrorToSentinel() to map proto error codes to sentinels, and
update CreateEntryWithResponse() to check error_code first, falling back
to the string-based path for backward compatibility with old servers.
* filer: return wrapped sentinel errors and set proto error codes
Replace fmt.Errorf string errors in filer.CreateEntry, UpdateEntry, and
ensureParentDirectoryEntry with wrapped filer_pb sentinel errors (using
%w). This preserves errors.Is() traversal on the server side.
In the gRPC CreateEntry handler, map sentinel errors to the
corresponding FilerError proto codes using errors.Is(), setting both
resp.Error (string, for backward compat) and resp.ErrorCode (enum).
* S3: use errors.Is() with filer sentinels instead of string matching
Replace fragile string-based error matching in filerErrorToS3Error and
other S3 API consumers with errors.Is() checks against filer_pb sentinel
errors. This works because the updated CreateEntryWithResponse helper
reconstructs sentinel errors from the proto FilerError code.
Update iceberg stage_create and metadata_files to check resp.ErrorCode
instead of parsing resp.Error strings. Update SSE-S3 to use errors.Is()
for the already-exists check.
String matching is retained only for non-filer errors (gRPC transport
errors, checksum validation) that don't go through CreateEntryResponse.
* filer: remove backward-compat string fallbacks for error codes
Clients and servers are always deployed together, so there is no need
for backward-compatibility fallback paths that parse resp.Error strings
when resp.ErrorCode is unset. Simplify all consumers to rely solely on
the structured error code.
* iceberg: ensure unknown non-OK error codes are not silently ignored
When FilerErrorToSentinel returns nil for an unrecognized error code,
return an error including the code and message rather than falling
through to return nil.
* filer: fix redundant error message and restore error wrapping in helper
Use request path instead of resp.Error in the sentinel error format
string to avoid duplicating the sentinel message (e.g. "entry already
exists: entry already exists"). Restore %w wrapping with errors.New()
in the fallback paths so callers can use errors.Is()/errors.As().
* filer: promote file to directory on path conflict instead of erroring
S3 allows both "foo/bar" (object) and "foo/bar/xyzzy" (another object)
to coexist because S3 has a flat key space. When ensureParentDirectoryEntry
finds a parent path that is a file instead of a directory, promote it to
a directory by setting ModeDir while preserving the original content and
chunks. Use Store.UpdateEntry directly to bypass the Filer.UpdateEntry
type-change guard.
This fixes the S3 compatibility test failures where creating overlapping
keys (e.g. "foo/bar" then "foo/bar/xyzzy") returned ExistingObjectIsFile.
AWS S3 policy conditions reference request headers with the s3: namespace
prefix (e.g., s3:x-amz-server-side-encryption). The extraction code was
storing these headers without the prefix, so bucket policy conditions
using the standard AWS key names would never match.
* S3: add KeyTooLongError error code
Add ErrKeyTooLongError (HTTP 400, code "KeyTooLongError") to match the
standard AWS S3 error for object keys that exceed length limits.
* S3: fix silent PutObject failure when entry name exceeds max_file_name_length
putToFiler called client.CreateEntry() directly and discarded the gRPC
response. The filer embeds application errors like "entry name too long"
in resp.Error (not as gRPC transport errors), so the error was silently
swallowed and clients received HTTP 200 with an ETag for objects that
were never stored.
Switch to the filer_pb.CreateEntry() helper which properly checks
resp.Error, and map "entry name too long" to KeyTooLongError (HTTP 400).
To avoid fragile string parsing across the gRPC boundary, define shared
error message constants in weed/util/constants and use them in both the
filer (producing errors) and S3 API (matching errors). Switch
filerErrorToS3Error to use strings.Contains/HasSuffix with these
constants so matches work regardless of any wrapper prefix. Apply
filerErrorToS3Error to the mkdir path for directory markers.
Fixes#8759
* S3: enforce 1024-byte maximum object key length
AWS S3 limits object keys to 1024 bytes. Add early validation on write
paths (PutObject, CopyObject, CreateMultipartUpload) to reject keys
exceeding the limit with the standard KeyTooLongError (HTTP 400).
The key length check runs before bucket auto-creation to prevent
overlong keys from triggering unnecessary side effects.
Also use filerErrorToS3Error for CopyObject's mkFile error paths so
name-too-long errors from the filer return KeyTooLongError instead of
InternalError.
Ref #8758
* S3: add handler-level tests for key length validation and error mapping
Add tests for filerErrorToS3Error mapping "entry name too long" to
KeyTooLongError, including a regression test for the CreateEntry-prefixed
"existing ... is a directory" form. Add handler-level integration tests
that exercise PutObjectHandler, CopyObjectHandler, and
NewMultipartUploadHandler via httptest, verifying HTTP 400 and
KeyTooLongError XML response for overlong keys and acceptance of keys at
the 1024-byte limit.
* fix: resolve Kafka gateway response deadlocks causing Sarama client hangs
Fix three bugs in the Kafka protocol handler that caused sequential
clients (notably Sarama) to hang during E2E tests:
1. Race condition in correlation queue ordering: the correlation ID was
added to the response ordering queue AFTER sending the request to
the processing channel. A fast processor (e.g. ApiVersions) could
finish and send its response before the ID was in the queue, causing
the response writer to miss it — permanently deadlocking the
connection. Now the ID is added BEFORE the channel send, with error
response injection on send failure.
2. Silent error response drops: when processRequestSync returned an
error, the response writer logged it but never sent anything back to
the client. The client would block forever waiting for bytes that
never arrived. Now sends a Kafka UNKNOWN_SERVER_ERROR response.
3. Produce V0/V1 missing timeout_ms parsing: the handler skipped the
4-byte timeout field, reading it as topicsCount instead. This caused
incorrect parsing of the entire produce request for V0/V1 clients.
* fix: API-versioned error responses, unsupported-version queue fix, V0V1 header alignment
1. errors.go — BuildAPIErrorResponse: emits a minimal-but-valid error
body whose layout matches the schema the client expects for each API
key and version (throttle_time position, array fields, etc.). The
old 2-byte generic body corrupted the protocol stream for APIs whose
response begins with throttle_time_ms or an array.
2. handler.go — unsupported-version path: the correlationID was never
added to correlationQueue before sending to responseChan, so the
response writer could never match it and the client hung. Now
appends the ID under correlationQueueMu before the send.
3. produce.go — handleProduceV0V1: requestBody is already post-header
(HandleConn strips client_id). The handler was erroneously parsing
acks bytes as a client_id length, misaligning all subsequent field
reads. Removed the client_id parsing; offset now starts at 0 with
acks(2) + timeout_ms(4) + topicsCount(4), matching handleProduceV2Plus.
* fix: free pooled message buffer per-iteration instead of deferring
The read loop allocated messageBuf via mem.Allocate and deferred
mem.Free. Since the defer only runs when HandleConn returns, pool
buffers accumulated for the entire connection lifetime — one per
request. Worse, the deferred frees ran in LIFO order before
wg.Wait(), so processing goroutines could read from already-freed
pool buffers.
Now: read into a pooled buffer, immediately copy to Go-managed
memory, and return the pool buffer. messageBuf is a regular slice
safe for async goroutine access with no defer accumulation.
* fix: cancel context before wg.Wait and on worker response-send timeout
Two related issues:
1. Cleanup defer ordering deadlock: defers run LIFO — the cleanup defer
(close channels, wg.Wait) ran before the cancel() defer. The
response writer is in the WaitGroup and exits only on ctx.Done() or
responseChan close, but both signals came after wg.Wait(). Deadlock
on every normal connection close (EOF, read error, queue-full).
Fix: call cancel() at the start of the cleanup defer, before
wg.Wait().
2. Worker 5s response-send timeout: when the timeout fired, the
response was silently dropped but the correlationID remained in the
ordered queue. The response writer could never advance past it,
stalling all subsequent responses permanently.
Fix: call cancel() to tear down the connection — if we cannot
deliver a response in 5s the connection is irrecoverable.
* chore: remove empty no-op ListOffsets conditional
The `if apiKey == 2 {}` block had no body — leftover debug code.
ListOffsets routing is handled by isDataPlaneAPI (returns false,
sending it to the control channel). No behavior change.
* mount: implement create for rsync temp files
* mount: move access implementation out of unsupported
* mount: tighten access checks
* mount: log access group lookup failures
* mount: reset dirty pages on truncate
* mount: tighten create and root access handling
* mount: handle existing creates before quota checks
* mount: restrict access fallback when group lookup fails
When lookupSupplementaryGroupIDs returns an error, the previous code
fell through to checking only the "other" permission bits, which could
overgrant access. Require both group and other permission classes to
satisfy the mask so access is never broader than intended.
* mount: guard against nil entry in Create existing-file path
maybeLoadEntry can return OK with a nil entry or nil Attributes in
edge cases. Check before dereferencing to prevent a panic.
* mount: reopen existing file on create race without O_EXCL
When createRegularFile returns EEXIST because another process won the
race, and O_EXCL is not set, reload the winner's entry and open it
instead of propagating the error to the caller.
* mount: check parent directory permission in createRegularFile
Verify the caller has write+search (W_OK|X_OK) permission on the
parent directory before creating a file. This applies to both
Create and Mknod. Update test fixture mount mode to 0o777 so the
existing tests pass with the new check.
* mount: enforce file permission bits in AcquireHandle
Map the open flags (O_RDONLY/O_WRONLY/O_RDWR) to an access mask and
call hasAccess before handing out a file handle. This makes
AcquireHandle the single source of truth for mode-based access
control across Open, Create-existing, and Create-new paths.
---------
Co-authored-by: Copilot <copilot@github.com>
* Add FUSE integration tests for POSIX file locking
Test flock() and fcntl() advisory locks over the FUSE mount:
- Exclusive and shared flock with conflict detection
- flock upgrade (shared to exclusive) and release on close
- fcntl F_SETLK write lock conflicts and shared read locks
- fcntl F_GETLK conflict reporting on overlapping byte ranges
- Non-overlapping byte-range locks held independently
- F_SETLKW blocking until conflicting lock is released
- Lock release on file descriptor close
- Concurrent lock contention with multiple workers
* Fix review feedback in POSIX lock integration tests
- Assert specific EAGAIN error on fcntl lock conflicts instead of generic Error
- Use O_APPEND in concurrent contention test so workers append rather than overwrite
- Verify exact line count (numWorkers * writesPerWorker) after concurrent test
- Check unlock error in F_SETLKW blocking test goroutine
* Refactor fcntl tests to use subprocesses for inter-process semantics
POSIX fcntl locks use the process's files_struct as lock owner, so all
fds in the same process share the same owner and never conflict. This
caused the fcntl tests to silently pass without exercising lock conflicts.
Changes:
- Add TestFcntlLockHelper subprocess entry point with hold/try/getlk actions
- Add lockHolder with channel-based coordination (no scanner race)
- Rewrite all fcntl tests to run contenders in separate subprocesses
- Fix F_UNLCK int16 cast in GetLk assertion for type-safe comparison
- Fix concurrent test: use non-blocking flock with retry to avoid
exhausting go-fuse server reader goroutines (blocking FUSE SETLKW
can starve unlock request processing, causing deadlock)
flock tests remain same-process since flock uses per-struct-file owners.
* Fix misleading comment and error handling in lock test subprocess
- Fix comment: tryLockInSubprocess tests a subprocess, not the test process
- Distinguish EAGAIN/EACCES from unexpected errors in subprocess try mode
so real failures aren't silently masked as lock conflicts
* Fix CI race in FcntlReleaseOnClose and increase flock retry budget
- FcntlReleaseOnClose: retry lock acquisition after subprocess exits
since the FUSE server may not process Release immediately
- ConcurrentLockContention: increase retry limit from 500 to 3000
(5s → 30s budget) to handle CI load
* separating flock and fcntl in the in-memory lock table and cleaning them up through the right release path: PID for POSIX locks, lock owner for flock
* ReleasePosixOwner
* weed/mount: flush before releasing posix close owner
* weed/mount: keep woken lock waiters from losing inode state
* test/fuse: make blocking fcntl helper state explicit
* test/fuse: assert flock contention never overlaps
* test/fuse: stabilize concurrent lock contention check
* test/fuse: make concurrent contention writes deterministic
* weed/mount: retry synchronous metadata flushes
* Add POSIX byte-range lock table for FUSE mount
Implement PosixLockTable with per-inode range lock tracking supporting:
- Shared (F_RDLCK) and exclusive (F_WRLCK) byte-range locks
- Conflict detection across different lock owners
- Lock coalescing for adjacent/overlapping same-owner same-type locks
- Lock splitting on partial-range unlock
- Blocking waiter support for SetLkw with cancellation
- Owner-based cleanup for Release
* Wire POSIX lock handlers into FUSE mount
Implement GetLk, SetLk, SetLkw on WFS delegating to PosixLockTable.
Add posixLocks field to WFS and initialize in constructor.
Clean up locks on Release via ReleaseOwner using ReleaseIn.LockOwner.
Remove ENOSYS stubs from weedfs_unsupported.go.
* Enable POSIX and flock lock capabilities in FUSE mount
Set EnableLocks: true in mount options to advertise
CAP_POSIX_LOCKS and CAP_FLOCK_LOCKS during FUSE INIT.
* Avoid thundering herd in lock waiter wake-up
Replace broadcast-all wakeWaiters with selective wakeEligibleWaiters
that checks each waiter's requested lock against remaining held locks.
Only waiters whose request no longer conflicts are woken; others stay
queued. Store the requested lockRange in each lockWaiter to enable this.
* Fix uint64 overflow in adjacency check for lock coalescing
Guard h.End+1 and lk.End+1 with < ^uint64(0) checks so that
End == math.MaxUint64 (EOF) does not wrap to 0 and falsely merge
non-adjacent locks.
* Add test for non-adjacent ranges with gap not being coalesced
The metadata subscription handler (updateBucketConfigCacheFromEntry) was
making a separate RPC call via loadCORSFromBucketContent to load CORS
configuration. This created a race window where a slow CreateBucket
subscription event could re-cache stale data after PutBucketCors had
already cleared the cache, causing subsequent GetBucketCors to return
404 NoSuchCORSConfiguration.
Parse CORS directly from the subscription entry's Content field instead
of making a separate RPC. Also fix getBucketConfig to parse CORS from
the already-fetched entry, eliminating a redundant RPC call.
Fix TestCORSCaching to use require.NoError to prevent nil pointer
dereference panics when GetBucketCors fails.
* fix: extend ignore404Error to match 404 Not Found string from S3 sink errors
* test: add unit tests for isIgnorable404 error matching
* improve: pre-compute ignorable 404 string and simplify isIgnorable404
* test: replace init() with TestMain for global HTTP client setup
* mount: async flush on close() when writebackCache is enabled
When -writebackCache is enabled, defer data upload and metadata flush
from Flush() (triggered by close()) to a background goroutine in
Release(). This allows processes like rsync that write many small files
to proceed to the next file immediately instead of blocking on two
network round-trips (volume upload + filer metadata) per file.
Fixes#8718
* mount: add retry with backoff for async metadata flush
The metadata flush in completeAsyncFlush now retries up to 3 times
with exponential backoff (1s, 2s, 4s) on transient gRPC errors.
Since the chunk data is already safely on volume servers at this point,
only the filer metadata reference needs persisting — retrying is both
safe and effective.
Data flush (FlushData) is not retried externally because
UploadWithRetry already handles transient HTTP/gRPC errors internally;
if it still fails, the chunk memory has been freed.
* test: add integration tests for writebackCache async flush
Add comprehensive FUSE integration tests for the writebackCache
async flush feature (issue #8718):
- Basic operations: write/read, sequential files, large files, empty
files, overwrites
- Fsync correctness: fsync forces synchronous flush even in writeback
mode, immediate read-after-fsync
- Concurrent small files: multi-worker parallel writes (rsync-like
workload), multi-directory, rapid create/close
- Data integrity: append after close, partial writes, file size
correctness, binary data preservation
- Performance comparison: writeback vs synchronous flush throughput
- Stress test: 16 workers x 100 files with content verification
- Mixed concurrent operations: reads, writes, creates running together
Also fix pre-existing test infrastructure issues:
- Rename framework.go to framework_test.go (fixes Go package conflict)
- Fix undefined totalSize variable in concurrent_operations_test.go
* ci: update fuse-integration workflow to run full test suite
The workflow previously only ran placeholder tests (simple_test.go,
working_demo_test.go) in a temp directory due to a Go module conflict.
Now that framework.go is renamed to framework_test.go, the full test
suite compiles and runs correctly from test/fuse_integration/.
Changes:
- Run go test directly in test/fuse_integration/ (no temp dir copy)
- Install weed binary to /usr/local/bin for test framework discovery
- Configure /etc/fuse.conf with user_allow_other for FUSE mounts
- Install fuse3 for modern FUSE support
- Stream test output to log file for artifact upload
* mount: fix three P1 races in async flush
P1-1: Reopen overwrites data still flushing in background
ReleaseByHandle removes the old handle from fhMap before the deferred
flush finishes. A reopen of the same inode during that window would
build from stale filer metadata, overwriting the async flush.
Fix: Track in-flight async flushes per inode via pendingAsyncFlush map.
AcquireHandle now calls waitForPendingAsyncFlush(inode) to block until
any pending flush completes before reading filer metadata.
P1-2: Deferred flush races rename and unlink after close
completeAsyncFlush captured the path once at entry, but rename or
unlink after close() could cause metadata to be written under the
wrong name or recreate a deleted file.
Fix: Re-resolve path from inode via GetPath right before metadata
flush. GetPath returns the current path (reflecting renames) or
ENOENT (if unlinked), in which case we skip the metadata flush.
P1-3: SIGINT/SIGTERM bypasses the async-flush drain
grace.OnInterrupt runs hooks then calls os.Exit(0), so
WaitForAsyncFlush after server.Serve() never executes on signal.
Fix: Add WaitForAsyncFlush (with 10s timeout) to the WFS interrupt
handler, before cache cleanup. The timeout prevents hanging on Ctrl-C
when the filer is unreachable.
* mount: fix P1 races — draining handle stays in fhMap
P1-1: Reopen TOCTOU
The gap between ReleaseByHandle removing from fhMap and
submitAsyncFlush registering in pendingAsyncFlush allowed a
concurrent AcquireHandle to slip through with stale metadata.
Fix: Hold pendingAsyncFlushMu across both the counter decrement
(ReleaseByHandle) and the pending registration. The handle is
registered as pending before the lock is released, so
waitForPendingAsyncFlush always sees it.
P1-2: Rename/unlink can't find draining handle
ReleaseByHandle deleted from fhMap immediately. Rename's
FindFileHandle(inode) at line 251 could not find the handle to
update entry.Name. Unlink could not coordinate either.
Fix: When asyncFlushPending is true, ReleaseByHandle/ReleaseByInode
leave the handle in fhMap (counter=0 but maps intact). The handle
stays visible to FindFileHandle so rename can update entry.Name.
completeAsyncFlush re-resolves the path from the inode (GetPath)
right before metadata flush for correctness after rename/unlink.
After drain, RemoveFileHandle cleans up the maps.
Double-return prevention: ReleaseByHandle/ReleaseByInode return nil
if counter is already <= 0, so Forget after Release doesn't start a
second drain goroutine.
P1-3: SIGINT deletes swap files under running goroutines
After the 10s timeout, os.RemoveAll deleted the write cache dir
(containing swap files) while FlushData goroutines were still
reading from them.
Fix: Increase timeout to 30s. If timeout expires, skip write cache
dir removal so in-flight goroutines can finish reading swap files.
The OS (or next mount) cleans them up. Read cache is always removed.
* mount: never skip metadata flush when Forget drops inode mapping
Forget removes the inode→path mapping when the kernel's lookup count
reaches zero, but this does NOT mean the file was unlinked — it only
means the kernel evicted its cache entry. completeAsyncFlush was
treating GetPath failure as "file unlinked" and skipping the metadata
flush, which orphaned the just-uploaded chunks for live files.
Fix: Save dir and name at doFlush defer time. In completeAsyncFlush,
try GetPath first to pick up renames; if the mapping is gone, fall
back to the saved dir/name. Always attempt the metadata flush — the
filer is the authority on whether the file exists, not the local
inode cache.
* mount: distinguish Forget from Unlink in async flush path fallback
The saved-path fallback (from the previous fix) always flushed
metadata when GetPath failed, which recreated files that were
explicitly unlinked after close(). The same stale fallback could
recreate the pre-rename path if Forget dropped the inode mapping
after a rename.
Root cause: GetPath failure has two meanings:
1. Forget — kernel evicted the cache entry (file still exists)
2. Unlink — file was explicitly deleted (should not recreate)
Fix (three coordinated changes):
Unlink (weedfs_file_mkrm.go): Before RemovePath, look up the inode
and find any draining handle via FindFileHandle. Set fh.isDeleted =
true so the async flush knows the file was explicitly removed.
Rename (weedfs_rename.go): When renaming a file with a draining
handle, update asyncFlushDir/asyncFlushName to the post-rename
location. This keeps the saved-path fallback current so Forget
after rename doesn't flush to the old (pre-rename) path.
completeAsyncFlush (weedfs_async_flush.go): Check fh.isDeleted
first — if true, skip metadata flush (file was unlinked, chunks
become orphans for volume.fsck). Otherwise, try GetPath for the
current path (renames); fall back to saved path if Forget dropped
the mapping (file is live, just evicted from kernel cache).
* test/ci: address PR review nitpicks
concurrent_operations_test.go:
- Restore precise totalSize assertion instead of info.Size() > 0
writeback_cache_test.go:
- Check rand.Read errors in all 3 locations (lines 310, 512, 757)
- Check os.MkdirAll error in stress test (line 752)
- Remove dead verifyErrors variable (line 332)
- Replace both time.Sleep(5s) with polling via waitForFileContent
to avoid flaky tests under CI load (lines 638, 700)
fuse-integration.yml:
- Add set -o pipefail so go test failures propagate through tee
* ci: fix fuse3/fuse package conflict on ubuntu-22.04 runner
fuse3 is pre-installed on ubuntu-22.04 runners and conflicts with
the legacy fuse package. Only install libfuse3-dev for the headers.
* mount/page_writer: remove debug println statements
Remove leftover debug println("read new data1/2") from
ReadDataAt in MemChunk and SwapFileChunk.
* test: fix findWeedBinary matching source directory instead of binary
findWeedBinary() matched ../../weed (the source directory) via
os.Stat before checking PATH, then tried to exec a directory
which fails with "permission denied" on the CI runner.
Fix: Check PATH first (reliable in CI where the binary is installed
to /usr/local/bin). For relative paths, verify the candidate is a
regular file (!info.IsDir()). Add ../../weed/weed as a candidate
for in-tree builds.
* test: fix framework — dynamic ports, output capture, data dirs
The integration test framework was failing in CI because:
1. All tests used hardcoded ports (19333/18080/18888), so sequential
tests could conflict when prior processes hadn't fully released
their ports yet.
2. Data subdirectories (data/master, data/volume) were not created
before starting processes.
3. Master was started with -peers=none which is not a valid address.
4. Process stdout/stderr was not captured, making failures opaque
("service not ready within timeout" with no diagnostics).
5. The unmount fallback used 'umount' instead of 'fusermount -u'.
6. The mount used -cacheSizeMB (nonexistent) instead of
-cacheCapacityMB and was missing -allowOthers=false for
unprivileged CI runners.
Fixes:
- Dynamic port allocation via freePort() (net.Listen ":0")
- Explicit gRPC ports via -port.grpc to avoid default port conflicts
- Create data/master and data/volume directories in Setup()
- Remove invalid -peers=none and -raftBootstrap flags
- Capture process output to logDir/*.log via startProcess() helper
- dumpLog() prints tail of log file on service startup failure
- Use fusermount3/fusermount -u for unmount
- Fix mount flag names (-cacheCapacityMB, -allowOthers=false)
* test: remove explicit -port.grpc flags from test framework
SeaweedFS convention: gRPC port = HTTP port + 10000. Volume and
filer discover the master gRPC port by this convention. Setting
explicit -port.grpc on master/volume/filer broke inter-service
communication because the volume server computed master gRPC as
HTTP+10000 but the actual gRPC was on a different port.
Remove all -port.grpc flags and let the default convention work.
Dynamic HTTP ports already ensure uniqueness; the derived gRPC
ports (HTTP+10000) will also be unique.
---------
Co-authored-by: Copilot <copilot@github.com>
* admin/plugin: delete job_detail files when jobs are pruned from memory
pruneTrackedJobsLocked evicts the oldest terminal jobs from the in-memory
tracker when the total exceeds maxTrackedJobsTotal (1000). However the
dedicated per-job detail files in jobs/job_details/ were never removed,
causing them to accumulate indefinitely on disk.
Add ConfigStore.DeleteJobDetail and call it from pruneTrackedJobsLocked so
that the file is cleaned up together with the in-memory entry. Deletion
errors are logged at verbosity level 2 and do not abort the prune.
* admin/plugin: add test for DeleteJobDetail
---------
Co-authored-by: Anton Ustyugov <anton@devops>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
loadPersistedMonitorState performed a backward-compatibility migration that
wrote every job with inline rich detail fields to a dedicated per-job detail
file synchronously during startup. On deployments with many historical jobs
(e.g. 1000+) stored on distributed block storage (e.g. Longhorn), each
individual file write requires an fsync round-trip, making startup
disproportionately slow and causing readiness/liveness probe failures.
The in-memory state is populated correctly before the goroutine is started
because stripTrackedJobDetailFields is still called in-place; only the disk
writes are deferred. A completion log message at V(1) is emitted once the
background migration finishes.
Co-authored-by: Anton Ustyugov <anton@devops>
RunPluginJobTypeAPI previously executed proposals with a naive sequential loop
calling ExecutePluginJob per proposal. This had two bugs:
1. Double-lock: RunPluginJobTypeAPI held pluginLock while calling ExecutePluginJob,
which tried to re-acquire the same lock for every job in the loop.
2. No capacity management: proposals were fired directly at workers without
reserveScheduledExecutor, so every job beyond the worker concurrency limit
received an immediate at_capacity error with no retry or backoff.
Fix: add Plugin.DispatchProposals which reuses dispatchScheduledProposals - the
same code path the scheduler loop uses - with executor reservation, configurable
concurrency, and per-job retry with backoff. RunPluginJobTypeAPI now calls
DispatchPluginProposals (a thin AdminServer wrapper) after holding pluginLock once.
Co-authored-by: Anton Ustyugov <anton@devops>
RequestLock used a bare println to report transient lock acquisition failures
('lock: already locked by ...'), which writes directly to stdout instead of
going through the structured logging pipeline. This causes log noise at the
wrong level and cannot be filtered with -v or redirected like glog output.
Changes:
- println("lock:", ...) -> glog.V(2).Infof for per-retry acquisition errors
(transient, high-frequency during startup when another instance still holds)
- Add glog.V(1).Infof when the lock is successfully acquired
- Add glog.V(2).Infof for successful renewals (replaces commented-out println)
- Errorf -> Warningf for renewal failures (the goroutine exits cleanly, it is
not a fatal error; the caller will re-acquire via RequestLock)
Co-authored-by: Anton Ustyugov <anton@devops>
* feat(shell): add volume.tier.compact command to reclaim cloud storage space
Adds a new shell command that automates compaction of cloud tier volumes.
When files are deleted from remote-tiered volumes, space is not reclaimed
on the cloud storage. This command orchestrates: download from remote,
compact locally, and re-upload to reclaim deleted space.
Closes#8563
* fix: log cleanup errors in compactVolumeOnServer instead of discarding them
Helps operators diagnose leftover temp files (.cpd/.cpx) if cleanup
fails after a compaction or commit failure.
* fix: return aggregate error from loop and use regex for collection filter
- Track and return error count when one or more volumes fail to compact,
so callers see partial failures instead of always getting nil.
- Use compileCollectionPattern for -collection in -volumeId mode too, so
regex patterns work consistently with the flag description. Empty
pattern (no -collection given) matches all collections.
* fix(telemetry): use correct TopologyId field in integration test
The proto field was renamed from cluster_id to topology_id but the
integration test was not updated, causing a compilation error.
* ci: add telemetry integration test workflow
Runs the telemetry integration test (server startup, protobuf
marshaling, client send, metrics/stats/instances API checks) on
changes to telemetry/ or weed/telemetry/.
* fix(telemetry): improve error message specificity in integration test
* fix(ci): pre-build telemetry server binary for integration test
go run compiles the server on the fly, which exceeds the 15s startup
timeout in CI. Build the binary first so the test starts instantly.
* fix(telemetry): fix ClusterId references in server and CI build path
- Replace ClusterId with TopologyId in server storage and API handler
(same rename as the integration test fix)
- Fix CI build: telemetry server has its own go.mod, so build from
within its directory
* ci(telemetry): add least-privilege permissions to workflow
Scope the workflow token to read-only repository contents, matching
the convention used in go.yml.
* fix(telemetry): set TopologyId in client integration test
The client only populates TopologyId when SetTopologyId has been
called. The test was missing this call, causing the server to reject
the request with 400 (missing required field).
* fix(telemetry): delete clusterInfo metric on instance cleanup
The cleanup loop removed all per-instance metrics except clusterInfo,
leaking that label set after eviction.
Key improvements:
- Fix concurrent map write panic in partition type cache
- Fix data races in yieldDataFiles and key map getter
- Fix response body leaks in REST catalog
- Fix index out of range in buildManifestEvaluator
- Table Metadata V3 support
- Schema evolution API
- Partitioned write throughput optimizations
- Gzipped metadata read/write support
* glog: add JSON structured logging mode
Add opt-in JSON output format for glog, enabling integration with
log aggregation systems like ELK, Loki, and Datadog.
- Add --log_json flag to enable JSON output at startup
- Add SetJSONMode()/IsJSONMode() for runtime toggle
- Add JSON branches in println, printDepth, printf, printWithFileLine
- Use manual JSON construction (no encoding/json) for performance
- Add jsonEscapeString() for safe string escaping
- Include 8 unit tests and 1 benchmark
Enabled via --log_json flag. Default behavior unchanged.
* glog: prevent lazy flag init from overriding SetJSONMode
If --log_json=true and SetJSONMode(false) was called at runtime,
a subsequent IsJSONMode() call would re-enable JSON mode via the
sync.Once lazy initialization. Mark jsonFlagOnce as done inside
SetJSONMode so the runtime API always takes precedence.
* glog: fix RuneError check to not misclassify valid U+FFFD
The condition r == utf8.RuneError matches both invalid UTF-8
sequences (size=1) and a valid U+FFFD replacement character
(size=3). Without checking size == 1, a valid U+FFFD input
would be incorrectly escaped and only advance by 1 byte,
corrupting the output.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
