Commit Graph

35 Commits

Author SHA1 Message Date
Chris Lu
51ec0d2122 fix(remote_gateway): prevent double-versioning when syncing to versioned central bucket (#8710)
* fix(remote_gateway): prevent double-versioning when syncing to versioned central bucket

When a file is uploaded to a versioned bucket on edge, SeaweedFS stores
it internally as {object}.versions/v_{versionId}. The remote_gateway was
syncing this internal path directly to the central S3 endpoint. When
central's bucket also has versioning enabled, this caused central to
apply its own versioning on top, producing corrupt paths like:
  object.versions/v_{edgeId}.versions/v_{centralId}

Fix: rewrite internal .versions/v_{id} paths to the original S3 object
key before uploading to the remote. Skip version file delete/update
events that are internal bookkeeping.

Fixes https://github.com/seaweedfs/seaweedfs/discussions/8481#discussioncomment-16209342

* fix(remote_gateway): propagate delete markers to remote as deletions

Delete markers are zero-content version entries (ExtDeleteMarkerKey=true)
created by S3 DELETE on a versioned bucket. Previously they were silently
dropped by the HasData() filter, so deletions on edge never reached
central.

Now: detect delete markers before the HasData check, rewrite the
.versions path to the original S3 key, and issue client.DeleteFile()
on the remote.

* fix(remote_gateway): tighten isVersionedPath to avoid false positives

Address PR review feedback:

- Add isDir parameter to isVersionedPath so it only matches the exact
  internal shapes: directories whose name ends with .versions (isDir=true),
  and files with the v_ prefix inside a .versions parent (isDir=false).
  Previously the function was too broad and could match user-created paths
  like "my.versions/data.txt".

- Update all 4 call sites to pass the entry's IsDirectory field.

- Rename TestVersionedDirectoryNotFilteredByHasData to
  TestVersionsDirectoryFilteredByHasData so the name reflects the
  actual assertion (directories ARE filtered by HasData).

- Expand TestIsVersionedPath with isDir cases and false-positive checks.

* fix(remote_gateway): persist sync marker after delete-marker propagation

The delete-marker branch was calling client.DeleteFile() and returning
without updating the local entry, making event replay re-issue the
remote delete. Now call updateLocalEntry after a successful DeleteFile
to stamp the delete-marker entry with a RemoteEntry, matching the
pattern used by the normal create path.

* refactor(remote_gateway): extract syncDeleteMarker and fix root path edge case

- Extract syncDeleteMarker() shared helper used by both bucketed and
  mounted-dir event processors, replacing the duplicated delete + persist
  local marker logic.

- Fix rewriteVersionedSourcePath for root-level objects: when lastSlash
  is 0 (e.g. "/file.xml.versions"), return "/" as the parent dir instead
  of an empty string.

- The strings.Contains(dir, ".versions/") condition flagged in review was
  already removed in a prior commit that tightened isVersionedPath.

* fix(remote_gateway): skip updateLocalEntry for versioned path rewrites

After rewriting a .versions/v_{id} path to the logical S3 key and
uploading, the code was calling updateLocalEntry on the original v_*
entry, stamping it with a RemoteEntry for the logical key. This is
semantically wrong: the logical object has no filer entry in versioned
buckets, and the internal v_* entry should not carry a RemoteEntry for
a different path.

Skip updateLocalEntry when the path was rewritten from a versioned
source. Replay safety is preserved because S3 PutObject is idempotent.

* fix(remote_gateway): scope versioning checks to /buckets/ namespace

isVersionedPath and rewriteVersionedSourcePath could wrongly match
paths in non-bucket mounts (e.g. /mnt/remote/file.xml.versions).

Add the same /buckets/ prefix guard used by isMultipartUploadDir so
the .versions / v_ logic only applies within the bucket namespace.
2026-03-19 21:18:52 -07:00
Chris Lu
8269dc136d simplify 2026-01-04 11:26:21 -08:00
Chris Lu
69553e5ba6 convert error fromating to %w everywhere (#6995) 2025-07-16 23:39:27 -07:00
Aleksey Kosov
165af32d6b added context to filer_client method calls (#6808)
Co-authored-by: akosov <a.kosov@kryptonite.ru>
2025-05-22 09:46:49 -07:00
chrislu
3a82f5ffad ensure metadata follow a specific folder
fix https://github.com/seaweedfs/seaweedfs/issues/5774
2024-07-12 11:17:30 -07:00
chrislu
a829f36d17 adjust error handling
fix https://github.com/seaweedfs/seaweedfs/issues/5677
2024-06-17 10:59:41 -07:00
chrislu
bb867694d8 skip updating multipart .upload files 2024-01-18 09:24:52 -08:00
chrislu
5c90b27b0a go fmt 2024-01-18 09:17:08 -08:00
chrislu
15b66a6633 refactor 2024-01-18 09:13:14 -08:00
Konstantin Lebedev
1169f94310 Fix filer sync set offset (#5197)
* fix: compose 2mount with sync

* fix: DATA RACE
https://github.com/seaweedfs/seaweedfs/issues/5194
https://github.com/seaweedfs/seaweedfs/issues/5195
2024-01-12 10:57:18 -08:00
chrislu
e0727071c8 go fmt 2023-12-22 11:33:50 -08:00
chrislu
915416a277 fix wrong parameter
fix https://github.com/seaweedfs/seaweedfs/pull/5102
2023-12-13 12:02:57 -08:00
chrislu
5db9fcccd4 refactoring 2023-03-21 23:01:49 -07:00
Konstantin Lebedev
6fa3d0cc46 fix delete key panic of remote sync dir (#3770) 2022-10-01 02:33:47 -07:00
chrislu
b834027c5a refactor 2022-09-11 21:53:15 -07:00
chrislu
2c6b68b40e simplify a bit 2022-09-10 14:18:28 -07:00
Konstantin Lebedev
b64674018a [sync] override amz storage class, None to delete (#3639)
* override amz storage class, None to delete
https://github.com/seaweedfs/seaweedfs/issues/3636

* use empty string to delete

* without nil check
2022-09-10 14:15:42 -07:00
chrislu
205ecb5d03 skip directories under ".uploads" directory
related to https://github.com/seaweedfs/seaweedfs/issues/3636

skipping all directories under ".uploads" directory.
2022-09-09 22:57:38 -07:00
chrislu
b4131a3a72 filer.remote.sync: parallelize uploading to remove object storage 2022-08-22 00:01:48 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu
202a29d014 refactoring 2022-02-25 01:17:26 -08:00
chrislu
5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
Chris Lu
ad16221a35 adjust error log 2021-11-28 22:06:17 -08:00
Chris Lu
cf1586a34d add logs for writing to remote file 2021-11-27 22:09:23 -08:00
Chris Lu
6c27845be0 add retries when writing to remote s3
fix https://github.com/chrislusf/seaweedfs/issues/2465
2021-11-22 21:48:04 -08:00
Chris Lu
63da4bbb54 separate filer.remote.gateway command to avoid confusion 2021-09-15 22:47:17 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
98c68d7a79 filer.remote.sync fix upload logic 2021-09-04 18:46:28 -07:00
Chris Lu
796b2eb929 remember sync time 2021-09-04 13:46:22 -07:00
Chris Lu
4b28c5f6c3 filer.remote.sync: split into buckets mode and single directory mode 2021-09-04 04:35:46 -07:00