Commit Graph

7 Commits

Author SHA1 Message Date
Chris Lu
29bdbb3c48 filer.sync: replace O(n) conflict check with O(depth) index lookups (#8772)
* filer.sync: replace O(n) conflict check with O(depth) index lookups

The MetadataProcessor.conflictsWith() scanned all active jobs linearly
for every new event dispatch. At high concurrency (256-1024), this O(n)
scan under the activeJobsLock became a bottleneck that throttled the
event dispatch pipeline, negating the benefit of higher -concurrency
values.

Replace the linear scan with three index maps:
- activeFilePaths: O(1) exact file path lookup
- activeDirPaths: O(1) directory path lookup per ancestor
- descendantCount: O(1) check for active jobs under a directory

Conflict check is now O(depth) where depth is the path depth (typically
3-6 levels), constant regardless of active job count. Benchmark confirms
~81ns per check whether there are 32 or 1024 active jobs.

Also replace the O(n) watermark scan with minActiveTs tracking so
non-oldest job completions are O(1).

Ref: #8771

* filer.sync: replace O(n) watermark rescan with min-heap lazy deletion

Address review feedback:
- Replace minActiveTs O(n) rescan with a tsMinHeap using lazy deletion.
  Each TsNs is pushed once and popped once, giving O(log n) amortized
  watermark tracking regardless of completion order.
- Fix benchmark to consume conflictsWith result via package-level sink
  variable to prevent compiler elision.

The watermark advancement semantics (conservative, sets to completing
job's TsNs) are unchanged from the original code. This is intentionally
safe for idempotent replay on restart.
2026-03-25 15:43:25 -07:00
promalert
9012069bd7 chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code

Signed-off-by: promalert <promalert@outlook.com>

* goimports -w .

---------

Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-07 13:06:08 -08:00
Konstantin Lebedev
1169f94310 Fix filer sync set offset (#5197)
* fix: compose 2mount with sync

* fix: DATA RACE
https://github.com/seaweedfs/seaweedfs/issues/5194
https://github.com/seaweedfs/seaweedfs/issues/5195
2024-01-12 10:57:18 -08:00
Konstantin Lebedev
b64674018a [sync] override amz storage class, None to delete (#3639)
* override amz storage class, None to delete
https://github.com/seaweedfs/seaweedfs/issues/3636

* use empty string to delete

* without nil check
2022-09-10 14:15:42 -07:00
chrislu
7ea2c24c5d metadata processing use retry instead of retryForEver 2022-08-22 11:25:40 -07:00
chrislu
f6b92b8561 refactor 2022-08-22 00:00:51 -07:00
chrislu
1a4bf0dcb5 filer.sync: parallelize the filer.sync 2022-08-07 00:56:15 -07:00