Files
seaweedFS/weed
Chris Lu 29bdbb3c48 filer.sync: replace O(n) conflict check with O(depth) index lookups (#8772)
* filer.sync: replace O(n) conflict check with O(depth) index lookups

The MetadataProcessor.conflictsWith() scanned all active jobs linearly
for every new event dispatch. At high concurrency (256-1024), this O(n)
scan under the activeJobsLock became a bottleneck that throttled the
event dispatch pipeline, negating the benefit of higher -concurrency
values.

Replace the linear scan with three index maps:
- activeFilePaths: O(1) exact file path lookup
- activeDirPaths: O(1) directory path lookup per ancestor
- descendantCount: O(1) check for active jobs under a directory

Conflict check is now O(depth) where depth is the path depth (typically
3-6 levels), constant regardless of active job count. Benchmark confirms
~81ns per check whether there are 32 or 1024 active jobs.

Also replace the O(n) watermark scan with minActiveTs tracking so
non-oldest job completions are O(1).

Ref: #8771

* filer.sync: replace O(n) watermark rescan with min-heap lazy deletion

Address review feedback:
- Replace minActiveTs O(n) rescan with a tsMinHeap using lazy deletion.
  Each TsNs is pushed once and popped once, giving O(log n) amortized
  watermark tracking regardless of completion order.
- Fix benchmark to consume conflictsWith result via package-level sink
  variable to prevent compiler elision.

The watermark advancement semantics (conservative, sets to completing
job's TsNs) are unchanged from the original code. This is intentionally
safe for idempotent replay on restart.
2026-03-25 15:43:25 -07:00
..
2026-02-20 18:42:00 -08:00
2024-02-14 08:26:38 -08:00