filer: improve FoundationDB performance by disabling batch by default (#7770)

* filer: improve FoundationDB performance by disabling batch by default

This PR addresses a performance issue where FoundationDB filer was achieving
only ~757 ops/sec with 12 concurrent S3 clients, despite FDB being capable
of 17,000+ ops/sec.

Root cause: The write batcher was waiting up to 5ms for each operation to
batch, even though S3 semantics require waiting for durability confirmation.
This added artificial latency that defeated the purpose of batching.

Changes:
- Disable write batching by default (batch_enabled = false)
- Each write now commits immediately in its own transaction
- Reduce batch interval from 5ms to 1ms when batching is enabled
- Add batch_enabled config option to toggle behavior
- Improve batcher to collect available ops without blocking
- Add benchmarks comparing batch vs no-batch performance

Benchmark results (16 concurrent goroutines):
- With batch:    2,924 ops/sec (342,032 ns/op)
- Without batch: 4,625 ops/sec (216,219 ns/op)
- Improvement:   +58% faster

Configuration:
- Default: batch_enabled = false (optimal for S3 PUT latency)
- For bulk ingestion: set batch_enabled = true

Also fixes ARM64 Docker test setup (shell compatibility, fdbserver path).

* fix: address review comments - use atomic counter and remove duplicate batcher

- Use sync/atomic.Uint64 for unique filenames in concurrent benchmarks
- Remove duplicate batcher creation in createBenchmarkStoreWithBatching
  (initialize() already creates batcher when batchEnabled=true)

* fix: add realistic default values to benchmark store helper

Set directoryPrefix, timeout, and maxRetryDelay to reasonable defaults
for more realistic benchmark conditions.
This commit is contained in:
Chris Lu
2025-12-15 13:03:34 -08:00
committed by GitHub
parent 44beb42eb9
commit 5a03b5538f
11 changed files with 321 additions and 36 deletions

View File

@@ -51,6 +51,9 @@ directory_prefix = "seaweedfs"
| `timeout` | Operation timeout duration | `5s` | No |
| `max_retry_delay` | Maximum retry delay | `1s` | No |
| `directory_prefix` | Directory prefix for organization | `seaweedfs` | No |
| `batch_enabled` | Enable write batching (see Performance section) | `false` | No |
| `batch_size` | Max operations per batch | `100` | No |
| `batch_interval` | Max time before batch flush | `1ms` | No |
### Path-Specific Configuration
@@ -109,12 +112,38 @@ make setup
## Performance Considerations
### Write Batching Configuration
By default, write batching is **disabled** (`batch_enabled = false`). Each write commits
immediately in its own transaction. This provides optimal latency for S3 PUT operations.
**When to enable batching:**
- High-throughput bulk ingestion workloads
- Scenarios where you can tolerate slightly higher per-operation latency
- Workloads with many concurrent small writes
**Batching configuration options:**
```toml
[foundationdb]
# Enable write batching (disabled by default for optimal S3 latency)
batch_enabled = true
# Maximum operations per batch
batch_size = 100
# Maximum time to wait before flushing a batch
batch_interval = "1ms"
```
**Performance comparison:**
- **Batching disabled**: Each S3 PUT commits immediately (~1-5ms per op depending on FDB latency)
- **Batching enabled**: Operations are grouped, reducing total commits but adding batch interval latency
### Optimal Configuration
- **API Version**: Use the latest stable API version (720+)
- **Directory Structure**: Use logical directory prefixes to isolate different SeaweedFS instances
- **Transaction Size**: Keep transactions under 10MB (FDB limit)
- **Batch Operations**: Use transactions for multiple related operations
- **Concurrency**: Use multiple client connections for parallel operations
### Monitoring