seaweedFS/test/foundationdb/README.md

# FoundationDB Integration Testing

This directory contains integration tests and setup scripts for the FoundationDB filer store in SeaweedFS.

## Quick Start

```bash
# ✅ GUARANTEED TO WORK - Run reliable tests (no FoundationDB dependencies)
make test-reliable      # Validation + Mock tests

# Run individual test types
make test-mock          # Mock FoundationDB tests (always work)
go test -v ./validation_test.go  # Package structure validation

# 🐳 FULL INTEGRATION (requires Docker + FoundationDB dependencies)
make setup              # Start FoundationDB cluster + SeaweedFS
make test               # Run all integration tests
make test-simple        # Simple containerized test environment

# Clean up
make clean              # Clean main environment
make clean-simple       # Clean simple test environment

# 🍎 ARM64 / APPLE SILICON SUPPORT
make setup-arm64        # Native ARM64 FoundationDB (builds from source)
make setup-emulated     # x86 emulation (faster setup)
make test-arm64         # Test with ARM64 native
make test-emulated      # Test with x86 emulation
```

### Test Levels

1. **✅ Validation Tests** (`validation_test.go`) - Always work, no dependencies
2. **✅ Mock Tests** (`mock_integration_test.go`) - Test FoundationDB store logic with mocks
3. **⚠️  Real Integration Tests** (`foundationdb_*_test.go`) - Require actual FoundationDB cluster

### ARM64 / Apple Silicon Support

**🍎 For M1/M2/M3 Mac users:** FoundationDB's official Docker images are AMD64-only. We provide three solutions:

- **Native ARM64** (`make setup-arm64`) - Downloads official FoundationDB ARM64 packages and builds SeaweedFS natively (≈2-3 min setup, best performance)
- **x86 Emulation** (`make setup-emulated`) - Uses Docker emulation (fast setup, slower runtime)  
- **Mock Testing** (`make test-mock`) - No FoundationDB needed (instant, tests logic only)

The ARM64 setup automatically builds both FoundationDB and SeaweedFS from source using `docker-compose.arm64.yml` and dedicated ARM64 Dockerfiles. No pre-built images required!

📖 **Detailed Guide:** See [README.ARM64.md](README.ARM64.md) for complete ARM64 documentation.

## Test Environment

The test environment includes:

- **3-node FoundationDB cluster** (fdb1, fdb2, fdb3) for realistic distributed testing
- **Database initialization service** (fdb-init) that configures the cluster
- **SeaweedFS service** configured to use the FoundationDB filer store
- **Automatic service orchestration** with proper startup dependencies

## Test Structure

### Integration Tests

#### `foundationdb_integration_test.go`
- Basic CRUD operations (Create, Read, Update, Delete)
- Directory operations and listing:
  - `ListDirectoryEntries` - List all entries in a directory
  - `ListDirectoryPrefixedEntries` - List entries matching a prefix
  - `DeleteFolderChildren` - Bulk deletion of directory contents
- Transaction handling (begin, commit, rollback)
- Key-Value operations
- Large entry handling with compression
- Error scenarios and edge cases

**Note:** These tests operate at the filer store level, testing the metadata index operations that underpin S3 bucket listing and directory tree operations.

#### `foundationdb_concurrent_test.go`
- Concurrent insert operations across multiple goroutines
- Concurrent read/write operations on shared files
- Concurrent transaction handling with conflict resolution
- Concurrent directory operations
- Concurrent key-value operations
- Stress testing under load

#### `test_fdb_s3.sh` - End-to-End S3 Integration Tests
- **S3 bucket creation** - Create buckets via S3 API
- **S3 file upload** - Upload files to buckets
- **S3 bucket listing** (`aws s3 ls`) - **Validates listing operations work correctly**
- **S3 file download** - Retrieve and verify file contents
- **S3 file deletion** - Delete objects and verify removal
- **FoundationDB backend verification** - Confirms data is stored in FDB
- **Filer directory operations** - Direct filer API calls for directory creation/listing

**This test validates the complete S3 workflow including the listing operations that were problematic in earlier versions.**

#### Unit Tests (`weed/filer/foundationdb/foundationdb_store_test.go`)
- Store initialization and configuration
- Key generation and directory prefixes
- Error handling and validation
- Performance benchmarks
- Configuration validation

## Configuration

### Environment Variables

The tests can be configured using environment variables:

```bash
export FDB_CLUSTER_FILE=/var/fdb/config/fdb.cluster
export WEED_FOUNDATIONDB_ENABLED=true
export WEED_FOUNDATIONDB_API_VERSION=740
export WEED_FOUNDATIONDB_TIMEOUT=10s
```

#### Docker Compose Environment Variables

The `docker-compose.yml` file supports the following optional environment variables with sensible defaults:

```bash
# FoundationDB image (default: foundationdb/foundationdb:7.1.61)
export FOUNDATIONDB_IMAGE=foundationdb/foundationdb:7.1.61

# FoundationDB port (default: 4500)
export FDB_PORT=4500

# FoundationDB cluster file contents (default: docker:docker@fdb1:4500,fdb2:4500,fdb3:4500)
export FDB_CLUSTER_FILE_CONTENTS="docker:docker@fdb1:4500,fdb2:4500,fdb3:4500"

# SeaweedFS image (default: chrislusf/seaweedfs:latest)
export SEAWEEDFS_IMAGE=chrislusf/seaweedfs:latest
```

**Note:** These variables are optional. If not set, the docker-compose will use the default values shown above, allowing `docker-compose up` to work out-of-the-box without any `.env` file or manual configuration.

### Docker Compose Configuration

The `docker-compose.yml` sets up:

1. **FoundationDB Cluster**: 3 coordinating nodes with data distribution
2. **Database Configuration**: Single SSD storage class for testing
3. **SeaweedFS Integration**: Automatic filer store configuration
4. **Volume Persistence**: Data persists between container restarts

### Test Configuration Files

- `filer.toml`: FoundationDB filer store configuration
- `s3.json`: S3 API credentials for end-to-end testing
- `Makefile`: Test automation and environment management

## Test Commands

### Setup Commands

```bash
make setup              # Full environment setup
make dev-fdb           # Just FoundationDB cluster
make install-deps      # Check dependencies
make check-env         # Validate configuration
```

### Test Commands

```bash
make test              # All tests
make test-unit         # Go unit tests
make test-integration  # Integration tests
make test-e2e         # End-to-end S3 tests (includes S3 bucket listing)
make test-crud        # Basic CRUD operations
make test-concurrent  # Concurrency tests
make test-benchmark   # Performance benchmarks
```

#### S3 and Listing Operation Coverage

**✅ Currently Tested:**
- **S3 bucket listing** (`aws s3 ls`) - Validated in `test_fdb_s3.sh`
- **Directory metadata listing** (`ListDirectoryEntries`) - Tested in `foundationdb_integration_test.go`
- **Prefix-based listing** (`ListDirectoryPrefixedEntries`) - Tested in `foundationdb_integration_test.go`
- **Filer directory operations** - Basic filer API calls in `test_fdb_s3.sh`
- **Metadata index operations** - All CRUD operations on directory entries

**⚠️ Limited/Future Coverage:**
- **Recursive tree operations** - Not explicitly tested (e.g., `weed filer.tree` command)
- **Large directory stress tests** - Listings with thousands of entries not currently benchmarked
- **Concurrent listing operations** - Multiple simultaneous directory listings under load
- **S3 ListObjectsV2 pagination** - Large bucket listing with continuation tokens

**Recommendation:** If experiencing issues with S3 listing operations in production, add stress tests for large directories and concurrent listing scenarios to validate FoundationDB's range scan performance at scale.

### Debug Commands

```bash
make status           # Show service status
make logs             # Show all logs
make logs-fdb         # FoundationDB logs only
make logs-seaweedfs   # SeaweedFS logs only
make debug            # Debug information
```

### Cleanup Commands

```bash
make clean            # Stop services and cleanup
```

## Test Data

Tests use isolated directory prefixes to avoid conflicts:

- **Unit tests**: `seaweedfs_test`
- **Integration tests**: `seaweedfs_test`
- **Concurrent tests**: `seaweedfs_concurrent_test_<timestamp>`
- **E2E tests**: `seaweedfs` (default)

## Expected Test Results

### Performance Expectations

Based on FoundationDB characteristics:
- **Single operations**: < 10ms latency
- **Batch operations**: High throughput with transactions
- **Concurrent operations**: Linear scaling with multiple clients
- **Directory listings**: Efficient range scans

### Reliability Expectations

- **ACID compliance**: All operations are atomic and consistent
- **Fault tolerance**: Automatic recovery from node failures
- **Concurrency**: No data corruption under concurrent load
- **Durability**: Data persists across restarts

## Troubleshooting

### Common Issues

1. **FoundationDB Connection Errors**
   ```bash
   # Check cluster status
   make status
   
   # Verify cluster file
   docker-compose exec fdb-init cat /var/fdb/config/fdb.cluster
   ```

2. **Test Failures**
   ```bash
   # Check service logs
   make logs-fdb
   make logs-seaweedfs
   
   # Run with verbose output
   go test -v -tags foundationdb ./...
   ```

3. **Performance Issues**
   ```bash
   # Check cluster health
   docker-compose exec fdb-init fdbcli --exec 'status details'
   
   # Monitor resource usage
   docker stats
   ```

4. **Docker Issues**
   ```bash
   # Clean Docker state
   make clean
   docker system prune -f
   
   # Restart from scratch
   make setup
   ```

### Debug Mode

Enable verbose logging for detailed troubleshooting:

```bash
# SeaweedFS debug logs
WEED_FILER_OPTIONS_V=2 make test

# FoundationDB debug logs (in fdbcli)
configure new single ssd; status details
```

### Manual Testing

For manual verification:

```bash
# Start environment
make dev-fdb

# Connect to FoundationDB
docker-compose exec fdb-init fdbcli

# FDB commands:
# status                    - Show cluster status  
# getrange "" \xFF          - Show all keys
# getrange seaweedfs seaweedfs\xFF  - Show SeaweedFS keys
```

### Listing Operations Return Empty Results

**Symptoms:** Uploads succeed, direct file reads work, but listing operations (`aws s3 ls`, `s3.bucket.list`, `weed filer.ls/tree`) return no results.

**Test Coverage:** The `test_fdb_s3.sh` script explicitly tests S3 bucket listing (`aws s3 ls`) to catch this class of issue. Integration tests cover the underlying `ListDirectoryEntries` operations.

**Diagnostic steps:**

```bash
# 1. Verify writes reached FoundationDB
docker-compose exec fdb-init fdbcli
> getrange seaweedfs seaweedfs\xFF
# If no keys appear, writes aren't reaching the store

# 2. Check SeaweedFS volume assignment
curl http://localhost:9333/cluster/status
# Look for "AssignVolume" errors in logs:
make logs-seaweedfs | grep -i "assignvolume\|writable"

# 3. Verify filer health and configuration
curl http://localhost:8888/statistics/health
make logs-seaweedfs | grep -i "store\|foundationdb"
```

**Interpretation:**
- No SeaweedFS keys in FDB: Directory index writes failing; check filer logs for write errors
- AssignVolume errors: Volume assignment blocked; check master status and disk space
- Filer health errors: Configuration or connectivity issue; restart services and verify filer.toml

**Recovery:**
- If fresh data: restart services (`make clean && make setup`)
- If production data: ensure volume assignment works, check disk space on data nodes

## CI Integration

For continuous integration:

```bash
# CI test suite
make ci-test    # Unit + integration tests
make ci-e2e     # Full end-to-end test suite
```

The tests are designed to be reliable in CI environments with:
- Automatic service startup and health checking
- Timeout handling for slow CI systems
- Proper cleanup and resource management
- Detailed error reporting and logs

## Performance Benchmarks

Run performance benchmarks:

```bash
make test-benchmark

# Sample expected results:
# BenchmarkFoundationDBStore_InsertEntry-8    1000    1.2ms per op
# BenchmarkFoundationDBStore_FindEntry-8      5000    0.5ms per op  
# BenchmarkFoundationDBStore_KvOperations-8   2000    0.8ms per op
```

## Contributing

When adding new tests:

1. Use the `//go:build foundationdb` build tag
2. Follow the existing test structure and naming
3. Include both success and error scenarios
4. Add appropriate cleanup and resource management
5. Update this README with new test descriptions