b244bb58aa
* full integration with iceberg-go * Table Commit Operations (handleUpdateTable) * s3tables: fix Iceberg v2 compliance and namespace properties This commit ensures SeaweedFS Iceberg REST Catalog is compliant with Iceberg Format Version 2 by: - Using iceberg-go's table.NewMetadataWithUUID for strict v2 compliance. - Explicitly initializing namespace properties to empty maps. - Removing omitempty from required Iceberg response fields. - Fixing CommitTableRequest unmarshaling using table.Requirements and table.Updates. * s3tables: automate Iceberg integration tests - Added Makefile for local test execution and cluster management. - Added docker-compose for PyIceberg compatibility kit. - Added Go integration test harness for PyIceberg. - Updated GitHub CI to run Iceberg catalog tests automatically. * s3tables: update PyIceberg test suite for compatibility - Updated test_rest_catalog.py to use latest PyIceberg transaction APIs. - Updated Dockerfile to include pyarrow and pandas dependencies. - Improved namespace and table handling in integration tests. * s3tables: address review feedback on Iceberg Catalog - Implemented robust metadata version parsing and incrementing. - Ensured table metadata changes are persisted during commit (handleUpdateTable). - Standardized namespace property initialization for consistency. - Fixed unused variable and incorrect struct field build errors. * s3tables: finalize Iceberg REST Catalog and optimize tests - Implemented robust metadata versioning and persistence. - Standardized namespace property initialization. - Optimized integration tests using pre-built Docker image. - Added strict property persistence validation to test suite. - Fixed build errors from previous partial updates. * Address PR review: fix Table UUID stability, implement S3Tables UpdateTable, and support full metadata persistence individually * fix: Iceberg catalog stable UUIDs, metadata persistence, and file writing - Ensure table UUIDs are stable (do not regenerate on load). - Persist full table metadata (Iceberg JSON) in s3tables extended attributes. - Add `MetadataVersion` to explicitly track version numbers, replacing regex parsing. - Implement `saveMetadataFile` to persist metadata JSON files to the Filer on commit. - Update `CreateTable` and `UpdateTable` handlers to use the new logic. * test: bind weed mini to 0.0.0.0 in integration tests to fix Docker connectivity * Iceberg: fix metadata handling in REST catalog - Add nil guard in createTable - Fix updateTable to correctly load existing metadata from storage - Ensure full metadata persistence on updates - Populate loadTable result with parsed metadata * S3Tables: add auth checks and fix response fields in UpdateTable - Add CheckPermissionWithContext to UpdateTable handler - Include TableARN and MetadataLocation in UpdateTable response - Use ErrCodeConflict (409) for version token mismatches * Tests: improve Iceberg catalog test infrastructure and cleanup - Makefile: use PID file for precise process killing - test_rest_catalog.py: remove unused variables and fix f-strings * Iceberg: fix variable shadowing in UpdateTable - Rename inner loop variable `req` to `requirement` to avoid shadowing outer request variable * S3Tables: simplify MetadataVersion initialization - Use `max(req.MetadataVersion, 1)` instead of anonymous function * Tests: remove unicode characters from S3 tables integration test logs - Remove unicode checkmarks from test output for cleaner logs * Iceberg: improve metadata persistence robustness - Fix MetadataLocation in LoadTableResult to fallback to generated location - Improve saveMetadataFile to ensure directory hierarchy existence and robust error handling
285 lines
7.7 KiB
YAML
285 lines
7.7 KiB
YAML
name: "S3 Tables Integration Tests"
|
|
|
|
on:
|
|
pull_request:
|
|
|
|
concurrency:
|
|
group: ${{ github.head_ref }}/s3-tables-tests
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
|
|
jobs:
|
|
s3-tables-tests:
|
|
name: S3 Tables Integration Tests
|
|
runs-on: ubuntu-22.04
|
|
timeout-minutes: 30
|
|
|
|
steps:
|
|
- name: Check out code
|
|
uses: actions/checkout@v6
|
|
|
|
- name: Set up Go
|
|
uses: actions/setup-go@v6
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
id: go
|
|
|
|
- name: Install SeaweedFS
|
|
run: |
|
|
go install -buildvcs=false ./weed
|
|
|
|
- name: Run S3 Tables Integration Tests
|
|
timeout-minutes: 25
|
|
working-directory: test/s3tables/table-buckets
|
|
run: |
|
|
set -x
|
|
set -o pipefail
|
|
echo "=== System Information ==="
|
|
uname -a
|
|
free -h
|
|
df -h
|
|
echo "=== Starting S3 Tables Tests ==="
|
|
|
|
# Run S3 Tables integration tests
|
|
go test -v -timeout 20m . 2>&1 | tee test-output.log || {
|
|
echo "S3 Tables integration tests failed"
|
|
exit 1
|
|
}
|
|
|
|
- name: Show test output on failure
|
|
if: failure()
|
|
working-directory: test/s3tables/table-buckets
|
|
run: |
|
|
echo "=== Test Output ==="
|
|
if [ -f test-output.log ]; then
|
|
tail -200 test-output.log
|
|
fi
|
|
|
|
echo "=== Process information ==="
|
|
ps aux | grep -E "(weed|test)" || true
|
|
|
|
- name: Upload test logs on failure
|
|
if: failure()
|
|
uses: actions/upload-artifact@v6
|
|
with:
|
|
name: s3-tables-test-logs
|
|
path: test/s3tables/table-buckets/test-output.log
|
|
retention-days: 3
|
|
|
|
iceberg-catalog-tests:
|
|
name: Iceberg Catalog Integration Tests
|
|
runs-on: ubuntu-22.04
|
|
timeout-minutes: 30
|
|
|
|
steps:
|
|
- name: Check out code
|
|
uses: actions/checkout@v6
|
|
|
|
- name: Set up Go
|
|
uses: actions/setup-go@v6
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
id: go
|
|
|
|
- name: Run Iceberg Catalog Integration Tests
|
|
timeout-minutes: 25
|
|
working-directory: test/s3tables/catalog
|
|
run: |
|
|
set -x
|
|
set -o pipefail
|
|
echo "=== System Information ==="
|
|
uname -a
|
|
free -h
|
|
df -h
|
|
echo "=== Starting Iceberg Catalog Tests ==="
|
|
|
|
# Run Iceberg catalog integration tests using Makefile (handles build)
|
|
make test 2>&1 | tee test-output.log || {
|
|
echo "Iceberg catalog integration tests failed"
|
|
exit 1
|
|
}
|
|
|
|
- name: Show test output on failure
|
|
if: failure()
|
|
working-directory: test/s3tables/catalog
|
|
run: |
|
|
echo "=== Test Output ==="
|
|
if [ -f test-output.log ]; then
|
|
tail -200 test-output.log
|
|
fi
|
|
|
|
echo "=== Process information ==="
|
|
ps aux | grep -E "(weed|test|docker)" || true
|
|
|
|
- name: Upload test logs on failure
|
|
if: failure()
|
|
uses: actions/upload-artifact@v6
|
|
with:
|
|
name: iceberg-catalog-test-logs
|
|
path: test/s3tables/catalog/test-output.log
|
|
retention-days: 3
|
|
|
|
s3-tables-build-verification:
|
|
name: S3 Tables Build Verification
|
|
runs-on: ubuntu-22.04
|
|
timeout-minutes: 15
|
|
|
|
steps:
|
|
- name: Check out code
|
|
uses: actions/checkout@v6
|
|
|
|
- name: Set up Go
|
|
uses: actions/setup-go@v6
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
id: go
|
|
|
|
- name: Verify S3 Tables Package Builds
|
|
run: |
|
|
set -x
|
|
echo "=== Building S3 Tables package ==="
|
|
go build ./weed/s3api/s3tables || {
|
|
echo "S3 Tables package build failed"
|
|
exit 1
|
|
}
|
|
echo "S3 Tables package built successfully"
|
|
|
|
- name: Verify S3 API Integration Builds
|
|
run: |
|
|
set -x
|
|
echo "=== Building S3 API with S3 Tables integration ==="
|
|
go build ./weed/s3api || {
|
|
echo "S3 API build with S3 Tables failed"
|
|
exit 1
|
|
}
|
|
echo "S3 API with S3 Tables integration built successfully"
|
|
|
|
- name: Run Go Tests for S3 Tables Package
|
|
run: |
|
|
set -x
|
|
echo "=== Running Go unit tests for S3 Tables ==="
|
|
go test -v -race -timeout 5m ./weed/s3api/s3tables/... || {
|
|
echo "S3 Tables unit tests failed"
|
|
exit 1
|
|
}
|
|
echo "S3 Tables unit tests passed"
|
|
|
|
- name: Verify Iceberg Package Builds
|
|
run: |
|
|
set -x
|
|
echo "=== Building Iceberg package ==="
|
|
go build ./weed/s3api/iceberg || {
|
|
echo "Iceberg package build failed"
|
|
exit 1
|
|
}
|
|
echo "Iceberg package built successfully"
|
|
|
|
- name: Run Go Tests for Iceberg Package
|
|
run: |
|
|
set -x
|
|
echo "=== Running Go unit tests for Iceberg ==="
|
|
go test -v -race -timeout 5m ./weed/s3api/iceberg/... || {
|
|
echo "Iceberg unit tests failed"
|
|
exit 1
|
|
}
|
|
echo "Iceberg unit tests passed"
|
|
|
|
s3-tables-fmt-check:
|
|
name: S3 Tables Format Check
|
|
runs-on: ubuntu-22.04
|
|
timeout-minutes: 10
|
|
|
|
steps:
|
|
- name: Check out code
|
|
uses: actions/checkout@v6
|
|
|
|
- name: Set up Go
|
|
uses: actions/setup-go@v6
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
id: go
|
|
|
|
- name: Check Go Format
|
|
run: |
|
|
set -x
|
|
echo "=== Checking S3 Tables Go format ==="
|
|
unformatted=$(gofmt -l ./weed/s3api/s3tables)
|
|
if [ -n "$unformatted" ]; then
|
|
echo "Go format check failed - files need formatting"
|
|
echo "$unformatted"
|
|
exit 1
|
|
fi
|
|
echo "All S3 Tables files are properly formatted"
|
|
|
|
- name: Check S3 Tables Test Format
|
|
run: |
|
|
set -x
|
|
echo "=== Checking S3 Tables test format ==="
|
|
unformatted=$(gofmt -l ./test/s3tables)
|
|
if [ -n "$unformatted" ]; then
|
|
echo "Go format check failed for tests"
|
|
echo "$unformatted"
|
|
exit 1
|
|
fi
|
|
echo "All S3 Tables test files are properly formatted"
|
|
|
|
- name: Check Iceberg Format
|
|
run: |
|
|
set -x
|
|
echo "=== Checking Iceberg Go format ==="
|
|
unformatted=$(gofmt -l ./weed/s3api/iceberg)
|
|
if [ -n "$unformatted" ]; then
|
|
echo "Go format check failed for Iceberg - files need formatting"
|
|
echo "$unformatted"
|
|
exit 1
|
|
fi
|
|
echo "All Iceberg files are properly formatted"
|
|
|
|
s3-tables-vet:
|
|
name: S3 Tables Go Vet Check
|
|
runs-on: ubuntu-22.04
|
|
timeout-minutes: 10
|
|
|
|
steps:
|
|
- name: Check out code
|
|
uses: actions/checkout@v6
|
|
|
|
- name: Set up Go
|
|
uses: actions/setup-go@v6
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
id: go
|
|
|
|
- name: Run Go Vet
|
|
run: |
|
|
set -x
|
|
echo "=== Running go vet on S3 Tables package ==="
|
|
go vet ./weed/s3api/s3tables/... || {
|
|
echo "go vet check failed"
|
|
exit 1
|
|
}
|
|
echo "go vet checks passed"
|
|
|
|
- name: Run Go Vet on Tests
|
|
run: |
|
|
set -x
|
|
echo "=== Running go vet on S3 Tables tests ==="
|
|
go vet ./test/s3tables/... || {
|
|
echo "go vet check failed for tests"
|
|
exit 1
|
|
}
|
|
echo "go vet checks passed for tests"
|
|
|
|
- name: Run Go Vet on Iceberg
|
|
run: |
|
|
set -x
|
|
echo "=== Running go vet on Iceberg package ==="
|
|
go vet ./weed/s3api/iceberg/... || {
|
|
echo "go vet check failed for Iceberg"
|
|
exit 1
|
|
}
|
|
echo "go vet checks passed for Iceberg"
|