b244bb58aa
* full integration with iceberg-go * Table Commit Operations (handleUpdateTable) * s3tables: fix Iceberg v2 compliance and namespace properties This commit ensures SeaweedFS Iceberg REST Catalog is compliant with Iceberg Format Version 2 by: - Using iceberg-go's table.NewMetadataWithUUID for strict v2 compliance. - Explicitly initializing namespace properties to empty maps. - Removing omitempty from required Iceberg response fields. - Fixing CommitTableRequest unmarshaling using table.Requirements and table.Updates. * s3tables: automate Iceberg integration tests - Added Makefile for local test execution and cluster management. - Added docker-compose for PyIceberg compatibility kit. - Added Go integration test harness for PyIceberg. - Updated GitHub CI to run Iceberg catalog tests automatically. * s3tables: update PyIceberg test suite for compatibility - Updated test_rest_catalog.py to use latest PyIceberg transaction APIs. - Updated Dockerfile to include pyarrow and pandas dependencies. - Improved namespace and table handling in integration tests. * s3tables: address review feedback on Iceberg Catalog - Implemented robust metadata version parsing and incrementing. - Ensured table metadata changes are persisted during commit (handleUpdateTable). - Standardized namespace property initialization for consistency. - Fixed unused variable and incorrect struct field build errors. * s3tables: finalize Iceberg REST Catalog and optimize tests - Implemented robust metadata versioning and persistence. - Standardized namespace property initialization. - Optimized integration tests using pre-built Docker image. - Added strict property persistence validation to test suite. - Fixed build errors from previous partial updates. * Address PR review: fix Table UUID stability, implement S3Tables UpdateTable, and support full metadata persistence individually * fix: Iceberg catalog stable UUIDs, metadata persistence, and file writing - Ensure table UUIDs are stable (do not regenerate on load). - Persist full table metadata (Iceberg JSON) in s3tables extended attributes. - Add `MetadataVersion` to explicitly track version numbers, replacing regex parsing. - Implement `saveMetadataFile` to persist metadata JSON files to the Filer on commit. - Update `CreateTable` and `UpdateTable` handlers to use the new logic. * test: bind weed mini to 0.0.0.0 in integration tests to fix Docker connectivity * Iceberg: fix metadata handling in REST catalog - Add nil guard in createTable - Fix updateTable to correctly load existing metadata from storage - Ensure full metadata persistence on updates - Populate loadTable result with parsed metadata * S3Tables: add auth checks and fix response fields in UpdateTable - Add CheckPermissionWithContext to UpdateTable handler - Include TableARN and MetadataLocation in UpdateTable response - Use ErrCodeConflict (409) for version token mismatches * Tests: improve Iceberg catalog test infrastructure and cleanup - Makefile: use PID file for precise process killing - test_rest_catalog.py: remove unused variables and fix f-strings * Iceberg: fix variable shadowing in UpdateTable - Rename inner loop variable `req` to `requirement` to avoid shadowing outer request variable * S3Tables: simplify MetadataVersion initialization - Use `max(req.MetadataVersion, 1)` instead of anonymous function * Tests: remove unicode characters from S3 tables integration test logs - Remove unicode checkmarks from test output for cleaner logs * Iceberg: improve metadata persistence robustness - Fix MetadataLocation in LoadTableResult to fallback to generated location - Improve saveMetadataFile to ensure directory hierarchy existence and robust error handling
38 lines
1.2 KiB
Makefile
38 lines
1.2 KiB
Makefile
# Makefile for Iceberg REST Catalog Integration Tests
|
|
#
|
|
# This Makefile provides easy-to-use targets for running integration tests
|
|
# and starting a local development environment.
|
|
|
|
.PHONY: build mini test test-pyiceberg clean
|
|
|
|
# Root directory of the SeaweedFS project
|
|
PROJECT_ROOT = ../../..
|
|
WEED_BIN = $(PROJECT_ROOT)/weed/weed
|
|
|
|
# Build the weed binary
|
|
build:
|
|
@echo "Building SeaweedFS binary..."
|
|
cd $(PROJECT_ROOT)/weed && go build -v
|
|
|
|
# Start SeaweedFS mini cluster for manual testing
|
|
# Uses default ports: S3=8333, Iceberg=8182
|
|
mini: build
|
|
@echo "Starting SeaweedFS mini cluster..."
|
|
$(WEED_BIN) mini -s3.port.iceberg=8182 & echo $$! > weed-mini.pid; wait
|
|
|
|
# Run all integration tests in this directory
|
|
test: build
|
|
@echo "Running all Iceberg catalog integration tests..."
|
|
go test -v .
|
|
|
|
# Run only the PyIceberg compatibility tests (requires Docker)
|
|
test-pyiceberg: build
|
|
@echo "Running PyIceberg compatibility tests..."
|
|
go test -v -run TestPyIcebergRestCatalog .
|
|
|
|
# Clean up temporary data and stop weed mini if running
|
|
clean:
|
|
@echo "Cleaning up..."
|
|
@test -f weed-mini.pid && kill $$(cat weed-mini.pid) && rm weed-mini.pid || true
|
|
@rm -rf /tmp/seaweed-iceberg-test-*
|