gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Enforce IAM for S3 Tables bucket creation (#8388)

Browse Source 
* Enforce IAM for s3tables bucket creation

* Prefer IAM path when policies exist

* Ensure IAM enforcement honors default allow

* address comments

* Reused the precomputed principal when setting tableBucketMetadata.OwnerAccountID, avoiding the redundant getAccountID call.

* get identity

* fix

* dedup

* fix

* comments

* fix tests

* update iam config

* go fmt

* fix ports

* fix flags

* mini clean shutdown

* Revert "update iam config"

This reverts commit ca48fdbb0afa45657823d98657556c0bbf24f239.

Revert "mini clean shutdown"

This reverts commit 9e17f6baffd5dd7cc404d831d18dd618b9fe5049.

Revert "fix flags"

This reverts commit e9e7b29d2f77ee5cb82147d50621255410695ee3.

Revert "go fmt"

This reverts commit bd3241960b1d9484b7900190773b0ecb3f762c9a.

* test/s3tables: share single weed mini per test package via TestMain

Previously each top-level test function in the catalog and s3tables
package started and stopped its own weed mini instance. This caused
failures when a prior instance wasn't cleanly stopped before the next
one started (port conflicts, leaked global state).

Changes:
- catalog/iceberg_catalog_test.go: introduce TestMain that starts one
  shared TestEnvironment (external weed binary) before all tests and
  tears it down after. All individual test functions now use sharedEnv.
  Added randomSuffix() for unique resource names across tests.
- catalog/pyiceberg_test.go: updated to use sharedEnv instead of
  per-test environments.
- catalog/pyiceberg_test_helpers.go -> pyiceberg_test_helpers_test.go:
  renamed to a _test.go file so it can access TestEnvironment which is
  defined in a test file.
- table-buckets/setup.go: add package-level sharedCluster variable.
- table-buckets/s3tables_integration_test.go: introduce TestMain that
  starts one shared TestCluster before all tests. TestS3TablesIntegration
  now uses sharedCluster. Extract startMiniClusterInDir (no *testing.T)
  for TestMain use. TestS3TablesCreateBucketIAMPolicy keeps its own
  cluster (different IAM config). Remove miniClusterMutex (no longer
  needed). Fix Stop() to not panic when t is nil."

* delete

* parse

* default allow should work with anonymous

* fix port

* iceberg route

The failures are from Iceberg REST using the default bucket warehouse when no prefix is provided. Your tests create random buckets, so /v1/namespaces was looking in warehouse and failing. I updated the tests to use the prefixed Iceberg routes (/v1/{bucket}/...) via a small helper.

* test(s3tables): fix port conflicts and IAM ARN matching in integration tests

- Pass -master.dir explicitly to prevent filer store directory collision
  between shared cluster and per-test clusters running in the same process
- Pass -volume.port.public and -volume.publicUrl to prevent the global
  publicPort flag (mutated from 0 → concrete port by first cluster) from
  being reused by a second cluster, causing 'address already in use'
- Remove the flag-reset loop in Stop() that reset global flag values while
  other goroutines were reading them (race → panic)
- Fix IAM policy Resource ARN in TestS3TablesCreateBucketIAMPolicy to use
  wildcards (arn:aws:s3tables:*:*:bucket/<name>) because the handler
  generates ARNs with its own DefaultRegion (us-east-1) and principal name
  ('admin'), not the test constants testRegion/testAccountID
This commit is contained in:
Chris Lu
2026-02-19 22:52:05 -08:00
committed by GitHub
parent a2005cb2a6
commit 36c469e34e
10 changed files with 671 additions and 172 deletions
Download Patch File Download Diff File Expand all files Collapse all files

210
test/s3tables/catalog/iceberg_catalog_test.go
View File

@@ -6,6 +6,7 @@ import (
"bytes"
"context"
"encoding/json"
"flag"
"fmt"
"io"
"net"
@@ -18,6 +19,37 @@ import (
"time"
)
// sharedEnv is the single TestEnvironment shared across all tests in this package.
var sharedEnv *TestEnvironment
// TestMain starts one weed mini instance for the whole package and tears it down
// after all tests have run.
func TestMain(m *testing.M) {
flag.Parse()
if os.Getenv("SHORT") != "" || testing.Short() {
// Let tests self-skip when run with -short.
os.Exit(m.Run())
}
env, err := newTestEnvironmentForMain()
if err != nil {
fmt.Fprintf(os.Stderr, "SKIP: setup failed: %v\n", err)
os.Exit(0) // Skip all tests rather than fail
}
sharedEnv = env
if startErr := sharedEnv.startSeaweedFSForMain(); startErr != nil {
fmt.Fprintf(os.Stderr, "SKIP: weed mini failed to start: %v\n", startErr)
sharedEnv.cleanupForMain()
os.Exit(0)
}
code := m.Run()
sharedEnv.cleanupForMain()
os.Exit(code)
}
// TestEnvironment contains the test environment configuration
type TestEnvironment struct {
seaweedDir string
@@ -54,17 +86,15 @@ func getFreePort() (int, net.Listener, error) {
return addr.Port, listener, nil
}
// NewTestEnvironment creates a new test environment
func NewTestEnvironment(t *testing.T) *TestEnvironment {
t.Helper()
// newTestEnvironmentForMain creates a TestEnvironment without calling t.Fatalf so it
// can be used from TestMain (which has no *testing.T).
func newTestEnvironmentForMain() (*TestEnvironment, error) {
// Find the SeaweedFS root directory
wd, err := os.Getwd()
if err != nil {
t.Fatalf("Failed to get working directory: %v", err)
return nil, fmt.Errorf("get working directory: %w", err)
}
// Navigate up to find the SeaweedFS root (contains go.mod)
seaweedDir := wd
for i := 0; i < 5; i++ {
if _, err := os.Stat(filepath.Join(seaweedDir, "go.mod")); err == nil {
@@ -76,82 +106,93 @@ func NewTestEnvironment(t *testing.T) *TestEnvironment {
// Check for weed binary
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
if _, err := os.Stat(weedBinary); os.IsNotExist(err) {
// Try system PATH
weedBinary = "weed"
if _, err := exec.LookPath(weedBinary); err != nil {
t.Skip("weed binary not found, skipping integration test")
return nil, fmt.Errorf("weed binary not found")
}
}
// Create temporary data directory
dataDir, err := os.MkdirTemp("", "seaweed-iceberg-test-*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
return nil, fmt.Errorf("create temp dir: %w", err)
}
// Allocate free ephemeral ports for each service
var listeners []net.Listener
defer func() {
closeListeners := func() {
for _, l := range listeners {
l.Close()
}
}()
}
var l net.Listener
s3Port, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for S3: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for S3: %w", err)
}
listeners = append(listeners, l)
icebergPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Iceberg: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Iceberg: %w", err)
}
listeners = append(listeners, l)
s3GrpcPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for S3 gRPC: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for S3 gRPC: %w", err)
}
listeners = append(listeners, l)
masterPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Master: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Master: %w", err)
}
listeners = append(listeners, l)
masterGrpcPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Master gRPC: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Master gRPC: %w", err)
}
listeners = append(listeners, l)
filerPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Filer: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Filer: %w", err)
}
listeners = append(listeners, l)
filerGrpcPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Filer gRPC: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Filer gRPC: %w", err)
}
listeners = append(listeners, l)
volumePort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Volume: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Volume: %w", err)
}
listeners = append(listeners, l)
volumeGrpcPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Volume gRPC: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Volume gRPC: %w", err)
}
listeners = append(listeners, l)
// Release the port reservations so weed mini can bind to them
closeListeners()
return &TestEnvironment{
seaweedDir: seaweedDir,
weedBinary: weedBinary,
@@ -166,13 +207,11 @@ func NewTestEnvironment(t *testing.T) *TestEnvironment {
volumePort: volumePort,
volumeGrpcPort: volumeGrpcPort,
dockerAvailable: hasDocker(),
}
}, nil
}
// StartSeaweedFS starts a SeaweedFS mini cluster
func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
t.Helper()
// startSeaweedFSForMain starts weed mini without a *testing.T (for use in TestMain).
func (env *TestEnvironment) startSeaweedFSForMain() error {
ctx, cancel := context.WithCancel(context.Background())
env.weedCancel = cancel
@@ -182,7 +221,8 @@ func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
for _, dir := range []string{masterDir, filerDir, volumeDir} {
if err := os.MkdirAll(dir, 0755); err != nil {
t.Fatalf("Failed to create directory %s: %v", dir, err)
cancel()
return fmt.Errorf("create directory %s: %w", dir, err)
}
}
@@ -203,13 +243,30 @@ func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
t.Fatalf("Failed to start SeaweedFS: %v", err)
cancel()
return fmt.Errorf("start SeaweedFS: %w", err)
}
env.weedProcess = cmd
// Wait for services to be ready
if !env.waitForService(fmt.Sprintf("http://127.0.0.1:%d/v1/config", env.icebergPort), 30*time.Second) {
t.Fatalf("Iceberg REST API did not become ready")
cancel()
cmd.Wait()
return fmt.Errorf("Iceberg REST API did not become ready")
}
return nil
}
// cleanupForMain stops SeaweedFS and cleans up resources (no *testing.T needed).
func (env *TestEnvironment) cleanupForMain() {
if env.weedCancel != nil {
env.weedCancel()
}
if env.weedProcess != nil {
time.Sleep(2 * time.Second)
env.weedProcess.Wait()
}
if env.dataDir != "" {
os.RemoveAll(env.dataDir)
}
}
@@ -230,25 +287,6 @@ func (env *TestEnvironment) waitForService(url string, timeout time.Duration) bo
return false
}
// Cleanup stops SeaweedFS and cleans up resources
func (env *TestEnvironment) Cleanup(t *testing.T) {
t.Helper()
if env.weedCancel != nil {
env.weedCancel()
}
if env.weedProcess != nil {
// Give process time to shut down gracefully
time.Sleep(2 * time.Second)
env.weedProcess.Wait()
}
if env.dataDir != "" {
os.RemoveAll(env.dataDir)
}
}
// IcebergURL returns the Iceberg REST Catalog URL
func (env *TestEnvironment) IcebergURL() string {
return fmt.Sprintf("http://127.0.0.1:%d", env.icebergPort)
@@ -260,10 +298,7 @@ func TestIcebergConfig(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env.StartSeaweedFS(t)
env := sharedEnv
// Test GET /v1/config
resp, err := http.Get(env.IcebergURL() + "/v1/config")
@@ -294,16 +329,14 @@ func TestIcebergNamespaces(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env.StartSeaweedFS(t)
env := sharedEnv
// Create the default table bucket first via S3
createTableBucket(t, env, "warehouse")
bucketName := "warehouse-ns-" + randomSuffix()
createTableBucket(t, env, bucketName)
// Test GET /v1/namespaces (should return empty list initially)
resp, err := http.Get(env.IcebergURL() + "/v1/namespaces")
resp, err := http.Get(env.IcebergURL() + icebergPath(bucketName, "/v1/namespaces"))
if err != nil {
t.Fatalf("Failed to list namespaces: %v", err)
}
@@ -321,16 +354,14 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env := sharedEnv
bucketName := "warehouse-stage-" + randomSuffix()
createTableBucket(t, env, bucketName)
env.StartSeaweedFS(t)
createTableBucket(t, env, "warehouse")
namespace := "stage_ns"
namespace := "stage_ns_" + randomSuffix()
tableName := "orders"
status, _, err := doIcebergJSONRequest(env, http.MethodPost, "/v1/namespaces", map[string]any{
status, _, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, "/v1/namespaces"), map[string]any{
"namespace": []string{namespace},
})
if err != nil {
@@ -340,7 +371,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("Create namespace status = %d, want 200 or 409", status)
}
status, badReqResp, err := doIcebergJSONRequest(env, http.MethodPost, fmt.Sprintf("/v1/namespaces/%s/tables", namespace), map[string]any{
status, badReqResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables", namespace)), map[string]any{
"stage-create": true,
})
if err != nil {
@@ -358,7 +389,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("error.message = %v, want it to include %q", errorObj["message"], "table name is required")
}
status, stageResp, err := doIcebergJSONRequest(env, http.MethodPost, fmt.Sprintf("/v1/namespaces/%s/tables", namespace), map[string]any{
status, stageResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables", namespace)), map[string]any{
"name": tableName,
"stage-create": true,
})
@@ -373,7 +404,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("stage metadata-location = %q, want suffix /metadata/v1.metadata.json", stageLocation)
}
status, _, err = doIcebergJSONRequest(env, http.MethodGet, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName), nil)
status, _, err = doIcebergJSONRequest(env, http.MethodGet, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), nil)
if err != nil {
t.Fatalf("Load staged table request failed: %v", err)
}
@@ -381,7 +412,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("Load staged table status = %d, want 404", status)
}
status, commitResp, err := doIcebergJSONRequest(env, http.MethodPost, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName), map[string]any{
status, commitResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), map[string]any{
"requirements": []map[string]any{
{"type": "assert-create"},
},
@@ -398,7 +429,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("final metadata-location = %q, want suffix /metadata/v1.metadata.json", commitLocation)
}
status, loadResp, err := doIcebergJSONRequest(env, http.MethodGet, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName), nil)
status, loadResp, err := doIcebergJSONRequest(env, http.MethodGet, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), nil)
if err != nil {
t.Fatalf("Load finalized table request failed: %v", err)
}
@@ -417,16 +448,14 @@ func TestCommitMissingTableWithoutAssertCreate(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env := sharedEnv
bucketName := "warehouse-missing-" + randomSuffix()
createTableBucket(t, env, bucketName)
env.StartSeaweedFS(t)
createTableBucket(t, env, "warehouse")
namespace := "stage_missing_assert_ns"
namespace := "stage_missing_assert_ns_" + randomSuffix()
tableName := "missing_table"
status, _, err := doIcebergJSONRequest(env, http.MethodPost, "/v1/namespaces", map[string]any{
status, _, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, "/v1/namespaces"), map[string]any{
"namespace": []string{namespace},
})
if err != nil {
@@ -436,7 +465,7 @@ func TestCommitMissingTableWithoutAssertCreate(t *testing.T) {
t.Fatalf("Create namespace status = %d, want 200 or 409", status)
}
status, _, err = doIcebergJSONRequest(env, http.MethodPost, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName), map[string]any{
status, _, err = doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), map[string]any{
"requirements": []any{},
"updates": []any{},
})
@@ -491,6 +520,21 @@ func doIcebergJSONRequest(env *TestEnvironment, method, path string, payload any
return resp.StatusCode, decoded, nil
}
// icebergPath inserts the table bucket prefix into Iceberg REST API paths.
// For example, "/v1/namespaces" with prefix "my-bucket" becomes
// "/v1/my-bucket/namespaces".
func icebergPath(prefix, path string) string {
if prefix == "" {
return path
}
const base = "/v1/"
if !strings.HasPrefix(path, base) {
return path
}
withPrefix := base + prefix + "/" + strings.TrimPrefix(path, base)
return withPrefix
}
// createTableBucket creates a table bucket via the S3Tables REST API
func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) {
t.Helper()
@@ -520,6 +564,11 @@ func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) {
t.Logf("Created table bucket %s", bucketName)
}
// randomSuffix returns a short random hex suffix for unique resource naming.
func randomSuffix() string {
return fmt.Sprintf("%x", time.Now().UnixNano()&0xffffffff)
}
// TestDuckDBIntegration tests Iceberg catalog operations using DuckDB
// This test requires Docker to be available
func TestDuckDBIntegration(t *testing.T) {
@@ -527,15 +576,12 @@ func TestDuckDBIntegration(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping DuckDB integration test")
}
env.StartSeaweedFS(t)
// Create a temporary SQL file for DuckDB to execute
sqlFile := filepath.Join(env.dataDir, "test.sql")
sqlContent := fmt.Sprintf(`

15
test/s3tables/catalog/pyiceberg_test.go
View File

@@ -26,17 +26,14 @@ func TestPyIcebergRestCatalog(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping PyIceberg integration test")
}
env.StartSeaweedFS(t)
// Create the test bucket first
bucketName := "pyiceberg-compat-test"
bucketName := "pyiceberg-compat-test-" + randomSuffix()
createTableBucket(t, env, bucketName)
// Build the test working directory path
@@ -84,8 +81,7 @@ func TestPyIcebergRestCatalogAuthenticated(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping PyIceberg integration test")
@@ -95,11 +91,8 @@ func TestPyIcebergRestCatalogAuthenticated(t *testing.T) {
testAccessKey := "admin"
testSecretKey := "admin"
// Start SeaweedFS (it will use default admin credentials from environment if set)
env.StartSeaweedFS(t)
// Create the test bucket first (using unauthenticated request, which works with DefaultAllow)
bucketName := "pyiceberg-auth-test"
bucketName := "pyiceberg-auth-test-" + randomSuffix()
createTableBucket(t, env, bucketName)
// Build the test working directory path

0
test/s3tables/catalog/pyiceberg_test_helpers.go → test/s3tables/catalog/pyiceberg_test_helpers_test.go
View File