Commit Graph

12673 Commits

Author SHA1 Message Date
Chris Lu
2b2ff008cd s3tables: improve resource resolution and error mapping for policies and tagging
Refactored resolveResourcePath to return resource type, enabling accurate
NoSuchBucket vs NoSuchTable error codes. Added existence checks before
deleting policies.
2026-01-28 13:25:32 -08:00
Chris Lu
f17ec59d46 s3tables: implement optimistic concurrency for table deletion
Added VersionToken validation to handleDeleteTable. Refactored table
listing to use request context for accurate ARN generation and fixed
cross-namespace pagination issues.
2026-01-28 13:25:32 -08:00
Chris Lu
12c1190a5c s3tables: update namespace handlers for multi-account support
Updated namespace creation to use authenticated account ID for ownership
and unified permission checks across all namespace operations to use the
correct account principal.
2026-01-28 13:25:27 -08:00
Chris Lu
922b0c3171 s3tables: update bucket handlers for multi-account support
Ensured bucket ownership is correctly attributed to the authenticated
account ID and updated ARNs to use the request-derived account ID. Added
standard S3 existence checks for bucket deletion.
2026-01-28 13:25:27 -08:00
Chris Lu
31867b6f75 s3tables: improve account ID handling and define missing error codes
Updated getPrincipalFromRequest to prioritize X-Amz-Account-ID header and
added getAccountID helper. Defined ErrVersionTokenMismatch and ErrCodeConflict
for better optimistic concurrency support.
2026-01-28 13:25:22 -08:00
Chris Lu
e381b81b47 s3tables: use crypto/rand for secure version token generation
Replaced math/rand with crypto/rand to ensure version tokens are
cryptographically secure and unpredictable for optimistic concurrency control.
2026-01-28 13:25:19 -08:00
Chris Lu
89b85bfd5e s3tables test: update integration tests for new client API 2026-01-28 12:46:21 -08:00
Chris Lu
783fe25eac s3tables test: expose pagination parameters in client list methods 2026-01-28 12:46:21 -08:00
Chris Lu
5c43f1e6a4 s3tables: fix cross-namespace pagination in listTablesInAllNamespaces 2026-01-28 12:46:21 -08:00
Chris Lu
5eed1874a9 s3tables: return 404 in handleDeleteNamespace if namespace not found 2026-01-28 12:46:20 -08:00
Chris Lu
ae7743304c s3tables: refine permission helpers to align with operation names 2026-01-28 12:46:20 -08:00
Chris Lu
babf1b06ac s3tables: implement token-based pagination for namespace listing 2026-01-28 12:30:33 -08:00
Chris Lu
6ff683a627 s3tables: implement token-based pagination for table buckets listing 2026-01-28 12:30:33 -08:00
Chris Lu
1fb3aefa95 s3tables: ensure root tables directory exists before bucket creation 2026-01-28 12:30:32 -08:00
Chris Lu
47ef8c3cce s3tables: add table name validation and cleanup duplicated logic in table handlers 2026-01-28 12:30:32 -08:00
Chris Lu
f83cef1259 s3tables: add table name validation and 404 propagation to policy handlers 2026-01-28 12:30:31 -08:00
Chris Lu
ab6351e1c9 s3tables: implement validateTableName helper 2026-01-28 12:30:31 -08:00
Chris Lu
9f1dd57939 s3tables test: improve error reporting on decoding failure 2026-01-28 12:30:30 -08:00
Chris Lu
1f70d82170 s3tables: remove duplicate comment in permissions.go 2026-01-28 12:30:30 -08:00
Chris Lu
6d01e42cef s3tables: improve principal extraction using identity context 2026-01-28 12:30:29 -08:00
Chris Lu
5cea00ff07 S3 Tables: use os.ModeDir constant in filer_ops.go
- Replace magic number 1<<31 with os.ModeDir for better readability.
- Added necessary os import.
2026-01-28 12:13:35 -08:00
Chris Lu
d8c7c16aad S3 Tables: fix gRPC stream loop handling in namespace handlers
- Correctly handle io.EOF in handleListNamespaces and handleDeleteNamespace.
- Propagate other errors to prevent silent failures or accidental data loss.
- Added necessary io import.
2026-01-28 12:13:29 -08:00
Chris Lu
b7c1eba0a1 S3 Tables: secure API router with IAM authentication
- Wrap S3 Tables handler with authenticateS3Tables.
- Use AuthSignatureOnly to enforce valid credentials while delegating granular authorization to handlers.
- Prevent anonymous access to all S3 Tables endpoints.
2026-01-28 12:09:15 -08:00
Chris Lu
ae19621230 S3 Tables: validate ARN namespace to prevent path traversal
- Enforce validation on decoded namespace in parseTableFromARN.
- Ensures path components are safe after URL unescaping.
2026-01-28 12:09:09 -08:00
Chris Lu
f13e250fc3 S3 Tables: fix gRPC stream loop handling for list operations
- Correctly handle io.EOF to terminate loops gracefully.
- Propagate other errors to prevent silent failures.
- Ensure all list results are processed effectively.
2026-01-28 12:09:04 -08:00
Chris Lu
dc4c62e742 s3tables: harden auth and error handling
- Add authorization checks to all S3 Tables handlers (policy, table ops) to enforce security
- Improve error handling to distinguish between NotFound (404) and InternalError (500)
- Fix directory FileMode usage in filer_ops
- Improve test randomness for version tokens
- Update permissions comments to acknowledge IAM gaps
2026-01-28 11:49:57 -08:00
Chris Lu
a3af5eb77a s3tables: optimize regex usage and improve version token uniqueness
- Pre-compile regex patterns as package-level variables to avoid re-compilation overhead on every call
- Add a random component to version token generation to reduce collision probability under high concurrency
2026-01-28 11:44:56 -08:00
Chris Lu
1c0d37e15a s3tables: improve error handling and permission logic
- Update handleGetNamespace to distinguish between 404 and 500 errors
- Refactor CanManagePolicy to use CheckPermission for consistent enforcement
- Ensure empty identities are correctly handled in policy management checks
2026-01-28 11:39:28 -08:00
Chris Lu
6fc170c645 test: add miniClusterMutex to prevent race conditions
- Introduce sync.Mutex to protect global state (os.Args, os.Chdir)
- Ensure serialized initialization of the mini cluster runner
- Fix intermittent race conditions during parallel test execution
2026-01-28 11:39:22 -08:00
Chris Lu
62a1178a0b s3tables: improve robustness, security, and error propagation in handlers
- Implement strict table name validation (prevention of path traversal and character enforcement)
- Add nil checks for entry.Entry in all listing loops to prevent panics
- Propagate backend errors instead of swallowing them or assuming 404
- Correctly map filer_pb.ErrNotFound to appropriate S3 error codes
- Standardize existence checks across bucket, namespace, and table handlers
2026-01-28 11:37:02 -08:00
Chris Lu
da15ee3e49 s3tables: harden namespace validation and correct ARN parsing
- Prohibit path traversal (".", "..") and "/" in namespaces
- Restrict namespace characters to [a-z0-9_] for consistency
- Switch to url.PathUnescape for correct decoding of ARN path components
- Align ARN parsing regex with single-segment namespace validation
2026-01-28 11:36:56 -08:00
Chris Lu
b4d8350936 s3tables: normalize filer errors and use standard helpers
- Migrate from custom ErrNotFound to filer_pb.ErrNotFound
- Use filer_pb.LookupEntry for automatic error normalization
- Normalize entryExists and attribute lookups
2026-01-28 11:36:49 -08:00
Chris Lu
1d1634c2a2 test: update integration tests to match refactored S3 Tables client
- Pass namespaces as []string to support hierarchical structures
- Adapt test calls to new client API signatures
2026-01-28 11:31:49 -08:00
Chris Lu
44f580c24e test: refactor S3 Tables client for DRYness and multi-segment namespaces
- Implement doRequestAndDecode to eliminate HTTP boilerplate
- Update client API to accept []string for namespaces to support hierarchy
- Standardize error response decoding across all client methods
2026-01-28 11:31:44 -08:00
Chris Lu
04514071a7 s3tables: implement granular authorization and refine error responses
- Remove mandatory ACTION_ADMIN at the router level
- Enforce granular permissions in bucket and namespace handlers
- Prioritize AccountID in ExtractPrincipalFromContext for ARN matching
- Distinguish between 404 (NoSuchBucket) and 500 (InternalError) in metadata lookups
- Clean up unused imports in s3api_tables.go
2026-01-28 11:31:38 -08:00
Chris Lu
580c2b4ad4 command: fix stale error variable logging in filer serving goroutines
- Use local 'err' variable instead of stale 'e' from outer scope
- Applied to both TLS and non-TLS paths for local listener
2026-01-28 11:27:18 -08:00
Chris Lu
c62a332c7f s3tables: align ARN formatting and optimize resource handling
- Update generateTableARN to match AWS S3 Tables specification
- Move defer r.Body.Close() to follow standard Go patterns
- Remove unused generateNamespaceARN helper
2026-01-28 10:36:38 -08:00
Chris Lu
2c551dad5d s3tables: fix pagination and enhance error handling in list/delete operations
- Fix InclusiveStartFrom logic to ensure exclusive start on continued pages
- Prevent duplicates in bucket, namespace, and table listings
- Fail fast on listing errors during bucket and namespace deletion
- Stop swallowing errors in handleListTables and return proper HTTP error responses
2026-01-28 10:36:28 -08:00
Chris Lu
01c17478ae command: implement graceful shutdown for mini cluster
- Introduce MiniClusterCtx to coordinate shutdown across mini services
- Update Master, Volume, Filer, S3, and WebDAV servers to respect context cancellation
- Ensure all resources are cleaned up properly during test teardown
- Integrate MiniClusterCtx in s3tables integration tests
2026-01-28 10:36:19 -08:00
Chris Lu
07002cf54c test: improve S3 Tables client error handling and cleanup
- Add detailed error reporting when decoding failure responses
- Remove orphaned comments and unused sections
2026-01-28 10:36:11 -08:00
Chris Lu
3aace37cf6 s3tables: further refinements to filer operations and utilities
- Add multi-segment namespace support to ARN parsing
- Refactor permission checking to use map lookup
- Wrap lookup errors with ErrNotFound in filer operations
- Standardize splitPath to use path package
2026-01-28 10:36:03 -08:00
Chris Lu
33da87452b Refine S3 Tables implementation to address code review feedback
- Standardize namespace representation to []string
- Improve listing logic with pagination and StartFromFileName
- Enhance error handling with sentinel errors and robust checks
- Add JSON encoding error logging
- Fix CI workflow to use gofmt -l
- Standardize timestamps in directory creation
- Validate single-level namespaces
2026-01-28 10:04:27 -08:00
Chris Lu
08ee4e37d8 s3tables: clean up unused code and improve error response formatting 2026-01-28 09:38:10 -08:00
Chris Lu
b30631c3b5 s3tables: propagate request context to filer operations 2026-01-28 09:38:01 -08:00
Chris Lu
6cdd34da77 s3tables: improve integration test stability and error reporting 2026-01-28 09:37:58 -08:00
Chris Lu
b01504649d s3tables: use path.Join for path construction and align namespace paths 2026-01-28 09:37:54 -08:00
Chris Lu
24c78d524c ci: fail s3 tables tests if any command in pipeline fails 2026-01-28 09:37:51 -08:00
Chris Lu
05c184b610 workflow: fix go install path to ./weed 2026-01-28 01:35:24 -08:00
Chris Lu
33c1a8251a test: format s3tables client.go 2026-01-28 01:30:07 -08:00
Chris Lu
96a6e4c551 workflow: remove emojis from echo statements 2026-01-28 01:29:42 -08:00