Commit Graph

12651 Commits

Author SHA1 Message Date
Chris Lu
b7c1eba0a1 S3 Tables: secure API router with IAM authentication
- Wrap S3 Tables handler with authenticateS3Tables.
- Use AuthSignatureOnly to enforce valid credentials while delegating granular authorization to handlers.
- Prevent anonymous access to all S3 Tables endpoints.
2026-01-28 12:09:15 -08:00
Chris Lu
ae19621230 S3 Tables: validate ARN namespace to prevent path traversal
- Enforce validation on decoded namespace in parseTableFromARN.
- Ensures path components are safe after URL unescaping.
2026-01-28 12:09:09 -08:00
Chris Lu
f13e250fc3 S3 Tables: fix gRPC stream loop handling for list operations
- Correctly handle io.EOF to terminate loops gracefully.
- Propagate other errors to prevent silent failures.
- Ensure all list results are processed effectively.
2026-01-28 12:09:04 -08:00
Chris Lu
dc4c62e742 s3tables: harden auth and error handling
- Add authorization checks to all S3 Tables handlers (policy, table ops) to enforce security
- Improve error handling to distinguish between NotFound (404) and InternalError (500)
- Fix directory FileMode usage in filer_ops
- Improve test randomness for version tokens
- Update permissions comments to acknowledge IAM gaps
2026-01-28 11:49:57 -08:00
Chris Lu
a3af5eb77a s3tables: optimize regex usage and improve version token uniqueness
- Pre-compile regex patterns as package-level variables to avoid re-compilation overhead on every call
- Add a random component to version token generation to reduce collision probability under high concurrency
2026-01-28 11:44:56 -08:00
Chris Lu
1c0d37e15a s3tables: improve error handling and permission logic
- Update handleGetNamespace to distinguish between 404 and 500 errors
- Refactor CanManagePolicy to use CheckPermission for consistent enforcement
- Ensure empty identities are correctly handled in policy management checks
2026-01-28 11:39:28 -08:00
Chris Lu
6fc170c645 test: add miniClusterMutex to prevent race conditions
- Introduce sync.Mutex to protect global state (os.Args, os.Chdir)
- Ensure serialized initialization of the mini cluster runner
- Fix intermittent race conditions during parallel test execution
2026-01-28 11:39:22 -08:00
Chris Lu
62a1178a0b s3tables: improve robustness, security, and error propagation in handlers
- Implement strict table name validation (prevention of path traversal and character enforcement)
- Add nil checks for entry.Entry in all listing loops to prevent panics
- Propagate backend errors instead of swallowing them or assuming 404
- Correctly map filer_pb.ErrNotFound to appropriate S3 error codes
- Standardize existence checks across bucket, namespace, and table handlers
2026-01-28 11:37:02 -08:00
Chris Lu
da15ee3e49 s3tables: harden namespace validation and correct ARN parsing
- Prohibit path traversal (".", "..") and "/" in namespaces
- Restrict namespace characters to [a-z0-9_] for consistency
- Switch to url.PathUnescape for correct decoding of ARN path components
- Align ARN parsing regex with single-segment namespace validation
2026-01-28 11:36:56 -08:00
Chris Lu
b4d8350936 s3tables: normalize filer errors and use standard helpers
- Migrate from custom ErrNotFound to filer_pb.ErrNotFound
- Use filer_pb.LookupEntry for automatic error normalization
- Normalize entryExists and attribute lookups
2026-01-28 11:36:49 -08:00
Chris Lu
1d1634c2a2 test: update integration tests to match refactored S3 Tables client
- Pass namespaces as []string to support hierarchical structures
- Adapt test calls to new client API signatures
2026-01-28 11:31:49 -08:00
Chris Lu
44f580c24e test: refactor S3 Tables client for DRYness and multi-segment namespaces
- Implement doRequestAndDecode to eliminate HTTP boilerplate
- Update client API to accept []string for namespaces to support hierarchy
- Standardize error response decoding across all client methods
2026-01-28 11:31:44 -08:00
Chris Lu
04514071a7 s3tables: implement granular authorization and refine error responses
- Remove mandatory ACTION_ADMIN at the router level
- Enforce granular permissions in bucket and namespace handlers
- Prioritize AccountID in ExtractPrincipalFromContext for ARN matching
- Distinguish between 404 (NoSuchBucket) and 500 (InternalError) in metadata lookups
- Clean up unused imports in s3api_tables.go
2026-01-28 11:31:38 -08:00
Chris Lu
580c2b4ad4 command: fix stale error variable logging in filer serving goroutines
- Use local 'err' variable instead of stale 'e' from outer scope
- Applied to both TLS and non-TLS paths for local listener
2026-01-28 11:27:18 -08:00
Chris Lu
c62a332c7f s3tables: align ARN formatting and optimize resource handling
- Update generateTableARN to match AWS S3 Tables specification
- Move defer r.Body.Close() to follow standard Go patterns
- Remove unused generateNamespaceARN helper
2026-01-28 10:36:38 -08:00
Chris Lu
2c551dad5d s3tables: fix pagination and enhance error handling in list/delete operations
- Fix InclusiveStartFrom logic to ensure exclusive start on continued pages
- Prevent duplicates in bucket, namespace, and table listings
- Fail fast on listing errors during bucket and namespace deletion
- Stop swallowing errors in handleListTables and return proper HTTP error responses
2026-01-28 10:36:28 -08:00
Chris Lu
01c17478ae command: implement graceful shutdown for mini cluster
- Introduce MiniClusterCtx to coordinate shutdown across mini services
- Update Master, Volume, Filer, S3, and WebDAV servers to respect context cancellation
- Ensure all resources are cleaned up properly during test teardown
- Integrate MiniClusterCtx in s3tables integration tests
2026-01-28 10:36:19 -08:00
Chris Lu
07002cf54c test: improve S3 Tables client error handling and cleanup
- Add detailed error reporting when decoding failure responses
- Remove orphaned comments and unused sections
2026-01-28 10:36:11 -08:00
Chris Lu
3aace37cf6 s3tables: further refinements to filer operations and utilities
- Add multi-segment namespace support to ARN parsing
- Refactor permission checking to use map lookup
- Wrap lookup errors with ErrNotFound in filer operations
- Standardize splitPath to use path package
2026-01-28 10:36:03 -08:00
Chris Lu
33da87452b Refine S3 Tables implementation to address code review feedback
- Standardize namespace representation to []string
- Improve listing logic with pagination and StartFromFileName
- Enhance error handling with sentinel errors and robust checks
- Add JSON encoding error logging
- Fix CI workflow to use gofmt -l
- Standardize timestamps in directory creation
- Validate single-level namespaces
2026-01-28 10:04:27 -08:00
Chris Lu
08ee4e37d8 s3tables: clean up unused code and improve error response formatting 2026-01-28 09:38:10 -08:00
Chris Lu
b30631c3b5 s3tables: propagate request context to filer operations 2026-01-28 09:38:01 -08:00
Chris Lu
6cdd34da77 s3tables: improve integration test stability and error reporting 2026-01-28 09:37:58 -08:00
Chris Lu
b01504649d s3tables: use path.Join for path construction and align namespace paths 2026-01-28 09:37:54 -08:00
Chris Lu
24c78d524c ci: fail s3 tables tests if any command in pipeline fails 2026-01-28 09:37:51 -08:00
Chris Lu
05c184b610 workflow: fix go install path to ./weed 2026-01-28 01:35:24 -08:00
Chris Lu
33c1a8251a test: format s3tables client.go 2026-01-28 01:30:07 -08:00
Chris Lu
96a6e4c551 workflow: remove emojis from echo statements 2026-01-28 01:29:42 -08:00
Chris Lu
f4e472d396 workflow: fix s3 tables tests path and working directory
The workflow was failing because it was running inside 'weed' directory,
but the tests are at the repository root. Removed working-directory
default and updated relative paths to weed source.
2026-01-28 01:28:47 -08:00
Chris Lu
fb0e12e985 s3tables: implement permission checking and authorization
- Add permissions.go with permission definitions and checks
- Define permissions for all 21 S3 Tables operations
- Add permission checking helper functions
- Add getPrincipalFromRequest to extract caller identity
- Implement access control in CreateTableBucket, GetTableBucket, DeleteTableBucket
- Return 403 Forbidden for unauthorized operations
- Only bucket owner can perform operations (extensible for future policies)
- Add AuthError type for authorization failures
2026-01-28 01:18:11 -08:00
Chris Lu
1b9c8b8614 s3api_tables: optimize action validation with map lookup
- Replace O(n) slice iteration with O(1) map lookup
- Move s3TablesActionsMap to package level
- Avoid recreating the map on every function call
- Improves performance for request validation
2026-01-28 01:14:31 -08:00
Chris Lu
450407fda1 s3tables: improve error handling specificity in ListTableBuckets
- Specifically check for 'not found' errors instead of catching all errors
- Return empty list only when directory doesn't exist
- Propagate other errors (network, permission) with context
- Prevents masking real errors
2026-01-28 01:14:14 -08:00
Chris Lu
b09d4d5d69 s3tables: replace custom splitPath with stdlib functions
- Remove custom splitPath implementation (23 lines)
- Use filepath.Dir and filepath.Base from stdlib
- More robust and handles edge cases correctly
- Reduces code duplication
2026-01-28 01:13:56 -08:00
Chris Lu
ef3873b616 s3tables: add error handling for json.Marshal calls
- Add error handling in handler_namespace.go (metadata marshaling)
- Add error handling in handler_table.go (metadata and tags marshaling)
- Add error handling in handler_policy.go (tag marshaling in TagResource and UntagResource)
- Return proper errors with context instead of silently ignoring failures
2026-01-28 01:13:42 -08:00
Chris Lu
dd7003a328 s3tables: add bucket name validation and fix error handling
- Add isValidBucketName validation function for [a-z0-9_-] characters
- Validate bucket name characters match ARN parsing regex
- Fix error handling in WithFilerClient closure - properly check for lookup errors
- Add error handling for json.Marshal calls (metadata and tags)
- Improve error messages and logging
2026-01-28 01:12:57 -08:00
Chris Lu
22a9fbf062 Delete client_methods.go 2026-01-28 01:07:53 -08:00
Chris Lu
06f90028f7 s3tables test: refactor to eliminate duplicate definitions
- Move all client methods to client.go
- Remove duplicate types/constants from s3tables_integration_test.go
- Keep setup.go for test infrastructure
- Keep integration test logic in s3tables_integration_test.go
- Clean up unused imports
- Test compiles successfully
2026-01-28 01:07:11 -08:00
Chris Lu
3b1920cf43 s3tables: add handler_ prefix to operation handler files
- Rename bucket_create.go → handler_bucket_create.go
- Rename bucket_get_list_delete.go → handler_bucket_get_list_delete.go
- Rename namespace.go → handler_namespace.go
- Rename table.go → handler_table.go
- Rename policy.go → handler_policy.go

Improves file organization by clearly identifying handler implementations.
No code changes, refactoring only.
2026-01-28 01:00:00 -08:00
Chris Lu
0dcb175514 ci: add S3 Tables integration tests to GitHub Actions
- Create new workflow for S3 Tables integration testing
- Add build verification job for s3tables package and s3api integration
- Add format checking for S3 Tables code
- Add go vet checks for code quality
- Workflow runs on all pull requests
- Includes test output logging and artifact upload on failure
2026-01-28 00:57:53 -08:00
Chris Lu
ddcdefb6b4 test: add S3 Tables integration tests
- Comprehensive integration tests for all 23 S3 Tables operations
- Test cluster setup based on existing S3 integration tests
- Tests cover:
  * Table bucket lifecycle (create, get, list, delete)
  * Namespace operations
  * Table CRUD with Iceberg schema
  * Table bucket and table policies
  * Resource tagging operations
- Ready for CI/CD pipeline integration
2026-01-28 00:55:50 -08:00
Chris Lu
d66e194284 test: add S3 Tables test infrastructure
- Create setup.go with TestCluster and S3TablesClient definitions
- Create client.go with HTTP client methods for all operations
- Test utilities and client methods organized for reusability
- Foundation for S3 Tables integration tests
2026-01-28 00:55:45 -08:00
Chris Lu
6a12438351 s3api: register S3 Tables routes in API server
- Add S3 Tables route registration in s3api_server.go registerRouter method
- Enable S3 Tables API operations to be routed through S3 API server
- Routes handled by s3api_tables.go integration layer
- Minimal changes to existing S3 API structure
2026-01-28 00:55:39 -08:00
Chris Lu
b1b922d757 s3api: add S3 Tables integration layer
- Create s3api_tables.go to integrate S3 Tables with S3 API server
- Implement S3 Tables route matcher for X-Amz-Target header
- Register S3 Tables routes with API router
- Provide gRPC filer client interface for S3 Tables handlers
- All S3 Tables operations accessible via S3 API endpoint
2026-01-28 00:55:34 -08:00
Chris Lu
0be47f9efd s3tables: complete s3tables package implementation
- namespace.go: namespace CRUD operations (310 lines)
- table.go: table CRUD operations with Iceberg schema support (409 lines)
- policy.go: resource policies and tagging operations (419 lines)
- types.go: request/response types and error definitions (290 lines)
- All handlers updated to use standalone utilities from utils.go
- All files follow single responsibility principle
2026-01-28 00:55:28 -08:00
Chris Lu
62fd4bd017 s3tables: simplify handler by removing duplicate utilities
- Reduce handler.go from 370 to 195 lines (47% reduction)
- Remove duplicate ARN parsing and path helper functions
- Remove filer operation methods moved to filer_ops.go
- Remove metadata structure definitions moved to utils.go
- Keep handler focused on request routing and response formatting
- Maintains all functionality with improved code organization
2026-01-28 00:55:23 -08:00
Chris Lu
09bb90e8dc s3tables: split table bucket operations into focused modules
- Create bucket_create.go for CreateTableBucket operation
- Create bucket_get_list_delete.go for Get, List, Delete operations
- Related operations grouped for better maintainability
- Each file has a single, clear responsibility
- Improves code clarity and makes it easier to test
2026-01-28 00:55:17 -08:00
Chris Lu
d730f81321 s3tables: extract utility and filer operations to separate modules
- Move ARN parsing, path helpers, and metadata structures to utils.go
- Extract all extended attribute and filer operations to filer_ops.go
- Reduces code duplication and improves modularity
- Improves code organization and maintainability
2026-01-28 00:55:11 -08:00
Chris Lu
6542d1e0aa Enable weed fuse on FreeBSD (#8146)
* Enable weed fuse on FreeBSD

* Update weed/command/fuse_notsupported.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/command/fuse_std.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-01-27 21:37:23 -08:00
Chris Lu
e86e65e5ab fix #8081: build latest container is missing latest_large_disk (#8145)
* fix #8081: build latest container is missing latest_large_disk

* fix: simplify QEMU setup condition in container_latest.yml matrix
2026-01-27 19:05:29 -08:00
Chris Lu
8e9c12e2c7 mount: apply UID/GID mapping in lookupEntry for cache misses (#8144)
* mount: apply UID/GID mapping in lookupEntry for cache misses

This fixes issue #8134 where rsync would fail with "Operation not permitted"
during chgrp. The issue was that entries fetched directly from the filer
(on cache miss) were not being mapped to local UIDs/GIDs.

* mount: add nil check for entry.Attributes in lookupEntry
2026-01-27 18:41:37 -08:00