* Add Trino blog operations test * Update test/s3tables/catalog_trino/trino_blog_operations_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * feat: add table bucket path helpers and filer operations - Add table object root and table location mapping directories - Implement ensureDirectory, upsertFile, deleteEntryIfExists helpers - Support table location bucket mapping for S3 access * feat: manage table bucket object roots on creation/deletion - Create .objects directory for table buckets on creation - Clean up table object bucket paths on deletion - Enable S3 operations on table bucket object roots * feat: add table location mapping for Iceberg REST - Track table location bucket mappings when tables are created/updated/deleted - Enable location-based routing for S3 operations on table data * feat: route S3 operations to table bucket object roots - Route table-s3 bucket names to mapped table paths - Route table buckets to object root directories - Support table location bucket mapping lookup * feat: emit table-s3 locations from Iceberg REST - Generate unique table-s3 bucket names with UUID suffix - Store table metadata under table bucket paths - Return table-s3 locations for Trino compatibility * fix: handle missing directories in S3 list operations - Propagate ErrNotFound from ListEntries for non-existent directories - Treat missing directories as empty results for list operations - Fixes Trino non-empty location checks on table creation * test: improve Trino CSV parsing for single-value results - Sanitize Trino output to skip jline warnings - Handle single-value CSV results without header rows - Strip quotes from numeric values in tests * refactor: use bucket path helpers throughout S3 API - Replace direct bucket path operations with helper functions - Leverage centralized table bucket routing logic - Improve maintainability with consistent path resolution * fix: add table bucket cache and improve filer error handling - Cache table bucket lookups to reduce filer overhead on repeated checks - Use filer_pb.CreateEntry and filer_pb.UpdateEntry helpers to check resp.Error - Fix delete order in handler_bucket_get_list_delete: delete table object before directory - Make location mapping errors best-effort: log and continue, don't fail API - Update table location mappings to delete stale prior bucket mappings on update - Add 1-second sleep before timestamp time travel query to ensure timestamps are in past - Fix CSV parsing: examine all lines, not skip first; handle single-value rows * fix: properly handle stale metadata location mapping cleanup - Capture oldMetadataLocation before mutation in handleUpdateTable - Update updateTableLocationMapping to accept both old and new locations - Use passed-in oldMetadataLocation to detect location changes - Delete stale mapping only when location actually changes - Pass empty string for oldLocation in handleCreateTable (new tables have no prior mapping) - Improve logging to show old -> new location transitions * refactor: cleanup imports and cache design - Remove unused 'sync' import from bucket_paths.go - Use filer_pb.UpdateEntry helper in setExtendedAttribute and deleteExtendedAttribute for consistent error handling - Add dedicated tableBucketCache map[string]bool to BucketRegistry instead of mixing concerns with metadataCache - Improve cache separation: table buckets cache is now separate from bucket metadata cache * fix: improve cache invalidation and add transient error handling Cache invalidation (critical fix): - Add tableLocationCache to BucketRegistry for location mapping lookups - Clear tableBucketCache and tableLocationCache in RemoveBucketMetadata - Prevents stale cache entries when buckets are deleted/recreated Transient error handling: - Only cache table bucket lookups when conclusive (found or ErrNotFound) - Skip caching on transient errors (network, permission, etc) - Prevents marking real table buckets as non-table due to transient failures Performance optimization: - Cache tableLocationDir results to avoid repeated filer RPCs on hot paths - tableLocationDir now checks cache before making expensive filer lookups - Cache stores empty string for 'not found' to avoid redundant lookups Code clarity: - Add comment to deleteDirectory explaining DeleteEntry response lacks Error field * go fmt * fix: mirror transient error handling in tableLocationDir and optimize bucketDir Transient error handling: - tableLocationDir now only caches definitive results - Mirrors isTableBucket behavior to prevent treating transient errors as permanent misses - Improves reliability on flaky systems or during recovery Performance optimization: - bucketDir avoids redundant isTableBucket call via bucketRoot - Directly use s3a.option.BucketsPath for regular buckets - Saves one cache lookup for every non-table bucket operation * fix: revert bucketDir optimization to preserve bucketRoot logic The optimization to directly use BucketsPath bypassed bucketRoot's logic and caused issues with S3 list operations on delimiter+prefix cases. Revert to using path.Join(s3a.bucketRoot(bucket), bucket) which properly handles all bucket types and ensures consistent path resolution across the codebase. The slight performance cost of an extra cache lookup is worth the correctness and consistency benefits. * feat: move table buckets under /buckets Add a table-bucket marker attribute, reuse bucket metadata cache for table bucket detection, and update list/validation/UI/test paths to treat table buckets as /buckets entries. * Fix S3 Tables code review issues - handler_bucket_create.go: Fix bucket existence check to properly validate entryResp.Entry before setting s3BucketExists flag (nil Entry should not indicate existing bucket) - bucket_paths.go: Add clarifying comment to bucketRoot() explaining unified buckets root path for all bucket types - file_browser_data.go: Optimize by extracting table bucket check early to avoid redundant WithFilerClient call * Fix list prefix delimiter handling * Handle list errors conservatively * Fix Trino FOR TIMESTAMP query - use past timestamp Iceberg requires the timestamp to be strictly in the past. Use current_timestamp - interval '1' second instead of current_timestamp. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
928 lines
31 KiB
Go
928 lines
31 KiB
Go
package s3tables
|
|
|
|
import (
|
|
"encoding/json"
|
|
"errors"
|
|
"fmt"
|
|
"net/http"
|
|
"strings"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
)
|
|
|
|
// extractResourceOwnerAndBucket extracts ownership info and bucket name from resource metadata.
|
|
// This helper consolidates the repeated pattern used in handleTagResource, handleListTagsForResource,
|
|
// and handleUntagResource.
|
|
func (h *S3TablesHandler) extractResourceOwnerAndBucket(
|
|
data []byte,
|
|
resourcePath string,
|
|
rType ResourceType,
|
|
) (ownerAccountID, bucketName string, err error) {
|
|
// Extract bucket name from resource path (format: /buckets/{bucket}/... for both tables and buckets)
|
|
parts := strings.Split(strings.Trim(resourcePath, "/"), "/")
|
|
if len(parts) >= 2 {
|
|
bucketName = parts[1]
|
|
}
|
|
|
|
if rType == ResourceTypeTable {
|
|
var meta tableMetadataInternal
|
|
if err := json.Unmarshal(data, &meta); err != nil {
|
|
return "", "", err
|
|
}
|
|
ownerAccountID = meta.OwnerAccountID
|
|
} else {
|
|
var meta tableBucketMetadata
|
|
if err := json.Unmarshal(data, &meta); err != nil {
|
|
return "", "", err
|
|
}
|
|
ownerAccountID = meta.OwnerAccountID
|
|
}
|
|
return ownerAccountID, bucketName, nil
|
|
}
|
|
|
|
// handlePutTableBucketPolicy puts a policy on a table bucket
|
|
func (h *S3TablesHandler) handlePutTableBucketPolicy(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
|
|
|
|
var req PutTableBucketPolicyRequest
|
|
if err := h.readRequestBody(r, &req); err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.TableBucketARN == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "tableBucketARN is required")
|
|
return fmt.Errorf("tableBucketARN is required")
|
|
}
|
|
|
|
if req.ResourcePolicy == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "resourcePolicy is required")
|
|
return fmt.Errorf("resourcePolicy is required")
|
|
}
|
|
|
|
bucketName, err := parseBucketNameFromARN(req.TableBucketARN)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
// Check if bucket exists and get metadata for ownership check
|
|
bucketPath := GetTableBucketPath(bucketName)
|
|
var bucketMetadata tableBucketMetadata
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
data, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyMetadata)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := json.Unmarshal(data, &bucketMetadata); err != nil {
|
|
return fmt.Errorf("failed to unmarshal bucket metadata: %w", err)
|
|
}
|
|
return nil
|
|
})
|
|
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchBucket, fmt.Sprintf("table bucket %s not found", bucketName))
|
|
} else {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, fmt.Sprintf("failed to check table bucket: %v", err))
|
|
}
|
|
return err
|
|
}
|
|
|
|
bucketARN := h.generateTableBucketARN(bucketMetadata.OwnerAccountID, bucketName)
|
|
principal := h.getAccountID(r)
|
|
identityActions := getIdentityActions(r)
|
|
if !CheckPermissionWithContext("PutTableBucketPolicy", principal, bucketMetadata.OwnerAccountID, "", bucketARN, &PolicyContext{
|
|
TableBucketName: bucketName,
|
|
IdentityActions: identityActions,
|
|
}) {
|
|
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to put table bucket policy")
|
|
return NewAuthError("PutTableBucketPolicy", principal, "not authorized to put table bucket policy")
|
|
}
|
|
|
|
// Write policy
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
return h.setExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy, []byte(req.ResourcePolicy))
|
|
})
|
|
|
|
if err != nil {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, "failed to put table bucket policy")
|
|
return err
|
|
}
|
|
|
|
h.writeJSON(w, http.StatusOK, nil)
|
|
return nil
|
|
}
|
|
|
|
// handleGetTableBucketPolicy gets the policy of a table bucket
|
|
func (h *S3TablesHandler) handleGetTableBucketPolicy(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
|
|
|
|
var req GetTableBucketPolicyRequest
|
|
if err := h.readRequestBody(r, &req); err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.TableBucketARN == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "tableBucketARN is required")
|
|
return fmt.Errorf("tableBucketARN is required")
|
|
}
|
|
|
|
bucketName, err := parseBucketNameFromARN(req.TableBucketARN)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
bucketPath := GetTableBucketPath(bucketName)
|
|
var policy []byte
|
|
var bucketMetadata tableBucketMetadata
|
|
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
// Get metadata for ownership check
|
|
data, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyMetadata)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := json.Unmarshal(data, &bucketMetadata); err != nil {
|
|
return fmt.Errorf("failed to unmarshal bucket metadata: %w", err)
|
|
}
|
|
|
|
// Get policy
|
|
policy, err = h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy)
|
|
return err
|
|
})
|
|
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchBucket, fmt.Sprintf("table bucket %s not found", bucketName))
|
|
return err
|
|
}
|
|
if errors.Is(err, ErrAttributeNotFound) {
|
|
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchPolicy, "table bucket policy not found")
|
|
return err
|
|
}
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, fmt.Sprintf("failed to get table bucket policy: %v", err))
|
|
return err
|
|
}
|
|
|
|
bucketARN := h.generateTableBucketARN(bucketMetadata.OwnerAccountID, bucketName)
|
|
principal := h.getAccountID(r)
|
|
identityActions := getIdentityActions(r)
|
|
if !CheckPermissionWithContext("GetTableBucketPolicy", principal, bucketMetadata.OwnerAccountID, string(policy), bucketARN, &PolicyContext{
|
|
TableBucketName: bucketName,
|
|
IdentityActions: identityActions,
|
|
}) {
|
|
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to get table bucket policy")
|
|
return NewAuthError("GetTableBucketPolicy", principal, "not authorized to get table bucket policy")
|
|
}
|
|
|
|
resp := &GetTableBucketPolicyResponse{
|
|
ResourcePolicy: string(policy),
|
|
}
|
|
|
|
h.writeJSON(w, http.StatusOK, resp)
|
|
return nil
|
|
}
|
|
|
|
// handleDeleteTableBucketPolicy deletes the policy of a table bucket
|
|
func (h *S3TablesHandler) handleDeleteTableBucketPolicy(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
|
|
|
|
var req DeleteTableBucketPolicyRequest
|
|
if err := h.readRequestBody(r, &req); err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.TableBucketARN == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "tableBucketARN is required")
|
|
return fmt.Errorf("tableBucketARN is required")
|
|
}
|
|
|
|
bucketName, err := parseBucketNameFromARN(req.TableBucketARN)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
bucketPath := GetTableBucketPath(bucketName)
|
|
|
|
// Check if bucket exists and get metadata for ownership check
|
|
var bucketMetadata tableBucketMetadata
|
|
var bucketPolicy string
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
data, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyMetadata)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := json.Unmarshal(data, &bucketMetadata); err != nil {
|
|
return fmt.Errorf("failed to unmarshal bucket metadata: %w", err)
|
|
}
|
|
|
|
// Fetch bucket policy if it exists
|
|
policyData, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy)
|
|
if err != nil {
|
|
if !errors.Is(err, ErrAttributeNotFound) {
|
|
return fmt.Errorf("failed to read bucket policy: %w", err)
|
|
}
|
|
// Policy not found is not an error; bucketPolicy remains empty
|
|
} else {
|
|
bucketPolicy = string(policyData)
|
|
}
|
|
|
|
return nil
|
|
})
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchBucket, fmt.Sprintf("table bucket %s not found", bucketName))
|
|
} else {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, fmt.Sprintf("failed to check table bucket: %v", err))
|
|
}
|
|
return err
|
|
}
|
|
|
|
bucketARN := h.generateTableBucketARN(bucketMetadata.OwnerAccountID, bucketName)
|
|
principal := h.getAccountID(r)
|
|
identityActions := getIdentityActions(r)
|
|
if !CheckPermissionWithContext("DeleteTableBucketPolicy", principal, bucketMetadata.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
|
|
TableBucketName: bucketName,
|
|
IdentityActions: identityActions,
|
|
}) {
|
|
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to delete table bucket policy")
|
|
return NewAuthError("DeleteTableBucketPolicy", principal, "not authorized to delete table bucket policy")
|
|
}
|
|
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
return h.deleteExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy)
|
|
})
|
|
|
|
if err != nil && !errors.Is(err, ErrAttributeNotFound) {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, "failed to delete table bucket policy")
|
|
return err
|
|
}
|
|
|
|
h.writeJSON(w, http.StatusOK, nil)
|
|
return nil
|
|
}
|
|
|
|
// handlePutTablePolicy puts a policy on a table
|
|
func (h *S3TablesHandler) handlePutTablePolicy(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
|
|
|
|
var req PutTablePolicyRequest
|
|
if err := h.readRequestBody(r, &req); err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.TableBucketARN == "" || len(req.Namespace) == 0 || req.Name == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "tableBucketARN, namespace, and name are required")
|
|
return fmt.Errorf("missing required parameters")
|
|
}
|
|
|
|
namespaceName, err := validateNamespace(req.Namespace)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.ResourcePolicy == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "resourcePolicy is required")
|
|
return fmt.Errorf("resourcePolicy is required")
|
|
}
|
|
|
|
bucketName, err := parseBucketNameFromARN(req.TableBucketARN)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
// Check if table exists
|
|
tableName, err := validateTableName(req.Name)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
tablePath := GetTablePath(bucketName, namespaceName, tableName)
|
|
bucketPath := GetTableBucketPath(bucketName)
|
|
|
|
var metadata tableMetadataInternal
|
|
var bucketPolicy string
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
data, err := h.getExtendedAttribute(r.Context(), client, tablePath, ExtendedKeyMetadata)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := json.Unmarshal(data, &metadata); err != nil {
|
|
return fmt.Errorf("failed to unmarshal table metadata: %w", err)
|
|
}
|
|
|
|
// Fetch bucket policy if it exists
|
|
policyData, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy)
|
|
if err != nil {
|
|
if !errors.Is(err, ErrAttributeNotFound) {
|
|
return fmt.Errorf("failed to read bucket policy: %w", err)
|
|
}
|
|
// Policy not found is not an error; bucketPolicy remains empty
|
|
} else {
|
|
bucketPolicy = string(policyData)
|
|
}
|
|
|
|
return nil
|
|
})
|
|
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchTable, fmt.Sprintf("table %s not found", tableName))
|
|
} else {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, fmt.Sprintf("failed to check table: %v", err))
|
|
}
|
|
return err
|
|
}
|
|
|
|
tableARN := h.generateTableARN(metadata.OwnerAccountID, bucketName, namespaceName+"/"+tableName)
|
|
principal := h.getAccountID(r)
|
|
identityActions := getIdentityActions(r)
|
|
if !CheckPermissionWithContext("PutTablePolicy", principal, metadata.OwnerAccountID, bucketPolicy, tableARN, &PolicyContext{
|
|
TableBucketName: bucketName,
|
|
Namespace: namespaceName,
|
|
TableName: tableName,
|
|
IdentityActions: identityActions,
|
|
}) {
|
|
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to put table policy")
|
|
return NewAuthError("PutTablePolicy", principal, "not authorized to put table policy")
|
|
}
|
|
|
|
// Write policy
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
return h.setExtendedAttribute(r.Context(), client, tablePath, ExtendedKeyPolicy, []byte(req.ResourcePolicy))
|
|
})
|
|
|
|
if err != nil {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, "failed to put table policy")
|
|
return err
|
|
}
|
|
|
|
h.writeJSON(w, http.StatusOK, nil)
|
|
return nil
|
|
}
|
|
|
|
// handleGetTablePolicy gets the policy of a table
|
|
func (h *S3TablesHandler) handleGetTablePolicy(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
|
|
|
|
var req GetTablePolicyRequest
|
|
if err := h.readRequestBody(r, &req); err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.TableBucketARN == "" || len(req.Namespace) == 0 || req.Name == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "tableBucketARN, namespace, and name are required")
|
|
return fmt.Errorf("missing required parameters")
|
|
}
|
|
|
|
namespaceName, err := validateNamespace(req.Namespace)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
bucketName, err := parseBucketNameFromARN(req.TableBucketARN)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
tableName, err := validateTableName(req.Name)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
tablePath := GetTablePath(bucketName, namespaceName, tableName)
|
|
bucketPath := GetTableBucketPath(bucketName)
|
|
var policy []byte
|
|
var metadata tableMetadataInternal
|
|
var bucketPolicy string
|
|
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
// Get metadata for ownership check
|
|
data, err := h.getExtendedAttribute(r.Context(), client, tablePath, ExtendedKeyMetadata)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := json.Unmarshal(data, &metadata); err != nil {
|
|
return fmt.Errorf("failed to unmarshal table metadata: %w", err)
|
|
}
|
|
|
|
// Get policy
|
|
policy, err = h.getExtendedAttribute(r.Context(), client, tablePath, ExtendedKeyPolicy)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// Fetch bucket policy if it exists
|
|
policyData, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy)
|
|
if err != nil {
|
|
if !errors.Is(err, ErrAttributeNotFound) {
|
|
return fmt.Errorf("failed to read bucket policy: %w", err)
|
|
}
|
|
// Policy not found is not an error; bucketPolicy remains empty
|
|
} else {
|
|
bucketPolicy = string(policyData)
|
|
}
|
|
|
|
return nil
|
|
})
|
|
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchTable, fmt.Sprintf("table %s not found", tableName))
|
|
return err
|
|
}
|
|
if errors.Is(err, ErrAttributeNotFound) {
|
|
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchPolicy, "table policy not found")
|
|
return err
|
|
}
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, fmt.Sprintf("failed to get table policy: %v", err))
|
|
return err
|
|
}
|
|
|
|
tableARN := h.generateTableARN(metadata.OwnerAccountID, bucketName, namespaceName+"/"+tableName)
|
|
principal := h.getAccountID(r)
|
|
identityActions := getIdentityActions(r)
|
|
if !CheckPermissionWithContext("GetTablePolicy", principal, metadata.OwnerAccountID, bucketPolicy, tableARN, &PolicyContext{
|
|
TableBucketName: bucketName,
|
|
Namespace: namespaceName,
|
|
TableName: tableName,
|
|
IdentityActions: identityActions,
|
|
}) {
|
|
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to get table policy")
|
|
return NewAuthError("GetTablePolicy", principal, "not authorized to get table policy")
|
|
}
|
|
|
|
resp := &GetTablePolicyResponse{
|
|
ResourcePolicy: string(policy),
|
|
}
|
|
|
|
h.writeJSON(w, http.StatusOK, resp)
|
|
return nil
|
|
}
|
|
|
|
// handleDeleteTablePolicy deletes the policy of a table
|
|
func (h *S3TablesHandler) handleDeleteTablePolicy(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
|
|
|
|
var req DeleteTablePolicyRequest
|
|
if err := h.readRequestBody(r, &req); err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.TableBucketARN == "" || len(req.Namespace) == 0 || req.Name == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "tableBucketARN, namespace, and name are required")
|
|
return fmt.Errorf("missing required parameters")
|
|
}
|
|
|
|
namespaceName, err := validateNamespace(req.Namespace)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
bucketName, err := parseBucketNameFromARN(req.TableBucketARN)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
tableName, err := validateTableName(req.Name)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
tablePath := GetTablePath(bucketName, namespaceName, tableName)
|
|
bucketPath := GetTableBucketPath(bucketName)
|
|
|
|
// Check if table exists
|
|
var metadata tableMetadataInternal
|
|
var bucketPolicy string
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
data, err := h.getExtendedAttribute(r.Context(), client, tablePath, ExtendedKeyMetadata)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := json.Unmarshal(data, &metadata); err != nil {
|
|
return fmt.Errorf("failed to unmarshal table metadata: %w", err)
|
|
}
|
|
|
|
// Fetch bucket policy if it exists
|
|
policyData, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy)
|
|
if err != nil {
|
|
if !errors.Is(err, ErrAttributeNotFound) {
|
|
return fmt.Errorf("failed to read bucket policy: %w", err)
|
|
}
|
|
// Policy not found is not an error; bucketPolicy remains empty
|
|
} else {
|
|
bucketPolicy = string(policyData)
|
|
}
|
|
|
|
return nil
|
|
})
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchTable, fmt.Sprintf("table %s not found", tableName))
|
|
} else {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, fmt.Sprintf("failed to check table: %v", err))
|
|
}
|
|
return err
|
|
}
|
|
|
|
tableARN := h.generateTableARN(metadata.OwnerAccountID, bucketName, namespaceName+"/"+tableName)
|
|
principal := h.getAccountID(r)
|
|
identityActions := getIdentityActions(r)
|
|
if !CheckPermissionWithContext("DeleteTablePolicy", principal, metadata.OwnerAccountID, bucketPolicy, tableARN, &PolicyContext{
|
|
TableBucketName: bucketName,
|
|
Namespace: namespaceName,
|
|
TableName: tableName,
|
|
IdentityActions: identityActions,
|
|
}) {
|
|
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to delete table policy")
|
|
return NewAuthError("DeleteTablePolicy", principal, "not authorized to delete table policy")
|
|
}
|
|
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
return h.deleteExtendedAttribute(r.Context(), client, tablePath, ExtendedKeyPolicy)
|
|
})
|
|
|
|
if err != nil && !errors.Is(err, ErrAttributeNotFound) {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, "failed to delete table policy")
|
|
return err
|
|
}
|
|
|
|
h.writeJSON(w, http.StatusOK, nil)
|
|
return nil
|
|
}
|
|
|
|
// handleTagResource adds tags to a resource
|
|
func (h *S3TablesHandler) handleTagResource(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
|
|
var req TagResourceRequest
|
|
if err := h.readRequestBody(r, &req); err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.ResourceARN == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "resourceArn is required")
|
|
return fmt.Errorf("resourceArn is required")
|
|
}
|
|
|
|
if len(req.Tags) == 0 {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "tags are required")
|
|
return fmt.Errorf("tags are required")
|
|
}
|
|
|
|
// Parse resource ARN to determine if it's a bucket or table
|
|
resourcePath, extendedKey, rType, err := h.resolveResourcePath(req.ResourceARN)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
// Read existing tags and merge, AND check permissions based on metadata ownership
|
|
existingTags := make(map[string]string)
|
|
var bucketPolicy string
|
|
var bucketTags map[string]string
|
|
requestTagKeys := mapKeys(req.Tags)
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
// Read metadata for ownership check
|
|
data, err := h.getExtendedAttribute(r.Context(), client, resourcePath, ExtendedKeyMetadata)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
ownerAccountID, bucketName, err := h.extractResourceOwnerAndBucket(data, resourcePath, rType)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// Fetch bucket policy if we have a bucket name
|
|
if bucketName != "" {
|
|
bucketPath := GetTableBucketPath(bucketName)
|
|
policyData, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy)
|
|
if err != nil {
|
|
if !errors.Is(err, ErrAttributeNotFound) {
|
|
return fmt.Errorf("failed to read bucket policy: %w", err)
|
|
}
|
|
// Policy not found is not an error; bucketPolicy remains empty
|
|
} else {
|
|
bucketPolicy = string(policyData)
|
|
}
|
|
bucketTags, err = h.readTags(r.Context(), client, bucketPath)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
}
|
|
|
|
// Read existing tags
|
|
data, err = h.getExtendedAttribute(r.Context(), client, resourcePath, extendedKey)
|
|
if err != nil {
|
|
if !errors.Is(err, ErrAttributeNotFound) {
|
|
return err
|
|
}
|
|
} else if err := json.Unmarshal(data, &existingTags); err != nil {
|
|
return err
|
|
}
|
|
|
|
resourceARN := req.ResourceARN
|
|
principal := h.getAccountID(r)
|
|
identityActions := getIdentityActions(r)
|
|
if !CheckPermissionWithContext("TagResource", principal, ownerAccountID, bucketPolicy, resourceARN, &PolicyContext{
|
|
TableBucketName: bucketName,
|
|
TableBucketTags: bucketTags,
|
|
RequestTags: req.Tags,
|
|
TagKeys: requestTagKeys,
|
|
ResourceTags: existingTags,
|
|
IdentityActions: identityActions,
|
|
}) {
|
|
return NewAuthError("TagResource", principal, "not authorized to tag resource")
|
|
}
|
|
return nil
|
|
})
|
|
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
errorCode := ErrCodeNoSuchBucket
|
|
if rType == ResourceTypeTable {
|
|
errorCode = ErrCodeNoSuchTable
|
|
}
|
|
h.writeError(w, http.StatusNotFound, errorCode, "resource not found")
|
|
} else if isAuthError(err) {
|
|
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, err.Error())
|
|
} else {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, fmt.Sprintf("failed to read existing tags: %v", err))
|
|
}
|
|
return err
|
|
}
|
|
|
|
// Merge new tags
|
|
for k, v := range req.Tags {
|
|
existingTags[k] = v
|
|
}
|
|
|
|
// Write merged tags
|
|
tagsBytes, err := json.Marshal(existingTags)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, "failed to marshal tags")
|
|
return fmt.Errorf("failed to marshal tags: %w", err)
|
|
}
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
return h.setExtendedAttribute(r.Context(), client, resourcePath, extendedKey, tagsBytes)
|
|
})
|
|
|
|
if err != nil {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, "failed to tag resource")
|
|
return err
|
|
}
|
|
|
|
h.writeJSON(w, http.StatusOK, nil)
|
|
return nil
|
|
}
|
|
|
|
// handleListTagsForResource lists tags for a resource
|
|
func (h *S3TablesHandler) handleListTagsForResource(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
|
|
|
|
var req ListTagsForResourceRequest
|
|
if err := h.readRequestBody(r, &req); err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.ResourceARN == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "resourceArn is required")
|
|
return fmt.Errorf("resourceArn is required")
|
|
}
|
|
|
|
resourcePath, extendedKey, rType, err := h.resolveResourcePath(req.ResourceARN)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
tags := make(map[string]string)
|
|
var bucketPolicy string
|
|
var bucketTags map[string]string
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
// Read metadata for ownership check
|
|
data, err := h.getExtendedAttribute(r.Context(), client, resourcePath, ExtendedKeyMetadata)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
ownerAccountID, bucketName, err := h.extractResourceOwnerAndBucket(data, resourcePath, rType)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// Fetch bucket policy if we have a bucket name
|
|
if bucketName != "" {
|
|
bucketPath := GetTableBucketPath(bucketName)
|
|
policyData, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy)
|
|
if err != nil {
|
|
if !errors.Is(err, ErrAttributeNotFound) {
|
|
return fmt.Errorf("failed to read bucket policy: %w", err)
|
|
}
|
|
// Policy not found is not an error; bucketPolicy remains empty
|
|
} else {
|
|
bucketPolicy = string(policyData)
|
|
}
|
|
bucketTags, err = h.readTags(r.Context(), client, bucketPath)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
}
|
|
|
|
data, err = h.getExtendedAttribute(r.Context(), client, resourcePath, extendedKey)
|
|
if err != nil {
|
|
if errors.Is(err, ErrAttributeNotFound) {
|
|
return nil // No tags is not an error.
|
|
}
|
|
return err // Propagate other errors.
|
|
}
|
|
if err := json.Unmarshal(data, &tags); err != nil {
|
|
return err
|
|
}
|
|
|
|
resourceARN := req.ResourceARN
|
|
principal := h.getAccountID(r)
|
|
identityActions := getIdentityActions(r)
|
|
if !CheckPermissionWithContext("ListTagsForResource", principal, ownerAccountID, bucketPolicy, resourceARN, &PolicyContext{
|
|
TableBucketName: bucketName,
|
|
TableBucketTags: bucketTags,
|
|
ResourceTags: tags,
|
|
IdentityActions: identityActions,
|
|
}) {
|
|
return NewAuthError("ListTagsForResource", principal, "not authorized to list tags for resource")
|
|
}
|
|
return nil
|
|
})
|
|
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
errorCode := ErrCodeNoSuchBucket
|
|
if rType == ResourceTypeTable {
|
|
errorCode = ErrCodeNoSuchTable
|
|
}
|
|
h.writeError(w, http.StatusNotFound, errorCode, "resource not found")
|
|
} else if isAuthError(err) {
|
|
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, err.Error())
|
|
} else {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, fmt.Sprintf("failed to list tags: %v", err))
|
|
}
|
|
return err
|
|
}
|
|
|
|
resp := &ListTagsForResourceResponse{
|
|
Tags: tags,
|
|
}
|
|
|
|
h.writeJSON(w, http.StatusOK, resp)
|
|
return nil
|
|
}
|
|
|
|
// handleUntagResource removes tags from a resource
|
|
func (h *S3TablesHandler) handleUntagResource(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
|
|
var req UntagResourceRequest
|
|
if err := h.readRequestBody(r, &req); err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
if req.ResourceARN == "" {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "resourceArn is required")
|
|
return fmt.Errorf("resourceArn is required")
|
|
}
|
|
|
|
if len(req.TagKeys) == 0 {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "tagKeys are required")
|
|
return fmt.Errorf("tagKeys are required")
|
|
}
|
|
|
|
resourcePath, extendedKey, rType, err := h.resolveResourcePath(req.ResourceARN)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, err.Error())
|
|
return err
|
|
}
|
|
|
|
// Read existing tags, check permission
|
|
tags := make(map[string]string)
|
|
var bucketPolicy string
|
|
var bucketTags map[string]string
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
// Read metadata for ownership check
|
|
data, err := h.getExtendedAttribute(r.Context(), client, resourcePath, ExtendedKeyMetadata)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
ownerAccountID, bucketName, err := h.extractResourceOwnerAndBucket(data, resourcePath, rType)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// Fetch bucket policy if we have a bucket name
|
|
if bucketName != "" {
|
|
bucketPath := GetTableBucketPath(bucketName)
|
|
policyData, err := h.getExtendedAttribute(r.Context(), client, bucketPath, ExtendedKeyPolicy)
|
|
if err != nil {
|
|
if !errors.Is(err, ErrAttributeNotFound) {
|
|
return fmt.Errorf("failed to read bucket policy: %w", err)
|
|
}
|
|
// Policy not found is not an error; bucketPolicy remains empty
|
|
} else {
|
|
bucketPolicy = string(policyData)
|
|
}
|
|
bucketTags, err = h.readTags(r.Context(), client, bucketPath)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
}
|
|
|
|
data, err = h.getExtendedAttribute(r.Context(), client, resourcePath, extendedKey)
|
|
if err != nil {
|
|
if errors.Is(err, ErrAttributeNotFound) {
|
|
return nil
|
|
}
|
|
return err
|
|
}
|
|
if err := json.Unmarshal(data, &tags); err != nil {
|
|
return err
|
|
}
|
|
|
|
resourceARN := req.ResourceARN
|
|
principal := h.getAccountID(r)
|
|
identityActions := getIdentityActions(r)
|
|
if !CheckPermissionWithContext("UntagResource", principal, ownerAccountID, bucketPolicy, resourceARN, &PolicyContext{
|
|
TableBucketName: bucketName,
|
|
TableBucketTags: bucketTags,
|
|
TagKeys: req.TagKeys,
|
|
ResourceTags: tags,
|
|
IdentityActions: identityActions,
|
|
}) {
|
|
return NewAuthError("UntagResource", principal, "not authorized to untag resource")
|
|
}
|
|
return nil
|
|
})
|
|
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
errorCode := ErrCodeNoSuchBucket
|
|
if rType == ResourceTypeTable {
|
|
errorCode = ErrCodeNoSuchTable
|
|
}
|
|
h.writeError(w, http.StatusNotFound, errorCode, "resource not found")
|
|
} else if isAuthError(err) {
|
|
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, err.Error())
|
|
} else {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, "failed to read existing tags")
|
|
}
|
|
return err
|
|
}
|
|
|
|
// Remove specified tags
|
|
for _, key := range req.TagKeys {
|
|
delete(tags, key)
|
|
}
|
|
|
|
// Write updated tags
|
|
tagsBytes, err := json.Marshal(tags)
|
|
if err != nil {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, "failed to marshal tags")
|
|
return fmt.Errorf("failed to marshal tags: %w", err)
|
|
}
|
|
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
return h.setExtendedAttribute(r.Context(), client, resourcePath, extendedKey, tagsBytes)
|
|
})
|
|
|
|
if err != nil {
|
|
h.writeError(w, http.StatusInternalServerError, ErrCodeInternalError, "failed to untag resource")
|
|
return err
|
|
}
|
|
|
|
h.writeJSON(w, http.StatusOK, nil)
|
|
return nil
|
|
}
|
|
|
|
// resolveResourcePath determines the resource path and extended attribute key from a resource ARN
|
|
func (h *S3TablesHandler) resolveResourcePath(resourceARN string) (path string, key string, rType ResourceType, err error) {
|
|
// Try parsing as table ARN first
|
|
bucketName, namespace, tableName, err := parseTableFromARN(resourceARN)
|
|
if err == nil {
|
|
return GetTablePath(bucketName, namespace, tableName), ExtendedKeyTags, ResourceTypeTable, nil
|
|
}
|
|
|
|
// Try parsing as bucket ARN
|
|
bucketName, err = parseBucketNameFromARN(resourceARN)
|
|
if err == nil {
|
|
return GetTableBucketPath(bucketName), ExtendedKeyTags, ResourceTypeBucket, nil
|
|
}
|
|
|
|
return "", "", "", fmt.Errorf("invalid resource ARN: %s", resourceARN)
|
|
}
|