* Support multiple filers for S3 and IAM servers with automatic failover
This change adds support for multiple filer addresses in the 'weed s3' and 'weed iam' commands, enabling high availability through automatic failover.
Key changes:
- Updated S3ApiServerOption.Filer to Filers ([]pb.ServerAddress)
- Updated IamServerOption.Filer to Filers ([]pb.ServerAddress)
- Modified -filer flag to accept comma-separated addresses
- Added getFilerAddress() helper methods for backward compatibility
- Updated all filer client calls to support multiple addresses
- Uses pb.WithOneOfGrpcFilerClients for automatic failover
Usage:
weed s3 -filer=localhost:8888,localhost:8889
weed iam -filer=localhost:8888,localhost:8889
The underlying FilerClient already supported multiple filers with health
tracking and automatic failover - this change exposes that capability
through the command-line interface.
* Add filer discovery: treat initial filers as seeds and discover peers from master
Enhances FilerClient to automatically discover additional filers in the same
filer group by querying the master server. This allows users to specify just
a few seed filers, and the client will discover all other filers in the cluster.
Key changes to wdclient/FilerClient:
- Added MasterClient, FilerGroup, and DiscoveryInterval fields
- Added thread-safe filer list management with RWMutex
- Implemented discoverFilers() background goroutine
- Uses cluster.ListExistingPeerUpdates() to query master for filers
- Automatically adds newly discovered filers to the list
- Added Close() method to clean up discovery goroutine
New FilerClientOption fields:
- MasterClient: enables filer discovery from master
- FilerGroup: specifies which filer group to discover
- DiscoveryInterval: how often to refresh (default 5 minutes)
Usage example:
masterClient := wdclient.NewMasterClient(...)
filerClient := wdclient.NewFilerClient(
[]pb.ServerAddress{"localhost:8888"}, // seed filers
grpcDialOption,
dataCenter,
&wdclient.FilerClientOption{
MasterClient: masterClient,
FilerGroup: "my-group",
},
)
defer filerClient.Close()
The initial filers act as seeds - the client discovers and adds all other
filers in the same group from the master. Discovered filers are added
dynamically without removing existing ones (relying on health checks for
unavailable filers).
* Address PR review comments: implement full failover for IAM operations
Critical fixes based on code review feedback:
1. **IAM API Failover (Critical)**:
- Replace pb.WithGrpcFilerClient with pb.WithOneOfGrpcFilerClients in:
* GetS3ApiConfigurationFromFiler()
* PutS3ApiConfigurationToFiler()
* GetPolicies()
* PutPolicies()
- Now all IAM operations support automatic failover across multiple filers
2. **Validation Improvements**:
- Add validation in NewIamApiServerWithStore() to require at least one filer
- Add validation in NewS3ApiServerWithStore() to require at least one filer
- Add warning log when no filers configured for credential store
3. **Error Logging**:
- Circuit breaker now logs when config load fails instead of silently ignoring
- Helps operators understand why circuit breaker limits aren't applied
4. **Code Quality**:
- Use ToGrpcAddress() for filer address in credential store setup
- More consistent with rest of codebase and future-proof
These changes ensure IAM operations have the same high availability guarantees
as S3 operations, completing the multi-filer failover implementation.
* Fix IAM manager initialization: remove code duplication, add TODO for HA
Addresses review comment on s3api_server.go:145
Changes:
- Remove duplicate code for getting first filer address
- Extract filerAddr variable once and reuse
- Add TODO comment documenting the HA limitation for IAM manager
- Document that loadIAMManagerFromConfig and NewS3IAMIntegration need
updates to support multiple filers for full HA
Note: This is a known limitation when using filer-backed IAM stores.
The interfaces need to be updated to accept multiple filer addresses.
For now, documenting this limitation clearly.
* Document credential store HA limitation with TODO
Addresses review comment on auth_credentials.go:149
Changes:
- Add TODO comment documenting that SetFilerClient interface needs update
for multi-filer support
- Add informative log message indicating HA limitation
- Document that this is a known limitation for filer-backed credential stores
The SetFilerClient interface currently only accepts a single filer address.
To properly support HA, the credential store interfaces need to be updated
to handle multiple filer addresses.
* Track current active filer in FilerClient for better HA
Add GetCurrentFiler() method to FilerClient that returns the currently
active filer based on the filerIndex which is updated on successful
operations. This provides better availability than always using the
first filer.
Changes:
- Add FilerClient.GetCurrentFiler() method that returns current active filer
- Update S3ApiServer.getFilerAddress() to use FilerClient's current filer
- Add fallback to first filer if FilerClient not yet initialized
- Document IAM limitation (doesn't have FilerClient access)
Benefits:
- Single-filer operations (URLs, ReadFilerConf, etc.) now use the
currently active/healthy filer
- Better distribution and failover behavior
- FilerClient's round-robin and health tracking automatically
determines which filer to use
* Document ReadFilerConf HA limitation in lifecycle handlers
Addresses review comment on s3api_bucket_handlers.go:880
Add comment documenting that ReadFilerConf uses the current active filer
from FilerClient (which is better than always using first filer), but
doesn't have built-in multi-filer failover.
Add TODO to update filer.ReadFilerConf to support multiple filers for
complete HA. For now, it uses the currently active/healthy filer tracked
by FilerClient which provides reasonable availability.
* Document multipart upload URL HA limitation
Addresses review comment on s3api_object_handlers_multipart.go:442
Add comment documenting that part upload URLs point to the current
active filer (tracked by FilerClient), which is better than always
using the first filer but still creates a potential point of failure
if that filer becomes unavailable during upload.
Suggest TODO solutions:
- Use virtual hostname/load balancer for filers
- Have S3 server proxy uploads to healthy filers
Current behavior provides reasonable availability by using the
currently active/healthy filer rather than being pinned to first filer.
* Document multipart completion Location URL limitation
Addresses review comment on filer_multipart.go:187
Add comment documenting that the Location URL in CompleteMultipartUpload
response points to the current active filer (tracked by FilerClient).
Note that clients should ideally use the S3 API endpoint rather than
this direct URL. If direct access is attempted and the specific filer
is unavailable, the request will fail.
Current behavior uses the currently active/healthy filer rather than
being pinned to the first filer, providing better availability.
* Make credential store use current active filer for HA
Update FilerEtcStore to use a function that returns the current active
filer instead of a fixed address, enabling high availability.
Changes:
- Add SetFilerAddressFunc() method to FilerEtcStore
- Store uses filerAddressFunc instead of fixed filerGrpcAddress
- withFilerClient() calls the function to get current active filer
- Keep SetFilerClient() for backward compatibility (marked deprecated)
- Update S3ApiServer to pass FilerClient.GetCurrentFiler to store
Benefits:
- Credential store now uses currently active/healthy filer
- Automatic failover when filer becomes unavailable
- True HA for credential operations
- Backward compatible with old SetFilerClient interface
This addresses the credential store limitation - no longer pinned to
first filer, uses FilerClient's tracked current active filer.
* Clarify multipart URL comments: filer address not used for uploads
Update comments to reflect that multipart upload URLs are not actually
used for upload traffic - uploads go directly to volume servers.
Key clarifications:
- genPartUploadUrl: Filer address is parsed out, only path is used
- CompleteMultipartUpload Location: Informational field per AWS S3 spec
- Actual uploads bypass filer proxy and go directly to volume servers
The filer address in these URLs is NOT a HA concern because:
1. Part uploads: URL is parsed for path, upload goes to volume servers
2. Location URL: Informational only, clients use S3 endpoint
This addresses the observation that S3 uploads don't go through filers,
only metadata operations do.
* Remove filer address from upload paths - pass path directly
Eliminate unnecessary filer address from upload URLs by passing file
paths directly instead of full URLs that get immediately parsed.
Changes:
- Rename genPartUploadUrl() → genPartUploadPath() (returns path only)
- Rename toFilerUrl() → toFilerPath() (returns path only)
- Update putToFiler() to accept filePath instead of uploadUrl
- Remove URL parsing code (no longer needed)
- Remove net/url import (no longer used)
- Keep old function names as deprecated wrappers for compatibility
Benefits:
- Cleaner code - no fake URL construction/parsing
- No dependency on filer address for internal operations
- More accurate naming (these are paths, not URLs)
- Eliminates confusion about HA concerns
This completely removes the filer address from upload operations - it was
never actually used for routing, only parsed for the path.
* Remove deprecated functions: use new path-based functions directly
Remove deprecated wrapper functions and update all callers to use the
new function names directly.
Removed:
- genPartUploadUrl() → all callers now use genPartUploadPath()
- toFilerUrl() → all callers now use toFilerPath()
- SetFilerClient() → removed along with fallback code
Updated:
- s3api_object_handlers_multipart.go: uploadUrl → filePath
- s3api_object_handlers_put.go: uploadUrl → filePath, versionUploadUrl → versionFilePath
- s3api_object_versioning.go: toFilerUrl → toFilerPath
- s3api_object_handlers_test.go: toFilerUrl → toFilerPath
- auth_credentials.go: removed SetFilerClient fallback
- filer_etc_store.go: removed deprecated SetFilerClient method
Benefits:
- Cleaner codebase with no deprecated functions
- All variable names accurately reflect that they're paths, not URLs
- Single interface for credential stores (SetFilerAddressFunc only)
All code now consistently uses the new path-based approach.
* Fix toFilerPath: remove URL escaping for raw file paths
The toFilerPath function should return raw file paths, not URL-escaped
paths. URL escaping was needed when the path was embedded in a URL
(old toFilerUrl), but now that we pass paths directly to putToFiler,
they should be unescaped.
This fixes S3 integration test failures:
- test_bucket_listv2_encoding_basic
- test_bucket_list_encoding_basic
- test_bucket_listv2_delimiter_whitespace
- test_bucket_list_delimiter_whitespace
The tests were failing because paths were double-encoded (escaped when
stored, then escaped again when listed), resulting in %252B instead of
%2B for '+' characters.
Root cause: When we removed URL parsing in putToFiler, we should have
also removed URL escaping in toFilerPath since paths are now used
directly without URL encoding/decoding.
* Add thread safety to FilerEtcStore and clarify credential store comments
Address review suggestions for better thread safety and code clarity:
1. **Thread Safety**: Add RWMutex to FilerEtcStore
- Protects filerAddressFunc and grpcDialOption from concurrent access
- Initialize() uses write lock when setting function
- SetFilerAddressFunc() uses write lock
- withFilerClient() uses read lock to get function and dial option
- GetPolicies() uses read lock to check if configured
2. **Improved Error Messages**:
- Prefix errors with "filer_etc:" for easier debugging
- "filer address not configured" → "filer_etc: filer address function not configured"
- "filer address is empty" → "filer_etc: filer address is empty"
3. **Clarified Comments**:
- auth_credentials.go: Clarify that initial setup is temporary
- Document that it's updated in s3api_server.go after FilerClient creation
- Remove ambiguity about when FilerClient.GetCurrentFiler is used
Benefits:
- Safe for concurrent credential operations
- Clear error messages for debugging
- Explicit documentation of initialization order
* Enable filer discovery: pass master addresses to FilerClient
Fix two critical issues:
1. **Filer Discovery Not Working**: Master client was not being passed to
FilerClient, so peer discovery couldn't work
2. **Credential Store Design**: Already uses FilerClient via GetCurrentFiler
function - this is the correct design for HA
Changes:
**Command (s3.go):**
- Read master addresses from GetFilerConfiguration response
- Pass masterAddresses to S3ApiServerOption
- Log master addresses for visibility
**S3ApiServerOption:**
- Add Masters []pb.ServerAddress field for discovery
**S3ApiServer:**
- Create MasterClient from Masters when available
- Pass MasterClient + FilerGroup to FilerClient via options
- Enable discovery with 5-minute refresh interval
- Log whether discovery is enabled or disabled
**Credential Store:**
- Already correctly uses filerClient.GetCurrentFiler via function
- This provides HA without tight coupling to FilerClient struct
- Function-based design is clean and thread-safe
Discovery Flow:
1. S3 command reads filer config → gets masters + filer group
2. S3ApiServer creates MasterClient from masters
3. FilerClient uses MasterClient to query for peer filers
4. Background goroutine refreshes peer list every 5 minutes
5. Credential store uses GetCurrentFiler to get active filer
Now filer discovery actually works! ��
* Use S3 endpoint in multipart Location instead of filer address
* Add multi-filer failover to ReadFilerConf
* Address CodeRabbit review: fix buffer reuse and improve lock safety
Address two code review suggestions:
1. **Fix buffer reuse in ReadFilerConfFromFilers**:
- Use local []byte data instead of shared buffer
- Prevents partial data from failed attempts affecting successful reads
- Creates fresh buffer inside callback for masterClient path
- More robust to future changes in read helpers
2. **Improve lock safety in FilerClient**:
- Add *WithHealth variants that accept health pointer
- Get health pointer while holding lock, then release before calling
- Eliminates potential for lock confusion (though no actual deadlock existed)
- Clearer separation: lock for data access, atomics for health ops
Changes:
- ReadFilerConfFromFilers: var data []byte, create buf inside callback
- shouldSkipUnhealthyFilerWithHealth(health *filerHealth)
- recordFilerSuccessWithHealth(health *filerHealth)
- recordFilerFailureWithHealth(health *filerHealth)
- Keep old functions for backward compatibility (marked deprecated)
- Update LookupVolumeIds to use WithHealth variants
Benefits:
- More robust multi-filer configuration reading
- Clearer lock vs atomic operation boundaries
- No lock held during health checks (even though atomics don't block)
- Better code organization and maintainability
* add constant
* Fix IAM manager and post policy to use current active filer
* Fix critical race condition and goroutine leak
* Update weed/s3api/filer_multipart.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Fix compilation error and address code review suggestions
Address remaining unresolved comments:
1. **Fix compilation error**: Add missing net/url import
- filer_multipart.go used url.PathEscape without import
- Added "net/url" to imports
2. **Fix Location URL formatting** (all 4 occurrences):
- Add missing slash between bucket and key
- Use url.PathEscape for bucket names
- Use urlPathEscape for object keys
- Handles special characters in bucket/key names
- Before: http://host/bucketkey
- After: http://host/bucket/key (properly escaped)
3. **Optimize discovery loop** (O(N*M) → O(N+M)):
- Use map for existing filers (O(1) lookup)
- Reduces time holding write lock
- Better performance with many filers
- Before: Nested loop for each discovered filer
- After: Build map once, then O(1) lookups
Changes:
- filer_multipart.go: Import net/url, fix all Location URLs
- filer_client.go: Use map for efficient filer discovery
Benefits:
- Compiles successfully
- Proper URL encoding (handles spaces, special chars)
- Faster discovery with less lock contention
- Production-ready URL formatting
* Fix race conditions and make Close() idempotent
Address CodeRabbit review #3512078995:
1. **Critical: Fix unsynchronized read in error message**
- Line 584 read len(fc.filerAddresses) without lock
- Race with refreshFilerList appending to slice
- Fixed: Take RLock to read length safely
- Prevents race detector warnings
2. **Important: Make Close() idempotent**
- Closing already-closed channel panics
- Can happen with layered cleanup in shutdown paths
- Fixed: Use sync.Once to ensure single close
- Safe to call Close() multiple times now
3. **Nitpick: Add warning for empty filer address**
- getFilerAddress() can return empty string
- Helps diagnose unexpected state
- Added: Warning log when no filers available
4. **Nitpick: Guard deprecated index-based helpers**
- shouldSkipUnhealthyFiler, recordFilerSuccess/Failure
- Accessed filerHealth without lock (races with discovery)
- Fixed: Take RLock and check bounds before array access
- Prevents index out of bounds and races
Changes:
- filer_client.go:
- Add closeDiscoveryOnce sync.Once field
- Use Do() in Close() for idempotent channel close
- Add RLock guards to deprecated index-based helpers
- Add bounds checking to prevent panics
- Synchronized read of filerAddresses length in error
- s3api_server.go:
- Add warning log when getFilerAddress returns empty
Benefits:
- No race conditions (passes race detector)
- No panic on double-close
- Better error diagnostics
- Safe with discovery enabled
- Production-hardened shutdown logic
* Fix hardcoded http scheme and add panic recovery
Address CodeRabbit review #3512114811:
1. **Major: Fix hardcoded http:// scheme in Location URLs**
- Location URLs always used http:// regardless of client connection
- HTTPS clients got http:// URLs (incorrect)
- Fixed: Detect scheme from request
- Check X-Forwarded-Proto header (for proxies) first
- Check r.TLS != nil for direct HTTPS
- Fallback to http for plain connections
- Applied to all 4 CompleteMultipartUploadResult locations
2. **Major: Add panic recovery to discovery goroutine**
- Long-running background goroutine could crash entire process
- Panic in refreshFilerList would terminate program
- Fixed: Add defer recover() with error logging
- Goroutine failures now logged, not fatal
3. **Note: Close() idempotency already implemented**
- Review flagged as duplicate issue
- Already fixed in commit 3d7a65c7e
- sync.Once (closeDiscoveryOnce) prevents double-close panic
- Safe to call Close() multiple times
Changes:
- filer_multipart.go:
- Add getRequestScheme() helper function
- Update all 4 Location URLs to use dynamic scheme
- Format: scheme://host/bucket/key (was: http://...)
- filer_client.go:
- Add panic recovery to discoverFilers()
- Log panics instead of crashing
Benefits:
- Correct scheme (https/http) in Location URLs
- Works behind proxies (X-Forwarded-Proto)
- No process crashes from discovery failures
- Production-hardened background goroutine
- Proper AWS S3 API compliance
* Fix S3 WithFilerClient to use filer failover
Critical fix for multi-filer deployments:
**Problem:**
- S3ApiServer.WithFilerClient() was creating direct connections to ONE filer
- Used pb.WithGrpcClient() with single filer address
- No failover - if that filer failed, ALL operations failed
- Caused test failures: "bucket directory not found"
- IAM Integration Tests failing with 500 Internal Error
**Root Cause:**
- WithFilerClient bypassed filerClient connection management
- Always connected to getFilerAddress() (current filer only)
- Didn't retry other filers on failure
- All getEntry(), updateEntry(), etc. operations failed if current filer down
**Solution:**
1. Added FilerClient.GetAllFilers() method
- Returns snapshot of all filer addresses
- Thread-safe copy to avoid races
2. Implemented withFilerClientFailover()
- Try current filer first (fast path)
- On failure, try all other filers
- Log successful failover
- Return error only if ALL filers fail
3. Updated WithFilerClient()
- Use filerClient for failover when available
- Fallback to direct connection for testing/init
**Impact:**
✅ All S3 operations now support multi-filer failover
✅ Bucket metadata reads work with any available filer
✅ Entry operations (getEntry, updateEntry) failover automatically
✅ IAM tests should pass now
✅ Production-ready HA support
**Files Changed:**
- wdclient/filer_client.go: Add GetAllFilers() method
- s3api/s3api_handlers.go: Implement failover logic
This fixes the test failure where bucket operations failed when
the primary filer was temporarily unavailable during cleanup.
* Update current filer after successful failover
Address code review: https://github.com/seaweedfs/seaweedfs/pull/7550#pullrequestreview-3512223723
**Issue:**
After successful failover, the current filer index was not updated.
This meant every subsequent request would still try the (potentially
unhealthy) original filer first, then failover again.
**Solution:**
1. Added FilerClient.SetCurrentFiler(addr) method:
- Finds the index of specified filer address
- Atomically updates filerIndex to point to it
- Thread-safe with RLock
2. Call SetCurrentFiler after successful failover:
- Update happens immediately after successful connection
- Future requests start with the known-healthy filer
- Reduces unnecessary failover attempts
**Benefits:**
✅ Subsequent requests use healthy filer directly
✅ No repeated failover to same unhealthy filer
✅ Better performance - fast path hits healthy filer
✅ Comment now matches actual behavior
* Integrate health tracking with S3 failover
Address code review suggestion to leverage existing health tracking
instead of simple iteration through all filers.
**Changes:**
1. Added address-based health tracking API to FilerClient:
- ShouldSkipUnhealthyFiler(addr) - check circuit breaker
- RecordFilerSuccess(addr) - reset failure count
- RecordFilerFailure(addr) - increment failure count
These methods find the filer by address and delegate to
existing *WithHealth methods for actual health management.
2. Updated withFilerClientFailover to use health tracking:
- Record success/failure for every filer attempt
- Skip unhealthy filers during failover (circuit breaker)
- Only try filers that haven't exceeded failure threshold
- Automatic re-check after reset timeout
**Benefits:**
✅ Circuit breaker prevents wasting time on known-bad filers
✅ Health tracking shared across all operations
✅ Automatic recovery when unhealthy filers come back
✅ Reduced latency - skip filers in failure state
✅ Better visibility with health metrics
**Behavior:**
- Try current filer first (fast path)
- If fails, record failure and try other HEALTHY filers
- Skip filers with failureCount >= threshold (default 3)
- Re-check unhealthy filers after resetTimeout (default 30s)
- Record all successes/failures for health tracking
* Update weed/wdclient/filer_client.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Enable filer discovery with empty filerGroup
Empty filerGroup is a valid value representing the default group.
The master client can discover filers even when filerGroup is empty.
**Change:**
- Remove the filerGroup != "" check in NewFilerClient
- Keep only masterClient != nil check
- Empty string will be passed to ListClusterNodes API as-is
This enables filer discovery to work with the default group.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1316 lines
45 KiB
Go
1316 lines
45 KiB
Go
package s3api
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"encoding/json"
|
|
"encoding/xml"
|
|
"errors"
|
|
"fmt"
|
|
"math"
|
|
"net/http"
|
|
"path"
|
|
"sort"
|
|
"strconv"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/util"
|
|
|
|
"github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3bucket"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/filer"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
|
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
|
|
|
"github.com/aws/aws-sdk-go/aws"
|
|
"github.com/aws/aws-sdk-go/service/s3"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/glog"
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
util_http "github.com/seaweedfs/seaweedfs/weed/util/http"
|
|
)
|
|
|
|
func (s3a *S3ApiServer) ListBucketsHandler(w http.ResponseWriter, r *http.Request) {
|
|
|
|
glog.V(3).Infof("ListBucketsHandler")
|
|
|
|
var identity *Identity
|
|
var s3Err s3err.ErrorCode
|
|
if s3a.iam.isEnabled() {
|
|
// Use authRequest instead of authUser for consistency with other endpoints
|
|
// This ensures the same authentication flow and any fixes (like prefix handling) are applied
|
|
identity, s3Err = s3a.iam.authRequest(r, s3_constants.ACTION_LIST)
|
|
if s3Err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, s3Err)
|
|
return
|
|
}
|
|
}
|
|
|
|
var response ListAllMyBucketsResult
|
|
|
|
entries, _, err := s3a.list(s3a.option.BucketsPath, "", "", false, math.MaxInt32)
|
|
|
|
if err != nil {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
|
|
// Get authenticated identity from context (secure, cannot be spoofed)
|
|
// For unauthenticated requests, this returns empty string
|
|
identityId := s3_constants.GetIdentityNameFromContext(r)
|
|
|
|
var listBuckets ListAllMyBucketsList
|
|
for _, entry := range entries {
|
|
if entry.IsDirectory {
|
|
// Check ownership: only show buckets owned by this user (unless admin)
|
|
if !isBucketVisibleToIdentity(entry, identity) {
|
|
continue
|
|
}
|
|
|
|
// Check permissions for each bucket
|
|
if identity != nil {
|
|
// For JWT-authenticated users, use IAM authorization
|
|
sessionToken := r.Header.Get("X-SeaweedFS-Session-Token")
|
|
if s3a.iam.iamIntegration != nil && sessionToken != "" {
|
|
// Use IAM authorization for JWT users
|
|
errCode := s3a.iam.authorizeWithIAM(r, identity, s3_constants.ACTION_LIST, entry.Name, "")
|
|
if errCode != s3err.ErrNone {
|
|
continue
|
|
}
|
|
} else {
|
|
// Use legacy authorization for non-JWT users
|
|
if !identity.canDo(s3_constants.ACTION_LIST, entry.Name, "") {
|
|
continue
|
|
}
|
|
}
|
|
}
|
|
listBuckets.Bucket = append(listBuckets.Bucket, ListAllMyBucketsEntry{
|
|
Name: entry.Name,
|
|
CreationDate: time.Unix(entry.Attributes.Crtime, 0).UTC(),
|
|
})
|
|
}
|
|
}
|
|
|
|
response = ListAllMyBucketsResult{
|
|
Owner: CanonicalUser{
|
|
ID: identityId,
|
|
DisplayName: identityId,
|
|
},
|
|
Buckets: listBuckets,
|
|
}
|
|
|
|
writeSuccessResponseXML(w, r, response)
|
|
}
|
|
|
|
// isBucketVisibleToIdentity checks if a bucket entry should be visible to the given identity
|
|
// based on ownership rules. Returns true if the bucket should be visible, false otherwise.
|
|
//
|
|
// Visibility rules:
|
|
// - Unauthenticated requests (identity == nil): no buckets visible
|
|
// - Admin users: all buckets visible
|
|
// - Non-admin users: only buckets they own (matching identity.Name) are visible
|
|
// - Buckets without owner metadata are hidden from non-admin users
|
|
func isBucketVisibleToIdentity(entry *filer_pb.Entry, identity *Identity) bool {
|
|
if !entry.IsDirectory {
|
|
return false
|
|
}
|
|
|
|
// Unauthenticated users should not see any buckets (standard S3 behavior)
|
|
if identity == nil {
|
|
return false
|
|
}
|
|
|
|
// Admin users bypass ownership check
|
|
if identity.isAdmin() {
|
|
return true
|
|
}
|
|
|
|
// Non-admin users with no name cannot own or see buckets.
|
|
// This prevents misconfigured identities from matching buckets with empty owner IDs.
|
|
if identity.Name == "" {
|
|
return false
|
|
}
|
|
|
|
// Non-admin users: check ownership
|
|
// Use the authenticated identity value directly (cannot be spoofed)
|
|
id, ok := entry.Extended[s3_constants.AmzIdentityId]
|
|
// Skip buckets that are not owned by the current user.
|
|
// Buckets without an owner are also skipped.
|
|
if !ok || string(id) != identity.Name {
|
|
return false
|
|
}
|
|
|
|
return true
|
|
}
|
|
|
|
func (s3a *S3ApiServer) PutBucketHandler(w http.ResponseWriter, r *http.Request) {
|
|
|
|
// collect parameters
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
|
|
// validate the bucket name
|
|
err := s3bucket.VerifyS3BucketName(bucket)
|
|
if err != nil {
|
|
glog.Errorf("put invalid bucket name: %v %v", bucket, err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidBucketName)
|
|
return
|
|
}
|
|
|
|
// Check if bucket already exists and handle ownership/settings
|
|
// Get authenticated identity from context (secure, cannot be spoofed)
|
|
currentIdentityId := s3_constants.GetIdentityNameFromContext(r)
|
|
|
|
// Check collection existence first
|
|
collectionExists := false
|
|
if err := s3a.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
if resp, err := client.CollectionList(context.Background(), &filer_pb.CollectionListRequest{
|
|
IncludeEcVolumes: true,
|
|
IncludeNormalVolumes: true,
|
|
}); err != nil {
|
|
glog.Errorf("list collection: %v", err)
|
|
return fmt.Errorf("list collections: %w", err)
|
|
} else {
|
|
for _, c := range resp.Collections {
|
|
if s3a.getCollectionName(bucket) == c.Name {
|
|
collectionExists = true
|
|
break
|
|
}
|
|
}
|
|
}
|
|
return nil
|
|
}); err != nil {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
|
|
// Check bucket directory existence and get metadata
|
|
if exist, err := s3a.exists(s3a.option.BucketsPath, bucket, true); err == nil && exist {
|
|
// Bucket exists, check ownership and settings
|
|
if entry, err := s3a.getEntry(s3a.option.BucketsPath, bucket); err == nil {
|
|
// Get existing bucket owner
|
|
var existingOwnerId string
|
|
if entry.Extended != nil {
|
|
if id, ok := entry.Extended[s3_constants.AmzIdentityId]; ok {
|
|
existingOwnerId = string(id)
|
|
}
|
|
}
|
|
|
|
// Check ownership
|
|
if existingOwnerId != "" && existingOwnerId != currentIdentityId {
|
|
// Different owner - always fail with BucketAlreadyExists
|
|
glog.V(3).Infof("PutBucketHandler: bucket %s owned by %s, requested by %s", bucket, existingOwnerId, currentIdentityId)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrBucketAlreadyExists)
|
|
return
|
|
}
|
|
|
|
// Same owner or no owner set - check for conflicting settings
|
|
objectLockRequested := strings.EqualFold(r.Header.Get(s3_constants.AmzBucketObjectLockEnabled), "true")
|
|
|
|
// Get current bucket configuration
|
|
bucketConfig, errCode := s3a.getBucketConfig(bucket)
|
|
if errCode != s3err.ErrNone {
|
|
glog.Errorf("PutBucketHandler: failed to get bucket config for %s: %v", bucket, errCode)
|
|
// If we can't get config, assume no conflict and allow recreation
|
|
} else {
|
|
// Check for Object Lock conflict
|
|
currentObjectLockEnabled := bucketConfig.ObjectLockConfig != nil &&
|
|
bucketConfig.ObjectLockConfig.ObjectLockEnabled == s3_constants.ObjectLockEnabled
|
|
|
|
if objectLockRequested != currentObjectLockEnabled {
|
|
// Conflicting Object Lock settings - fail with BucketAlreadyExists
|
|
glog.V(3).Infof("PutBucketHandler: bucket %s has conflicting Object Lock settings (requested: %v, current: %v)",
|
|
bucket, objectLockRequested, currentObjectLockEnabled)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrBucketAlreadyExists)
|
|
return
|
|
}
|
|
}
|
|
|
|
// Bucket already exists - always return BucketAlreadyExists per S3 specification
|
|
// The S3 tests expect BucketAlreadyExists in all cases, not BucketAlreadyOwnedByYou
|
|
glog.V(3).Infof("PutBucketHandler: bucket %s already exists", bucket)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrBucketAlreadyExists)
|
|
return
|
|
}
|
|
}
|
|
|
|
// If collection exists but bucket directory doesn't, this is an inconsistent state
|
|
if collectionExists {
|
|
glog.Errorf("PutBucketHandler: collection exists but bucket directory missing for %s", bucket)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrBucketAlreadyExists)
|
|
return
|
|
}
|
|
|
|
// create the folder for bucket, but lazily create actual collection
|
|
if err := s3a.mkdir(s3a.option.BucketsPath, bucket, setBucketOwner(r)); err != nil {
|
|
glog.Errorf("PutBucketHandler mkdir: %v", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
|
|
// Remove bucket from negative cache after successful creation
|
|
if s3a.bucketConfigCache != nil {
|
|
s3a.bucketConfigCache.RemoveNegativeCache(bucket)
|
|
}
|
|
|
|
// Check for x-amz-bucket-object-lock-enabled header (S3 standard compliance)
|
|
if objectLockHeaderValue := r.Header.Get(s3_constants.AmzBucketObjectLockEnabled); strings.EqualFold(objectLockHeaderValue, "true") {
|
|
glog.V(3).Infof("PutBucketHandler: enabling Object Lock and Versioning for bucket %s due to x-amz-bucket-object-lock-enabled header", bucket)
|
|
|
|
// Atomically update the configuration of the specified bucket. See the updateBucketConfig
|
|
// function definition for detailed documentation on parameters and behavior.
|
|
errCode := s3a.updateBucketConfig(bucket, func(bucketConfig *BucketConfig) error {
|
|
// Enable versioning (required for Object Lock)
|
|
bucketConfig.Versioning = s3_constants.VersioningEnabled
|
|
|
|
// Create basic Object Lock configuration (enabled without default retention)
|
|
objectLockConfig := &ObjectLockConfiguration{
|
|
ObjectLockEnabled: s3_constants.ObjectLockEnabled,
|
|
}
|
|
|
|
// Set the cached Object Lock configuration
|
|
bucketConfig.ObjectLockConfig = objectLockConfig
|
|
glog.V(3).Infof("PutBucketHandler: set ObjectLockConfig for bucket %s: %+v", bucket, objectLockConfig)
|
|
|
|
return nil
|
|
})
|
|
|
|
if errCode != s3err.ErrNone {
|
|
glog.Errorf("PutBucketHandler: failed to enable Object Lock for bucket %s: %v", bucket, errCode)
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
glog.V(3).Infof("PutBucketHandler: enabled Object Lock and Versioning for bucket %s", bucket)
|
|
}
|
|
|
|
w.Header().Set("Location", "/"+bucket)
|
|
writeSuccessResponseEmpty(w, r)
|
|
}
|
|
|
|
func (s3a *S3ApiServer) DeleteBucketHandler(w http.ResponseWriter, r *http.Request) {
|
|
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("DeleteBucketHandler %s", bucket)
|
|
|
|
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, err)
|
|
return
|
|
}
|
|
|
|
// Check if bucket has object lock enabled
|
|
bucketConfig, errCode := s3a.getBucketConfig(bucket)
|
|
if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
|
|
// If object lock is enabled, check for objects with active locks
|
|
if bucketConfig.ObjectLockConfig != nil {
|
|
hasLockedObjects, checkErr := s3a.hasObjectsWithActiveLocks(bucket)
|
|
if checkErr != nil {
|
|
glog.Errorf("DeleteBucketHandler: failed to check for locked objects in bucket %s: %v", bucket, checkErr)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
if hasLockedObjects {
|
|
glog.V(3).Infof("DeleteBucketHandler: bucket %s has objects with active object locks, cannot delete", bucket)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrBucketNotEmpty)
|
|
return
|
|
}
|
|
}
|
|
|
|
err := s3a.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
if !s3a.option.AllowDeleteBucketNotEmpty {
|
|
entries, _, err := s3a.list(s3a.option.BucketsPath+"/"+bucket, "", "", false, 2)
|
|
if err != nil {
|
|
return fmt.Errorf("failed to list bucket %s: %v", bucket, err)
|
|
}
|
|
for _, entry := range entries {
|
|
// Allow bucket deletion if only special directories remain
|
|
if entry.Name != s3_constants.MultipartUploadsFolder &&
|
|
!strings.HasSuffix(entry.Name, s3_constants.VersionsFolder) {
|
|
return errors.New(s3err.GetAPIError(s3err.ErrBucketNotEmpty).Code)
|
|
}
|
|
}
|
|
}
|
|
|
|
// delete collection
|
|
deleteCollectionRequest := &filer_pb.DeleteCollectionRequest{
|
|
Collection: s3a.getCollectionName(bucket),
|
|
}
|
|
|
|
glog.V(1).Infof("delete collection: %v", deleteCollectionRequest)
|
|
if _, err := client.DeleteCollection(context.Background(), deleteCollectionRequest); err != nil {
|
|
return fmt.Errorf("delete collection %s: %v", bucket, err)
|
|
}
|
|
|
|
return nil
|
|
})
|
|
|
|
if err != nil {
|
|
s3ErrorCode := s3err.ErrInternalError
|
|
if err.Error() == s3err.GetAPIError(s3err.ErrBucketNotEmpty).Code {
|
|
s3ErrorCode = s3err.ErrBucketNotEmpty
|
|
}
|
|
s3err.WriteErrorResponse(w, r, s3ErrorCode)
|
|
return
|
|
}
|
|
|
|
err = s3a.rm(s3a.option.BucketsPath, bucket, false, true)
|
|
|
|
if err != nil {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
|
|
// Clean up bucket-related caches and locks after successful deletion
|
|
s3a.invalidateBucketConfigCache(bucket)
|
|
|
|
s3err.WriteEmptyResponse(w, r, http.StatusNoContent)
|
|
}
|
|
|
|
// hasObjectsWithActiveLocks checks if any objects in the bucket have active retention or legal hold
|
|
func (s3a *S3ApiServer) hasObjectsWithActiveLocks(bucket string) (bool, error) {
|
|
bucketPath := s3a.option.BucketsPath + "/" + bucket
|
|
|
|
// Check all objects including versions for active locks
|
|
// Establish current time once at the start for consistency across the entire scan
|
|
hasLocks := false
|
|
currentTime := time.Now()
|
|
err := s3a.recursivelyCheckLocks(bucketPath, "", &hasLocks, currentTime)
|
|
if err != nil {
|
|
return false, fmt.Errorf("error checking for locked objects: %w", err)
|
|
}
|
|
|
|
return hasLocks, nil
|
|
}
|
|
|
|
const (
|
|
// lockCheckPaginationSize is the page size for listing directories during lock checks
|
|
lockCheckPaginationSize = 10000
|
|
)
|
|
|
|
// errStopPagination is a sentinel error to signal early termination of pagination
|
|
var errStopPagination = errors.New("stop pagination")
|
|
|
|
// paginateEntries iterates through directory entries with pagination
|
|
// Calls fn for each page of entries. If fn returns errStopPagination, iteration stops successfully.
|
|
func (s3a *S3ApiServer) paginateEntries(dir string, fn func(entries []*filer_pb.Entry) error) error {
|
|
startFrom := ""
|
|
for {
|
|
entries, isLast, err := s3a.list(dir, "", startFrom, false, lockCheckPaginationSize)
|
|
if err != nil {
|
|
// Fail-safe: propagate error to prevent incorrect bucket deletion
|
|
return fmt.Errorf("failed to list directory %s: %w", dir, err)
|
|
}
|
|
|
|
if err := fn(entries); err != nil {
|
|
if errors.Is(err, errStopPagination) {
|
|
return nil
|
|
}
|
|
return err
|
|
}
|
|
|
|
if isLast || len(entries) == 0 {
|
|
break
|
|
}
|
|
// Use the last entry name as the start point for next page
|
|
startFrom = entries[len(entries)-1].Name
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// recursivelyCheckLocks recursively checks all objects and versions for active locks
|
|
// Uses pagination to handle directories with more than 10,000 entries
|
|
func (s3a *S3ApiServer) recursivelyCheckLocks(dir string, relativePath string, hasLocks *bool, currentTime time.Time) error {
|
|
if *hasLocks {
|
|
// Early exit if we've already found a locked object
|
|
return nil
|
|
}
|
|
|
|
// Process entries in the current directory with pagination
|
|
err := s3a.paginateEntries(dir, func(entries []*filer_pb.Entry) error {
|
|
for _, entry := range entries {
|
|
if *hasLocks {
|
|
// Early exit if we've already found a locked object
|
|
return errStopPagination
|
|
}
|
|
|
|
// Skip special directories (multipart uploads, etc)
|
|
if entry.Name == s3_constants.MultipartUploadsFolder {
|
|
continue
|
|
}
|
|
|
|
if entry.IsDirectory {
|
|
subDir := path.Join(dir, entry.Name)
|
|
if strings.HasSuffix(entry.Name, s3_constants.VersionsFolder) {
|
|
// If it's a .versions directory, check all version files with pagination
|
|
err := s3a.paginateEntries(subDir, func(versionEntries []*filer_pb.Entry) error {
|
|
for _, versionEntry := range versionEntries {
|
|
if s3a.entryHasActiveLock(versionEntry, currentTime) {
|
|
*hasLocks = true
|
|
glog.V(2).Infof("Found object with active lock in versions: %s/%s", subDir, versionEntry.Name)
|
|
return errStopPagination
|
|
}
|
|
}
|
|
return nil
|
|
})
|
|
if err != nil {
|
|
return err
|
|
}
|
|
} else {
|
|
// Recursively check other subdirectories
|
|
subRelativePath := path.Join(relativePath, entry.Name)
|
|
if err := s3a.recursivelyCheckLocks(subDir, subRelativePath, hasLocks, currentTime); err != nil {
|
|
return err
|
|
}
|
|
// Early exit if a locked object was found in the subdirectory
|
|
if *hasLocks {
|
|
return errStopPagination
|
|
}
|
|
}
|
|
} else {
|
|
// Check regular files for locks
|
|
if s3a.entryHasActiveLock(entry, currentTime) {
|
|
*hasLocks = true
|
|
objectPath := path.Join(relativePath, entry.Name)
|
|
glog.V(2).Infof("Found object with active lock: %s", objectPath)
|
|
return errStopPagination
|
|
}
|
|
}
|
|
}
|
|
return nil
|
|
})
|
|
|
|
return err
|
|
}
|
|
|
|
// entryHasActiveLock checks if an entry has an active retention or legal hold
|
|
func (s3a *S3ApiServer) entryHasActiveLock(entry *filer_pb.Entry, currentTime time.Time) bool {
|
|
if entry.Extended == nil {
|
|
return false
|
|
}
|
|
|
|
// Check for active legal hold
|
|
if legalHoldBytes, exists := entry.Extended[s3_constants.ExtLegalHoldKey]; exists {
|
|
if string(legalHoldBytes) == s3_constants.LegalHoldOn {
|
|
return true
|
|
}
|
|
}
|
|
|
|
// Check for active retention
|
|
if modeBytes, exists := entry.Extended[s3_constants.ExtObjectLockModeKey]; exists {
|
|
mode := string(modeBytes)
|
|
if mode == s3_constants.RetentionModeCompliance || mode == s3_constants.RetentionModeGovernance {
|
|
// Check if retention is still active
|
|
if dateBytes, dateExists := entry.Extended[s3_constants.ExtRetentionUntilDateKey]; dateExists {
|
|
timestamp, err := strconv.ParseInt(string(dateBytes), 10, 64)
|
|
if err != nil {
|
|
// Fail-safe: if we can't parse the retention date, assume the object is locked
|
|
// to prevent accidental data loss
|
|
glog.Warningf("Failed to parse retention date '%s' for entry, assuming locked: %v", string(dateBytes), err)
|
|
return true
|
|
}
|
|
retainUntil := time.Unix(timestamp, 0)
|
|
if retainUntil.After(currentTime) {
|
|
return true
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
return false
|
|
}
|
|
|
|
func (s3a *S3ApiServer) HeadBucketHandler(w http.ResponseWriter, r *http.Request) {
|
|
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("HeadBucketHandler %s", bucket)
|
|
|
|
if entry, err := s3a.getEntry(s3a.option.BucketsPath, bucket); entry == nil || errors.Is(err, filer_pb.ErrNotFound) {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket)
|
|
return
|
|
}
|
|
|
|
writeSuccessResponseEmpty(w, r)
|
|
}
|
|
|
|
func (s3a *S3ApiServer) checkBucket(r *http.Request, bucket string) s3err.ErrorCode {
|
|
// Use cached bucket config instead of direct getEntry call (optimization)
|
|
config, errCode := s3a.getBucketConfig(bucket)
|
|
if errCode != s3err.ErrNone {
|
|
return errCode
|
|
}
|
|
|
|
//if iam is enabled, the access was already checked before
|
|
if s3a.iam.isEnabled() {
|
|
return s3err.ErrNone
|
|
}
|
|
if !s3a.hasAccess(r, config.Entry) {
|
|
return s3err.ErrAccessDenied
|
|
}
|
|
return s3err.ErrNone
|
|
}
|
|
|
|
// ErrAutoCreatePermissionDenied is returned when a user lacks permission to auto-create buckets
|
|
var ErrAutoCreatePermissionDenied = errors.New("permission denied - requires Admin permission")
|
|
|
|
// ErrInvalidBucketName is returned when a bucket name doesn't meet S3 naming requirements
|
|
var ErrInvalidBucketName = errors.New("invalid bucket name")
|
|
|
|
// setBucketOwner creates a function that sets the bucket owner from the request context
|
|
func setBucketOwner(r *http.Request) func(entry *filer_pb.Entry) {
|
|
currentIdentityId := s3_constants.GetIdentityNameFromContext(r)
|
|
return func(entry *filer_pb.Entry) {
|
|
if currentIdentityId != "" {
|
|
if entry.Extended == nil {
|
|
entry.Extended = make(map[string][]byte)
|
|
}
|
|
entry.Extended[s3_constants.AmzIdentityId] = []byte(currentIdentityId)
|
|
}
|
|
}
|
|
}
|
|
|
|
// autoCreateBucket creates a bucket if it doesn't exist, setting the owner from the request context
|
|
// Only users with admin permissions are allowed to auto-create buckets
|
|
func (s3a *S3ApiServer) autoCreateBucket(r *http.Request, bucket string) error {
|
|
// Validate the bucket name before auto-creating
|
|
if err := s3bucket.VerifyS3BucketName(bucket); err != nil {
|
|
return fmt.Errorf("auto-create bucket %s: %w", bucket, errors.Join(ErrInvalidBucketName, err))
|
|
}
|
|
|
|
// Check if user has admin permissions
|
|
if !s3a.isUserAdmin(r) {
|
|
return fmt.Errorf("auto-create bucket %s: %w", bucket, ErrAutoCreatePermissionDenied)
|
|
}
|
|
|
|
if err := s3a.mkdir(s3a.option.BucketsPath, bucket, setBucketOwner(r)); err != nil {
|
|
// In case of a race condition where another request created the bucket
|
|
// in the meantime, check for existence before returning an error.
|
|
if exist, err2 := s3a.exists(s3a.option.BucketsPath, bucket, true); err2 != nil {
|
|
glog.Warningf("autoCreateBucket: failed to check existence for bucket %s: %v", bucket, err2)
|
|
return fmt.Errorf("failed to auto-create bucket %s: %w", bucket, errors.Join(err, err2))
|
|
} else if exist {
|
|
// The bucket exists, which is fine. However, we should ensure it has an owner.
|
|
// If it was created by a concurrent request that didn't set an owner,
|
|
// we'll set it here to ensure consistency.
|
|
if entry, getErr := s3a.getEntry(s3a.option.BucketsPath, bucket); getErr == nil {
|
|
if entry.Extended == nil || len(entry.Extended[s3_constants.AmzIdentityId]) == 0 {
|
|
// No owner set, assign current admin as owner
|
|
setBucketOwner(r)(entry)
|
|
if updateErr := s3a.updateEntry(s3a.option.BucketsPath, entry); updateErr != nil {
|
|
glog.Warningf("autoCreateBucket: failed to set owner for existing bucket %s: %v", bucket, updateErr)
|
|
} else {
|
|
glog.V(1).Infof("Set owner for existing bucket %s (created by concurrent request)", bucket)
|
|
}
|
|
}
|
|
} else {
|
|
glog.Warningf("autoCreateBucket: failed to get entry for existing bucket %s: %v", bucket, getErr)
|
|
}
|
|
return nil
|
|
}
|
|
return fmt.Errorf("failed to auto-create bucket %s: %w", bucket, err)
|
|
}
|
|
|
|
// Remove bucket from negative cache after successful creation
|
|
if s3a.bucketConfigCache != nil {
|
|
s3a.bucketConfigCache.RemoveNegativeCache(bucket)
|
|
}
|
|
|
|
glog.V(1).Infof("Auto-created bucket %s", bucket)
|
|
return nil
|
|
}
|
|
|
|
// handleAutoCreateBucket attempts to auto-create a bucket and writes appropriate error responses
|
|
// Returns true if the bucket was created successfully or already exists, false if an error was written
|
|
func (s3a *S3ApiServer) handleAutoCreateBucket(w http.ResponseWriter, r *http.Request, bucket, handlerName string) bool {
|
|
if err := s3a.autoCreateBucket(r, bucket); err != nil {
|
|
glog.Warningf("%s: %v", handlerName, err)
|
|
// Check for specific errors to return appropriate S3 error codes
|
|
if errors.Is(err, ErrInvalidBucketName) {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidBucketName)
|
|
} else if errors.Is(err, ErrAutoCreatePermissionDenied) {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrAccessDenied)
|
|
} else {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
}
|
|
return false
|
|
}
|
|
return true
|
|
}
|
|
|
|
func (s3a *S3ApiServer) hasAccess(r *http.Request, entry *filer_pb.Entry) bool {
|
|
// Check if user is properly authenticated as admin through IAM system
|
|
if s3a.isUserAdmin(r) {
|
|
return true
|
|
}
|
|
|
|
if entry.Extended == nil {
|
|
return true
|
|
}
|
|
|
|
// Get authenticated identity from context (secure, cannot be spoofed)
|
|
identityId := s3_constants.GetIdentityNameFromContext(r)
|
|
if id, ok := entry.Extended[s3_constants.AmzIdentityId]; ok {
|
|
if identityId != string(id) {
|
|
glog.V(3).Infof("hasAccess: %s != %s (entry.Extended = %v)", identityId, id, entry.Extended)
|
|
return false
|
|
}
|
|
}
|
|
return true
|
|
}
|
|
|
|
// isUserAdmin securely checks if the authenticated user is an admin
|
|
// This validates admin status through proper IAM authentication, not spoofable headers
|
|
func (s3a *S3ApiServer) isUserAdmin(r *http.Request) bool {
|
|
// Use a minimal admin action to authenticate and check admin status
|
|
adminAction := Action("Admin")
|
|
identity, errCode := s3a.iam.authRequest(r, adminAction)
|
|
if errCode != s3err.ErrNone {
|
|
return false
|
|
}
|
|
|
|
// Check if the authenticated identity has admin privileges
|
|
return identity != nil && identity.isAdmin()
|
|
}
|
|
|
|
// isBucketPublicRead checks if a bucket allows anonymous read access based on its cached ACL status
|
|
func (s3a *S3ApiServer) isBucketPublicRead(bucket string) bool {
|
|
// Get bucket configuration which contains cached public-read status
|
|
config, errCode := s3a.getBucketConfig(bucket)
|
|
if errCode != s3err.ErrNone {
|
|
glog.V(4).Infof("isBucketPublicRead: failed to get bucket config for %s: %v", bucket, errCode)
|
|
return false
|
|
}
|
|
|
|
glog.V(4).Infof("isBucketPublicRead: bucket=%s, IsPublicRead=%v", bucket, config.IsPublicRead)
|
|
// Return the cached public-read status (no JSON parsing needed)
|
|
return config.IsPublicRead
|
|
}
|
|
|
|
// isPublicReadGrants checks if the grants allow public read access
|
|
func isPublicReadGrants(grants []*s3.Grant) bool {
|
|
for _, grant := range grants {
|
|
if grant.Grantee != nil && grant.Grantee.URI != nil && grant.Permission != nil {
|
|
// Check for AllUsers group with Read permission
|
|
if *grant.Grantee.URI == s3_constants.GranteeGroupAllUsers &&
|
|
(*grant.Permission == s3_constants.PermissionRead || *grant.Permission == s3_constants.PermissionFullControl) {
|
|
return true
|
|
}
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// buildResourceARN builds a resource ARN from bucket and object
|
|
// Used by the policy engine wrapper
|
|
func buildResourceARN(bucket, object string) string {
|
|
if object == "" || object == "/" {
|
|
return fmt.Sprintf("arn:aws:s3:::%s", bucket)
|
|
}
|
|
// Remove leading slash if present
|
|
object = strings.TrimPrefix(object, "/")
|
|
return fmt.Sprintf("arn:aws:s3:::%s/%s", bucket, object)
|
|
}
|
|
|
|
// AuthWithPublicRead creates an auth wrapper that allows anonymous access for public-read buckets
|
|
func (s3a *S3ApiServer) AuthWithPublicRead(handler http.HandlerFunc, action Action) http.HandlerFunc {
|
|
return func(w http.ResponseWriter, r *http.Request) {
|
|
bucket, object := s3_constants.GetBucketAndObject(r)
|
|
authType := getRequestAuthType(r)
|
|
isAnonymous := authType == authTypeAnonymous
|
|
|
|
glog.V(4).Infof("AuthWithPublicRead: bucket=%s, object=%s, authType=%v, isAnonymous=%v", bucket, object, authType, isAnonymous)
|
|
|
|
// For anonymous requests, check if bucket allows public read via ACLs or bucket policies
|
|
if isAnonymous {
|
|
// First check ACL-based public access
|
|
isPublic := s3a.isBucketPublicRead(bucket)
|
|
glog.V(4).Infof("AuthWithPublicRead: bucket=%s, isPublicACL=%v", bucket, isPublic)
|
|
if isPublic {
|
|
glog.V(3).Infof("AuthWithPublicRead: allowing anonymous access to public-read bucket %s (ACL)", bucket)
|
|
handler(w, r)
|
|
return
|
|
}
|
|
|
|
// Check bucket policy for anonymous access using the policy engine
|
|
principal := "*" // Anonymous principal
|
|
// Use context-aware policy evaluation to get the correct S3 action
|
|
allowed, evaluated, err := s3a.policyEngine.EvaluatePolicyWithContext(bucket, object, string(action), principal, r)
|
|
if err != nil {
|
|
// SECURITY: Fail-close on policy evaluation errors
|
|
// If we can't evaluate the policy, deny access rather than falling through to IAM
|
|
glog.Errorf("AuthWithPublicRead: error evaluating bucket policy for %s/%s: %v - denying access", bucket, object, err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrAccessDenied)
|
|
return
|
|
} else if evaluated {
|
|
// A bucket policy exists and was evaluated with a matching statement
|
|
if allowed {
|
|
// Policy explicitly allows anonymous access
|
|
glog.V(3).Infof("AuthWithPublicRead: allowing anonymous access to bucket %s (bucket policy)", bucket)
|
|
handler(w, r)
|
|
return
|
|
} else {
|
|
// Policy explicitly denies anonymous access
|
|
glog.V(3).Infof("AuthWithPublicRead: bucket policy explicitly denies anonymous access to %s/%s", bucket, object)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrAccessDenied)
|
|
return
|
|
}
|
|
}
|
|
// No matching policy statement - fall through to check ACLs and then IAM auth
|
|
glog.V(3).Infof("AuthWithPublicRead: no bucket policy match for %s, checking ACLs", bucket)
|
|
}
|
|
|
|
// For all authenticated requests and anonymous requests to non-public buckets,
|
|
// use normal IAM auth to enforce policies
|
|
s3a.iam.Auth(handler, action)(w, r)
|
|
}
|
|
}
|
|
|
|
// GetBucketAclHandler Get Bucket ACL
|
|
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAcl.html
|
|
func (s3a *S3ApiServer) GetBucketAclHandler(w http.ResponseWriter, r *http.Request) {
|
|
// collect parameters
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("GetBucketAclHandler %s", bucket)
|
|
|
|
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, err)
|
|
return
|
|
}
|
|
|
|
amzAccountId := r.Header.Get(s3_constants.AmzAccountId)
|
|
amzDisplayName := s3a.iam.GetAccountNameById(amzAccountId)
|
|
response := AccessControlPolicy{
|
|
Owner: CanonicalUser{
|
|
ID: amzAccountId,
|
|
DisplayName: amzDisplayName,
|
|
},
|
|
}
|
|
response.AccessControlList.Grant = append(response.AccessControlList.Grant, Grant{
|
|
Grantee: Grantee{
|
|
ID: amzAccountId,
|
|
DisplayName: amzDisplayName,
|
|
Type: "CanonicalUser",
|
|
XMLXSI: "CanonicalUser",
|
|
XMLNS: "http://www.w3.org/2001/XMLSchema-instance"},
|
|
Permission: s3.PermissionFullControl,
|
|
})
|
|
writeSuccessResponseXML(w, r, response)
|
|
}
|
|
|
|
// PutBucketAclHandler Put bucket ACL
|
|
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketAcl.html //
|
|
func (s3a *S3ApiServer) PutBucketAclHandler(w http.ResponseWriter, r *http.Request) {
|
|
// collect parameters
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("PutBucketAclHandler %s", bucket)
|
|
|
|
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, err)
|
|
return
|
|
}
|
|
|
|
// Get account information for ACL processing
|
|
amzAccountId := r.Header.Get(s3_constants.AmzAccountId)
|
|
|
|
// Get bucket ownership settings (these would be used for ownership validation in a full implementation)
|
|
bucketOwnership := "" // Default/simplified for now - in a full implementation this would be retrieved from bucket config
|
|
bucketOwnerId := amzAccountId // Simplified - bucket owner is current account
|
|
|
|
// Use the existing ACL parsing logic to handle both canned ACLs and XML body
|
|
grants, errCode := ExtractAcl(r, s3a.iam, bucketOwnership, bucketOwnerId, amzAccountId, amzAccountId)
|
|
if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
|
|
glog.V(3).Infof("PutBucketAclHandler: bucket=%s, extracted %d grants", bucket, len(grants))
|
|
isPublic := isPublicReadGrants(grants)
|
|
glog.V(3).Infof("PutBucketAclHandler: bucket=%s, isPublicReadGrants=%v", bucket, isPublic)
|
|
|
|
// Store the bucket ACL in bucket metadata
|
|
errCode = s3a.updateBucketConfig(bucket, func(config *BucketConfig) error {
|
|
if len(grants) > 0 {
|
|
grantsBytes, err := json.Marshal(grants)
|
|
if err != nil {
|
|
glog.Errorf("PutBucketAclHandler: failed to marshal grants: %v", err)
|
|
return err
|
|
}
|
|
config.ACL = grantsBytes
|
|
// Cache the public-read status to avoid JSON parsing on every request
|
|
config.IsPublicRead = isPublicReadGrants(grants)
|
|
glog.V(4).Infof("PutBucketAclHandler: bucket=%s, setting IsPublicRead=%v", bucket, config.IsPublicRead)
|
|
} else {
|
|
config.ACL = nil
|
|
config.IsPublicRead = false
|
|
}
|
|
config.Owner = amzAccountId
|
|
return nil
|
|
})
|
|
|
|
if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
|
|
glog.V(3).Infof("PutBucketAclHandler: Successfully stored ACL for bucket %s with %d grants", bucket, len(grants))
|
|
|
|
// Small delay to ensure ACL propagation across distributed caches
|
|
// This prevents race conditions in tests where anonymous access is attempted immediately after ACL change
|
|
time.Sleep(50 * time.Millisecond)
|
|
|
|
writeSuccessResponseEmpty(w, r)
|
|
}
|
|
|
|
// GetBucketLifecycleConfigurationHandler Get Bucket Lifecycle configuration
|
|
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLifecycleConfiguration.html
|
|
func (s3a *S3ApiServer) GetBucketLifecycleConfigurationHandler(w http.ResponseWriter, r *http.Request) {
|
|
// collect parameters
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("GetBucketLifecycleConfigurationHandler %s", bucket)
|
|
|
|
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, err)
|
|
return
|
|
}
|
|
// ReadFilerConfFromFilers provides multi-filer failover
|
|
fc, err := filer.ReadFilerConfFromFilers(s3a.option.Filers, s3a.option.GrpcDialOption, nil)
|
|
if err != nil {
|
|
glog.Errorf("GetBucketLifecycleConfigurationHandler: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
ttls := fc.GetCollectionTtls(s3a.getCollectionName(bucket))
|
|
if len(ttls) == 0 {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchLifecycleConfiguration)
|
|
return
|
|
}
|
|
|
|
response := Lifecycle{}
|
|
// Sort locationPrefixes to ensure consistent ordering of lifecycle rules
|
|
var locationPrefixes []string
|
|
for locationPrefix := range ttls {
|
|
locationPrefixes = append(locationPrefixes, locationPrefix)
|
|
}
|
|
sort.Strings(locationPrefixes)
|
|
|
|
for _, locationPrefix := range locationPrefixes {
|
|
internalTtl := ttls[locationPrefix]
|
|
ttl, _ := needle.ReadTTL(internalTtl)
|
|
days := int(ttl.Minutes() / 60 / 24)
|
|
if days == 0 {
|
|
continue
|
|
}
|
|
prefix, found := strings.CutPrefix(locationPrefix, fmt.Sprintf("%s/%s/", s3a.option.BucketsPath, bucket))
|
|
if !found {
|
|
continue
|
|
}
|
|
response.Rules = append(response.Rules, Rule{
|
|
ID: prefix,
|
|
Status: Enabled,
|
|
Prefix: Prefix{val: prefix, set: true},
|
|
Expiration: Expiration{Days: days, set: true},
|
|
})
|
|
}
|
|
|
|
writeSuccessResponseXML(w, r, response)
|
|
}
|
|
|
|
// PutBucketLifecycleConfigurationHandler Put Bucket Lifecycle configuration
|
|
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html
|
|
func (s3a *S3ApiServer) PutBucketLifecycleConfigurationHandler(w http.ResponseWriter, r *http.Request) {
|
|
// collect parameters
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("PutBucketLifecycleConfigurationHandler %s", bucket)
|
|
|
|
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, err)
|
|
return
|
|
}
|
|
|
|
lifeCycleConfig := Lifecycle{}
|
|
if err := xmlDecoder(r.Body, &lifeCycleConfig, r.ContentLength); err != nil {
|
|
glog.Warningf("PutBucketLifecycleConfigurationHandler xml decode: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrMalformedXML)
|
|
return
|
|
}
|
|
|
|
fc, err := filer.ReadFilerConfFromFilers(s3a.option.Filers, s3a.option.GrpcDialOption, nil)
|
|
if err != nil {
|
|
glog.Errorf("PutBucketLifecycleConfigurationHandler read filer config: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
collectionName := s3a.getCollectionName(bucket)
|
|
collectionTtls := fc.GetCollectionTtls(collectionName)
|
|
changed := false
|
|
|
|
for _, rule := range lifeCycleConfig.Rules {
|
|
if rule.Status != Enabled {
|
|
continue
|
|
}
|
|
var rulePrefix string
|
|
switch {
|
|
case rule.Filter.Prefix.set:
|
|
rulePrefix = rule.Filter.Prefix.val
|
|
case rule.Prefix.set:
|
|
rulePrefix = rule.Prefix.val
|
|
case !rule.Expiration.Date.IsZero() || rule.Transition.Days > 0 || !rule.Transition.Date.IsZero():
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrNotImplemented)
|
|
return
|
|
}
|
|
|
|
if rule.Expiration.Days == 0 {
|
|
continue
|
|
}
|
|
locationPrefix := fmt.Sprintf("%s/%s/%s", s3a.option.BucketsPath, bucket, rulePrefix)
|
|
locConf := &filer_pb.FilerConf_PathConf{
|
|
LocationPrefix: locationPrefix,
|
|
Collection: collectionName,
|
|
Ttl: fmt.Sprintf("%dd", rule.Expiration.Days),
|
|
}
|
|
if ttl, ok := collectionTtls[locConf.LocationPrefix]; ok && ttl == locConf.Ttl {
|
|
continue
|
|
}
|
|
if err := fc.AddLocationConf(locConf); err != nil {
|
|
glog.Errorf("PutBucketLifecycleConfigurationHandler add location config: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
ttlSec := int32((time.Duration(rule.Expiration.Days) * util.LifeCycleInterval).Seconds())
|
|
glog.V(2).Infof("Start updating TTL for %s", locationPrefix)
|
|
if updErr := s3a.updateEntriesTTL(locationPrefix, ttlSec); updErr != nil {
|
|
glog.Errorf("PutBucketLifecycleConfigurationHandler update TTL for %s: %s", locationPrefix, updErr)
|
|
} else {
|
|
glog.V(2).Infof("Finished updating TTL for %s", locationPrefix)
|
|
}
|
|
changed = true
|
|
}
|
|
|
|
if changed {
|
|
var buf bytes.Buffer
|
|
if err := fc.ToText(&buf); err != nil {
|
|
glog.Errorf("PutBucketLifecycleConfigurationHandler save config to text: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
}
|
|
if err := s3a.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
return filer.SaveInsideFiler(client, filer.DirectoryEtcSeaweedFS, filer.FilerConfName, buf.Bytes())
|
|
}); err != nil {
|
|
glog.Errorf("PutBucketLifecycleConfigurationHandler save config inside filer: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
}
|
|
|
|
writeSuccessResponseEmpty(w, r)
|
|
}
|
|
|
|
// DeleteBucketLifecycleHandler Delete Bucket Lifecycle
|
|
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketLifecycle.html
|
|
func (s3a *S3ApiServer) DeleteBucketLifecycleHandler(w http.ResponseWriter, r *http.Request) {
|
|
// collect parameters
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("DeleteBucketLifecycleHandler %s", bucket)
|
|
|
|
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, err)
|
|
return
|
|
}
|
|
|
|
fc, err := filer.ReadFilerConfFromFilers(s3a.option.Filers, s3a.option.GrpcDialOption, nil)
|
|
if err != nil {
|
|
glog.Errorf("DeleteBucketLifecycleHandler read filer config: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
collectionTtls := fc.GetCollectionTtls(s3a.getCollectionName(bucket))
|
|
changed := false
|
|
for prefix, ttl := range collectionTtls {
|
|
bucketPrefix := fmt.Sprintf("%s/%s/", s3a.option.BucketsPath, bucket)
|
|
if strings.HasPrefix(prefix, bucketPrefix) && strings.HasSuffix(ttl, "d") {
|
|
pathConf, found := fc.GetLocationConf(prefix)
|
|
if found {
|
|
pathConf.Ttl = ""
|
|
fc.SetLocationConf(pathConf)
|
|
}
|
|
changed = true
|
|
}
|
|
}
|
|
|
|
if changed {
|
|
var buf bytes.Buffer
|
|
if err := fc.ToText(&buf); err != nil {
|
|
glog.Errorf("DeleteBucketLifecycleHandler save config to text: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
}
|
|
if err := s3a.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
|
return filer.SaveInsideFiler(client, filer.DirectoryEtcSeaweedFS, filer.FilerConfName, buf.Bytes())
|
|
}); err != nil {
|
|
glog.Errorf("DeleteBucketLifecycleHandler save config inside filer: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
}
|
|
|
|
s3err.WriteEmptyResponse(w, r, http.StatusNoContent)
|
|
}
|
|
|
|
// GetBucketLocationHandler Get bucket location
|
|
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html
|
|
func (s3a *S3ApiServer) GetBucketLocationHandler(w http.ResponseWriter, r *http.Request) {
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
|
|
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, err)
|
|
return
|
|
}
|
|
|
|
writeSuccessResponseXML(w, r, CreateBucketConfiguration{})
|
|
}
|
|
|
|
// GetBucketRequestPaymentHandler Get bucket location
|
|
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketRequestPayment.html
|
|
func (s3a *S3ApiServer) GetBucketRequestPaymentHandler(w http.ResponseWriter, r *http.Request) {
|
|
writeSuccessResponseXML(w, r, RequestPaymentConfiguration{Payer: "BucketOwner"})
|
|
}
|
|
|
|
// PutBucketOwnershipControls https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketOwnershipControls.html
|
|
func (s3a *S3ApiServer) PutBucketOwnershipControls(w http.ResponseWriter, r *http.Request) {
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("PutBucketOwnershipControls %s", bucket)
|
|
|
|
errCode := s3a.checkAccessByOwnership(r, bucket)
|
|
if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
|
|
if r.Body == nil || r.Body == http.NoBody {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
|
|
return
|
|
}
|
|
|
|
var v s3.OwnershipControls
|
|
defer util_http.CloseRequest(r)
|
|
|
|
err := xmlutil.UnmarshalXML(&v, xml.NewDecoder(r.Body), "")
|
|
if err != nil {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
|
|
return
|
|
}
|
|
|
|
if len(v.Rules) != 1 {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
|
|
return
|
|
}
|
|
|
|
printOwnership := true
|
|
ownership := *v.Rules[0].ObjectOwnership
|
|
switch ownership {
|
|
case s3_constants.OwnershipObjectWriter:
|
|
case s3_constants.OwnershipBucketOwnerPreferred:
|
|
case s3_constants.OwnershipBucketOwnerEnforced:
|
|
printOwnership = false
|
|
default:
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
|
|
return
|
|
}
|
|
|
|
// Check if ownership needs to be updated
|
|
currentOwnership, errCode := s3a.getBucketOwnership(bucket)
|
|
if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
|
|
if currentOwnership != ownership {
|
|
errCode = s3a.setBucketOwnership(bucket, ownership)
|
|
if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
}
|
|
|
|
if printOwnership {
|
|
result := &s3.PutBucketOwnershipControlsInput{
|
|
OwnershipControls: &v,
|
|
}
|
|
s3err.WriteAwsXMLResponse(w, r, http.StatusOK, result)
|
|
} else {
|
|
writeSuccessResponseEmpty(w, r)
|
|
}
|
|
}
|
|
|
|
// GetBucketOwnershipControls https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketOwnershipControls.html
|
|
func (s3a *S3ApiServer) GetBucketOwnershipControls(w http.ResponseWriter, r *http.Request) {
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("GetBucketOwnershipControls %s", bucket)
|
|
|
|
errCode := s3a.checkAccessByOwnership(r, bucket)
|
|
if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
|
|
// Get ownership using new bucket config system
|
|
ownership, errCode := s3a.getBucketOwnership(bucket)
|
|
if errCode == s3err.ErrNoSuchBucket {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket)
|
|
return
|
|
} else if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, s3err.OwnershipControlsNotFoundError)
|
|
return
|
|
}
|
|
|
|
result := &s3.PutBucketOwnershipControlsInput{
|
|
OwnershipControls: &s3.OwnershipControls{
|
|
Rules: []*s3.OwnershipControlsRule{
|
|
{
|
|
ObjectOwnership: &ownership,
|
|
},
|
|
},
|
|
},
|
|
}
|
|
|
|
s3err.WriteAwsXMLResponse(w, r, http.StatusOK, result)
|
|
}
|
|
|
|
// DeleteBucketOwnershipControls https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketOwnershipControls.html
|
|
func (s3a *S3ApiServer) DeleteBucketOwnershipControls(w http.ResponseWriter, r *http.Request) {
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("PutBucketOwnershipControls %s", bucket)
|
|
|
|
errCode := s3a.checkAccessByOwnership(r, bucket)
|
|
if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
|
|
bucketEntry, err := s3a.getEntry(s3a.option.BucketsPath, bucket)
|
|
if err != nil {
|
|
if errors.Is(err, filer_pb.ErrNotFound) {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket)
|
|
return
|
|
}
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
|
|
_, ok := bucketEntry.Extended[s3_constants.ExtOwnershipKey]
|
|
if !ok {
|
|
s3err.WriteErrorResponse(w, r, s3err.OwnershipControlsNotFoundError)
|
|
return
|
|
}
|
|
|
|
delete(bucketEntry.Extended, s3_constants.ExtOwnershipKey)
|
|
err = s3a.updateEntry(s3a.option.BucketsPath, bucketEntry)
|
|
if err != nil {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
|
|
return
|
|
}
|
|
|
|
emptyOwnershipControls := &s3.OwnershipControls{
|
|
Rules: []*s3.OwnershipControlsRule{},
|
|
}
|
|
s3err.WriteAwsXMLResponse(w, r, http.StatusOK, emptyOwnershipControls)
|
|
}
|
|
|
|
// GetBucketVersioningHandler Get Bucket Versioning status
|
|
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketVersioning.html
|
|
func (s3a *S3ApiServer) GetBucketVersioningHandler(w http.ResponseWriter, r *http.Request) {
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("GetBucketVersioning %s", bucket)
|
|
|
|
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, err)
|
|
return
|
|
}
|
|
|
|
// Get versioning status using new bucket config system
|
|
versioningStatus, errCode := s3a.getBucketVersioningStatus(bucket)
|
|
if errCode != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
|
|
// AWS S3 behavior: If versioning was never configured, don't return Status field
|
|
var response *s3.PutBucketVersioningInput
|
|
if versioningStatus == "" {
|
|
// No versioning configuration - return empty response (no Status field)
|
|
response = &s3.PutBucketVersioningInput{
|
|
VersioningConfiguration: &s3.VersioningConfiguration{},
|
|
}
|
|
} else {
|
|
// Versioning was explicitly configured - return the status
|
|
response = &s3.PutBucketVersioningInput{
|
|
VersioningConfiguration: &s3.VersioningConfiguration{
|
|
Status: aws.String(versioningStatus),
|
|
},
|
|
}
|
|
}
|
|
s3err.WriteAwsXMLResponse(w, r, http.StatusOK, response)
|
|
}
|
|
|
|
// PutBucketVersioningHandler Put bucket Versioning
|
|
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketVersioning.html
|
|
func (s3a *S3ApiServer) PutBucketVersioningHandler(w http.ResponseWriter, r *http.Request) {
|
|
bucket, _ := s3_constants.GetBucketAndObject(r)
|
|
glog.V(3).Infof("PutBucketVersioning %s", bucket)
|
|
|
|
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
|
|
s3err.WriteErrorResponse(w, r, err)
|
|
return
|
|
}
|
|
|
|
if r.Body == nil || r.Body == http.NoBody {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
|
|
return
|
|
}
|
|
|
|
var versioningConfig s3.VersioningConfiguration
|
|
defer util_http.CloseRequest(r)
|
|
|
|
err := xmlutil.UnmarshalXML(&versioningConfig, xml.NewDecoder(r.Body), "")
|
|
if err != nil {
|
|
glog.Warningf("PutBucketVersioningHandler xml decode: %s", err)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrMalformedXML)
|
|
return
|
|
}
|
|
|
|
if versioningConfig.Status == nil {
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
|
|
return
|
|
}
|
|
|
|
status := *versioningConfig.Status
|
|
if status != s3_constants.VersioningEnabled && status != s3_constants.VersioningSuspended {
|
|
glog.Errorf("PutBucketVersioningHandler: invalid status '%s' for bucket %s", status, bucket)
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
|
|
return
|
|
}
|
|
|
|
// Check if trying to suspend versioning on a bucket with object lock enabled
|
|
if status == s3_constants.VersioningSuspended {
|
|
// Get bucket configuration to check for object lock
|
|
bucketConfig, errCode := s3a.getBucketConfig(bucket)
|
|
if errCode == s3err.ErrNone && bucketConfig.ObjectLockConfig != nil {
|
|
// Object lock is enabled, cannot suspend versioning
|
|
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidBucketState)
|
|
return
|
|
}
|
|
}
|
|
|
|
// Update bucket versioning configuration using new bucket config system
|
|
if errCode := s3a.setBucketVersioningStatus(bucket, status); errCode != s3err.ErrNone {
|
|
glog.Errorf("PutBucketVersioningHandler save config: bucket=%s, status='%s', errCode=%d", bucket, status, errCode)
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
|
|
writeSuccessResponseEmpty(w, r)
|
|
}
|