S3: Directly read write volume servers (#7481)

* Lazy Versioning Check, Conditional SSE Entry Fetch, HEAD Request Optimization

* revert

Reverted the conditional versioning check to always check versioning status
Reverted the conditional SSE entry fetch to always fetch entry metadata
Reverted the conditional versioning check to always check versioning status
Reverted the conditional SSE entry fetch to always fetch entry metadata

* Lazy Entry Fetch for SSE, Skip Conditional Header Check

* SSE-KMS headers are present, this is not an SSE-C request (mutually exclusive)

* SSE-C is mutually exclusive with SSE-S3 and SSE-KMS

* refactor

* Removed Premature Mutual Exclusivity Check

* check for the presence of the X-Amz-Server-Side-Encryption header

* not used

* fmt

* directly read write volume servers

* HTTP Range Request Support

* set header

* md5

* copy object

* fix sse

* fmt

* implement sse

* sse continue

* fixed the suffix range bug (bytes=-N for "last N bytes")

* debug logs

* Missing PartsCount Header

* profiling

* url encoding

* test_multipart_get_part

* headers

* debug

* adjust log level

* handle part number

* Update s3api_object_handlers.go

* nil safety

* set ModifiedTsNs

* remove

* nil check

* fix sse header

* same logic as filer

* decode values

* decode ivBase64

* s3: Fix SSE decryption JWT authentication and streaming errors

Critical fix for SSE (Server-Side Encryption) test failures:

1. **JWT Authentication Bug** (Root Cause):
   - Changed from GenJwtForFilerServer to GenJwtForVolumeServer
   - S3 API now uses correct JWT when directly reading from volume servers
   - Matches filer's authentication pattern for direct volume access
   - Fixes 'unexpected EOF' and 500 errors in SSE tests

2. **Streaming Error Handling**:
   - Added error propagation in getEncryptedStreamFromVolumes goroutine
   - Use CloseWithError() to properly communicate stream failures
   - Added debug logging for streaming errors

3. **Response Header Timing**:
   - Removed premature WriteHeader(http.StatusOK) call
   - Let Go's http package write status automatically on first write
   - Prevents header lock when errors occur during streaming

4. **Enhanced SSE Decryption Debugging**:
   - Added IV/Key validation and logging for SSE-C, SSE-KMS, SSE-S3
   - Better error messages for missing or invalid encryption metadata
   - Added glog.V(2) debugging for decryption setup

This fixes SSE integration test failures where encrypted objects
could not be retrieved due to volume server authentication failures.
The JWT bug was causing volume servers to reject requests, resulting
in truncated/empty streams (EOF) or internal errors.

* s3: Fix SSE multipart upload metadata preservation

Critical fix for SSE multipart upload test failures (SSE-C and SSE-KMS):

**Root Cause - Incomplete SSE Metadata Copying**:
The old code only tried to copy 'SeaweedFSSSEKMSKey' from the first
part to the completed object. This had TWO bugs:

1. **Wrong Constant Name** (Key Mismatch Bug):
   - Storage uses: SeaweedFSSSEKMSKeyHeader = 'X-SeaweedFS-SSE-KMS-Key'
   - Old code read: SeaweedFSSSEKMSKey = 'x-seaweedfs-sse-kms-key'
   - Result: SSE-KMS metadata was NEVER copied → 500 errors

2. **Missing SSE-C and SSE-S3 Headers**:
   - SSE-C requires: IV, Algorithm, KeyMD5
   - SSE-S3 requires: encrypted key data + standard headers
   - Old code: copied nothing for SSE-C/SSE-S3 → decryption failures

**Fix - Complete SSE Header Preservation**:
Now copies ALL SSE headers from first part to completed object:

- SSE-C: SeaweedFSSSEIV, CustomerAlgorithm, CustomerKeyMD5
- SSE-KMS: SeaweedFSSSEKMSKeyHeader, AwsKmsKeyId, ServerSideEncryption
- SSE-S3: SeaweedFSSSES3Key, ServerSideEncryption

Applied consistently to all 3 code paths:
1. Versioned buckets (creates version file)
2. Suspended versioning (creates main object with null versionId)
3. Non-versioned buckets (creates main object)

**Why This Is Correct**:
The headers copied EXACTLY match what putToFiler stores during part
upload (lines 496-521 in s3api_object_handlers_put.go). This ensures
detectPrimarySSEType() can correctly identify encrypted multipart
objects and trigger inline decryption with proper metadata.

Fixes: TestSSEMultipartUploadIntegration (SSE-C and SSE-KMS subtests)

* s3: Add debug logging for versioning state diagnosis

Temporary debug logging to diagnose test_versioning_obj_plain_null_version_overwrite_suspended failure.

Added glog.V(0) logging to show:
1. setBucketVersioningStatus: when versioning status is changed
2. PutObjectHandler: what versioning state is detected (Enabled/Suspended/none)
3. PutObjectHandler: which code path is taken (putVersionedObject vs putSuspendedVersioningObject)

This will help identify if:
- The versioning status is being set correctly in bucket config
- The cache is returning stale/incorrect versioning state
- The switch statement is correctly routing to suspended vs enabled handlers

* s3: Enhanced versioning state tracing for suspended versioning diagnosis

Added comprehensive logging across the entire versioning state flow:

PutBucketVersioningHandler:
- Log requested status (Enabled/Suspended)
- Log when calling setBucketVersioningStatus
- Log success/failure of status change

setBucketVersioningStatus:
- Log bucket and status being set
- Log when config is updated
- Log completion with error code

updateBucketConfig:
- Log versioning state being written to cache
- Immediate cache verification after Set
- Log if cache verification fails

getVersioningState:
- Log bucket name and state being returned
- Log if object lock forces VersioningEnabled
- Log errors

This will reveal:
1. If PutBucketVersioning(Suspended) is reaching the handler
2. If the cache update succeeds
3. What state getVersioningState returns during PUT
4. Any cache consistency issues

Expected to show why bucket still reports 'Enabled' after 'Suspended' call.

* s3: Add SSE chunk detection debugging for multipart uploads

Added comprehensive logging to diagnose why TestSSEMultipartUploadIntegration fails:

detectPrimarySSEType now logs:
1. Total chunk count and extended header count
2. All extended headers with 'sse'/'SSE'/'encryption' in the name
3. For each chunk: index, SseType, and whether it has metadata
4. Final SSE type counts (SSE-C, SSE-KMS, SSE-S3)

This will reveal if:
- Chunks are missing SSE metadata after multipart completion
- Extended headers are copied correctly from first part
- The SSE detection logic is working correctly

Expected to show if chunks have SseType=0 (none) or proper SSE types set.

* s3: Trace SSE chunk metadata through multipart completion and retrieval

Added end-to-end logging to track SSE chunk metadata lifecycle:

**During Multipart Completion (filer_multipart.go)**:
1. Log finalParts chunks BEFORE mkFile - shows SseType and metadata
2. Log versionEntry.Chunks INSIDE mkFile callback - shows if mkFile preserves SSE info
3. Log success after mkFile completes

**During GET Retrieval (s3api_object_handlers.go)**:
1. Log retrieved entry chunks - shows SseType and metadata after retrieval
2. Log detected SSE type result

This will reveal at which point SSE chunk metadata is lost:
- If finalParts have SSE metadata but versionEntry.Chunks don't → mkFile bug
- If versionEntry.Chunks have SSE metadata but retrieved chunks don't → storage/retrieval bug
- If chunks never have SSE metadata → multipart completion SSE processing bug

Expected to show chunks with SseType=NONE during retrieval even though
they were created with proper SseType during multipart completion.

* s3: Fix SSE-C multipart IV base64 decoding bug

**Critical Bug Found**: SSE-C multipart uploads were failing because:

Root Cause:
- entry.Extended[SeaweedFSSSEIV] stores base64-encoded IV (24 bytes for 16-byte IV)
- SerializeSSECMetadata expects raw IV bytes (16 bytes)
- During multipart completion, we were passing base64 IV directly → serialization error

Error Message:
"Failed to serialize SSE-C metadata for chunk in part X: invalid IV length: expected 16 bytes, got 24"

Fix:
- Base64-decode IV before passing to SerializeSSECMetadata
- Added error handling for decode failures

Impact:
- SSE-C multipart uploads will now correctly serialize chunk metadata
- Chunks will have proper SSE metadata for decryption during GET

This fixes the SSE-C subtest of TestSSEMultipartUploadIntegration.
SSE-KMS still has a separate issue (error code 23) being investigated.

* fixes

* kms sse

* handle retry if not found in .versions folder and should read the normal object

* quick check (no retries) to see if the .versions/ directory exists

* skip retry if object is not found

* explicit update to avoid sync delay

* fix map update lock

* Remove fmt.Printf debug statements

* Fix SSE-KMS multipart base IV fallback to fail instead of regenerating

* fmt

* Fix ACL grants storage logic

* header handling

* nil handling

* range read for sse content

* test range requests for sse objects

* fmt

* unused code

* upload in chunks

* header case

* fix url

* bucket policy error vs bucket not found

* jwt handling

* fmt

* jwt in request header

* Optimize Case-Insensitive Prefix Check

* dead code

* Eliminated Unnecessary Stream Prefetch for Multipart SSE

* range sse

* sse

* refactor

* context

* fmt

* fix type

* fix SSE-C IV Mismatch

* Fix Headers Being Set After WriteHeader

* fix url parsing

* propergate sse headers

* multipart sse-s3

* aws sig v4 authen

* sse kms

* set content range

* better errors

* Update s3api_object_handlers_copy.go

* Update s3api_object_handlers.go

* Update s3api_object_handlers.go

* avoid magic number

* clean up

* Update s3api_bucket_policy_handlers.go

* fix url parsing

* context

* data and metadata both use background context

* adjust the offset

* SSE Range Request IV Calculation

* adjust logs

* IV relative to offset in each part, not the whole file

* collect logs

* offset

* fix offset

* fix url

* logs

* variable

* jwt

* Multipart ETag semantics: conditionally set object-level Md5 for single-chunk uploads only.

* sse

* adjust IV and offset

* multipart boundaries

* ensures PUT and GET operations return consistent ETags

* Metadata Header Case

* CommonPrefixes Sorting with URL Encoding

* always sort

* remove the extra PathUnescape call

* fix the multipart get part ETag

* the FileChunk is created without setting ModifiedTsNs

* Sort CommonPrefixes lexicographically to match AWS S3 behavior

* set md5 for multipart uploads

* prevents any potential data loss or corruption in the small-file inline storage path

* compiles correctly

* decryptedReader will now be properly closed after use

* Fixed URL encoding and sort order for CommonPrefixes

* Update s3api_object_handlers_list.go

* SSE-x Chunk View Decryption

* Different IV offset calculations for single-part vs multipart objects

* still too verbose in logs

* less logs

* ensure correct conversion

* fix listing

* nil check

* minor fixes

* nil check

* single character delimiter

* optimize

* range on empty object or zero-length

* correct IV based on its position within that part, not its position in the entire object

* adjust offset

* offset

Fetch FULL encrypted chunk (not just the range)
Adjust IV by PartOffset/ChunkOffset only
Decrypt full chunk
Skip in the DECRYPTED stream to reach OffsetInChunk

* look breaking

* refactor

* error on no content

* handle intra-block byte skipping

* Incomplete HTTP Response Error Handling

* multipart SSE

* Update s3api_object_handlers.go

* address comments

* less logs

* handling directory

* Optimized rejectDirectoryObjectWithoutSlash() to avoid unnecessary lookups

* Revert "handling directory"

This reverts commit 3a335f0ac33c63f51975abc63c40e5328857a74b.

* constant

* Consolidate nil entry checks in GetObjectHandler

* add range tests

* Consolidate redundant nil entry checks in HeadObjectHandler

* adjust logs

* SSE type

* large files

* large files

Reverted the plain-object range test

* ErrNoEncryptionConfig

* Fixed SSERangeReader Infinite Loop Vulnerability

* Fixed SSE-KMS Multipart ChunkReader HTTP Body Leak

* handle empty directory in S3, added PyArrow tests

* purge unused code

* Update s3_parquet_test.py

* Update requirements.txt

* According to S3 specifications, when both partNumber and Range are present, the Range should apply within the selected part's boundaries, not to the full object.

* handle errors

* errors after writing header

* https

* fix: Wait for volume assignment readiness before running Parquet tests

The test-implicit-dir-with-server test was failing with an Internal Error
because volume assignment was not ready when tests started. This fix adds
a check that attempts a volume assignment and waits for it to succeed
before proceeding with tests.

This ensures that:
1. Volume servers are registered with the master
2. Volume growth is triggered if needed
3. The system can successfully assign volumes for writes

Fixes the timeout issue where boto3 would retry 4 times and fail with
'We encountered an internal error, please try again.'

* sse tests

* store derived IV

* fix: Clean up gRPC ports between tests to prevent port conflicts

The second test (test-implicit-dir-with-server) was failing because the
volume server's gRPC port (18080 = VOLUME_PORT + 10000) was still in use
from the first test. The cleanup code only killed HTTP port processes,
not gRPC port processes.

Added cleanup for gRPC ports in all stop targets:
- Master gRPC: MASTER_PORT + 10000 (19333)
- Volume gRPC: VOLUME_PORT + 10000 (18080)
- Filer gRPC: FILER_PORT + 10000 (18888)

This ensures clean state between test runs in CI.

* add import

* address comments

* docs: Add placeholder documentation files for Parquet test suite

Added three missing documentation files referenced in test/s3/parquet/README.md:

1. TEST_COVERAGE.md - Documents 43 total test cases (17 Go unit tests,
   6 Python integration tests, 20 Python end-to-end tests)

2. FINAL_ROOT_CAUSE_ANALYSIS.md - Explains the s3fs compatibility issue
   with PyArrow, the implicit directory problem, and how the fix works

3. MINIO_DIRECTORY_HANDLING.md - Compares MinIO's directory handling
   approach with SeaweedFS's implementation

Each file contains:
- Title and overview
- Key technical details relevant to the topic
- TODO sections for future expansion

These placeholder files resolve the broken README links and provide
structure for future detailed documentation.

* clean up if metadata operation failed

* Update s3_parquet_test.py

* clean up

* Update Makefile

* Update s3_parquet_test.py

* Update Makefile

* Handle ivSkip for non-block-aligned offsets

* Update README.md

* stop volume server faster

* stop volume server in 1 second

* different IV for each chunk in SSE-S3 and SSE-KMS

* clean up if fails

* testing upload

* error propagation

* fmt

* simplify

* fix copying

* less logs

* endian

* Added marshaling error handling

* handling invalid ranges

* error handling for adding to log buffer

* fix logging

* avoid returning too quickly and ensure proper cleaning up

* Activity Tracking for Disk Reads

* Cleanup Unused Parameters

* Activity Tracking for Kafka Publishers

* Proper Test Error Reporting

* refactoring

* less logs

* less logs

* go fmt

* guard it with if entry.Attributes.TtlSec > 0 to match the pattern used elsewhere.

* Handle bucket-default encryption config errors explicitly for multipart

* consistent activity tracking

* obsolete code for s3 on filer read/write handlers

* Update weed/s3api/s3api_object_handlers_list.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
Chris Lu
2025-11-18 23:18:35 -08:00
committed by GitHub
parent 7c8896c3ff
commit ca84a8a713
74 changed files with 8343 additions and 1274 deletions

View File

@@ -53,7 +53,7 @@ type IdentityAccessManagement struct {
// IAM Integration for advanced features
iamIntegration *S3IAMIntegration
// Bucket policy engine for evaluating bucket policies
policyEngine *BucketPolicyEngine
}
@@ -178,7 +178,7 @@ func NewIdentityAccessManagementWithStore(option *S3ApiServerOption, explicitSto
secretAccessKey := os.Getenv("AWS_SECRET_ACCESS_KEY")
if accessKeyId != "" && secretAccessKey != "" {
glog.V(0).Infof("No S3 configuration found, using AWS environment variables as fallback")
glog.V(1).Infof("No S3 configuration found, using AWS environment variables as fallback")
// Create environment variable identity name
identityNameSuffix := accessKeyId
@@ -210,7 +210,7 @@ func NewIdentityAccessManagementWithStore(option *S3ApiServerOption, explicitSto
}
iam.m.Unlock()
glog.V(0).Infof("Added admin identity from AWS environment variables: %s", envIdentity.Name)
glog.V(1).Infof("Added admin identity from AWS environment variables: %s", envIdentity.Name)
}
}
@@ -464,7 +464,7 @@ func (iam *IdentityAccessManagement) authRequest(r *http.Request, action Action)
identity, s3Err = iam.authenticateJWTWithIAM(r)
authType = "Jwt"
} else {
glog.V(0).Infof("IAM integration is nil, returning ErrNotImplemented")
glog.V(2).Infof("IAM integration is nil, returning ErrNotImplemented")
return identity, s3err.ErrNotImplemented
}
case authTypeAnonymous:
@@ -501,7 +501,7 @@ func (iam *IdentityAccessManagement) authRequest(r *http.Request, action Action)
// For ListBuckets, authorization is performed in the handler by iterating
// through buckets and checking permissions for each. Skip the global check here.
policyAllows := false
if action == s3_constants.ACTION_LIST && bucket == "" {
// ListBuckets operation - authorization handled per-bucket in the handler
} else {
@@ -515,7 +515,7 @@ func (iam *IdentityAccessManagement) authRequest(r *http.Request, action Action)
principal := buildPrincipalARN(identity)
// Use context-aware policy evaluation to get the correct S3 action
allowed, evaluated, err := iam.policyEngine.EvaluatePolicyWithContext(bucket, object, string(action), principal, r)
if err != nil {
// SECURITY: Fail-close on policy evaluation errors
// If we can't evaluate the policy, deny access rather than falling through to IAM
@@ -537,7 +537,7 @@ func (iam *IdentityAccessManagement) authRequest(r *http.Request, action Action)
}
// If not evaluated (no policy or no matching statements), fall through to IAM/identity checks
}
// Only check IAM if bucket policy didn't explicitly allow
// This ensures bucket policies can independently grant access (AWS semantics)
if !policyAllows {
@@ -617,26 +617,26 @@ func buildPrincipalARN(identity *Identity) string {
if identity == nil {
return "*" // Anonymous
}
// Check if this is the anonymous user identity (authenticated as anonymous)
// S3 policies expect Principal: "*" for anonymous access
if identity.Name == s3_constants.AccountAnonymousId ||
(identity.Account != nil && identity.Account.Id == s3_constants.AccountAnonymousId) {
if identity.Name == s3_constants.AccountAnonymousId ||
(identity.Account != nil && identity.Account.Id == s3_constants.AccountAnonymousId) {
return "*" // Anonymous user
}
// Build an AWS-compatible principal ARN
// Format: arn:aws:iam::account-id:user/user-name
accountId := identity.Account.Id
if accountId == "" {
accountId = "000000000000" // Default account ID
}
userName := identity.Name
if userName == "" {
userName = "unknown"
}
return fmt.Sprintf("arn:aws:iam::%s:user/%s", accountId, userName)
}

View File

@@ -52,7 +52,7 @@ func (s3a *S3ApiServer) subscribeMetaEvents(clientName string, lastTsNs int64, p
metadataFollowOption.ClientEpoch++
return pb.WithFilerClientFollowMetadata(s3a, metadataFollowOption, processEventFn)
}, func(err error) bool {
glog.V(0).Infof("iam follow metadata changes: %v", err)
glog.V(1).Infof("iam follow metadata changes: %v", err)
return true
})
}
@@ -63,7 +63,7 @@ func (s3a *S3ApiServer) onIamConfigUpdate(dir, filename string, content []byte)
if err := s3a.iam.LoadS3ApiConfigurationFromBytes(content); err != nil {
return err
}
glog.V(0).Infof("updated %s/%s", dir, filename)
glog.V(1).Infof("updated %s/%s", dir, filename)
}
return nil
}
@@ -74,7 +74,7 @@ func (s3a *S3ApiServer) onCircuitBreakerConfigUpdate(dir, filename string, conte
if err := s3a.cb.LoadS3ApiConfigurationFromBytes(content); err != nil {
return err
}
glog.V(0).Infof("updated %s/%s", dir, filename)
glog.V(1).Infof("updated %s/%s", dir, filename)
}
return nil
}
@@ -85,14 +85,14 @@ func (s3a *S3ApiServer) onBucketMetadataChange(dir string, oldEntry *filer_pb.En
if newEntry != nil {
// Update bucket registry (existing functionality)
s3a.bucketRegistry.LoadBucketMetadata(newEntry)
glog.V(0).Infof("updated bucketMetadata %s/%s", dir, newEntry.Name)
glog.V(1).Infof("updated bucketMetadata %s/%s", dir, newEntry.Name)
// Update bucket configuration cache with new entry
s3a.updateBucketConfigCacheFromEntry(newEntry)
} else if oldEntry != nil {
// Remove from bucket registry (existing functionality)
s3a.bucketRegistry.RemoveBucketMetadata(oldEntry)
glog.V(0).Infof("remove bucketMetadata %s/%s", dir, oldEntry.Name)
glog.V(1).Infof("remove bucketMetadata %s/%s", dir, oldEntry.Name)
// Remove from bucket configuration cache
s3a.invalidateBucketConfigCache(oldEntry.Name)
@@ -145,7 +145,7 @@ func (s3a *S3ApiServer) updateBucketConfigCacheFromEntry(entry *filer_pb.Entry)
} else {
glog.V(3).Infof("updateBucketConfigCacheFromEntry: no Object Lock configuration found for bucket %s", bucket)
}
// Load bucket policy if present (for performance optimization)
config.BucketPolicy = loadBucketPolicyFromExtended(entry, bucket)
}

View File

@@ -10,6 +10,6 @@ const s3TimeFormat = "2006-01-02T15:04:05.999Z07:00"
// ConditionalHeaderResult holds the result of conditional header checking
type ConditionalHeaderResult struct {
ErrorCode s3err.ErrorCode
ETag string // ETag of the object (for 304 responses)
Entry *filer_pb.Entry // Entry fetched during conditional check (nil if not fetched or object doesn't exist)
ETag string // ETag of the object (for 304 responses)
Entry *filer_pb.Entry // Entry fetched during conditional check (nil if not fetched or object doesn't exist)
}

View File

@@ -5,7 +5,9 @@ import (
"crypto/rand"
"encoding/base64"
"encoding/hex"
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"math"
"path/filepath"
@@ -71,7 +73,7 @@ func (s3a *S3ApiServer) createMultipartUpload(r *http.Request, input *s3.CreateM
// Prepare and apply encryption configuration within directory creation
// This ensures encryption resources are only allocated if directory creation succeeds
encryptionConfig, prepErr := s3a.prepareMultipartEncryptionConfig(r, uploadIdString)
encryptionConfig, prepErr := s3a.prepareMultipartEncryptionConfig(r, *input.Bucket, uploadIdString)
if prepErr != nil {
encryptionError = prepErr
return // Exit callback, letting mkdir handle the error
@@ -118,6 +120,36 @@ type CompleteMultipartUploadResult struct {
VersionId *string `xml:"-"`
}
// copySSEHeadersFromFirstPart copies all SSE-related headers from the first part to the destination entry
// This is critical for detectPrimarySSEType to work correctly and ensures encryption metadata is preserved
func copySSEHeadersFromFirstPart(dst *filer_pb.Entry, firstPart *filer_pb.Entry, context string) {
if firstPart == nil || firstPart.Extended == nil {
return
}
// Copy ALL SSE-related headers (not just SeaweedFSSSEKMSKey)
sseKeys := []string{
// SSE-C headers
s3_constants.SeaweedFSSSEIV,
s3_constants.AmzServerSideEncryptionCustomerAlgorithm,
s3_constants.AmzServerSideEncryptionCustomerKeyMD5,
// SSE-KMS headers
s3_constants.SeaweedFSSSEKMSKey,
s3_constants.AmzServerSideEncryptionAwsKmsKeyId,
// SSE-S3 headers
s3_constants.SeaweedFSSSES3Key,
// Common SSE header (for SSE-KMS and SSE-S3)
s3_constants.AmzServerSideEncryption,
}
for _, key := range sseKeys {
if value, exists := firstPart.Extended[key]; exists {
dst.Extended[key] = value
glog.V(4).Infof("completeMultipartUpload: copied SSE header %s from first part (%s)", key, context)
}
}
}
func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.CompleteMultipartUploadInput, parts *CompleteMultipartUpload) (output *CompleteMultipartUploadResult, code s3err.ErrorCode) {
glog.V(2).Infof("completeMultipartUpload input %v", input)
@@ -231,6 +263,16 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
mime := pentry.Attributes.Mime
var finalParts []*filer_pb.FileChunk
var offset int64
// Track part boundaries for later retrieval with PartNumber parameter
type PartBoundary struct {
PartNumber int `json:"part"`
StartChunk int `json:"start"`
EndChunk int `json:"end"` // exclusive
ETag string `json:"etag"`
}
var partBoundaries []PartBoundary
for _, partNumber := range completedPartNumbers {
partEntriesByNumber, ok := partEntries[partNumber]
if !ok {
@@ -251,42 +293,18 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
continue
}
// Track within-part offset for SSE-KMS IV calculation
var withinPartOffset int64 = 0
// Record the start chunk index for this part
partStartChunk := len(finalParts)
// Calculate the part's ETag (for GetObject with PartNumber)
partETag := filer.ETag(entry)
for _, chunk := range entry.GetChunks() {
// Update SSE metadata with correct within-part offset (unified approach for KMS and SSE-C)
sseKmsMetadata := chunk.SseMetadata
if chunk.SseType == filer_pb.SSEType_SSE_KMS && len(chunk.SseMetadata) > 0 {
// Deserialize, update offset, and re-serialize SSE-KMS metadata
if kmsKey, err := DeserializeSSEKMSMetadata(chunk.SseMetadata); err == nil {
kmsKey.ChunkOffset = withinPartOffset
if updatedMetadata, serErr := SerializeSSEKMSMetadata(kmsKey); serErr == nil {
sseKmsMetadata = updatedMetadata
glog.V(4).Infof("Updated SSE-KMS metadata for chunk in part %d: withinPartOffset=%d", partNumber, withinPartOffset)
}
}
} else if chunk.SseType == filer_pb.SSEType_SSE_C {
// For SSE-C chunks, create per-chunk metadata using the part's IV
if ivData, exists := entry.Extended[s3_constants.SeaweedFSSSEIV]; exists {
// Get keyMD5 from entry metadata if available
var keyMD5 string
if keyMD5Data, keyExists := entry.Extended[s3_constants.AmzServerSideEncryptionCustomerKeyMD5]; keyExists {
keyMD5 = string(keyMD5Data)
}
// Create SSE-C metadata with the part's IV and this chunk's within-part offset
if ssecMetadata, serErr := SerializeSSECMetadata(ivData, keyMD5, withinPartOffset); serErr == nil {
sseKmsMetadata = ssecMetadata // Reuse the same field for unified handling
glog.V(4).Infof("Created SSE-C metadata for chunk in part %d: withinPartOffset=%d", partNumber, withinPartOffset)
} else {
glog.Errorf("Failed to serialize SSE-C metadata for chunk in part %d: %v", partNumber, serErr)
}
} else {
glog.Errorf("SSE-C chunk in part %d missing IV in entry metadata", partNumber)
}
}
// CRITICAL: Do NOT modify SSE metadata offsets during assembly!
// The encrypted data was created with the offset stored in chunk.SseMetadata.
// Changing the offset here would cause decryption to fail because CTR mode
// uses the offset to initialize the counter. We must decrypt with the same
// offset that was used during encryption.
p := &filer_pb.FileChunk{
FileId: chunk.GetFileIdString(),
@@ -296,14 +314,23 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
CipherKey: chunk.CipherKey,
ETag: chunk.ETag,
IsCompressed: chunk.IsCompressed,
// Preserve SSE metadata with updated within-part offset
// Preserve SSE metadata UNCHANGED - do not modify the offset!
SseType: chunk.SseType,
SseMetadata: sseKmsMetadata,
SseMetadata: chunk.SseMetadata,
}
finalParts = append(finalParts, p)
offset += int64(chunk.Size)
withinPartOffset += int64(chunk.Size)
}
// Record the part boundary
partEndChunk := len(finalParts)
partBoundaries = append(partBoundaries, PartBoundary{
PartNumber: partNumber,
StartChunk: partStartChunk,
EndChunk: partEndChunk,
ETag: partETag,
})
found = true
}
}
@@ -325,6 +352,12 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
}
versionEntry.Extended[s3_constants.ExtVersionIdKey] = []byte(versionId)
versionEntry.Extended[s3_constants.SeaweedFSUploadId] = []byte(*input.UploadId)
// Store parts count for x-amz-mp-parts-count header
versionEntry.Extended[s3_constants.SeaweedFSMultipartPartsCount] = []byte(fmt.Sprintf("%d", len(completedPartNumbers)))
// Store part boundaries for GetObject with PartNumber
if partBoundariesJSON, err := json.Marshal(partBoundaries); err == nil {
versionEntry.Extended[s3_constants.SeaweedFSMultipartPartBoundaries] = partBoundariesJSON
}
// Set object owner for versioned multipart objects
amzAccountId := r.Header.Get(s3_constants.AmzAccountId)
@@ -338,17 +371,11 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
}
}
// Preserve SSE-KMS metadata from the first part (if any)
// SSE-KMS metadata is stored in individual parts, not the upload directory
// Preserve ALL SSE metadata from the first part (if any)
// SSE metadata is stored in individual parts, not the upload directory
if len(completedPartNumbers) > 0 && len(partEntries[completedPartNumbers[0]]) > 0 {
firstPartEntry := partEntries[completedPartNumbers[0]][0]
if firstPartEntry.Extended != nil {
// Copy SSE-KMS metadata from the first part
if kmsMetadata, exists := firstPartEntry.Extended[s3_constants.SeaweedFSSSEKMSKey]; exists {
versionEntry.Extended[s3_constants.SeaweedFSSSEKMSKey] = kmsMetadata
glog.V(3).Infof("completeMultipartUpload: preserved SSE-KMS metadata from first part (versioned)")
}
}
copySSEHeadersFromFirstPart(versionEntry, firstPartEntry, "versioned")
}
if pentry.Attributes.Mime != "" {
versionEntry.Attributes.Mime = pentry.Attributes.Mime
@@ -387,6 +414,12 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
entry.Extended = make(map[string][]byte)
}
entry.Extended[s3_constants.ExtVersionIdKey] = []byte("null")
// Store parts count for x-amz-mp-parts-count header
entry.Extended[s3_constants.SeaweedFSMultipartPartsCount] = []byte(fmt.Sprintf("%d", len(completedPartNumbers)))
// Store part boundaries for GetObject with PartNumber
if partBoundariesJSON, jsonErr := json.Marshal(partBoundaries); jsonErr == nil {
entry.Extended[s3_constants.SeaweedFSMultipartPartBoundaries] = partBoundariesJSON
}
// Set object owner for suspended versioning multipart objects
amzAccountId := r.Header.Get(s3_constants.AmzAccountId)
@@ -400,17 +433,11 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
}
}
// Preserve SSE-KMS metadata from the first part (if any)
// SSE-KMS metadata is stored in individual parts, not the upload directory
// Preserve ALL SSE metadata from the first part (if any)
// SSE metadata is stored in individual parts, not the upload directory
if len(completedPartNumbers) > 0 && len(partEntries[completedPartNumbers[0]]) > 0 {
firstPartEntry := partEntries[completedPartNumbers[0]][0]
if firstPartEntry.Extended != nil {
// Copy SSE-KMS metadata from the first part
if kmsMetadata, exists := firstPartEntry.Extended[s3_constants.SeaweedFSSSEKMSKey]; exists {
entry.Extended[s3_constants.SeaweedFSSSEKMSKey] = kmsMetadata
glog.V(3).Infof("completeMultipartUpload: preserved SSE-KMS metadata from first part (suspended versioning)")
}
}
copySSEHeadersFromFirstPart(entry, firstPartEntry, "suspended versioning")
}
if pentry.Attributes.Mime != "" {
entry.Attributes.Mime = pentry.Attributes.Mime
@@ -440,6 +467,12 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
entry.Extended = make(map[string][]byte)
}
entry.Extended[s3_constants.SeaweedFSUploadId] = []byte(*input.UploadId)
// Store parts count for x-amz-mp-parts-count header
entry.Extended[s3_constants.SeaweedFSMultipartPartsCount] = []byte(fmt.Sprintf("%d", len(completedPartNumbers)))
// Store part boundaries for GetObject with PartNumber
if partBoundariesJSON, err := json.Marshal(partBoundaries); err == nil {
entry.Extended[s3_constants.SeaweedFSMultipartPartBoundaries] = partBoundariesJSON
}
// Set object owner for non-versioned multipart objects
amzAccountId := r.Header.Get(s3_constants.AmzAccountId)
@@ -453,17 +486,11 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
}
}
// Preserve SSE-KMS metadata from the first part (if any)
// SSE-KMS metadata is stored in individual parts, not the upload directory
// Preserve ALL SSE metadata from the first part (if any)
// SSE metadata is stored in individual parts, not the upload directory
if len(completedPartNumbers) > 0 && len(partEntries[completedPartNumbers[0]]) > 0 {
firstPartEntry := partEntries[completedPartNumbers[0]][0]
if firstPartEntry.Extended != nil {
// Copy SSE-KMS metadata from the first part
if kmsMetadata, exists := firstPartEntry.Extended[s3_constants.SeaweedFSSSEKMSKey]; exists {
entry.Extended[s3_constants.SeaweedFSSSEKMSKey] = kmsMetadata
glog.V(3).Infof("completeMultipartUpload: preserved SSE-KMS metadata from first part")
}
}
copySSEHeadersFromFirstPart(entry, firstPartEntry, "non-versioned")
}
if pentry.Attributes.Mime != "" {
entry.Attributes.Mime = pentry.Attributes.Mime
@@ -510,15 +537,11 @@ func (s3a *S3ApiServer) getEntryNameAndDir(input *s3.CompleteMultipartUploadInpu
if dirName == "." {
dirName = ""
}
if strings.HasPrefix(dirName, "/") {
dirName = dirName[1:]
}
dirName = strings.TrimPrefix(dirName, "/")
dirName = fmt.Sprintf("%s/%s/%s", s3a.option.BucketsPath, *input.Bucket, dirName)
// remove suffix '/'
if strings.HasSuffix(dirName, "/") {
dirName = dirName[:len(dirName)-1]
}
dirName = strings.TrimSuffix(dirName, "/")
return entryName, dirName
}
@@ -664,18 +687,23 @@ func (s3a *S3ApiServer) listObjectParts(input *s3.ListPartsInput) (output *ListP
glog.Errorf("listObjectParts %s %s parse %s: %v", *input.Bucket, *input.UploadId, entry.Name, err)
continue
}
output.Part = append(output.Part, &s3.Part{
partETag := filer.ETag(entry)
part := &s3.Part{
PartNumber: aws.Int64(int64(partNumber)),
LastModified: aws.Time(time.Unix(entry.Attributes.Mtime, 0).UTC()),
Size: aws.Int64(int64(filer.FileSize(entry))),
ETag: aws.String("\"" + filer.ETag(entry) + "\""),
})
ETag: aws.String("\"" + partETag + "\""),
}
output.Part = append(output.Part, part)
glog.V(3).Infof("listObjectParts: Added part %d, size=%d, etag=%s",
partNumber, filer.FileSize(entry), partETag)
if !isLast {
output.NextPartNumberMarker = aws.Int64(int64(partNumber))
}
}
}
glog.V(2).Infof("listObjectParts: Returning %d parts for uploadId=%s", len(output.Part), *input.UploadId)
return
}
@@ -704,11 +732,16 @@ type MultipartEncryptionConfig struct {
// prepareMultipartEncryptionConfig prepares encryption configuration with proper error handling
// This eliminates the need for criticalError variable in callback functions
func (s3a *S3ApiServer) prepareMultipartEncryptionConfig(r *http.Request, uploadIdString string) (*MultipartEncryptionConfig, error) {
// Updated to support bucket-default encryption (matches putToFiler behavior)
func (s3a *S3ApiServer) prepareMultipartEncryptionConfig(r *http.Request, bucket string, uploadIdString string) (*MultipartEncryptionConfig, error) {
config := &MultipartEncryptionConfig{}
// Prepare SSE-KMS configuration
if IsSSEKMSRequest(r) {
// Check for explicit encryption headers first (priority over bucket defaults)
hasExplicitSSEKMS := IsSSEKMSRequest(r)
hasExplicitSSES3 := IsSSES3RequestInternal(r)
// Prepare SSE-KMS configuration (explicit request headers)
if hasExplicitSSEKMS {
config.IsSSEKMS = true
config.KMSKeyID = r.Header.Get(s3_constants.AmzServerSideEncryptionAwsKmsKeyId)
config.BucketKeyEnabled = strings.ToLower(r.Header.Get(s3_constants.AmzServerSideEncryptionBucketKeyEnabled)) == "true"
@@ -721,11 +754,11 @@ func (s3a *S3ApiServer) prepareMultipartEncryptionConfig(r *http.Request, upload
return nil, fmt.Errorf("failed to generate secure IV for SSE-KMS multipart upload: %v (read %d/%d bytes)", err, n, len(baseIV))
}
config.KMSBaseIVEncoded = base64.StdEncoding.EncodeToString(baseIV)
glog.V(4).Infof("Generated base IV %x for SSE-KMS multipart upload %s", baseIV[:8], uploadIdString)
glog.V(4).Infof("Generated base IV %x for explicit SSE-KMS multipart upload %s", baseIV[:8], uploadIdString)
}
// Prepare SSE-S3 configuration
if IsSSES3RequestInternal(r) {
// Prepare SSE-S3 configuration (explicit request headers)
if hasExplicitSSES3 {
config.IsSSES3 = true
// Generate and encode base IV with proper error handling
@@ -735,7 +768,7 @@ func (s3a *S3ApiServer) prepareMultipartEncryptionConfig(r *http.Request, upload
return nil, fmt.Errorf("failed to generate secure IV for SSE-S3 multipart upload: %v (read %d/%d bytes)", err, n, len(baseIV))
}
config.S3BaseIVEncoded = base64.StdEncoding.EncodeToString(baseIV)
glog.V(4).Infof("Generated base IV %x for SSE-S3 multipart upload %s", baseIV[:8], uploadIdString)
glog.V(4).Infof("Generated base IV %x for explicit SSE-S3 multipart upload %s", baseIV[:8], uploadIdString)
// Generate and serialize SSE-S3 key with proper error handling
keyManager := GetSSES3KeyManager()
@@ -753,7 +786,77 @@ func (s3a *S3ApiServer) prepareMultipartEncryptionConfig(r *http.Request, upload
// Store key in manager for later retrieval
keyManager.StoreKey(sseS3Key)
glog.V(4).Infof("Stored SSE-S3 key %s for multipart upload %s", sseS3Key.KeyID, uploadIdString)
glog.V(4).Infof("Stored SSE-S3 key %s for explicit multipart upload %s", sseS3Key.KeyID, uploadIdString)
}
// If no explicit encryption headers, check bucket-default encryption
// This matches AWS S3 behavior and putToFiler() implementation
if !hasExplicitSSEKMS && !hasExplicitSSES3 {
encryptionConfig, err := s3a.GetBucketEncryptionConfig(bucket)
if err != nil {
// Check if this is just "no encryption configured" vs a real error
if !errors.Is(err, ErrNoEncryptionConfig) {
// Real error - propagate to prevent silent encryption bypass
return nil, fmt.Errorf("failed to read bucket encryption config for multipart upload: %v", err)
}
// No default encryption configured, continue without encryption
} else if encryptionConfig != nil && encryptionConfig.SseAlgorithm != "" {
glog.V(3).Infof("prepareMultipartEncryptionConfig: applying bucket-default encryption %s for bucket %s, upload %s",
encryptionConfig.SseAlgorithm, bucket, uploadIdString)
switch encryptionConfig.SseAlgorithm {
case EncryptionTypeKMS:
// Apply SSE-KMS as bucket default
config.IsSSEKMS = true
config.KMSKeyID = encryptionConfig.KmsKeyId
config.BucketKeyEnabled = encryptionConfig.BucketKeyEnabled
// No encryption context for bucket defaults
// Generate and encode base IV
baseIV := make([]byte, s3_constants.AESBlockSize)
n, readErr := rand.Read(baseIV)
if readErr != nil || n != len(baseIV) {
return nil, fmt.Errorf("failed to generate secure IV for bucket-default SSE-KMS multipart upload: %v (read %d/%d bytes)", readErr, n, len(baseIV))
}
config.KMSBaseIVEncoded = base64.StdEncoding.EncodeToString(baseIV)
glog.V(4).Infof("Generated base IV %x for bucket-default SSE-KMS multipart upload %s", baseIV[:8], uploadIdString)
case EncryptionTypeAES256:
// Apply SSE-S3 (AES256) as bucket default
config.IsSSES3 = true
// Generate and encode base IV
baseIV := make([]byte, s3_constants.AESBlockSize)
n, readErr := rand.Read(baseIV)
if readErr != nil || n != len(baseIV) {
return nil, fmt.Errorf("failed to generate secure IV for bucket-default SSE-S3 multipart upload: %v (read %d/%d bytes)", readErr, n, len(baseIV))
}
config.S3BaseIVEncoded = base64.StdEncoding.EncodeToString(baseIV)
glog.V(4).Infof("Generated base IV %x for bucket-default SSE-S3 multipart upload %s", baseIV[:8], uploadIdString)
// Generate and serialize SSE-S3 key
keyManager := GetSSES3KeyManager()
sseS3Key, keyErr := keyManager.GetOrCreateKey("")
if keyErr != nil {
return nil, fmt.Errorf("failed to generate SSE-S3 key for bucket-default multipart upload: %v", keyErr)
}
keyData, serErr := SerializeSSES3Metadata(sseS3Key)
if serErr != nil {
return nil, fmt.Errorf("failed to serialize SSE-S3 metadata for bucket-default multipart upload: %v", serErr)
}
config.S3KeyDataEncoded = base64.StdEncoding.EncodeToString(keyData)
// Store key in manager for later retrieval
keyManager.StoreKey(sseS3Key)
glog.V(4).Infof("Stored SSE-S3 key %s for bucket-default multipart upload %s", sseS3Key.KeyID, uploadIdString)
default:
glog.V(3).Infof("prepareMultipartEncryptionConfig: unsupported bucket-default encryption algorithm %s for bucket %s",
encryptionConfig.SseAlgorithm, bucket)
}
}
}
return config, nil

View File

@@ -68,7 +68,7 @@ func doDeleteEntry(client filer_pb.SeaweedFilerClient, parentDirectoryPath strin
glog.V(1).Infof("delete entry %v/%v: %v", parentDirectoryPath, entryName, request)
if resp, err := client.DeleteEntry(context.Background(), request); err != nil {
glog.V(0).Infof("delete entry %v: %v", request, err)
glog.V(1).Infof("delete entry %v: %v", request, err)
return fmt.Errorf("delete entry %s/%s: %v", parentDirectoryPath, entryName, err)
} else {
if resp.Error != "" {
@@ -137,9 +137,9 @@ func (s3a *S3ApiServer) updateEntriesTTL(parentDirectoryPath string, ttlSec int3
}
// processDirectoryTTL processes a single directory in paginated batches
func (s3a *S3ApiServer) processDirectoryTTL(ctx context.Context, client filer_pb.SeaweedFilerClient,
func (s3a *S3ApiServer) processDirectoryTTL(ctx context.Context, client filer_pb.SeaweedFilerClient,
dir string, ttlSec int32, dirsToProcess *[]string, updateErrors *[]error) error {
const batchSize = filer.PaginationSize
startFrom := ""

View File

@@ -140,13 +140,13 @@ func convertPrincipal(principal interface{}) (*policy_engine.StringOrStringSlice
// Handle AWS-style principal with service/user keys
// Example: {"AWS": "arn:aws:iam::123456789012:user/Alice"}
// Only AWS principals are supported for now. Other types like Service or Federated need special handling.
awsPrincipals, ok := p["AWS"]
if !ok || len(p) != 1 {
glog.Warningf("unsupported principal map, only a single 'AWS' key is supported: %v", p)
return nil, fmt.Errorf("unsupported principal map, only a single 'AWS' key is supported, got keys: %v", getMapKeys(p))
}
// Recursively convert the AWS principal value
res, err := convertPrincipal(awsPrincipals)
if err != nil {
@@ -236,4 +236,3 @@ func getMapKeys(m map[string]interface{}) []string {
}
return keys
}

View File

@@ -13,10 +13,10 @@ func TestConvertPolicyDocumentWithMixedTypes(t *testing.T) {
Version: "2012-10-17",
Statement: []policy.Statement{
{
Sid: "TestMixedTypes",
Effect: "Allow",
Action: []string{"s3:GetObject"},
Resource: []string{"arn:aws:s3:::bucket/*"},
Sid: "TestMixedTypes",
Effect: "Allow",
Action: []string{"s3:GetObject"},
Resource: []string{"arn:aws:s3:::bucket/*"},
Principal: []interface{}{"user1", 123, true}, // Mixed types
Condition: map[string]map[string]interface{}{
"NumericEquals": {
@@ -90,7 +90,7 @@ func TestConvertPolicyDocumentWithMixedTypes(t *testing.T) {
}
}
// Check StringEquals condition
// Check StringEquals condition
stringCond, ok := stmt.Condition["StringEquals"]
if !ok {
t.Fatal("Expected StringEquals condition")
@@ -116,7 +116,7 @@ func TestConvertPrincipalWithMapAndMixedTypes(t *testing.T) {
principalMap := map[string]interface{}{
"AWS": []interface{}{
"arn:aws:iam::123456789012:user/Alice",
456, // User ID as number
456, // User ID as number
true, // Some boolean value
},
}
@@ -125,7 +125,7 @@ func TestConvertPrincipalWithMapAndMixedTypes(t *testing.T) {
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if result == nil {
t.Fatal("Expected non-nil result")
}
@@ -230,7 +230,7 @@ func TestConvertPrincipalWithNilValues(t *testing.T) {
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if result == nil {
t.Fatal("Expected non-nil result")
}
@@ -296,7 +296,7 @@ func TestConvertPrincipalMapWithNilValues(t *testing.T) {
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if result == nil {
t.Fatal("Expected non-nil result")
}
@@ -322,11 +322,11 @@ func TestConvertPrincipalMapWithNilValues(t *testing.T) {
func TestConvertToStringUnsupportedType(t *testing.T) {
// Test that unsupported types (e.g., nested maps/slices) return empty string
// This should trigger a warning log and return an error
type customStruct struct {
Field string
}
testCases := []struct {
name string
input interface{}
@@ -494,7 +494,7 @@ func TestConvertPrincipalEmptyStrings(t *testing.T) {
func TestConvertStatementWithUnsupportedFields(t *testing.T) {
// Test that errors are returned for unsupported fields
// These fields are critical for policy semantics and ignoring them would be a security risk
testCases := []struct {
name string
statement *policy.Statement
@@ -544,7 +544,7 @@ func TestConvertStatementWithUnsupportedFields(t *testing.T) {
} else if !strings.Contains(err.Error(), tc.wantError) {
t.Errorf("Expected error containing %q, got: %v", tc.wantError, err)
}
// Verify zero-value struct is returned on error
if result.Sid != "" || result.Effect != "" {
t.Error("Expected zero-value struct on error")
@@ -611,4 +611,3 @@ func TestConvertPolicyDocumentWithId(t *testing.T) {
t.Errorf("Expected 1 statement, got %d", len(dest.Statement))
}
}

View File

@@ -2,6 +2,7 @@ package s3api
import (
"encoding/xml"
"errors"
"fmt"
"io"
"net/http"
@@ -12,6 +13,9 @@ import (
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
)
// ErrNoEncryptionConfig is returned when a bucket has no encryption configuration
var ErrNoEncryptionConfig = errors.New("no encryption configuration found")
// ServerSideEncryptionConfiguration represents the bucket encryption configuration
type ServerSideEncryptionConfiguration struct {
XMLName xml.Name `xml:"ServerSideEncryptionConfiguration"`
@@ -186,7 +190,7 @@ func (s3a *S3ApiServer) GetBucketEncryptionConfig(bucket string) (*s3_pb.Encrypt
config, errCode := s3a.getEncryptionConfiguration(bucket)
if errCode != s3err.ErrNone {
if errCode == s3err.ErrNoSuchBucketEncryptionConfiguration {
return nil, fmt.Errorf("no encryption configuration found")
return nil, ErrNoEncryptionConfig
}
return nil, fmt.Errorf("failed to get encryption configuration")
}
@@ -251,7 +255,11 @@ func (s3a *S3ApiServer) removeEncryptionConfiguration(bucket string) s3err.Error
// IsDefaultEncryptionEnabled checks if default encryption is enabled for a bucket
func (s3a *S3ApiServer) IsDefaultEncryptionEnabled(bucket string) bool {
config, err := s3a.GetBucketEncryptionConfig(bucket)
if err != nil || config == nil {
if err != nil {
glog.V(4).Infof("IsDefaultEncryptionEnabled: failed to get encryption config for bucket %s: %v", bucket, err)
return false
}
if config == nil {
return false
}
return config.SseAlgorithm != ""
@@ -260,7 +268,11 @@ func (s3a *S3ApiServer) IsDefaultEncryptionEnabled(bucket string) bool {
// GetDefaultEncryptionHeaders returns the default encryption headers for a bucket
func (s3a *S3ApiServer) GetDefaultEncryptionHeaders(bucket string) map[string]string {
config, err := s3a.GetBucketEncryptionConfig(bucket)
if err != nil || config == nil {
if err != nil {
glog.V(4).Infof("GetDefaultEncryptionHeaders: failed to get encryption config for bucket %s: %v", bucket, err)
return nil
}
if config == nil {
return nil
}

View File

@@ -39,10 +39,13 @@ const (
AmzObjectTaggingDirective = "X-Amz-Tagging-Directive"
AmzTagCount = "x-amz-tagging-count"
SeaweedFSIsDirectoryKey = "X-Seaweedfs-Is-Directory-Key"
SeaweedFSPartNumber = "X-Seaweedfs-Part-Number"
SeaweedFSUploadId = "X-Seaweedfs-Upload-Id"
SeaweedFSExpiresS3 = "X-Seaweedfs-Expires-S3"
SeaweedFSIsDirectoryKey = "X-Seaweedfs-Is-Directory-Key"
SeaweedFSPartNumber = "X-Seaweedfs-Part-Number"
SeaweedFSUploadId = "X-Seaweedfs-Upload-Id"
SeaweedFSMultipartPartsCount = "X-Seaweedfs-Multipart-Parts-Count"
SeaweedFSMultipartPartBoundaries = "X-Seaweedfs-Multipart-Part-Boundaries" // JSON: [{part:1,start:0,end:2,etag:"abc"},{part:2,start:2,end:3,etag:"def"}]
SeaweedFSExpiresS3 = "X-Seaweedfs-Expires-S3"
AmzMpPartsCount = "x-amz-mp-parts-count"
// S3 ACL headers
AmzCannedAcl = "X-Amz-Acl"
@@ -70,8 +73,6 @@ const (
AmzCopySourceIfModifiedSince = "X-Amz-Copy-Source-If-Modified-Since"
AmzCopySourceIfUnmodifiedSince = "X-Amz-Copy-Source-If-Unmodified-Since"
AmzMpPartsCount = "X-Amz-Mp-Parts-Count"
// S3 Server-Side Encryption with Customer-provided Keys (SSE-C)
AmzServerSideEncryptionCustomerAlgorithm = "X-Amz-Server-Side-Encryption-Customer-Algorithm"
AmzServerSideEncryptionCustomerKey = "X-Amz-Server-Side-Encryption-Customer-Key"

View File

@@ -452,7 +452,7 @@ func minInt(a, b int) int {
func (s3a *S3ApiServer) SetIAMIntegration(iamManager *integration.IAMManager) {
if s3a.iam != nil {
s3a.iam.iamIntegration = NewS3IAMIntegration(iamManager, "localhost:8888")
glog.V(0).Infof("IAM integration successfully set on S3ApiServer")
glog.V(1).Infof("IAM integration successfully set on S3ApiServer")
} else {
glog.Errorf("Cannot set IAM integration: s3a.iam is nil")
}

View File

@@ -83,7 +83,7 @@ func (iam *IdentityAccessManagement) ValidateMultipartOperationWithIAM(r *http.R
// This header is set during initial authentication and contains the correct assumed role ARN
principalArn := r.Header.Get("X-SeaweedFS-Principal")
if principalArn == "" {
glog.V(0).Info("IAM authorization for multipart operation failed: missing principal ARN in request header")
glog.V(2).Info("IAM authorization for multipart operation failed: missing principal ARN in request header")
return s3err.ErrAccessDenied
}

View File

@@ -16,6 +16,20 @@ import (
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
)
// decryptReaderCloser wraps a cipher.StreamReader with proper Close() support
// This ensures the underlying io.ReadCloser (like http.Response.Body) is properly closed
type decryptReaderCloser struct {
io.Reader
underlyingCloser io.Closer
}
func (d *decryptReaderCloser) Close() error {
if d.underlyingCloser != nil {
return d.underlyingCloser.Close()
}
return nil
}
// SSECCopyStrategy represents different strategies for copying SSE-C objects
type SSECCopyStrategy int
@@ -197,8 +211,17 @@ func CreateSSECDecryptedReader(r io.Reader, customerKey *SSECustomerKey, iv []by
// Create CTR mode cipher using the IV from metadata
stream := cipher.NewCTR(block, iv)
decryptReader := &cipher.StreamReader{S: stream, R: r}
return &cipher.StreamReader{S: stream, R: r}, nil
// Wrap with closer if the underlying reader implements io.Closer
if closer, ok := r.(io.Closer); ok {
return &decryptReaderCloser{
Reader: decryptReader,
underlyingCloser: closer,
}, nil
}
return decryptReader, nil
}
// CreateSSECEncryptedReaderWithOffset creates an encrypted reader with a specific counter offset

View File

@@ -0,0 +1,307 @@
package s3api
import (
"bytes"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"io"
"testing"
)
// TestCalculateIVWithOffset tests the calculateIVWithOffset function
func TestCalculateIVWithOffset(t *testing.T) {
baseIV := make([]byte, 16)
rand.Read(baseIV)
tests := []struct {
name string
offset int64
expectedSkip int
expectedBlock int64
}{
{"BlockAligned_0", 0, 0, 0},
{"BlockAligned_16", 16, 0, 1},
{"BlockAligned_32", 32, 0, 2},
{"BlockAligned_48", 48, 0, 3},
{"NonAligned_1", 1, 1, 0},
{"NonAligned_5", 5, 5, 0},
{"NonAligned_10", 10, 10, 0},
{"NonAligned_15", 15, 15, 0},
{"NonAligned_17", 17, 1, 1},
{"NonAligned_21", 21, 5, 1},
{"NonAligned_33", 33, 1, 2},
{"NonAligned_47", 47, 15, 2},
{"LargeOffset", 1000, 1000 % 16, 1000 / 16},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
adjustedIV, skip := calculateIVWithOffset(baseIV, tt.offset)
// Verify skip is correct
if skip != tt.expectedSkip {
t.Errorf("calculateIVWithOffset(%d) skip = %d, want %d", tt.offset, skip, tt.expectedSkip)
}
// Verify IV length is preserved
if len(adjustedIV) != 16 {
t.Errorf("calculateIVWithOffset(%d) IV length = %d, want 16", tt.offset, len(adjustedIV))
}
// Verify IV was adjusted correctly (last 8 bytes incremented by blockOffset)
if tt.expectedBlock == 0 {
if !bytes.Equal(adjustedIV, baseIV) {
t.Errorf("calculateIVWithOffset(%d) IV changed when blockOffset=0", tt.offset)
}
} else {
// IV should be different for non-zero block offsets
if bytes.Equal(adjustedIV, baseIV) {
t.Errorf("calculateIVWithOffset(%d) IV not changed when blockOffset=%d", tt.offset, tt.expectedBlock)
}
}
})
}
}
// TestCTRDecryptionWithNonBlockAlignedOffset tests that CTR decryption works correctly
// for non-block-aligned offsets (the critical bug fix)
func TestCTRDecryptionWithNonBlockAlignedOffset(t *testing.T) {
// Generate test data
plaintext := make([]byte, 1024)
for i := range plaintext {
plaintext[i] = byte(i % 256)
}
// Generate random key and IV
key := make([]byte, 32) // AES-256
iv := make([]byte, 16)
rand.Read(key)
rand.Read(iv)
// Encrypt the entire plaintext
block, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
ciphertext := make([]byte, len(plaintext))
stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(ciphertext, plaintext)
// Test various offsets (both block-aligned and non-block-aligned)
testOffsets := []int64{0, 1, 5, 10, 15, 16, 17, 21, 32, 33, 47, 48, 100, 500}
for _, offset := range testOffsets {
t.Run(string(rune('A'+offset)), func(t *testing.T) {
// Calculate adjusted IV and skip
adjustedIV, skip := calculateIVWithOffset(iv, offset)
// CRITICAL: Start from the block-aligned offset, not the user offset
// CTR mode works on 16-byte blocks, so we need to decrypt from the block start
blockAlignedOffset := offset - int64(skip)
// Decrypt from the block-aligned offset
decryptBlock, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create decrypt cipher: %v", err)
}
decryptStream := cipher.NewCTR(decryptBlock, adjustedIV)
// Create a reader for the ciphertext starting at block-aligned offset
ciphertextFromBlockStart := ciphertext[blockAlignedOffset:]
decryptedFromBlockStart := make([]byte, len(ciphertextFromBlockStart))
decryptStream.XORKeyStream(decryptedFromBlockStart, ciphertextFromBlockStart)
// CRITICAL: Skip the intra-block bytes to get to the user-requested offset
if skip > 0 {
if skip > len(decryptedFromBlockStart) {
t.Fatalf("Skip %d exceeds decrypted data length %d", skip, len(decryptedFromBlockStart))
}
decryptedFromBlockStart = decryptedFromBlockStart[skip:]
}
// Rename for consistency
decryptedFromOffset := decryptedFromBlockStart
// Verify decrypted data matches original plaintext
expectedPlaintext := plaintext[offset:]
if !bytes.Equal(decryptedFromOffset, expectedPlaintext) {
t.Errorf("Decryption mismatch at offset %d (skip=%d)", offset, skip)
previewLen := 32
if len(expectedPlaintext) < previewLen {
previewLen = len(expectedPlaintext)
}
t.Errorf(" Expected first 32 bytes: %x", expectedPlaintext[:previewLen])
previewLen2 := 32
if len(decryptedFromOffset) < previewLen2 {
previewLen2 = len(decryptedFromOffset)
}
t.Errorf(" Got first 32 bytes: %x", decryptedFromOffset[:previewLen2])
// Find first mismatch
for i := 0; i < len(expectedPlaintext) && i < len(decryptedFromOffset); i++ {
if expectedPlaintext[i] != decryptedFromOffset[i] {
t.Errorf(" First mismatch at byte %d: expected %02x, got %02x", i, expectedPlaintext[i], decryptedFromOffset[i])
break
}
}
}
})
}
}
// TestCTRRangeRequestSimulation simulates a real-world S3 range request scenario
func TestCTRRangeRequestSimulation(t *testing.T) {
// Simulate uploading a 5MB object
objectSize := 5 * 1024 * 1024
plaintext := make([]byte, objectSize)
for i := range plaintext {
plaintext[i] = byte(i % 256)
}
// Encrypt the object
key := make([]byte, 32)
iv := make([]byte, 16)
rand.Read(key)
rand.Read(iv)
block, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
ciphertext := make([]byte, len(plaintext))
stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(ciphertext, plaintext)
// Simulate various S3 range requests
rangeTests := []struct {
name string
start int64
end int64
}{
{"First byte", 0, 0},
{"First 100 bytes", 0, 99},
{"Mid-block range", 5, 100}, // Critical: starts at non-aligned offset
{"Single mid-block byte", 17, 17}, // Critical: single byte at offset 17
{"Cross-block range", 10, 50}, // Spans multiple blocks
{"Large range", 1000, 10000},
{"Tail range", int64(objectSize - 1000), int64(objectSize - 1)},
}
for _, rt := range rangeTests {
t.Run(rt.name, func(t *testing.T) {
rangeSize := rt.end - rt.start + 1
// Calculate adjusted IV and skip for the range start
adjustedIV, skip := calculateIVWithOffset(iv, rt.start)
// CRITICAL: Start decryption from block-aligned offset
blockAlignedStart := rt.start - int64(skip)
// Create decryption stream
decryptBlock, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create decrypt cipher: %v", err)
}
decryptStream := cipher.NewCTR(decryptBlock, adjustedIV)
// Decrypt from block-aligned start through the end of range
ciphertextFromBlock := ciphertext[blockAlignedStart : rt.end+1]
decryptedFromBlock := make([]byte, len(ciphertextFromBlock))
decryptStream.XORKeyStream(decryptedFromBlock, ciphertextFromBlock)
// CRITICAL: Skip intra-block bytes to get to user-requested start
if skip > 0 {
decryptedFromBlock = decryptedFromBlock[skip:]
}
decryptedRange := decryptedFromBlock
// Verify decrypted range matches original plaintext
expectedPlaintext := plaintext[rt.start : rt.end+1]
if !bytes.Equal(decryptedRange, expectedPlaintext) {
t.Errorf("Range decryption mismatch for %s (offset=%d, size=%d, skip=%d)",
rt.name, rt.start, rangeSize, skip)
previewLen := 64
if len(expectedPlaintext) < previewLen {
previewLen = len(expectedPlaintext)
}
t.Errorf(" Expected: %x", expectedPlaintext[:previewLen])
previewLen2 := previewLen
if len(decryptedRange) < previewLen2 {
previewLen2 = len(decryptedRange)
}
t.Errorf(" Got: %x", decryptedRange[:previewLen2])
}
})
}
}
// TestCTRDecryptionWithIOReader tests the integration with io.Reader
func TestCTRDecryptionWithIOReader(t *testing.T) {
plaintext := []byte("Hello, World! This is a test of CTR mode decryption with non-aligned offsets.")
key := make([]byte, 32)
iv := make([]byte, 16)
rand.Read(key)
rand.Read(iv)
// Encrypt
block, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
ciphertext := make([]byte, len(plaintext))
stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(ciphertext, plaintext)
// Test reading from various offsets using io.Reader
testOffsets := []int64{0, 5, 10, 16, 17, 30}
for _, offset := range testOffsets {
t.Run(string(rune('A'+offset)), func(t *testing.T) {
// Calculate adjusted IV and skip
adjustedIV, skip := calculateIVWithOffset(iv, offset)
// CRITICAL: Start reading from block-aligned offset in ciphertext
blockAlignedOffset := offset - int64(skip)
// Create decrypted reader
decryptBlock, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create decrypt cipher: %v", err)
}
decryptStream := cipher.NewCTR(decryptBlock, adjustedIV)
ciphertextReader := bytes.NewReader(ciphertext[blockAlignedOffset:])
decryptedReader := &cipher.StreamReader{S: decryptStream, R: ciphertextReader}
// Skip intra-block bytes to get to user-requested offset
if skip > 0 {
_, err := io.CopyN(io.Discard, decryptedReader, int64(skip))
if err != nil {
t.Fatalf("Failed to skip %d bytes: %v", skip, err)
}
}
// Read decrypted data
decryptedData, err := io.ReadAll(decryptedReader)
if err != nil {
t.Fatalf("Failed to read decrypted data: %v", err)
}
// Verify
expectedPlaintext := plaintext[offset:]
if !bytes.Equal(decryptedData, expectedPlaintext) {
t.Errorf("Decryption mismatch at offset %d (skip=%d)", offset, skip)
t.Errorf(" Expected: %q", expectedPlaintext)
t.Errorf(" Got: %q", decryptedData)
}
})
}
}

View File

@@ -164,7 +164,8 @@ func CreateSSEKMSEncryptedReaderWithBaseIVAndOffset(r io.Reader, keyID string, e
defer clearKMSDataKey(dataKeyResult)
// Calculate unique IV using base IV and offset to prevent IV reuse in multipart uploads
iv := calculateIVWithOffset(baseIV, offset)
// Skip is not used here because we're encrypting from the start (not reading a range)
iv, _ := calculateIVWithOffset(baseIV, offset)
// Create CTR mode cipher stream
stream := cipher.NewCTR(dataKeyResult.Block, iv)
@@ -420,9 +421,11 @@ func CreateSSEKMSDecryptedReader(r io.Reader, sseKey *SSEKMSKey) (io.Reader, err
}
// Calculate the correct IV for this chunk's offset within the original part
// Note: The skip bytes must be discarded by the caller before reading from the returned reader
var iv []byte
if sseKey.ChunkOffset > 0 {
iv = calculateIVWithOffset(sseKey.IV, sseKey.ChunkOffset)
iv, _ = calculateIVWithOffset(sseKey.IV, sseKey.ChunkOffset)
// Skip value is ignored here; caller must handle intra-block byte skipping
} else {
iv = sseKey.IV
}
@@ -436,9 +439,18 @@ func CreateSSEKMSDecryptedReader(r io.Reader, sseKey *SSEKMSKey) (io.Reader, err
// Create CTR mode cipher stream for decryption
// Note: AES-CTR is used for object data decryption to match the encryption mode
stream := cipher.NewCTR(block, iv)
decryptReader := &cipher.StreamReader{S: stream, R: r}
// Wrap with closer if the underlying reader implements io.Closer
if closer, ok := r.(io.Closer); ok {
return &decryptReaderCloser{
Reader: decryptReader,
underlyingCloser: closer,
}, nil
}
// Return the decrypted reader
return &cipher.StreamReader{S: stream, R: r}, nil
return decryptReader, nil
}
// ParseSSEKMSHeaders parses SSE-KMS headers from an HTTP request

View File

@@ -109,8 +109,17 @@ func CreateSSES3DecryptedReader(reader io.Reader, key *SSES3Key, iv []byte) (io.
// Create CTR mode cipher with the provided IV
stream := cipher.NewCTR(block, iv)
decryptReader := &cipher.StreamReader{S: stream, R: reader}
return &cipher.StreamReader{S: stream, R: reader}, nil
// Wrap with closer if the underlying reader implements io.Closer
if closer, ok := reader.(io.Closer); ok {
return &decryptReaderCloser{
Reader: decryptReader,
underlyingCloser: closer,
}, nil
}
return decryptReader, nil
}
// GetSSES3Headers returns the headers for SSE-S3 encrypted objects
@@ -531,7 +540,8 @@ func CreateSSES3EncryptedReaderWithBaseIV(reader io.Reader, key *SSES3Key, baseI
// Calculate the proper IV with offset to ensure unique IV per chunk/part
// This prevents the severe security vulnerability of IV reuse in CTR mode
iv := calculateIVWithOffset(baseIV, offset)
// Skip is not used here because we're encrypting from the start (not reading a range)
iv, _ := calculateIVWithOffset(baseIV, offset)
stream := cipher.NewCTR(block, iv)
encryptedReader := &cipher.StreamReader{S: stream, R: reader}

View File

@@ -0,0 +1,266 @@
package s3api
import (
"bytes"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
)
// TestSSES3MultipartChunkViewDecryption tests that multipart SSE-S3 objects use per-chunk IVs
func TestSSES3MultipartChunkViewDecryption(t *testing.T) {
// Generate test key and base IV
key := make([]byte, 32)
rand.Read(key)
baseIV := make([]byte, 16)
rand.Read(baseIV)
// Create test plaintext
plaintext := []byte("This is test data for SSE-S3 multipart encryption testing")
// Simulate multipart upload with 2 parts at different offsets
testCases := []struct {
name string
partNumber int
partOffset int64
data []byte
}{
{"Part 1", 1, 0, plaintext[:30]},
{"Part 2", 2, 5 * 1024 * 1024, plaintext[30:]}, // 5MB offset
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
// Calculate IV with offset (simulating upload encryption)
adjustedIV, _ := calculateIVWithOffset(baseIV, tc.partOffset)
// Encrypt the part data
block, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
ciphertext := make([]byte, len(tc.data))
stream := cipher.NewCTR(block, adjustedIV)
stream.XORKeyStream(ciphertext, tc.data)
// SSE-S3 stores the offset-adjusted IV directly in chunk metadata
// (unlike SSE-C which stores base IV + PartOffset)
chunkIV := adjustedIV
// Verify the IV is offset-adjusted for non-zero offsets
if tc.partOffset == 0 {
if !bytes.Equal(chunkIV, baseIV) {
t.Error("IV should equal base IV when offset is 0")
}
} else {
if bytes.Equal(chunkIV, baseIV) {
t.Error("Chunk IV should be offset-adjusted, not base IV")
}
}
// Verify decryption works with the chunk's IV
decryptedData := make([]byte, len(ciphertext))
decryptBlock, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create decrypt cipher: %v", err)
}
decryptStream := cipher.NewCTR(decryptBlock, chunkIV)
decryptStream.XORKeyStream(decryptedData, ciphertext)
if !bytes.Equal(decryptedData, tc.data) {
t.Errorf("Decryption failed: expected %q, got %q", tc.data, decryptedData)
}
})
}
}
// TestSSES3SinglePartChunkViewDecryption tests single-part SSE-S3 objects use object-level IV
func TestSSES3SinglePartChunkViewDecryption(t *testing.T) {
// Generate test key and IV
key := make([]byte, 32)
rand.Read(key)
iv := make([]byte, 16)
rand.Read(iv)
// Create test plaintext
plaintext := []byte("This is test data for SSE-S3 single-part encryption testing")
// Encrypt the data
block, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
ciphertext := make([]byte, len(plaintext))
stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(ciphertext, plaintext)
// Create a mock file chunk WITHOUT per-chunk metadata (single-part path)
fileChunk := &filer_pb.FileChunk{
FileId: "test-file-id",
Offset: 0,
Size: uint64(len(ciphertext)),
SseType: filer_pb.SSEType_SSE_S3,
SseMetadata: nil, // No per-chunk metadata for single-part
}
// Verify the chunk does NOT have per-chunk metadata
if len(fileChunk.GetSseMetadata()) > 0 {
t.Error("Single-part chunk should not have per-chunk metadata")
}
// For single-part, the object-level IV is used
objectLevelIV := iv
// Verify decryption works with the object-level IV
decryptedData := make([]byte, len(ciphertext))
decryptBlock, _ := aes.NewCipher(key)
decryptStream := cipher.NewCTR(decryptBlock, objectLevelIV)
decryptStream.XORKeyStream(decryptedData, ciphertext)
if !bytes.Equal(decryptedData, plaintext) {
t.Errorf("Decryption failed: expected %q, got %q", plaintext, decryptedData)
}
}
// TestSSES3IVOffsetCalculation verifies IV offset calculation for multipart uploads
func TestSSES3IVOffsetCalculation(t *testing.T) {
baseIV := make([]byte, 16)
rand.Read(baseIV)
testCases := []struct {
name string
partNumber int
partSize int64
offset int64
}{
{"Part 1", 1, 5 * 1024 * 1024, 0},
{"Part 2", 2, 5 * 1024 * 1024, 5 * 1024 * 1024},
{"Part 3", 3, 5 * 1024 * 1024, 10 * 1024 * 1024},
{"Part 10", 10, 5 * 1024 * 1024, 45 * 1024 * 1024},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
// Calculate IV with offset
adjustedIV, skip := calculateIVWithOffset(baseIV, tc.offset)
// Verify IV is different from base (except for offset 0)
if tc.offset == 0 {
if !bytes.Equal(adjustedIV, baseIV) {
t.Error("IV should equal base IV when offset is 0")
}
if skip != 0 {
t.Errorf("Skip should be 0 when offset is 0, got %d", skip)
}
} else {
if bytes.Equal(adjustedIV, baseIV) {
t.Error("IV should be different from base IV when offset > 0")
}
}
// Verify skip is calculated correctly
expectedSkip := int(tc.offset % 16)
if skip != expectedSkip {
t.Errorf("Skip mismatch: expected %d, got %d", expectedSkip, skip)
}
// Verify IV adjustment is deterministic
adjustedIV2, skip2 := calculateIVWithOffset(baseIV, tc.offset)
if !bytes.Equal(adjustedIV, adjustedIV2) || skip != skip2 {
t.Error("IV calculation is not deterministic")
}
})
}
}
// TestSSES3ChunkMetadataDetection tests detection of per-chunk vs object-level metadata
func TestSSES3ChunkMetadataDetection(t *testing.T) {
// Test data for multipart chunk
mockMetadata := []byte("mock-serialized-metadata")
testCases := []struct {
name string
chunk *filer_pb.FileChunk
expectedMultipart bool
}{
{
name: "Multipart chunk with metadata",
chunk: &filer_pb.FileChunk{
SseType: filer_pb.SSEType_SSE_S3,
SseMetadata: mockMetadata,
},
expectedMultipart: true,
},
{
name: "Single-part chunk without metadata",
chunk: &filer_pb.FileChunk{
SseType: filer_pb.SSEType_SSE_S3,
SseMetadata: nil,
},
expectedMultipart: false,
},
{
name: "Non-SSE-S3 chunk",
chunk: &filer_pb.FileChunk{
SseType: filer_pb.SSEType_NONE,
SseMetadata: nil,
},
expectedMultipart: false,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
hasPerChunkMetadata := tc.chunk.GetSseType() == filer_pb.SSEType_SSE_S3 && len(tc.chunk.GetSseMetadata()) > 0
if hasPerChunkMetadata != tc.expectedMultipart {
t.Errorf("Expected multipart=%v, got hasPerChunkMetadata=%v", tc.expectedMultipart, hasPerChunkMetadata)
}
})
}
}
// TestSSES3EncryptionConsistency verifies encryption/decryption roundtrip
func TestSSES3EncryptionConsistency(t *testing.T) {
plaintext := []byte("Test data for SSE-S3 encryption consistency verification")
key := make([]byte, 32)
rand.Read(key)
iv := make([]byte, 16)
rand.Read(iv)
// Encrypt
block, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
ciphertext := make([]byte, len(plaintext))
encryptStream := cipher.NewCTR(block, iv)
encryptStream.XORKeyStream(ciphertext, plaintext)
// Decrypt
decrypted := make([]byte, len(ciphertext))
decryptBlock, _ := aes.NewCipher(key)
decryptStream := cipher.NewCTR(decryptBlock, iv)
decryptStream.XORKeyStream(decrypted, ciphertext)
// Verify
if !bytes.Equal(decrypted, plaintext) {
t.Errorf("Decryption mismatch: expected %q, got %q", plaintext, decrypted)
}
// Verify idempotency - decrypt again should give garbage
decrypted2 := make([]byte, len(ciphertext))
decryptStream2 := cipher.NewCTR(decryptBlock, iv)
decryptStream2.XORKeyStream(decrypted2, ciphertext)
if !bytes.Equal(decrypted2, plaintext) {
t.Error("Second decryption should also work with fresh stream")
}
}

View File

@@ -4,19 +4,22 @@ import "github.com/seaweedfs/seaweedfs/weed/glog"
// calculateIVWithOffset calculates a unique IV by combining a base IV with an offset.
// This ensures each chunk/part uses a unique IV, preventing CTR mode IV reuse vulnerabilities.
// Returns the adjusted IV and the number of bytes to skip from the decrypted stream.
// The skip is needed because CTR mode operates on 16-byte blocks, but the offset may not be block-aligned.
// This function is shared between SSE-KMS and SSE-S3 implementations for consistency.
func calculateIVWithOffset(baseIV []byte, offset int64) []byte {
func calculateIVWithOffset(baseIV []byte, offset int64) ([]byte, int) {
if len(baseIV) != 16 {
glog.Errorf("Invalid base IV length: expected 16, got %d", len(baseIV))
return baseIV // Return original IV as fallback
return baseIV, 0 // Return original IV as fallback
}
// Create a copy of the base IV to avoid modifying the original
iv := make([]byte, 16)
copy(iv, baseIV)
// Calculate the block offset (AES block size is 16 bytes)
// Calculate the block offset (AES block size is 16 bytes) and intra-block skip
blockOffset := offset / 16
skip := int(offset % 16)
originalBlockOffset := blockOffset
// Add the block offset to the IV counter (last 8 bytes, big-endian)
@@ -36,7 +39,7 @@ func calculateIVWithOffset(baseIV []byte, offset int64) []byte {
}
// Single consolidated debug log to avoid performance impact in high-throughput scenarios
glog.V(4).Infof("calculateIVWithOffset: baseIV=%x, offset=%d, blockOffset=%d, derivedIV=%x",
baseIV, offset, originalBlockOffset, iv)
return iv
glog.V(4).Infof("calculateIVWithOffset: baseIV=%x, offset=%d, blockOffset=%d, skip=%d, derivedIV=%x",
baseIV, offset, originalBlockOffset, skip, iv)
return iv, skip
}

View File

@@ -290,8 +290,8 @@ func (bcc *BucketConfigCache) Clear() {
// IsNegativelyCached checks if a bucket is in the negative cache (doesn't exist)
func (bcc *BucketConfigCache) IsNegativelyCached(bucket string) bool {
bcc.mutex.RLock()
defer bcc.mutex.RUnlock()
bcc.mutex.Lock()
defer bcc.mutex.Unlock()
if cachedTime, exists := bcc.negativeCache[bucket]; exists {
// Check if the negative cache entry is still valid
@@ -400,7 +400,7 @@ func (s3a *S3ApiServer) getBucketConfig(bucket string) (*BucketConfig, s3err.Err
} else {
glog.V(3).Infof("getBucketConfig: no Object Lock config found in extended attributes for bucket %s", bucket)
}
// Load bucket policy if present (for performance optimization)
config.BucketPolicy = loadBucketPolicyFromExtended(entry, bucket)
}
@@ -479,7 +479,6 @@ func (s3a *S3ApiServer) updateBucketConfig(bucket string, updateFn func(*BucketC
glog.V(3).Infof("updateBucketConfig: saved entry to filer for bucket %s", bucket)
// Update cache
glog.V(3).Infof("updateBucketConfig: updating cache for bucket %s, ObjectLockConfig=%+v", bucket, config.ObjectLockConfig)
s3a.bucketConfigCache.Set(bucket, config)
return s3err.ErrNone
@@ -522,6 +521,7 @@ func (s3a *S3ApiServer) getVersioningState(bucket string) (string, error) {
if errCode == s3err.ErrNoSuchBucket {
return "", nil
}
glog.Errorf("getVersioningState: failed to get bucket config for %s: %v", bucket, errCode)
return "", fmt.Errorf("failed to get bucket config: %v", errCode)
}
@@ -548,10 +548,11 @@ func (s3a *S3ApiServer) getBucketVersioningStatus(bucket string) (string, s3err.
// setBucketVersioningStatus sets the versioning status for a bucket
func (s3a *S3ApiServer) setBucketVersioningStatus(bucket, status string) s3err.ErrorCode {
return s3a.updateBucketConfig(bucket, func(config *BucketConfig) error {
errCode := s3a.updateBucketConfig(bucket, func(config *BucketConfig) error {
config.Versioning = status
return nil
})
return errCode
}
// getBucketOwnership returns the ownership setting for a bucket

View File

@@ -1159,6 +1159,7 @@ func (s3a *S3ApiServer) PutBucketVersioningHandler(w http.ResponseWriter, r *htt
status := *versioningConfig.Status
if status != s3_constants.VersioningEnabled && status != s3_constants.VersioningSuspended {
glog.Errorf("PutBucketVersioningHandler: invalid status '%s' for bucket %s", status, bucket)
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
return
}
@@ -1176,7 +1177,7 @@ func (s3a *S3ApiServer) PutBucketVersioningHandler(w http.ResponseWriter, r *htt
// Update bucket versioning configuration using new bucket config system
if errCode := s3a.setBucketVersioningStatus(bucket, status); errCode != s3err.ErrNone {
glog.Errorf("PutBucketVersioningHandler save config: %d", errCode)
glog.Errorf("PutBucketVersioningHandler save config: bucket=%s, status='%s', errCode=%d", bucket, status, errCode)
s3err.WriteErrorResponse(w, r, errCode)
return
}

View File

@@ -2,7 +2,7 @@ package s3api
import (
"testing"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
)
@@ -123,4 +123,3 @@ func TestBuildPrincipalARN(t *testing.T) {
})
}
}

View File

@@ -64,7 +64,7 @@ func (bpe *BucketPolicyEngine) LoadBucketPolicyFromCache(bucket string, policyDo
glog.Errorf("Failed to convert bucket policy for %s: %v", bucket, err)
return fmt.Errorf("failed to convert bucket policy: %w", err)
}
// Marshal the converted policy to JSON for storage in the engine
policyJSON, err := json.Marshal(enginePolicyDoc)
if err != nil {
@@ -152,7 +152,7 @@ func (bpe *BucketPolicyEngine) EvaluatePolicyWithContext(bucket, object, action,
// Build resource ARN
resource := buildResourceARN(bucket, object)
glog.V(4).Infof("EvaluatePolicyWithContext: bucket=%s, resource=%s, action=%s (from %s), principal=%s",
glog.V(4).Infof("EvaluatePolicyWithContext: bucket=%s, resource=%s, action=%s (from %s), principal=%s",
bucket, resource, s3Action, action, principal)
// Evaluate using the policy engine

View File

@@ -3,6 +3,7 @@ package s3api
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
@@ -18,17 +19,37 @@ import (
// Bucket policy metadata key for storing policies in filer
const BUCKET_POLICY_METADATA_KEY = "s3-bucket-policy"
// Sentinel errors for bucket policy operations
var (
ErrPolicyNotFound = errors.New("bucket policy not found")
// ErrBucketNotFound is already defined in s3api_object_retention.go
)
// GetBucketPolicyHandler handles GET bucket?policy requests
func (s3a *S3ApiServer) GetBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
bucket, _ := s3_constants.GetBucketAndObject(r)
glog.V(3).Infof("GetBucketPolicyHandler: bucket=%s", bucket)
// Validate bucket exists first for correct error mapping
_, err := s3a.getEntry(s3a.option.BucketsPath, bucket)
if err != nil {
if errors.Is(err, filer_pb.ErrNotFound) {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket)
} else {
glog.Errorf("Failed to check bucket existence for %s: %v", bucket, err)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
}
return
}
// Get bucket policy from filer metadata
policyDocument, err := s3a.getBucketPolicy(bucket)
if err != nil {
if strings.Contains(err.Error(), "not found") {
if errors.Is(err, ErrPolicyNotFound) {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucketPolicy)
} else if errors.Is(err, ErrBucketNotFound) {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket)
} else {
glog.Errorf("Failed to get bucket policy for %s: %v", bucket, err)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
@@ -89,6 +110,15 @@ func (s3a *S3ApiServer) PutBucketPolicyHandler(w http.ResponseWriter, r *http.Re
return
}
// Immediately load into policy engine to avoid race condition
// (The subscription system will also do this async, but we want immediate effect)
if s3a.policyEngine != nil {
if err := s3a.policyEngine.LoadBucketPolicyFromCache(bucket, &policyDoc); err != nil {
glog.Warningf("Failed to immediately load bucket policy into engine for %s: %v", bucket, err)
// Don't fail the request since the subscription will eventually sync it
}
}
// Update IAM integration with new bucket policy
if s3a.iam.iamIntegration != nil {
if err := s3a.updateBucketPolicyInIAM(bucket, &policyDoc); err != nil {
@@ -106,10 +136,24 @@ func (s3a *S3ApiServer) DeleteBucketPolicyHandler(w http.ResponseWriter, r *http
glog.V(3).Infof("DeleteBucketPolicyHandler: bucket=%s", bucket)
// Validate bucket exists first for correct error mapping
_, err := s3a.getEntry(s3a.option.BucketsPath, bucket)
if err != nil {
if errors.Is(err, filer_pb.ErrNotFound) {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket)
} else {
glog.Errorf("Failed to check bucket existence for %s: %v", bucket, err)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
}
return
}
// Check if bucket policy exists
if _, err := s3a.getBucketPolicy(bucket); err != nil {
if strings.Contains(err.Error(), "not found") {
if errors.Is(err, ErrPolicyNotFound) {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucketPolicy)
} else if errors.Is(err, ErrBucketNotFound) {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket)
} else {
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
}
@@ -123,6 +167,15 @@ func (s3a *S3ApiServer) DeleteBucketPolicyHandler(w http.ResponseWriter, r *http
return
}
// Immediately remove from policy engine to avoid race condition
// (The subscription system will also do this async, but we want immediate effect)
if s3a.policyEngine != nil {
if err := s3a.policyEngine.DeleteBucketPolicy(bucket); err != nil {
glog.Warningf("Failed to immediately remove bucket policy from engine for %s: %v", bucket, err)
// Don't fail the request since the subscription will eventually sync it
}
}
// Update IAM integration to remove bucket policy
if s3a.iam.iamIntegration != nil {
if err := s3a.removeBucketPolicyFromIAM(bucket); err != nil {
@@ -146,16 +199,17 @@ func (s3a *S3ApiServer) getBucketPolicy(bucket string) (*policy.PolicyDocument,
Name: bucket,
})
if err != nil {
return fmt.Errorf("bucket not found: %v", err)
// Return sentinel error for bucket not found
return fmt.Errorf("%w: %v", ErrBucketNotFound, err)
}
if resp.Entry == nil {
return fmt.Errorf("bucket policy not found: no entry")
return ErrPolicyNotFound
}
policyJSON, exists := resp.Entry.Extended[BUCKET_POLICY_METADATA_KEY]
if !exists || len(policyJSON) == 0 {
return fmt.Errorf("bucket policy not found: no policy metadata")
return ErrPolicyNotFound
}
if err := json.Unmarshal(policyJSON, &policyDoc); err != nil {

View File

@@ -0,0 +1,285 @@
package s3api
import (
"io"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
)
// TestImplicitDirectoryBehaviorLogic tests the core logic for implicit directory detection
// This tests the decision logic without requiring a full S3 server setup
func TestImplicitDirectoryBehaviorLogic(t *testing.T) {
tests := []struct {
name string
objectPath string
hasTrailingSlash bool
fileSize uint64
isDirectory bool
hasChildren bool
versioningEnabled bool
shouldReturn404 bool
description string
}{
{
name: "Implicit directory: 0-byte file with children, no trailing slash",
objectPath: "dataset",
hasTrailingSlash: false,
fileSize: 0,
isDirectory: false,
hasChildren: true,
versioningEnabled: false,
shouldReturn404: true,
description: "Should return 404 to force s3fs LIST-based discovery",
},
{
name: "Implicit directory: actual directory with children, no trailing slash",
objectPath: "dataset",
hasTrailingSlash: false,
fileSize: 0,
isDirectory: true,
hasChildren: true,
versioningEnabled: false,
shouldReturn404: true,
description: "Should return 404 for directory with children",
},
{
name: "Explicit directory request: trailing slash",
objectPath: "dataset/",
hasTrailingSlash: true,
fileSize: 0,
isDirectory: true,
hasChildren: true,
versioningEnabled: false,
shouldReturn404: false,
description: "Should return 200 for explicit directory request (trailing slash)",
},
{
name: "Empty file: 0-byte file without children",
objectPath: "empty.txt",
hasTrailingSlash: false,
fileSize: 0,
isDirectory: false,
hasChildren: false,
versioningEnabled: false,
shouldReturn404: false,
description: "Should return 200 for legitimate empty file",
},
{
name: "Empty directory: 0-byte directory without children",
objectPath: "empty-dir",
hasTrailingSlash: false,
fileSize: 0,
isDirectory: true,
hasChildren: false,
versioningEnabled: false,
shouldReturn404: false,
description: "Should return 200 for empty directory",
},
{
name: "Regular file: non-zero size",
objectPath: "file.txt",
hasTrailingSlash: false,
fileSize: 100,
isDirectory: false,
hasChildren: false,
versioningEnabled: false,
shouldReturn404: false,
description: "Should return 200 for regular file with content",
},
{
name: "Versioned bucket: implicit directory should return 200",
objectPath: "dataset",
hasTrailingSlash: false,
fileSize: 0,
isDirectory: false,
hasChildren: true,
versioningEnabled: true,
shouldReturn404: false,
description: "Should return 200 for versioned buckets (skip implicit dir check)",
},
{
name: "PyArrow directory marker: 0-byte with children",
objectPath: "dataset",
hasTrailingSlash: false,
fileSize: 0,
isDirectory: false,
hasChildren: true,
versioningEnabled: false,
shouldReturn404: true,
description: "Should return 404 for PyArrow-created directory markers",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Test the logic: should we return 404?
// Logic from HeadObjectHandler:
// if !versioningConfigured && !strings.HasSuffix(object, "/") {
// if isZeroByteFile || isActualDirectory {
// if hasChildren {
// return 404
// }
// }
// }
isZeroByteFile := tt.fileSize == 0 && !tt.isDirectory
isActualDirectory := tt.isDirectory
shouldReturn404 := false
if !tt.versioningEnabled && !tt.hasTrailingSlash {
if isZeroByteFile || isActualDirectory {
if tt.hasChildren {
shouldReturn404 = true
}
}
}
if shouldReturn404 != tt.shouldReturn404 {
t.Errorf("Logic mismatch for %s:\n Expected shouldReturn404=%v\n Got shouldReturn404=%v\n Description: %s",
tt.name, tt.shouldReturn404, shouldReturn404, tt.description)
} else {
t.Logf("✓ %s: correctly returns %d", tt.name, map[bool]int{true: 404, false: 200}[shouldReturn404])
}
})
}
}
// TestHasChildrenLogic tests the hasChildren helper function logic
func TestHasChildrenLogic(t *testing.T) {
tests := []struct {
name string
bucket string
prefix string
listResponse *filer_pb.ListEntriesResponse
listError error
expectedResult bool
description string
}{
{
name: "Directory with children",
bucket: "test-bucket",
prefix: "dataset",
listResponse: &filer_pb.ListEntriesResponse{
Entry: &filer_pb.Entry{
Name: "file.parquet",
IsDirectory: false,
},
},
listError: nil,
expectedResult: true,
description: "Should return true when at least one child exists",
},
{
name: "Empty directory",
bucket: "test-bucket",
prefix: "empty-dir",
listResponse: nil,
listError: io.EOF,
expectedResult: false,
description: "Should return false when no children exist (EOF)",
},
{
name: "Directory with leading slash in prefix",
bucket: "test-bucket",
prefix: "/dataset",
listResponse: &filer_pb.ListEntriesResponse{
Entry: &filer_pb.Entry{
Name: "file.parquet",
IsDirectory: false,
},
},
listError: nil,
expectedResult: true,
description: "Should handle leading slashes correctly",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Test the hasChildren logic:
// 1. It should trim leading slashes from prefix
// 2. It should list with Limit=1
// 3. It should return true if any entry is received
// 4. It should return false if EOF is received
hasChildren := false
if tt.listError == nil && tt.listResponse != nil {
hasChildren = true
} else if tt.listError == io.EOF {
hasChildren = false
}
if hasChildren != tt.expectedResult {
t.Errorf("hasChildren logic mismatch for %s:\n Expected: %v\n Got: %v\n Description: %s",
tt.name, tt.expectedResult, hasChildren, tt.description)
} else {
t.Logf("✓ %s: correctly returns %v", tt.name, hasChildren)
}
})
}
}
// TestImplicitDirectoryEdgeCases tests edge cases in the implicit directory detection
func TestImplicitDirectoryEdgeCases(t *testing.T) {
tests := []struct {
name string
scenario string
expectation string
}{
{
name: "PyArrow write_dataset creates 0-byte files",
scenario: "PyArrow creates 'dataset' as 0-byte file, then writes 'dataset/file.parquet'",
expectation: "HEAD dataset → 404 (has children), s3fs uses LIST → correctly identifies as directory",
},
{
name: "Filer creates actual directories",
scenario: "Filer creates 'dataset' as actual directory with IsDirectory=true",
expectation: "HEAD dataset → 404 (has children), s3fs uses LIST → correctly identifies as directory",
},
{
name: "Empty file edge case",
scenario: "User creates 'empty.txt' as 0-byte file with no children",
expectation: "HEAD empty.txt → 200 (no children), s3fs correctly reports as file",
},
{
name: "Explicit directory request",
scenario: "User requests 'dataset/' with trailing slash",
expectation: "HEAD dataset/ → 200 (explicit directory request), normal directory behavior",
},
{
name: "Versioned bucket",
scenario: "Bucket has versioning enabled",
expectation: "HEAD dataset → 200 (skip implicit dir check), versioned semantics apply",
},
{
name: "AWS S3 compatibility",
scenario: "Only 'dataset/file.txt' exists, no marker at 'dataset'",
expectation: "HEAD dataset → 404 (object doesn't exist), matches AWS S3 behavior",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Logf("Scenario: %s", tt.scenario)
t.Logf("Expected: %s", tt.expectation)
})
}
}
// TestImplicitDirectoryIntegration is an integration test placeholder
// Run with: cd test/s3/parquet && make test-implicit-dir-with-server
func TestImplicitDirectoryIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
t.Skip("Integration test - run manually with: cd test/s3/parquet && make test-implicit-dir-with-server")
}
// Benchmark for hasChildren performance
func BenchmarkHasChildrenCheck(b *testing.B) {
// This benchmark would measure the performance impact of the hasChildren check
// Expected: ~1-5ms per call (one gRPC LIST request with Limit=1)
b.Skip("Benchmark - requires full filer setup")
}

File diff suppressed because it is too large Load Diff

View File

@@ -36,13 +36,14 @@ func (s3a *S3ApiServer) CopyObjectHandler(w http.ResponseWriter, r *http.Request
dstBucket, dstObject := s3_constants.GetBucketAndObject(r)
// Copy source path.
cpSrcPath, err := url.QueryUnescape(r.Header.Get("X-Amz-Copy-Source"))
rawCopySource := r.Header.Get("X-Amz-Copy-Source")
cpSrcPath, err := url.QueryUnescape(rawCopySource)
if err != nil {
// Save unescaped string as is.
cpSrcPath = r.Header.Get("X-Amz-Copy-Source")
cpSrcPath = rawCopySource
}
srcBucket, srcObject, srcVersionId := pathToBucketObjectAndVersion(cpSrcPath)
srcBucket, srcObject, srcVersionId := pathToBucketObjectAndVersion(rawCopySource, cpSrcPath)
glog.V(3).Infof("CopyObjectHandler %s %s (version: %s) => %s %s", srcBucket, srcObject, srcVersionId, dstBucket, dstObject)
@@ -84,7 +85,7 @@ func (s3a *S3ApiServer) CopyObjectHandler(w http.ResponseWriter, r *http.Request
return
}
writeSuccessResponseXML(w, r, CopyObjectResult{
ETag: fmt.Sprintf("%x", entry.Attributes.Md5),
ETag: filer.ETag(entry),
LastModified: time.Now().UTC(),
})
return
@@ -339,23 +340,46 @@ func (s3a *S3ApiServer) CopyObjectHandler(w http.ResponseWriter, r *http.Request
}
func pathToBucketAndObject(path string) (bucket, object string) {
// Remove leading slash if present
path = strings.TrimPrefix(path, "/")
// Split by first slash to separate bucket and object
parts := strings.SplitN(path, "/", 2)
if len(parts) == 2 {
return parts[0], "/" + parts[1]
bucket = parts[0]
object = "/" + parts[1]
return bucket, object
} else if len(parts) == 1 && parts[0] != "" {
// Only bucket provided, no object
return parts[0], ""
}
return parts[0], "/"
// Empty path
return "", ""
}
func pathToBucketObjectAndVersion(path string) (bucket, object, versionId string) {
// Parse versionId from query string if present
// Format: /bucket/object?versionId=version-id
if idx := strings.Index(path, "?versionId="); idx != -1 {
versionId = path[idx+len("?versionId="):] // dynamically calculate length
path = path[:idx]
func pathToBucketObjectAndVersion(rawPath, decodedPath string) (bucket, object, versionId string) {
pathForBucket := decodedPath
if rawPath != "" {
if idx := strings.Index(rawPath, "?"); idx != -1 {
queryPart := rawPath[idx+1:]
if values, err := url.ParseQuery(queryPart); err == nil && values.Has("versionId") {
versionId = values.Get("versionId")
rawPathNoQuery := rawPath[:idx]
if unescaped, err := url.QueryUnescape(rawPathNoQuery); err == nil {
pathForBucket = unescaped
} else {
pathForBucket = rawPathNoQuery
}
bucket, object = pathToBucketAndObject(pathForBucket)
return bucket, object, versionId
}
}
}
bucket, object = pathToBucketAndObject(path)
bucket, object = pathToBucketAndObject(pathForBucket)
return bucket, object, versionId
}
@@ -370,15 +394,28 @@ func (s3a *S3ApiServer) CopyObjectPartHandler(w http.ResponseWriter, r *http.Req
dstBucket, dstObject := s3_constants.GetBucketAndObject(r)
// Copy source path.
cpSrcPath, err := url.QueryUnescape(r.Header.Get("X-Amz-Copy-Source"))
rawCopySource := r.Header.Get("X-Amz-Copy-Source")
glog.V(4).Infof("CopyObjectPart: Raw copy source header=%q", rawCopySource)
// Try URL unescaping - AWS SDK sends URL-encoded copy sources
cpSrcPath, err := url.QueryUnescape(rawCopySource)
if err != nil {
// Save unescaped string as is.
cpSrcPath = r.Header.Get("X-Amz-Copy-Source")
// If unescaping fails, log and use original
glog.V(4).Infof("CopyObjectPart: Failed to unescape copy source %q: %v, using as-is", rawCopySource, err)
cpSrcPath = rawCopySource
}
srcBucket, srcObject, srcVersionId := pathToBucketObjectAndVersion(cpSrcPath)
srcBucket, srcObject, srcVersionId := pathToBucketObjectAndVersion(rawCopySource, cpSrcPath)
glog.V(4).Infof("CopyObjectPart: Parsed srcBucket=%q, srcObject=%q, srcVersionId=%q",
srcBucket, srcObject, srcVersionId)
// If source object is empty or bucket is empty, reply back invalid copy source.
// Note: srcObject can be "/" for root-level objects, but empty string means parsing failed
if srcObject == "" || srcBucket == "" {
glog.Errorf("CopyObjectPart: Invalid copy source - srcBucket=%q, srcObject=%q (original header: %q)",
srcBucket, srcObject, r.Header.Get("X-Amz-Copy-Source"))
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidCopySource)
return
}
@@ -471,9 +508,15 @@ func (s3a *S3ApiServer) CopyObjectPartHandler(w http.ResponseWriter, r *http.Req
}
// Create new entry for the part
// Calculate part size, avoiding underflow for invalid ranges
partSize := uint64(0)
if endOffset >= startOffset {
partSize = uint64(endOffset - startOffset + 1)
}
dstEntry := &filer_pb.Entry{
Attributes: &filer_pb.FuseAttributes{
FileSize: uint64(endOffset - startOffset + 1),
FileSize: partSize,
Mtime: time.Now().Unix(),
Crtime: time.Now().Unix(),
Mime: entry.Attributes.Mime,
@@ -483,7 +526,8 @@ func (s3a *S3ApiServer) CopyObjectPartHandler(w http.ResponseWriter, r *http.Req
// Handle zero-size files or empty ranges
if entry.Attributes.FileSize == 0 || endOffset < startOffset {
// For zero-size files or invalid ranges, create an empty part
// For zero-size files or invalid ranges, create an empty part with size 0
dstEntry.Attributes.FileSize = 0
dstEntry.Chunks = nil
} else {
// Copy chunks that overlap with the range
@@ -660,15 +704,37 @@ func processMetadataBytes(reqHeader http.Header, existing map[string][]byte, rep
if replaceMeta {
for header, values := range reqHeader {
if strings.HasPrefix(header, s3_constants.AmzUserMetaPrefix) {
// Go's HTTP server canonicalizes headers (e.g., x-amz-meta-foo → X-Amz-Meta-Foo)
// We store them as they come in (after canonicalization) to preserve the user's intent
for _, value := range values {
metadata[header] = []byte(value)
}
}
}
} else {
// Copy existing metadata as-is
// Note: Metadata should already be normalized during storage (X-Amz-Meta-*),
// but we handle legacy non-canonical formats for backward compatibility
for k, v := range existing {
if strings.HasPrefix(k, s3_constants.AmzUserMetaPrefix) {
// Already in canonical format
metadata[k] = v
} else if len(k) >= 11 && strings.EqualFold(k[:11], "x-amz-meta-") {
// Backward compatibility: migrate old non-canonical format to canonical format
// This ensures gradual migration of metadata to consistent format
suffix := k[11:] // Extract suffix after "x-amz-meta-"
canonicalKey := s3_constants.AmzUserMetaPrefix + suffix
if glog.V(3) {
glog.Infof("Migrating legacy user metadata key %q to canonical format %q during copy", k, canonicalKey)
}
// Check for collision with canonical key
if _, exists := metadata[canonicalKey]; exists {
glog.Warningf("User metadata key collision during copy migration: canonical key %q already exists, skipping legacy key %q", canonicalKey, k)
} else {
metadata[canonicalKey] = v
}
}
}
}
@@ -1272,6 +1338,7 @@ func (s3a *S3ApiServer) copyMultipartSSEKMSChunk(chunk *filer_pb.FileChunk, dest
}
// Encrypt with destination key
originalSize := len(finalData)
encryptedReader, destSSEKey, encErr := CreateSSEKMSEncryptedReaderWithBucketKey(bytes.NewReader(finalData), destKeyID, encryptionContext, bucketKeyEnabled)
if encErr != nil {
return nil, fmt.Errorf("create SSE-KMS encrypted reader: %w", encErr)
@@ -1296,7 +1363,7 @@ func (s3a *S3ApiServer) copyMultipartSSEKMSChunk(chunk *filer_pb.FileChunk, dest
dstChunk.SseType = filer_pb.SSEType_SSE_KMS
dstChunk.SseMetadata = kmsMetadata
glog.V(4).Infof("Re-encrypted multipart SSE-KMS chunk: %d bytes → %d bytes", len(finalData)-len(reencryptedData)+len(finalData), len(finalData))
glog.V(4).Infof("Re-encrypted multipart SSE-KMS chunk: %d bytes → %d bytes", originalSize, len(finalData))
}
// Upload the final data
@@ -1360,10 +1427,12 @@ func (s3a *S3ApiServer) copyMultipartSSECChunk(chunk *filer_pb.FileChunk, copySo
// Calculate the correct IV for this chunk using within-part offset
var chunkIV []byte
var ivSkip int
if ssecMetadata.PartOffset > 0 {
chunkIV = calculateIVWithOffset(chunkBaseIV, ssecMetadata.PartOffset)
chunkIV, ivSkip = calculateIVWithOffset(chunkBaseIV, ssecMetadata.PartOffset)
} else {
chunkIV = chunkBaseIV
ivSkip = 0
}
// Decrypt the chunk data
@@ -1372,6 +1441,14 @@ func (s3a *S3ApiServer) copyMultipartSSECChunk(chunk *filer_pb.FileChunk, copySo
return nil, nil, fmt.Errorf("create decrypted reader: %w", decErr)
}
// CRITICAL: Skip intra-block bytes from CTR decryption (non-block-aligned offset handling)
if ivSkip > 0 {
_, skipErr := io.CopyN(io.Discard, decryptedReader, int64(ivSkip))
if skipErr != nil {
return nil, nil, fmt.Errorf("failed to skip intra-block bytes (%d): %w", ivSkip, skipErr)
}
}
decryptedData, readErr := io.ReadAll(decryptedReader)
if readErr != nil {
return nil, nil, fmt.Errorf("decrypt chunk data: %w", readErr)
@@ -1393,6 +1470,7 @@ func (s3a *S3ApiServer) copyMultipartSSECChunk(chunk *filer_pb.FileChunk, copySo
destIV = newIV
// Encrypt with new key and IV
originalSize := len(finalData)
encryptedReader, iv, encErr := CreateSSECEncryptedReader(bytes.NewReader(finalData), destKey)
if encErr != nil {
return nil, nil, fmt.Errorf("create encrypted reader: %w", encErr)
@@ -1415,7 +1493,7 @@ func (s3a *S3ApiServer) copyMultipartSSECChunk(chunk *filer_pb.FileChunk, copySo
dstChunk.SseType = filer_pb.SSEType_SSE_C
dstChunk.SseMetadata = ssecMetadata // Use unified metadata field
glog.V(4).Infof("Re-encrypted multipart SSE-C chunk: %d bytes → %d bytes", len(finalData)-len(reencryptedData)+len(finalData), len(finalData))
glog.V(4).Infof("Re-encrypted multipart SSE-C chunk: %d bytes → %d bytes", originalSize, len(finalData))
}
// Upload the final data
@@ -1580,10 +1658,12 @@ func (s3a *S3ApiServer) copyCrossEncryptionChunk(chunk *filer_pb.FileChunk, sour
// Calculate the correct IV for this chunk using within-part offset
var chunkIV []byte
var ivSkip int
if ssecMetadata.PartOffset > 0 {
chunkIV = calculateIVWithOffset(chunkBaseIV, ssecMetadata.PartOffset)
chunkIV, ivSkip = calculateIVWithOffset(chunkBaseIV, ssecMetadata.PartOffset)
} else {
chunkIV = chunkBaseIV
ivSkip = 0
}
decryptedReader, decErr := CreateSSECDecryptedReader(bytes.NewReader(encryptedData), sourceSSECKey, chunkIV)
@@ -1591,6 +1671,14 @@ func (s3a *S3ApiServer) copyCrossEncryptionChunk(chunk *filer_pb.FileChunk, sour
return nil, fmt.Errorf("create SSE-C decrypted reader: %w", decErr)
}
// CRITICAL: Skip intra-block bytes from CTR decryption (non-block-aligned offset handling)
if ivSkip > 0 {
_, skipErr := io.CopyN(io.Discard, decryptedReader, int64(ivSkip))
if skipErr != nil {
return nil, fmt.Errorf("failed to skip intra-block bytes (%d): %w", ivSkip, skipErr)
}
}
decryptedData, readErr := io.ReadAll(decryptedReader)
if readErr != nil {
return nil, fmt.Errorf("decrypt SSE-C chunk data: %w", readErr)

View File

@@ -7,6 +7,7 @@ import (
"io"
"net/http"
"net/url"
"sort"
"strconv"
"strings"
@@ -206,13 +207,15 @@ func (s3a *S3ApiServer) listFilerEntries(bucket string, originalPrefix string, m
nextMarker, doErr = s3a.doListFilerEntries(client, reqDir, prefix, cursor, marker, delimiter, false, func(dir string, entry *filer_pb.Entry) {
empty = false
dirName, entryName, prefixName := entryUrlEncode(dir, entry.Name, encodingTypeUrl)
dirName, entryName, _ := entryUrlEncode(dir, entry.Name, encodingTypeUrl)
if entry.IsDirectory {
// When delimiter is specified, apply delimiter logic to directory key objects too
if delimiter != "" && entry.IsDirectoryKeyObject() {
// Apply the same delimiter logic as for regular files
var delimiterFound bool
undelimitedPath := fmt.Sprintf("%s/%s/", dirName, entryName)[len(bucketPrefix):]
// Use raw dir and entry.Name (not encoded) to ensure consistent handling
// Encoding will be applied after sorting if encodingTypeUrl is set
undelimitedPath := fmt.Sprintf("%s/%s/", dir, entry.Name)[len(bucketPrefix):]
// take into account a prefix if supplied while delimiting.
undelimitedPath = strings.TrimPrefix(undelimitedPath, originalPrefix)
@@ -257,8 +260,10 @@ func (s3a *S3ApiServer) listFilerEntries(bucket string, originalPrefix string, m
lastEntryWasCommonPrefix = false
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
} else if delimiter == "/" { // A response can contain CommonPrefixes only if you specify a delimiter.
// Use raw dir and entry.Name (not encoded) to ensure consistent handling
// Encoding will be applied after sorting if encodingTypeUrl is set
commonPrefixes = append(commonPrefixes, PrefixEntry{
Prefix: fmt.Sprintf("%s/%s/", dirName, prefixName)[len(bucketPrefix):],
Prefix: fmt.Sprintf("%s/%s/", dir, entry.Name)[len(bucketPrefix):],
})
//All of the keys (up to 1,000) rolled up into a common prefix count as a single return when calculating the number of returns.
cursor.maxKeys--
@@ -350,10 +355,21 @@ func (s3a *S3ApiServer) listFilerEntries(bucket string, originalPrefix string, m
Contents: contents,
CommonPrefixes: commonPrefixes,
}
// Sort CommonPrefixes to match AWS S3 behavior
// AWS S3 treats the delimiter character specially for sorting common prefixes.
// For example, with delimiter '/', 'foo/' should come before 'foo+1/' even though '+' (ASCII 43) < '/' (ASCII 47).
// This custom comparison ensures correct S3-compatible lexicographical ordering.
sort.Slice(response.CommonPrefixes, func(i, j int) bool {
return compareWithDelimiter(response.CommonPrefixes[i].Prefix, response.CommonPrefixes[j].Prefix, delimiter)
})
// URL-encode CommonPrefixes AFTER sorting (if EncodingType=url)
// This ensures proper sort order (on decoded values) and correct encoding in response
if encodingTypeUrl {
// Todo used for pass test_bucket_listv2_encoding_basic
// sort.Slice(response.CommonPrefixes, func(i, j int) bool { return response.CommonPrefixes[i].Prefix < response.CommonPrefixes[j].Prefix })
response.EncodingType = s3.EncodingTypeUrl
for i := range response.CommonPrefixes {
response.CommonPrefixes[i].Prefix = urlPathEscape(response.CommonPrefixes[i].Prefix)
}
}
return nil
})
@@ -728,6 +744,57 @@ func (s3a *S3ApiServer) getLatestVersionEntryForListOperation(bucket, object str
return logicalEntry, nil
}
// compareWithDelimiter compares two strings for sorting, treating the delimiter character
// as having lower precedence than other characters to match AWS S3 behavior.
// For example, with delimiter '/', 'foo/' should come before 'foo+1/' even though '+' < '/' in ASCII.
// Note: This function assumes delimiter is a single character. Multi-character delimiters will fall back to standard comparison.
func compareWithDelimiter(a, b, delimiter string) bool {
if delimiter == "" {
return a < b
}
// Multi-character delimiters are not supported by AWS S3 in practice,
// but if encountered, fall back to standard byte-wise comparison
if len(delimiter) != 1 {
return a < b
}
delimByte := delimiter[0]
minLen := len(a)
if len(b) < minLen {
minLen = len(b)
}
// Compare character by character
for i := 0; i < minLen; i++ {
charA := a[i]
charB := b[i]
if charA == charB {
continue
}
// Check if either character is the delimiter
isDelimA := charA == delimByte
isDelimB := charB == delimByte
if isDelimA && !isDelimB {
// Delimiter in 'a' should come first
return true
}
if !isDelimA && isDelimB {
// Delimiter in 'b' should come first
return false
}
// Neither or both are delimiters, use normal comparison
return charA < charB
}
// If we get here, one string is a prefix of the other
return len(a) < len(b)
}
// adjustMarkerForDelimiter handles delimiter-ending markers by incrementing them to skip entries with that prefix.
// For example, when continuation token is "boo/", this returns "boo~" to skip all "boo/*" entries
// but still finds any "bop" or later entries. We add a high ASCII character rather than incrementing

View File

@@ -1,7 +1,6 @@
package s3api
import (
"crypto/rand"
"crypto/sha1"
"encoding/base64"
"encoding/json"
@@ -308,6 +307,7 @@ func (s3a *S3ApiServer) PutObjectPartHandler(w http.ResponseWriter, r *http.Requ
dataReader, s3ErrCode := getRequestDataReader(s3a, r)
if s3ErrCode != s3err.ErrNone {
glog.Errorf("PutObjectPartHandler: getRequestDataReader failed with code %v", s3ErrCode)
s3err.WriteErrorResponse(w, r, s3ErrCode)
return
}
@@ -349,21 +349,19 @@ func (s3a *S3ApiServer) PutObjectPartHandler(w http.ResponseWriter, r *http.Requ
if baseIVBytes, exists := uploadEntry.Extended[s3_constants.SeaweedFSSSEKMSBaseIV]; exists {
// Decode the base64 encoded base IV
decodedIV, decodeErr := base64.StdEncoding.DecodeString(string(baseIVBytes))
if decodeErr == nil && len(decodedIV) == 16 {
if decodeErr == nil && len(decodedIV) == s3_constants.AESBlockSize {
baseIV = decodedIV
glog.V(4).Infof("Using stored base IV %x for multipart upload %s", baseIV[:8], uploadID)
} else {
glog.Errorf("Failed to decode base IV for multipart upload %s: %v", uploadID, decodeErr)
glog.Errorf("Failed to decode base IV for multipart upload %s: %v (expected %d bytes, got %d)", uploadID, decodeErr, s3_constants.AESBlockSize, len(decodedIV))
}
}
// Base IV is required for SSE-KMS multipart uploads - fail if missing or invalid
if len(baseIV) == 0 {
glog.Errorf("No valid base IV found for SSE-KMS multipart upload %s", uploadID)
// Generate a new base IV as fallback
baseIV = make([]byte, 16)
if _, err := rand.Read(baseIV); err != nil {
glog.Errorf("Failed to generate fallback base IV: %v", err)
}
glog.Errorf("No valid base IV found for SSE-KMS multipart upload %s - cannot proceed with encryption", uploadID)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
return
}
// Add SSE-KMS headers to the request for putToFiler to handle encryption
@@ -390,7 +388,9 @@ func (s3a *S3ApiServer) PutObjectPartHandler(w http.ResponseWriter, r *http.Requ
}
}
}
} else {
} else if !errors.Is(err, filer_pb.ErrNotFound) {
// Log unexpected errors (but not "not found" which is normal for non-SSE uploads)
glog.V(3).Infof("Could not retrieve upload entry for %s/%s: %v (may be non-SSE upload)", bucket, uploadID, err)
}
}
@@ -399,16 +399,26 @@ func (s3a *S3ApiServer) PutObjectPartHandler(w http.ResponseWriter, r *http.Requ
if partID == 1 && r.Header.Get("Content-Type") == "" {
dataReader = mimeDetect(r, dataReader)
}
destination := fmt.Sprintf("%s/%s%s", s3a.option.BucketsPath, bucket, object)
etag, errCode, _ := s3a.putToFiler(r, uploadUrl, dataReader, destination, bucket, partID)
glog.V(2).Infof("PutObjectPart: bucket=%s, object=%s, uploadId=%s, partNumber=%d, size=%d",
bucket, object, uploadID, partID, r.ContentLength)
etag, errCode, sseMetadata := s3a.putToFiler(r, uploadUrl, dataReader, bucket, partID)
if errCode != s3err.ErrNone {
glog.Errorf("PutObjectPart: putToFiler failed with error code %v for bucket=%s, object=%s, partNumber=%d",
errCode, bucket, object, partID)
s3err.WriteErrorResponse(w, r, errCode)
return
}
glog.V(2).Infof("PutObjectPart: SUCCESS - bucket=%s, object=%s, partNumber=%d, etag=%s, sseType=%s",
bucket, object, partID, etag, sseMetadata.SSEType)
setEtag(w, etag)
// Set SSE response headers for multipart uploads
s3a.setSSEResponseHeaders(w, r, sseMetadata)
writeSuccessResponseEmpty(w, r)
}

View File

@@ -136,7 +136,7 @@ func (s3a *S3ApiServer) PostPolicyBucketHandler(w http.ResponseWriter, r *http.R
}
}
etag, errCode, _ := s3a.putToFiler(r, uploadUrl, fileBody, "", bucket, 1)
etag, errCode, sseMetadata := s3a.putToFiler(r, uploadUrl, fileBody, bucket, 1)
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
@@ -152,6 +152,8 @@ func (s3a *S3ApiServer) PostPolicyBucketHandler(w http.ResponseWriter, r *http.R
}
setEtag(w, etag)
// Include SSE response headers (important for bucket-default encryption)
s3a.setSSEResponseHeaders(w, r, sseMetadata)
// Decide what http response to send depending on success_action_status parameter
switch successStatus {

View File

@@ -1,25 +1,28 @@
package s3api
import (
"crypto/md5"
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/pquerna/cachecontrol/cacheobject"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/operation"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/s3_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
"github.com/seaweedfs/seaweedfs/weed/security"
weed_server "github.com/seaweedfs/seaweedfs/weed/server"
stats_collect "github.com/seaweedfs/seaweedfs/weed/stats"
"github.com/seaweedfs/seaweedfs/weed/util/constants"
)
@@ -60,6 +63,13 @@ type BucketDefaultEncryptionResult struct {
SSEKMSKey *SSEKMSKey
}
// SSEResponseMetadata holds encryption metadata needed for HTTP response headers
type SSEResponseMetadata struct {
SSEType string
KMSKeyID string
BucketKeyEnabled bool
}
func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
// http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
@@ -135,7 +145,7 @@ func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request)
versioningEnabled := (versioningState == s3_constants.VersioningEnabled)
versioningConfigured := (versioningState != "")
glog.V(2).Infof("PutObjectHandler: bucket=%s, object=%s, versioningState='%s', versioningEnabled=%v, versioningConfigured=%v", bucket, object, versioningState, versioningEnabled, versioningConfigured)
glog.V(3).Infof("PutObjectHandler: bucket=%s, object=%s, versioningState='%s', versioningEnabled=%v, versioningConfigured=%v", bucket, object, versioningState, versioningEnabled, versioningConfigured)
// Validate object lock headers before processing
if err := s3a.validateObjectLockHeaders(r, versioningEnabled); err != nil {
@@ -158,29 +168,34 @@ func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request)
switch versioningState {
case s3_constants.VersioningEnabled:
// Handle enabled versioning - create new versions with real version IDs
glog.V(0).Infof("PutObjectHandler: ENABLED versioning detected for %s/%s, calling putVersionedObject", bucket, object)
versionId, etag, errCode := s3a.putVersionedObject(r, bucket, object, dataReader, objectContentType)
glog.V(3).Infof("PutObjectHandler: ENABLED versioning detected for %s/%s, calling putVersionedObject", bucket, object)
versionId, etag, errCode, sseMetadata := s3a.putVersionedObject(r, bucket, object, dataReader, objectContentType)
if errCode != s3err.ErrNone {
glog.Errorf("PutObjectHandler: putVersionedObject failed with errCode=%v for %s/%s", errCode, bucket, object)
s3err.WriteErrorResponse(w, r, errCode)
return
}
glog.V(0).Infof("PutObjectHandler: putVersionedObject returned versionId=%s, etag=%s for %s/%s", versionId, etag, bucket, object)
glog.V(3).Infof("PutObjectHandler: putVersionedObject returned versionId=%s, etag=%s for %s/%s", versionId, etag, bucket, object)
// Set version ID in response header
if versionId != "" {
w.Header().Set("x-amz-version-id", versionId)
glog.V(0).Infof("PutObjectHandler: set x-amz-version-id header to %s for %s/%s", versionId, bucket, object)
glog.V(3).Infof("PutObjectHandler: set x-amz-version-id header to %s for %s/%s", versionId, bucket, object)
} else {
glog.Errorf("PutObjectHandler: CRITICAL - versionId is EMPTY for versioned bucket %s, object %s", bucket, object)
}
// Set ETag in response
setEtag(w, etag)
// Set SSE response headers for versioned objects
s3a.setSSEResponseHeaders(w, r, sseMetadata)
case s3_constants.VersioningSuspended:
// Handle suspended versioning - overwrite with "null" version ID but preserve existing versions
etag, errCode := s3a.putSuspendedVersioningObject(r, bucket, object, dataReader, objectContentType)
glog.V(3).Infof("PutObjectHandler: SUSPENDED versioning detected for %s/%s, calling putSuspendedVersioningObject", bucket, object)
etag, errCode, sseMetadata := s3a.putSuspendedVersioningObject(r, bucket, object, dataReader, objectContentType)
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
@@ -191,6 +206,9 @@ func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request)
// Set ETag in response
setEtag(w, etag)
// Set SSE response headers for suspended versioning
s3a.setSSEResponseHeaders(w, r, sseMetadata)
default:
// Handle regular PUT (never configured versioning)
uploadUrl := s3a.toFilerUrl(bucket, object)
@@ -198,7 +216,7 @@ func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request)
dataReader = mimeDetect(r, dataReader)
}
etag, errCode, sseType := s3a.putToFiler(r, uploadUrl, dataReader, "", bucket, 1)
etag, errCode, sseMetadata := s3a.putToFiler(r, uploadUrl, dataReader, bucket, 1)
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
@@ -209,9 +227,7 @@ func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request)
setEtag(w, etag)
// Set SSE response headers based on encryption type used
if sseType == s3_constants.SSETypeS3 {
w.Header().Set(s3_constants.AmzServerSideEncryption, s3_constants.SSEAlgorithmAES256)
}
s3a.setSSEResponseHeaders(w, r, sseMetadata)
}
}
stats_collect.RecordBucketActiveTime(bucket)
@@ -220,15 +236,18 @@ func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request)
writeSuccessResponseEmpty(w, r)
}
func (s3a *S3ApiServer) putToFiler(r *http.Request, uploadUrl string, dataReader io.Reader, destination string, bucket string, partNumber int) (etag string, code s3err.ErrorCode, sseType string) {
// Calculate unique offset for each part to prevent IV reuse in multipart uploads
// This is critical for CTR mode encryption security
partOffset := calculatePartOffset(partNumber)
func (s3a *S3ApiServer) putToFiler(r *http.Request, uploadUrl string, dataReader io.Reader, bucket string, partNumber int) (etag string, code s3err.ErrorCode, sseMetadata SSEResponseMetadata) {
// NEW OPTIMIZATION: Write directly to volume servers, bypassing filer proxy
// This eliminates the filer proxy overhead for PUT operations
// Handle all SSE encryption types in a unified manner to eliminate repetitive dataReader assignments
// For SSE, encrypt with offset=0 for all parts
// Each part is encrypted independently, then decrypted using metadata during GET
partOffset := int64(0)
// Handle all SSE encryption types in a unified manner
sseResult, sseErrorCode := s3a.handleAllSSEEncryption(r, dataReader, partOffset)
if sseErrorCode != s3err.ErrNone {
return "", sseErrorCode, ""
return "", sseErrorCode, SSEResponseMetadata{}
}
// Extract results from unified SSE handling
@@ -239,6 +258,7 @@ func (s3a *S3ApiServer) putToFiler(r *http.Request, uploadUrl string, dataReader
sseKMSMetadata := sseResult.SSEKMSMetadata
sseS3Key := sseResult.SSES3Key
sseS3Metadata := sseResult.SSES3Metadata
sseType := sseResult.SSEType
// Apply bucket default encryption if no explicit encryption was provided
// This implements AWS S3 behavior where bucket default encryption automatically applies
@@ -249,7 +269,7 @@ func (s3a *S3ApiServer) putToFiler(r *http.Request, uploadUrl string, dataReader
encryptionResult, applyErr := s3a.applyBucketDefaultEncryption(bucket, r, dataReader)
if applyErr != nil {
glog.Errorf("Failed to apply bucket default encryption: %v", applyErr)
return "", s3err.ErrInternalError, ""
return "", s3err.ErrInternalError, SSEResponseMetadata{}
}
// Update variables based on the result
@@ -257,121 +277,357 @@ func (s3a *S3ApiServer) putToFiler(r *http.Request, uploadUrl string, dataReader
sseS3Key = encryptionResult.SSES3Key
sseKMSKey = encryptionResult.SSEKMSKey
// If bucket-default encryption selected an algorithm, reflect it in SSE type
if sseType == "" {
if sseS3Key != nil {
sseType = s3_constants.SSETypeS3
} else if sseKMSKey != nil {
sseType = s3_constants.SSETypeKMS
}
}
// If SSE-S3 was applied by bucket default, prepare metadata (if not already done)
if sseS3Key != nil && len(sseS3Metadata) == 0 {
var metaErr error
sseS3Metadata, metaErr = SerializeSSES3Metadata(sseS3Key)
if metaErr != nil {
glog.Errorf("Failed to serialize SSE-S3 metadata for bucket default encryption: %v", metaErr)
return "", s3err.ErrInternalError, ""
return "", s3err.ErrInternalError, SSEResponseMetadata{}
}
}
} else {
glog.V(4).Infof("putToFiler: explicit encryption already applied, skipping bucket default encryption")
}
hash := md5.New()
var body = io.TeeReader(dataReader, hash)
proxyReq, err := http.NewRequest(http.MethodPut, uploadUrl, body)
if err != nil {
glog.Errorf("NewRequest %s: %v", uploadUrl, err)
return "", s3err.ErrInternalError, ""
// Parse the upload URL to extract the file path
// uploadUrl format: http://filer:8888/path/to/bucket/object (or https://, IPv6, etc.)
// Use proper URL parsing instead of string manipulation for robustness
parsedUrl, parseErr := url.Parse(uploadUrl)
if parseErr != nil {
glog.Errorf("putToFiler: failed to parse uploadUrl %q: %v", uploadUrl, parseErr)
return "", s3err.ErrInternalError, SSEResponseMetadata{}
}
proxyReq.Header.Set("X-Forwarded-For", r.RemoteAddr)
if destination != "" {
proxyReq.Header.Set(s3_constants.SeaweedStorageDestinationHeader, destination)
}
// Use parsedUrl.Path directly - it's already decoded by url.Parse()
// Per Go documentation: "Path is stored in decoded form: /%47%6f%2f becomes /Go/"
// Calling PathUnescape again would double-decode and fail on keys like "b%ar"
filePath := parsedUrl.Path
// Step 1 & 2: Use auto-chunking to handle large files without OOM
// This splits large uploads into 8MB chunks, preventing memory issues on both S3 API and volume servers
const chunkSize = 8 * 1024 * 1024 // 8MB chunks (S3 standard)
const smallFileLimit = 256 * 1024 // 256KB - store inline in filer
collection := ""
if s3a.option.FilerGroup != "" {
query := proxyReq.URL.Query()
query.Add("collection", s3a.getCollectionName(bucket))
proxyReq.URL.RawQuery = query.Encode()
collection = s3a.getCollectionName(bucket)
}
for header, values := range r.Header {
for _, value := range values {
proxyReq.Header.Add(header, value)
// Create assign function for chunked upload
assignFunc := func(ctx context.Context, count int) (*operation.VolumeAssignRequest, *operation.AssignResult, error) {
var assignResult *filer_pb.AssignVolumeResponse
err := s3a.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
resp, err := client.AssignVolume(ctx, &filer_pb.AssignVolumeRequest{
Count: int32(count),
Replication: "",
Collection: collection,
DiskType: "",
DataCenter: s3a.option.DataCenter,
Path: filePath,
})
if err != nil {
return fmt.Errorf("assign volume: %w", err)
}
if resp.Error != "" {
return fmt.Errorf("assign volume: %v", resp.Error)
}
assignResult = resp
return nil
})
if err != nil {
return nil, nil, err
}
// Convert filer_pb.AssignVolumeResponse to operation.AssignResult
return nil, &operation.AssignResult{
Fid: assignResult.FileId,
Url: assignResult.Location.Url,
PublicUrl: assignResult.Location.PublicUrl,
Count: uint64(count),
Auth: security.EncodedJwt(assignResult.Auth),
}, nil
}
// Upload with auto-chunking
// Use context.Background() to ensure chunk uploads complete even if HTTP request is cancelled
// This prevents partial uploads and data corruption
chunkResult, err := operation.UploadReaderInChunks(context.Background(), dataReader, &operation.ChunkedUploadOption{
ChunkSize: chunkSize,
SmallFileLimit: smallFileLimit,
Collection: collection,
DataCenter: s3a.option.DataCenter,
SaveSmallInline: false, // S3 API always creates chunks, never stores inline
MimeType: r.Header.Get("Content-Type"),
AssignFunc: assignFunc,
})
if err != nil {
glog.Errorf("putToFiler: chunked upload failed: %v", err)
// CRITICAL: Cleanup orphaned chunks before returning error
// UploadReaderInChunks now returns partial results even on error,
// allowing us to cleanup any chunks that were successfully uploaded
// before the failure occurred
if chunkResult != nil && len(chunkResult.FileChunks) > 0 {
glog.Warningf("putToFiler: Upload failed, attempting to cleanup %d orphaned chunks", len(chunkResult.FileChunks))
s3a.deleteOrphanedChunks(chunkResult.FileChunks)
}
if strings.Contains(err.Error(), s3err.ErrMsgPayloadChecksumMismatch) {
return "", s3err.ErrInvalidDigest, SSEResponseMetadata{}
}
return "", s3err.ErrInternalError, SSEResponseMetadata{}
}
// Step 3: Calculate MD5 hash and add SSE metadata to chunks
md5Sum := chunkResult.Md5Hash.Sum(nil)
glog.V(4).Infof("putToFiler: Chunked upload SUCCESS - path=%s, chunks=%d, size=%d",
filePath, len(chunkResult.FileChunks), chunkResult.TotalSize)
// Log chunk details for debugging (verbose only - high frequency)
if glog.V(4) {
for i, chunk := range chunkResult.FileChunks {
glog.Infof(" PUT Chunk[%d]: fid=%s, offset=%d, size=%d", i, chunk.GetFileIdString(), chunk.Offset, chunk.Size)
}
}
// Log version ID header for debugging
if versionIdHeader := proxyReq.Header.Get(s3_constants.ExtVersionIdKey); versionIdHeader != "" {
glog.V(0).Infof("putToFiler: version ID header set: %s=%s for %s", s3_constants.ExtVersionIdKey, versionIdHeader, uploadUrl)
// Add SSE metadata to all chunks if present
for _, chunk := range chunkResult.FileChunks {
switch {
case customerKey != nil:
// SSE-C: Create per-chunk metadata (matches filer logic)
chunk.SseType = filer_pb.SSEType_SSE_C
if len(sseIV) > 0 {
// PartOffset tracks position within the encrypted stream
// Since ALL uploads (single-part and multipart parts) encrypt starting from offset 0,
// PartOffset = chunk.Offset represents where this chunk is in that encrypted stream
// - Single-part: chunk.Offset is position in the file's encrypted stream
// - Multipart: chunk.Offset is position in this part's encrypted stream
ssecMetadataStruct := struct {
Algorithm string `json:"algorithm"`
IV string `json:"iv"`
KeyMD5 string `json:"keyMD5"`
PartOffset int64 `json:"partOffset"`
}{
Algorithm: "AES256",
IV: base64.StdEncoding.EncodeToString(sseIV),
KeyMD5: customerKey.KeyMD5,
PartOffset: chunk.Offset, // Position within the encrypted stream (always encrypted from 0)
}
if ssecMetadata, serErr := json.Marshal(ssecMetadataStruct); serErr == nil {
chunk.SseMetadata = ssecMetadata
}
}
case sseKMSKey != nil:
// SSE-KMS: Create per-chunk metadata with chunk-specific offsets
// Each chunk needs its own metadata with ChunkOffset set for proper IV calculation during decryption
chunk.SseType = filer_pb.SSEType_SSE_KMS
// Create a copy of the SSE-KMS key with chunk-specific offset
chunkSSEKey := &SSEKMSKey{
KeyID: sseKMSKey.KeyID,
EncryptedDataKey: sseKMSKey.EncryptedDataKey,
EncryptionContext: sseKMSKey.EncryptionContext,
BucketKeyEnabled: sseKMSKey.BucketKeyEnabled,
IV: sseKMSKey.IV,
ChunkOffset: chunk.Offset, // Set chunk-specific offset for IV calculation
}
// Serialize per-chunk metadata
if chunkMetadata, serErr := SerializeSSEKMSMetadata(chunkSSEKey); serErr == nil {
chunk.SseMetadata = chunkMetadata
} else {
glog.Errorf("Failed to serialize SSE-KMS metadata for chunk at offset %d: %v", chunk.Offset, serErr)
}
case sseS3Key != nil:
// SSE-S3: Create per-chunk metadata with chunk-specific IVs
// Each chunk needs its own IV calculated from the base IV + chunk offset
chunk.SseType = filer_pb.SSEType_SSE_S3
// Calculate chunk-specific IV using base IV and chunk offset
chunkIV, _ := calculateIVWithOffset(sseS3Key.IV, chunk.Offset)
// Create a copy of the SSE-S3 key with chunk-specific IV
chunkSSEKey := &SSES3Key{
Key: sseS3Key.Key,
KeyID: sseS3Key.KeyID,
Algorithm: sseS3Key.Algorithm,
IV: chunkIV, // Use chunk-specific IV
}
// Serialize per-chunk metadata
if chunkMetadata, serErr := SerializeSSES3Metadata(chunkSSEKey); serErr == nil {
chunk.SseMetadata = chunkMetadata
} else {
glog.Errorf("Failed to serialize SSE-S3 metadata for chunk at offset %d: %v", chunk.Offset, serErr)
}
}
}
// Set object owner header for filer to extract
// Step 4: Create metadata entry
now := time.Now()
mimeType := r.Header.Get("Content-Type")
if mimeType == "" {
mimeType = "application/octet-stream"
}
// Create entry
entry := &filer_pb.Entry{
Name: filepath.Base(filePath),
IsDirectory: false,
Attributes: &filer_pb.FuseAttributes{
Crtime: now.Unix(),
Mtime: now.Unix(),
FileMode: 0660,
Uid: 0,
Gid: 0,
Mime: mimeType,
FileSize: uint64(chunkResult.TotalSize),
},
Chunks: chunkResult.FileChunks, // All chunks from auto-chunking
Extended: make(map[string][]byte),
}
// Set Md5 attribute based on context:
// 1. For multipart upload PARTS (stored in .uploads/ directory): ALWAYS set Md5
// - Parts must use simple MD5 ETags, never composite format
// - Even if a part has multiple chunks internally, its ETag is MD5 of entire part
// 2. For regular object uploads: only set Md5 for single-chunk uploads
// - Multi-chunk regular objects use composite "md5-count" format
isMultipartPart := strings.Contains(filePath, "/"+s3_constants.MultipartUploadsFolder+"/")
if isMultipartPart || len(chunkResult.FileChunks) == 1 {
entry.Attributes.Md5 = md5Sum
}
// Calculate ETag using the same logic as GET to ensure consistency
// For single chunk: uses entry.Attributes.Md5
// For multiple chunks: uses filer.ETagChunks() which returns "<hash>-<count>"
etag = filer.ETag(entry)
glog.V(4).Infof("putToFiler: Calculated ETag=%s for %d chunks", etag, len(chunkResult.FileChunks))
// Set object owner
amzAccountId := r.Header.Get(s3_constants.AmzAccountId)
if amzAccountId != "" {
proxyReq.Header.Set(s3_constants.ExtAmzOwnerKey, amzAccountId)
glog.V(2).Infof("putToFiler: setting owner header %s for object %s", amzAccountId, uploadUrl)
entry.Extended[s3_constants.ExtAmzOwnerKey] = []byte(amzAccountId)
glog.V(2).Infof("putToFiler: setting owner %s for object %s", amzAccountId, filePath)
}
// Set SSE-C metadata headers for the filer if encryption was applied
if customerKey != nil && len(sseIV) > 0 {
proxyReq.Header.Set(s3_constants.AmzServerSideEncryptionCustomerAlgorithm, "AES256")
proxyReq.Header.Set(s3_constants.AmzServerSideEncryptionCustomerKeyMD5, customerKey.KeyMD5)
// Store IV in a custom header that the filer can use to store in entry metadata
proxyReq.Header.Set(s3_constants.SeaweedFSSSEIVHeader, base64.StdEncoding.EncodeToString(sseIV))
// Set version ID if present
if versionIdHeader := r.Header.Get(s3_constants.ExtVersionIdKey); versionIdHeader != "" {
entry.Extended[s3_constants.ExtVersionIdKey] = []byte(versionIdHeader)
glog.V(3).Infof("putToFiler: setting version ID %s for object %s", versionIdHeader, filePath)
}
// Set SSE-KMS metadata headers for the filer if KMS encryption was applied
if sseKMSKey != nil {
// Use already-serialized SSE-KMS metadata from helper function
// Store serialized KMS metadata in a custom header that the filer can use
proxyReq.Header.Set(s3_constants.SeaweedFSSSEKMSKeyHeader, base64.StdEncoding.EncodeToString(sseKMSMetadata))
glog.V(3).Infof("putToFiler: storing SSE-KMS metadata for object %s with keyID %s", uploadUrl, sseKMSKey.KeyID)
} else {
glog.V(4).Infof("putToFiler: no SSE-KMS encryption detected")
// Set TTL-based S3 expiry flag only if object has a TTL
if entry.Attributes.TtlSec > 0 {
entry.Extended[s3_constants.SeaweedFSExpiresS3] = []byte("true")
}
// Set SSE-S3 metadata headers for the filer if S3 encryption was applied
if sseS3Key != nil && len(sseS3Metadata) > 0 {
// Store serialized S3 metadata in a custom header that the filer can use
proxyReq.Header.Set(s3_constants.SeaweedFSSSES3Key, base64.StdEncoding.EncodeToString(sseS3Metadata))
glog.V(3).Infof("putToFiler: storing SSE-S3 metadata for object %s with keyID %s", uploadUrl, sseS3Key.KeyID)
}
// Set TTL-based S3 expiry (modification time)
proxyReq.Header.Set(s3_constants.SeaweedFSExpiresS3, "true")
// ensure that the Authorization header is overriding any previous
// Authorization header which might be already present in proxyReq
s3a.maybeAddFilerJwtAuthorization(proxyReq, true)
resp, postErr := s3a.client.Do(proxyReq)
if postErr != nil {
glog.Errorf("post to filer: %v", postErr)
if strings.Contains(postErr.Error(), s3err.ErrMsgPayloadChecksumMismatch) {
return "", s3err.ErrInvalidDigest, ""
// Copy user metadata and standard headers
for k, v := range r.Header {
if len(v) > 0 && len(v[0]) > 0 {
if strings.HasPrefix(k, s3_constants.AmzUserMetaPrefix) {
// Go's HTTP server canonicalizes headers (e.g., x-amz-meta-foo → X-Amz-Meta-Foo)
// We store them as they come in (after canonicalization) to preserve the user's intent
entry.Extended[k] = []byte(v[0])
} else if k == "Cache-Control" || k == "Expires" || k == "Content-Disposition" {
entry.Extended[k] = []byte(v[0])
}
if k == "Response-Content-Disposition" {
entry.Extended["Content-Disposition"] = []byte(v[0])
}
}
return "", s3err.ErrInternalError, ""
}
defer resp.Body.Close()
etag = fmt.Sprintf("%x", hash.Sum(nil))
resp_body, ra_err := io.ReadAll(resp.Body)
if ra_err != nil {
glog.Errorf("upload to filer response read %d: %v", resp.StatusCode, ra_err)
return etag, s3err.ErrInternalError, ""
}
var ret weed_server.FilerPostResult
unmarshal_err := json.Unmarshal(resp_body, &ret)
if unmarshal_err != nil {
glog.Errorf("failing to read upload to %s : %v", uploadUrl, string(resp_body))
return "", s3err.ErrInternalError, ""
}
if ret.Error != "" {
glog.Errorf("upload to filer error: %v", ret.Error)
return "", filerErrorToS3Error(ret.Error), ""
}
BucketTrafficReceived(ret.Size, r)
// Set SSE-C metadata
if customerKey != nil && len(sseIV) > 0 {
// Store IV as RAW bytes (matches filer behavior - filer decodes base64 headers and stores raw bytes)
entry.Extended[s3_constants.SeaweedFSSSEIV] = sseIV
entry.Extended[s3_constants.AmzServerSideEncryptionCustomerAlgorithm] = []byte("AES256")
entry.Extended[s3_constants.AmzServerSideEncryptionCustomerKeyMD5] = []byte(customerKey.KeyMD5)
glog.V(3).Infof("putToFiler: storing SSE-C metadata - IV len=%d", len(sseIV))
}
// Return the SSE type determined by the unified handler
return etag, s3err.ErrNone, sseResult.SSEType
// Set SSE-KMS metadata
if sseKMSKey != nil {
// Store metadata as RAW bytes (matches filer behavior - filer decodes base64 headers and stores raw bytes)
entry.Extended[s3_constants.SeaweedFSSSEKMSKey] = sseKMSMetadata
// Set standard SSE headers for detection
entry.Extended[s3_constants.AmzServerSideEncryption] = []byte("aws:kms")
entry.Extended[s3_constants.AmzServerSideEncryptionAwsKmsKeyId] = []byte(sseKMSKey.KeyID)
glog.V(3).Infof("putToFiler: storing SSE-KMS metadata - keyID=%s, raw len=%d", sseKMSKey.KeyID, len(sseKMSMetadata))
}
// Set SSE-S3 metadata
if sseS3Key != nil && len(sseS3Metadata) > 0 {
// Store metadata as RAW bytes (matches filer behavior - filer decodes base64 headers and stores raw bytes)
entry.Extended[s3_constants.SeaweedFSSSES3Key] = sseS3Metadata
// Set standard SSE header for detection
entry.Extended[s3_constants.AmzServerSideEncryption] = []byte("AES256")
glog.V(3).Infof("putToFiler: storing SSE-S3 metadata - keyID=%s, raw len=%d", sseS3Key.KeyID, len(sseS3Metadata))
}
// Step 4: Save metadata to filer via gRPC
// Use context.Background() to ensure metadata save completes even if HTTP request is cancelled
// This matches the chunk upload behavior and prevents orphaned chunks
glog.V(3).Infof("putToFiler: About to create entry - dir=%s, name=%s, chunks=%d, extended keys=%d",
filepath.Dir(filePath), filepath.Base(filePath), len(entry.Chunks), len(entry.Extended))
createErr := s3a.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
req := &filer_pb.CreateEntryRequest{
Directory: filepath.Dir(filePath),
Entry: entry,
}
glog.V(3).Infof("putToFiler: Calling CreateEntry for %s", filePath)
_, err := client.CreateEntry(context.Background(), req)
if err != nil {
glog.Errorf("putToFiler: CreateEntry returned error: %v", err)
}
return err
})
if createErr != nil {
glog.Errorf("putToFiler: failed to create entry for %s: %v", filePath, createErr)
// CRITICAL: Cleanup orphaned chunks before returning error
// If CreateEntry fails, the uploaded chunks are orphaned and must be deleted
// to prevent resource leaks and wasted storage
if len(chunkResult.FileChunks) > 0 {
glog.Warningf("putToFiler: CreateEntry failed, attempting to cleanup %d orphaned chunks", len(chunkResult.FileChunks))
s3a.deleteOrphanedChunks(chunkResult.FileChunks)
}
return "", filerErrorToS3Error(createErr.Error()), SSEResponseMetadata{}
}
glog.V(3).Infof("putToFiler: CreateEntry SUCCESS for %s", filePath)
glog.V(2).Infof("putToFiler: Metadata saved SUCCESS - path=%s, etag(hex)=%s, size=%d, partNumber=%d",
filePath, etag, entry.Attributes.FileSize, partNumber)
BucketTrafficReceived(chunkResult.TotalSize, r)
// Build SSE response metadata with encryption details
responseMetadata := SSEResponseMetadata{
SSEType: sseType,
}
// For SSE-KMS, include key ID and bucket-key-enabled flag from stored metadata
if sseKMSKey != nil {
responseMetadata.KMSKeyID = sseKMSKey.KeyID
responseMetadata.BucketKeyEnabled = sseKMSKey.BucketKeyEnabled
glog.V(4).Infof("putToFiler: returning SSE-KMS metadata - keyID=%s, bucketKeyEnabled=%v",
sseKMSKey.KeyID, sseKMSKey.BucketKeyEnabled)
}
return etag, s3err.ErrNone, responseMetadata
}
func setEtag(w http.ResponseWriter, etag string) {
@@ -384,6 +640,43 @@ func setEtag(w http.ResponseWriter, etag string) {
}
}
// setSSEResponseHeaders sets appropriate SSE response headers based on encryption type
func (s3a *S3ApiServer) setSSEResponseHeaders(w http.ResponseWriter, r *http.Request, sseMetadata SSEResponseMetadata) {
switch sseMetadata.SSEType {
case s3_constants.SSETypeS3:
// SSE-S3: Return the encryption algorithm
w.Header().Set(s3_constants.AmzServerSideEncryption, s3_constants.SSEAlgorithmAES256)
case s3_constants.SSETypeC:
// SSE-C: Echo back the customer-provided algorithm and key MD5
if algo := r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerAlgorithm); algo != "" {
w.Header().Set(s3_constants.AmzServerSideEncryptionCustomerAlgorithm, algo)
}
if keyMD5 := r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerKeyMD5); keyMD5 != "" {
w.Header().Set(s3_constants.AmzServerSideEncryptionCustomerKeyMD5, keyMD5)
}
case s3_constants.SSETypeKMS:
// SSE-KMS: Return the KMS key ID and algorithm
w.Header().Set(s3_constants.AmzServerSideEncryption, "aws:kms")
// Use metadata from stored encryption config (for bucket-default encryption)
// or fall back to request headers (for explicit encryption)
if sseMetadata.KMSKeyID != "" {
w.Header().Set(s3_constants.AmzServerSideEncryptionAwsKmsKeyId, sseMetadata.KMSKeyID)
} else if keyID := r.Header.Get(s3_constants.AmzServerSideEncryptionAwsKmsKeyId); keyID != "" {
w.Header().Set(s3_constants.AmzServerSideEncryptionAwsKmsKeyId, keyID)
}
// Set bucket-key-enabled header if it was enabled
if sseMetadata.BucketKeyEnabled {
w.Header().Set(s3_constants.AmzServerSideEncryptionBucketKeyEnabled, "true")
} else if bucketKeyEnabled := r.Header.Get(s3_constants.AmzServerSideEncryptionBucketKeyEnabled); bucketKeyEnabled == "true" {
w.Header().Set(s3_constants.AmzServerSideEncryptionBucketKeyEnabled, "true")
}
}
}
func filerErrorToS3Error(errString string) s3err.ErrorCode {
switch {
case errString == constants.ErrMsgBadDigest:
@@ -400,26 +693,6 @@ func filerErrorToS3Error(errString string) s3err.ErrorCode {
}
}
func (s3a *S3ApiServer) maybeAddFilerJwtAuthorization(r *http.Request, isWrite bool) {
encodedJwt := s3a.maybeGetFilerJwtAuthorizationToken(isWrite)
if encodedJwt == "" {
return
}
r.Header.Set("Authorization", "BEARER "+string(encodedJwt))
}
func (s3a *S3ApiServer) maybeGetFilerJwtAuthorizationToken(isWrite bool) string {
var encodedJwt security.EncodedJwt
if isWrite {
encodedJwt = security.GenJwtForFilerServer(s3a.filerGuard.SigningKey, s3a.filerGuard.ExpiresAfterSec)
} else {
encodedJwt = security.GenJwtForFilerServer(s3a.filerGuard.ReadSigningKey, s3a.filerGuard.ReadExpiresAfterSec)
}
return string(encodedJwt)
}
// setObjectOwnerFromRequest sets the object owner metadata based on the authenticated user
func (s3a *S3ApiServer) setObjectOwnerFromRequest(r *http.Request, entry *filer_pb.Entry) {
amzAccountId := r.Header.Get(s3_constants.AmzAccountId)
@@ -446,19 +719,12 @@ func (s3a *S3ApiServer) setObjectOwnerFromRequest(r *http.Request, entry *filer_
//
// For suspended versioning, objects are stored as regular files (version ID "null") in the bucket directory,
// while existing versions from when versioning was enabled remain preserved in the .versions subdirectory.
func (s3a *S3ApiServer) putSuspendedVersioningObject(r *http.Request, bucket, object string, dataReader io.Reader, objectContentType string) (etag string, errCode s3err.ErrorCode) {
func (s3a *S3ApiServer) putSuspendedVersioningObject(r *http.Request, bucket, object string, dataReader io.Reader, objectContentType string) (etag string, errCode s3err.ErrorCode, sseMetadata SSEResponseMetadata) {
// Normalize object path to ensure consistency with toFilerUrl behavior
normalizedObject := removeDuplicateSlashes(object)
// Enable detailed logging for testobjbar
isTestObj := (normalizedObject == "testobjbar")
glog.V(0).Infof("putSuspendedVersioningObject: START bucket=%s, object=%s, normalized=%s, isTestObj=%v",
bucket, object, normalizedObject, isTestObj)
if isTestObj {
glog.V(0).Infof("=== TESTOBJBAR: putSuspendedVersioningObject START ===")
}
glog.V(3).Infof("putSuspendedVersioningObject: START bucket=%s, object=%s, normalized=%s",
bucket, object, normalizedObject)
bucketDir := s3a.option.BucketsPath + "/" + bucket
@@ -470,20 +736,20 @@ func (s3a *S3ApiServer) putSuspendedVersioningObject(r *http.Request, bucket, ob
entries, _, err := s3a.list(versionsDir, "", "", false, 1000)
if err == nil {
// .versions directory exists
glog.V(0).Infof("putSuspendedVersioningObject: found %d entries in .versions for %s/%s", len(entries), bucket, object)
glog.V(3).Infof("putSuspendedVersioningObject: found %d entries in .versions for %s/%s", len(entries), bucket, object)
for _, entry := range entries {
if entry.Extended != nil {
if versionIdBytes, ok := entry.Extended[s3_constants.ExtVersionIdKey]; ok {
versionId := string(versionIdBytes)
glog.V(0).Infof("putSuspendedVersioningObject: found version '%s' in .versions", versionId)
glog.V(3).Infof("putSuspendedVersioningObject: found version '%s' in .versions", versionId)
if versionId == "null" {
// Only delete null version - preserve real versioned entries
glog.V(0).Infof("putSuspendedVersioningObject: deleting null version from .versions")
glog.V(3).Infof("putSuspendedVersioningObject: deleting null version from .versions")
err := s3a.rm(versionsDir, entry.Name, true, false)
if err != nil {
glog.Warningf("putSuspendedVersioningObject: failed to delete null version: %v", err)
} else {
glog.V(0).Infof("putSuspendedVersioningObject: successfully deleted null version")
glog.V(3).Infof("putSuspendedVersioningObject: successfully deleted null version")
}
break
}
@@ -491,13 +757,12 @@ func (s3a *S3ApiServer) putSuspendedVersioningObject(r *http.Request, bucket, ob
}
}
} else {
glog.V(0).Infof("putSuspendedVersioningObject: no .versions directory for %s/%s", bucket, object)
glog.V(3).Infof("putSuspendedVersioningObject: no .versions directory for %s/%s", bucket, object)
}
uploadUrl := s3a.toFilerUrl(bucket, normalizedObject)
hash := md5.New()
var body = io.TeeReader(dataReader, hash)
body := dataReader
if objectContentType == "" {
body = mimeDetect(r, body)
}
@@ -508,10 +773,6 @@ func (s3a *S3ApiServer) putSuspendedVersioningObject(r *http.Request, bucket, ob
// Set version ID to "null" for suspended versioning
r.Header.Set(s3_constants.ExtVersionIdKey, "null")
if isTestObj {
glog.V(0).Infof("=== TESTOBJBAR: set version header before putToFiler, r.Header[%s]=%s ===",
s3_constants.ExtVersionIdKey, r.Header.Get(s3_constants.ExtVersionIdKey))
}
// Extract and set object lock metadata as headers
// This handles retention mode, retention date, and legal hold
@@ -528,7 +789,7 @@ func (s3a *S3ApiServer) putSuspendedVersioningObject(r *http.Request, bucket, ob
parsedTime, err := time.Parse(time.RFC3339, explicitRetainUntilDate)
if err != nil {
glog.Errorf("putSuspendedVersioningObject: failed to parse retention until date: %v", err)
return "", s3err.ErrInvalidRequest
return "", s3err.ErrInvalidRequest, SSEResponseMetadata{}
}
r.Header.Set(s3_constants.ExtRetentionUntilDateKey, strconv.FormatInt(parsedTime.Unix(), 10))
glog.V(2).Infof("putSuspendedVersioningObject: setting retention until date header (timestamp: %d)", parsedTime.Unix())
@@ -540,7 +801,7 @@ func (s3a *S3ApiServer) putSuspendedVersioningObject(r *http.Request, bucket, ob
glog.V(2).Infof("putSuspendedVersioningObject: setting legal hold header: %s", legalHold)
} else {
glog.Errorf("putSuspendedVersioningObject: invalid legal hold value: %s", legalHold)
return "", s3err.ErrInvalidRequest
return "", s3err.ErrInvalidRequest, SSEResponseMetadata{}
}
}
@@ -562,43 +823,10 @@ func (s3a *S3ApiServer) putSuspendedVersioningObject(r *http.Request, bucket, ob
}
// Upload the file using putToFiler - this will create the file with version metadata
if isTestObj {
glog.V(0).Infof("=== TESTOBJBAR: calling putToFiler ===")
}
etag, errCode, _ = s3a.putToFiler(r, uploadUrl, body, "", bucket, 1)
etag, errCode, sseMetadata = s3a.putToFiler(r, uploadUrl, body, bucket, 1)
if errCode != s3err.ErrNone {
glog.Errorf("putSuspendedVersioningObject: failed to upload object: %v", errCode)
return "", errCode
}
if isTestObj {
glog.V(0).Infof("=== TESTOBJBAR: putToFiler completed, etag=%s ===", etag)
}
// Verify the metadata was set correctly during file creation
if isTestObj {
// Read back the entry to verify
maxRetries := 3
for attempt := 1; attempt <= maxRetries; attempt++ {
verifyEntry, verifyErr := s3a.getEntry(bucketDir, normalizedObject)
if verifyErr == nil {
glog.V(0).Infof("=== TESTOBJBAR: verify attempt %d, entry.Extended=%v ===", attempt, verifyEntry.Extended)
if verifyEntry.Extended != nil {
if versionIdBytes, ok := verifyEntry.Extended[s3_constants.ExtVersionIdKey]; ok {
glog.V(0).Infof("=== TESTOBJBAR: verification SUCCESSFUL, version=%s ===", string(versionIdBytes))
} else {
glog.V(0).Infof("=== TESTOBJBAR: verification FAILED, ExtVersionIdKey not found ===")
}
} else {
glog.V(0).Infof("=== TESTOBJBAR: verification FAILED, Extended is nil ===")
}
break
} else {
glog.V(0).Infof("=== TESTOBJBAR: getEntry failed on attempt %d: %v ===", attempt, verifyErr)
}
if attempt < maxRetries {
time.Sleep(time.Millisecond * 10)
}
}
return "", errCode, SSEResponseMetadata{}
}
// Update all existing versions/delete markers to set IsLatest=false since "null" is now latest
@@ -609,10 +837,8 @@ func (s3a *S3ApiServer) putSuspendedVersioningObject(r *http.Request, bucket, ob
}
glog.V(2).Infof("putSuspendedVersioningObject: successfully created null version for %s/%s", bucket, object)
if isTestObj {
glog.V(0).Infof("=== TESTOBJBAR: putSuspendedVersioningObject COMPLETED ===")
}
return etag, s3err.ErrNone
return etag, s3err.ErrNone, sseMetadata
}
// updateIsLatestFlagsForSuspendedVersioning sets IsLatest=false on all existing versions/delete markers
@@ -684,7 +910,7 @@ func (s3a *S3ApiServer) updateIsLatestFlagsForSuspendedVersioning(bucket, object
return nil
}
func (s3a *S3ApiServer) putVersionedObject(r *http.Request, bucket, object string, dataReader io.Reader, objectContentType string) (versionId string, etag string, errCode s3err.ErrorCode) {
func (s3a *S3ApiServer) putVersionedObject(r *http.Request, bucket, object string, dataReader io.Reader, objectContentType string) (versionId string, etag string, errCode s3err.ErrorCode, sseMetadata SSEResponseMetadata) {
// Generate version ID
versionId = generateVersionId()
@@ -709,21 +935,20 @@ func (s3a *S3ApiServer) putVersionedObject(r *http.Request, bucket, object strin
})
if err != nil {
glog.Errorf("putVersionedObject: failed to create .versions directory: %v", err)
return "", "", s3err.ErrInternalError
return "", "", s3err.ErrInternalError, SSEResponseMetadata{}
}
hash := md5.New()
var body = io.TeeReader(dataReader, hash)
body := dataReader
if objectContentType == "" {
body = mimeDetect(r, body)
}
glog.V(2).Infof("putVersionedObject: uploading %s/%s version %s to %s", bucket, object, versionId, versionUploadUrl)
etag, errCode, _ = s3a.putToFiler(r, versionUploadUrl, body, "", bucket, 1)
etag, errCode, sseMetadata = s3a.putToFiler(r, versionUploadUrl, body, bucket, 1)
if errCode != s3err.ErrNone {
glog.Errorf("putVersionedObject: failed to upload version: %v", errCode)
return "", "", errCode
return "", "", errCode, SSEResponseMetadata{}
}
// Get the uploaded entry to add versioning metadata
@@ -745,7 +970,7 @@ func (s3a *S3ApiServer) putVersionedObject(r *http.Request, bucket, object strin
if err != nil {
glog.Errorf("putVersionedObject: failed to get version entry after %d attempts: %v", maxRetries, err)
return "", "", s3err.ErrInternalError
return "", "", s3err.ErrInternalError, SSEResponseMetadata{}
}
// Add versioning metadata to this version
@@ -766,7 +991,7 @@ func (s3a *S3ApiServer) putVersionedObject(r *http.Request, bucket, object strin
// Extract and store object lock metadata from request headers
if err := s3a.extractObjectLockMetadataFromRequest(r, versionEntry); err != nil {
glog.Errorf("putVersionedObject: failed to extract object lock metadata: %v", err)
return "", "", s3err.ErrInvalidRequest
return "", "", s3err.ErrInvalidRequest, SSEResponseMetadata{}
}
// Update the version entry with metadata
@@ -777,17 +1002,17 @@ func (s3a *S3ApiServer) putVersionedObject(r *http.Request, bucket, object strin
})
if err != nil {
glog.Errorf("putVersionedObject: failed to update version metadata: %v", err)
return "", "", s3err.ErrInternalError
return "", "", s3err.ErrInternalError, SSEResponseMetadata{}
}
// Update the .versions directory metadata to indicate this is the latest version
err = s3a.updateLatestVersionInDirectory(bucket, normalizedObject, versionId, versionFileName)
if err != nil {
glog.Errorf("putVersionedObject: failed to update latest version in directory: %v", err)
return "", "", s3err.ErrInternalError
return "", "", s3err.ErrInternalError, SSEResponseMetadata{}
}
glog.V(2).Infof("putVersionedObject: successfully created version %s for %s/%s (normalized: %s)", versionId, bucket, object, normalizedObject)
return versionId, etag, s3err.ErrNone
return versionId, etag, s3err.ErrNone, sseMetadata
}
// updateLatestVersionInDirectory updates the .versions directory metadata to indicate the latest version
@@ -897,7 +1122,16 @@ func (s3a *S3ApiServer) extractObjectLockMetadataFromRequest(r *http.Request, en
func (s3a *S3ApiServer) applyBucketDefaultEncryption(bucket string, r *http.Request, dataReader io.Reader) (*BucketDefaultEncryptionResult, error) {
// Check if bucket has default encryption configured
encryptionConfig, err := s3a.GetBucketEncryptionConfig(bucket)
if err != nil || encryptionConfig == nil {
if err != nil {
// Check if this is just "no encryption configured" vs a real error
if errors.Is(err, ErrNoEncryptionConfig) {
// No default encryption configured, return original reader
return &BucketDefaultEncryptionResult{DataReader: dataReader}, nil
}
// Real error - propagate to prevent silent encryption bypass
return nil, fmt.Errorf("failed to read bucket encryption config: %v", err)
}
if encryptionConfig == nil {
// No default encryption configured, return original reader
return &BucketDefaultEncryptionResult{DataReader: dataReader}, nil
}
@@ -963,7 +1197,8 @@ func (s3a *S3ApiServer) applySSEKMSDefaultEncryption(bucket string, r *http.Requ
bucketKeyEnabled := encryptionConfig.BucketKeyEnabled
// Build encryption context for KMS
bucket, object := s3_constants.GetBucketAndObject(r)
// Use bucket parameter passed to function (not from request parsing)
_, object := s3_constants.GetBucketAndObject(r)
encryptionContext := BuildEncryptionContext(bucket, object, bucketKeyEnabled)
// Create SSE-KMS encrypted reader
@@ -1474,3 +1709,88 @@ func (s3a *S3ApiServer) checkConditionalHeadersForReadsWithGetter(getter EntryGe
func (s3a *S3ApiServer) checkConditionalHeadersForReads(r *http.Request, bucket, object string) ConditionalHeaderResult {
return s3a.checkConditionalHeadersForReadsWithGetter(s3a, r, bucket, object)
}
// deleteOrphanedChunks attempts to delete chunks that were uploaded but whose entry creation failed
// This prevents resource leaks and wasted storage. Errors are logged but don't prevent cleanup attempts.
func (s3a *S3ApiServer) deleteOrphanedChunks(chunks []*filer_pb.FileChunk) {
if len(chunks) == 0 {
return
}
// Extract file IDs from chunks
var fileIds []string
for _, chunk := range chunks {
if chunk.GetFileIdString() != "" {
fileIds = append(fileIds, chunk.GetFileIdString())
}
}
if len(fileIds) == 0 {
glog.Warningf("deleteOrphanedChunks: no valid file IDs found in %d chunks", len(chunks))
return
}
glog.V(3).Infof("deleteOrphanedChunks: attempting to delete %d file IDs: %v", len(fileIds), fileIds)
// Create a lookup function that queries the filer for volume locations
// This is similar to createLookupFileIdFunction but returns the format needed by DeleteFileIdsWithLookupVolumeId
lookupFunc := func(vids []string) (map[string]*operation.LookupResult, error) {
results := make(map[string]*operation.LookupResult)
err := s3a.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
// Query filer for all volume IDs at once
resp, err := client.LookupVolume(context.Background(), &filer_pb.LookupVolumeRequest{
VolumeIds: vids,
})
if err != nil {
return err
}
// Convert filer response to operation.LookupResult format
for vid, locs := range resp.LocationsMap {
result := &operation.LookupResult{
VolumeOrFileId: vid,
}
for _, loc := range locs.Locations {
result.Locations = append(result.Locations, operation.Location{
Url: loc.Url,
PublicUrl: loc.PublicUrl,
DataCenter: loc.DataCenter,
GrpcPort: int(loc.GrpcPort),
})
}
results[vid] = result
}
return nil
})
return results, err
}
// Attempt deletion using the operation package's batch delete with custom lookup
deleteResults := operation.DeleteFileIdsWithLookupVolumeId(s3a.option.GrpcDialOption, fileIds, lookupFunc)
// Log results - track successes and failures
successCount := 0
failureCount := 0
for _, result := range deleteResults {
if result.Error != "" {
glog.Warningf("deleteOrphanedChunks: failed to delete chunk %s: %s (status: %d)",
result.FileId, result.Error, result.Status)
failureCount++
} else {
glog.V(4).Infof("deleteOrphanedChunks: successfully deleted chunk %s (size: %d bytes)",
result.FileId, result.Size)
successCount++
}
}
if failureCount > 0 {
glog.Warningf("deleteOrphanedChunks: cleanup completed with %d successes and %d failures out of %d chunks",
successCount, failureCount, len(fileIds))
} else {
glog.V(3).Infof("deleteOrphanedChunks: successfully deleted all %d orphaned chunks", successCount)
}
}

View File

@@ -147,3 +147,112 @@ func TestS3ApiServer_toFilerUrl(t *testing.T) {
})
}
}
func TestPartNumberWithRangeHeader(t *testing.T) {
tests := []struct {
name string
partStartOffset int64 // Part's start offset in the object
partEndOffset int64 // Part's end offset in the object
clientRangeHeader string
expectedStart int64 // Expected absolute start offset
expectedEnd int64 // Expected absolute end offset
expectError bool
}{
{
name: "No client range - full part",
partStartOffset: 1000,
partEndOffset: 1999,
clientRangeHeader: "",
expectedStart: 1000,
expectedEnd: 1999,
expectError: false,
},
{
name: "Range within part - start and end",
partStartOffset: 1000,
partEndOffset: 1999, // Part size: 1000 bytes
clientRangeHeader: "bytes=0-99",
expectedStart: 1000, // 1000 + 0
expectedEnd: 1099, // 1000 + 99
expectError: false,
},
{
name: "Range within part - start to end",
partStartOffset: 1000,
partEndOffset: 1999,
clientRangeHeader: "bytes=100-",
expectedStart: 1100, // 1000 + 100
expectedEnd: 1999, // 1000 + 999 (end of part)
expectError: false,
},
{
name: "Range suffix - last 100 bytes",
partStartOffset: 1000,
partEndOffset: 1999, // Part size: 1000 bytes
clientRangeHeader: "bytes=-100",
expectedStart: 1900, // 1000 + (1000 - 100)
expectedEnd: 1999, // 1000 + 999
expectError: false,
},
{
name: "Range suffix larger than part",
partStartOffset: 1000,
partEndOffset: 1999, // Part size: 1000 bytes
clientRangeHeader: "bytes=-2000",
expectedStart: 1000, // Start of part (clamped)
expectedEnd: 1999, // End of part
expectError: false,
},
{
name: "Range start beyond part size",
partStartOffset: 1000,
partEndOffset: 1999,
clientRangeHeader: "bytes=1000-1100",
expectedStart: 0,
expectedEnd: 0,
expectError: true,
},
{
name: "Range end clamped to part size",
partStartOffset: 1000,
partEndOffset: 1999,
clientRangeHeader: "bytes=0-2000",
expectedStart: 1000, // 1000 + 0
expectedEnd: 1999, // Clamped to end of part
expectError: false,
},
{
name: "Single byte range at start",
partStartOffset: 5000,
partEndOffset: 9999, // Part size: 5000 bytes
clientRangeHeader: "bytes=0-0",
expectedStart: 5000,
expectedEnd: 5000,
expectError: false,
},
{
name: "Single byte range in middle",
partStartOffset: 5000,
partEndOffset: 9999,
clientRangeHeader: "bytes=100-100",
expectedStart: 5100,
expectedEnd: 5100,
expectError: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Test the actual range adjustment logic from GetObjectHandler
startOffset, endOffset, err := adjustRangeForPart(tt.partStartOffset, tt.partEndOffset, tt.clientRangeHeader)
if tt.expectError {
assert.Error(t, err, "Expected error for range %s", tt.clientRangeHeader)
} else {
assert.NoError(t, err, "Unexpected error for range %s: %v", tt.clientRangeHeader, err)
assert.Equal(t, tt.expectedStart, startOffset, "Start offset mismatch")
assert.Equal(t, tt.expectedEnd, endOffset, "End offset mismatch")
}
})
}
}

View File

@@ -328,7 +328,7 @@ func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string
seenVersionIds[versionKey] = true
if version.IsDeleteMarker {
glog.V(0).Infof("Adding delete marker from .versions: objectKey=%s, versionId=%s, isLatest=%v, versionKey=%s",
glog.V(4).Infof("Adding delete marker from .versions: objectKey=%s, versionId=%s, isLatest=%v, versionKey=%s",
normalizedObjectKey, version.VersionId, version.IsLatest, versionKey)
deleteMarker := &DeleteMarkerEntry{
Key: normalizedObjectKey, // Use normalized key for consistency
@@ -339,7 +339,7 @@ func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string
}
*allVersions = append(*allVersions, deleteMarker)
} else {
glog.V(0).Infof("Adding version from .versions: objectKey=%s, versionId=%s, isLatest=%v, versionKey=%s",
glog.V(4).Infof("Adding version from .versions: objectKey=%s, versionId=%s, isLatest=%v, versionKey=%s",
normalizedObjectKey, version.VersionId, version.IsLatest, versionKey)
versionEntry := &VersionEntry{
Key: normalizedObjectKey, // Use normalized key for consistency
@@ -401,12 +401,12 @@ func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string
// Skip if this object already has a .versions directory (already processed)
// Check both normalized and original keys for backward compatibility
if processedObjects[objectKey] || processedObjects[normalizedObjectKey] {
glog.V(0).Infof("Skipping already processed object: objectKey=%s, normalizedObjectKey=%s, processedObjects[objectKey]=%v, processedObjects[normalizedObjectKey]=%v",
glog.V(4).Infof("Skipping already processed object: objectKey=%s, normalizedObjectKey=%s, processedObjects[objectKey]=%v, processedObjects[normalizedObjectKey]=%v",
objectKey, normalizedObjectKey, processedObjects[objectKey], processedObjects[normalizedObjectKey])
continue
}
glog.V(0).Infof("Processing regular file: objectKey=%s, normalizedObjectKey=%s, NOT in processedObjects", objectKey, normalizedObjectKey)
glog.V(4).Infof("Processing regular file: objectKey=%s, normalizedObjectKey=%s, NOT in processedObjects", objectKey, normalizedObjectKey)
// This is a pre-versioning or suspended-versioning object
// Check if this file has version metadata (ExtVersionIdKey)
@@ -414,7 +414,7 @@ func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string
if entry.Extended != nil {
if versionIdBytes, ok := entry.Extended[s3_constants.ExtVersionIdKey]; ok {
hasVersionMeta = true
glog.V(0).Infof("Regular file %s has version metadata: %s", normalizedObjectKey, string(versionIdBytes))
glog.V(4).Infof("Regular file %s has version metadata: %s", normalizedObjectKey, string(versionIdBytes))
}
}
@@ -423,12 +423,12 @@ func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string
_, versionsErr := s3a.getEntry(currentPath, versionsObjectPath)
if versionsErr == nil {
// .versions directory exists
glog.V(0).Infof("Found .versions directory for regular file %s, hasVersionMeta=%v", normalizedObjectKey, hasVersionMeta)
glog.V(4).Infof("Found .versions directory for regular file %s, hasVersionMeta=%v", normalizedObjectKey, hasVersionMeta)
// If this file has version metadata, it's a suspended versioning null version
// Include it and it will be the latest
if hasVersionMeta {
glog.V(0).Infof("Including suspended versioning file %s (has version metadata)", normalizedObjectKey)
glog.V(4).Infof("Including suspended versioning file %s (has version metadata)", normalizedObjectKey)
// Continue to add it below
} else {
// No version metadata - this is a pre-versioning file
@@ -443,16 +443,16 @@ func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string
}
}
if hasNullVersion {
glog.V(0).Infof("Skipping pre-versioning file %s, null version exists in .versions", normalizedObjectKey)
glog.V(4).Infof("Skipping pre-versioning file %s, null version exists in .versions", normalizedObjectKey)
processedObjects[objectKey] = true
processedObjects[normalizedObjectKey] = true
continue
}
}
glog.V(0).Infof("Including pre-versioning file %s (no null version in .versions)", normalizedObjectKey)
glog.V(4).Infof("Including pre-versioning file %s (no null version in .versions)", normalizedObjectKey)
}
} else {
glog.V(0).Infof("No .versions directory for regular file %s, hasVersionMeta=%v", normalizedObjectKey, hasVersionMeta)
glog.V(4).Infof("No .versions directory for regular file %s, hasVersionMeta=%v", normalizedObjectKey, hasVersionMeta)
}
// Add this file as a null version with IsLatest=true
@@ -469,7 +469,7 @@ func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string
etag := s3a.calculateETagFromChunks(entry.Chunks)
glog.V(0).Infof("Adding null version from regular file: objectKey=%s, normalizedObjectKey=%s, versionKey=%s, isLatest=%v, hasVersionMeta=%v",
glog.V(4).Infof("Adding null version from regular file: objectKey=%s, normalizedObjectKey=%s, versionKey=%s, isLatest=%v, hasVersionMeta=%v",
objectKey, normalizedObjectKey, versionKey, isLatest, hasVersionMeta)
versionEntry := &VersionEntry{

View File

@@ -100,20 +100,28 @@ func (s3a *S3ApiServer) handleSSEKMSEncryption(r *http.Request, dataReader io.Re
if baseIVHeader != "" {
// Decode the base IV from the header
baseIV, decodeErr := base64.StdEncoding.DecodeString(baseIVHeader)
if decodeErr != nil || len(baseIV) != 16 {
if decodeErr != nil {
glog.Errorf("handleSSEKMSEncryption: failed to decode base IV: %v", decodeErr)
return nil, nil, nil, s3err.ErrInternalError
}
if len(baseIV) != 16 {
glog.Errorf("handleSSEKMSEncryption: invalid base IV length: %d (expected 16)", len(baseIV))
return nil, nil, nil, s3err.ErrInternalError
}
// Use the provided base IV with unique part offset for multipart upload consistency
glog.V(4).Infof("handleSSEKMSEncryption: creating encrypted reader with baseIV=%x, partOffset=%d", baseIV[:8], partOffset)
encryptedReader, sseKey, encErr = CreateSSEKMSEncryptedReaderWithBaseIVAndOffset(dataReader, keyID, encryptionContext, bucketKeyEnabled, baseIV, partOffset)
glog.V(4).Infof("Using provided base IV %x for SSE-KMS encryption", baseIV[:8])
} else {
// Generate a new IV for single-part uploads
glog.V(4).Infof("handleSSEKMSEncryption: creating encrypted reader for single-part (no base IV)")
encryptedReader, sseKey, encErr = CreateSSEKMSEncryptedReaderWithBucketKey(dataReader, keyID, encryptionContext, bucketKeyEnabled)
}
if encErr != nil {
glog.Errorf("handleSSEKMSEncryption: encryption failed: %v", encErr)
return nil, nil, nil, s3err.ErrInternalError
}
glog.V(3).Infof("handleSSEKMSEncryption: encryption successful, keyID=%s", keyID)
// Prepare SSE-KMS metadata for later header setting
sseKMSMetadata, metaErr := SerializeSSEKMSMetadata(sseKey)
@@ -151,12 +159,20 @@ func (s3a *S3ApiServer) handleSSES3MultipartEncryption(r *http.Request, dataRead
}
// Use the provided base IV with unique part offset for multipart upload consistency
encryptedReader, _, encErr := CreateSSES3EncryptedReaderWithBaseIV(dataReader, key, baseIV, partOffset)
// CRITICAL: Capture the derived IV returned by CreateSSES3EncryptedReaderWithBaseIV
// This function calculates adjustedIV = calculateIVWithOffset(baseIV, partOffset)
// We MUST store this derived IV in metadata, not the base IV, for decryption to work
encryptedReader, derivedIV, encErr := CreateSSES3EncryptedReaderWithBaseIV(dataReader, key, baseIV, partOffset)
if encErr != nil {
return nil, nil, s3err.ErrInternalError
}
glog.V(4).Infof("handleSSES3MultipartEncryption: using provided base IV %x", baseIV[:8])
// Update the key with the derived IV so it gets serialized into chunk metadata
// This ensures decryption uses the correct offset-adjusted IV
key.IV = derivedIV
glog.V(4).Infof("handleSSES3MultipartEncryption: using base IV %x, derived IV %x for offset %d",
baseIV[:8], derivedIV[:8], partOffset)
return encryptedReader, key, s3err.ErrNone
}

View File

@@ -90,7 +90,7 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
// Initialize bucket policy engine first
policyEngine := NewBucketPolicyEngine()
s3ApiServer = &S3ApiServer{
option: option,
iam: iam,
@@ -108,7 +108,7 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
// Initialize advanced IAM system if config is provided
if option.IamConfig != "" {
glog.V(0).Infof("Loading advanced IAM configuration from: %s", option.IamConfig)
glog.V(1).Infof("Loading advanced IAM configuration from: %s", option.IamConfig)
iamManager, err := loadIAMManagerFromConfig(option.IamConfig, func() string {
return string(option.Filer)
@@ -125,7 +125,7 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
// Set the integration in the traditional IAM for compatibility
iam.SetIAMIntegration(s3iam)
glog.V(0).Infof("Advanced IAM system initialized successfully")
glog.V(1).Infof("Advanced IAM system initialized successfully")
}
}
@@ -134,7 +134,7 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
if err := s3ApiServer.iam.loadS3ApiConfigurationFromFile(option.Config); err != nil {
glog.Errorf("fail to load config file %s: %v", option.Config, err)
} else {
glog.V(0).Infof("Loaded %d identities from config file %s", len(s3ApiServer.iam.identities), option.Config)
glog.V(1).Infof("Loaded %d identities from config file %s", len(s3ApiServer.iam.identities), option.Config)
}
})
}
@@ -168,6 +168,10 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
// This helper method centralizes the logic for loading bucket policies into the engine
// to avoid duplication and ensure consistent error handling
func (s3a *S3ApiServer) syncBucketPolicyToEngine(bucket string, policyDoc *policy.PolicyDocument) {
if s3a.policyEngine == nil {
return
}
if policyDoc != nil {
if err := s3a.policyEngine.LoadBucketPolicyFromCache(bucket, policyDoc); err != nil {
glog.Errorf("Failed to sync bucket policy for %s to policy engine: %v", bucket, err)
@@ -498,7 +502,7 @@ func loadIAMManagerFromConfig(configPath string, filerAddressProvider func() str
if configRoot.Policy == nil {
// Provide a secure default if not specified in the config file
// Default to Deny with in-memory store so that JSON-defined policies work without filer
glog.V(0).Infof("No policy engine config provided; using defaults (DefaultEffect=%s, StoreType=%s)", sts.EffectDeny, sts.StoreTypeMemory)
glog.V(1).Infof("No policy engine config provided; using defaults (DefaultEffect=%s, StoreType=%s)", sts.EffectDeny, sts.StoreTypeMemory)
configRoot.Policy = &policy.PolicyEngineConfig{
DefaultEffect: sts.EffectDeny,
StoreType: sts.StoreTypeMemory,
@@ -556,7 +560,7 @@ func loadIAMManagerFromConfig(configPath string, filerAddressProvider func() str
}
}
glog.V(0).Infof("Loaded %d providers, %d policies and %d roles from config", len(configRoot.Providers), len(configRoot.Policies), len(configRoot.Roles))
glog.V(1).Infof("Loaded %d providers, %d policies and %d roles from config", len(configRoot.Providers), len(configRoot.Policies), len(configRoot.Roles))
return iamManager, nil
}

View File

@@ -0,0 +1,361 @@
package s3api
import (
"bytes"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"encoding/json"
"io"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
)
// TestSSEKMSChunkMetadataAssignment tests that SSE-KMS creates per-chunk metadata
// with correct ChunkOffset values for each chunk (matching the fix in putToFiler)
func TestSSEKMSChunkMetadataAssignment(t *testing.T) {
kmsKey := SetupTestKMS(t)
defer kmsKey.Cleanup()
// Generate SSE-KMS key by encrypting test data (this gives us a real SSEKMSKey)
encryptionContext := BuildEncryptionContext("test-bucket", "test-object", false)
testData := "Test data for SSE-KMS chunk metadata validation"
encryptedReader, sseKMSKey, err := CreateSSEKMSEncryptedReader(bytes.NewReader([]byte(testData)), kmsKey.KeyID, encryptionContext)
if err != nil {
t.Fatalf("Failed to create encrypted reader: %v", err)
}
// Read to complete encryption setup
io.ReadAll(encryptedReader)
// Serialize the base metadata (what putToFiler receives before chunking)
baseMetadata, err := SerializeSSEKMSMetadata(sseKMSKey)
if err != nil {
t.Fatalf("Failed to serialize base SSE-KMS metadata: %v", err)
}
// Simulate multi-chunk upload scenario (what putToFiler does after UploadReaderInChunks)
simulatedChunks := []*filer_pb.FileChunk{
{FileId: "chunk1", Offset: 0, Size: 8 * 1024 * 1024}, // 8MB chunk at offset 0
{FileId: "chunk2", Offset: 8 * 1024 * 1024, Size: 8 * 1024 * 1024}, // 8MB chunk at offset 8MB
{FileId: "chunk3", Offset: 16 * 1024 * 1024, Size: 4 * 1024 * 1024}, // 4MB chunk at offset 16MB
}
// THIS IS THE CRITICAL FIX: Create per-chunk metadata (lines 421-443 in putToFiler)
for _, chunk := range simulatedChunks {
chunk.SseType = filer_pb.SSEType_SSE_KMS
// Create a copy of the SSE-KMS key with chunk-specific offset
chunkSSEKey := &SSEKMSKey{
KeyID: sseKMSKey.KeyID,
EncryptedDataKey: sseKMSKey.EncryptedDataKey,
EncryptionContext: sseKMSKey.EncryptionContext,
BucketKeyEnabled: sseKMSKey.BucketKeyEnabled,
IV: sseKMSKey.IV,
ChunkOffset: chunk.Offset, // Set chunk-specific offset
}
// Serialize per-chunk metadata
chunkMetadata, serErr := SerializeSSEKMSMetadata(chunkSSEKey)
if serErr != nil {
t.Fatalf("Failed to serialize SSE-KMS metadata for chunk at offset %d: %v", chunk.Offset, serErr)
}
chunk.SseMetadata = chunkMetadata
}
// VERIFICATION 1: Each chunk should have different metadata (due to different ChunkOffset)
metadataSet := make(map[string]bool)
for i, chunk := range simulatedChunks {
metadataStr := string(chunk.SseMetadata)
if metadataSet[metadataStr] {
t.Errorf("Chunk %d has duplicate metadata (should be unique per chunk)", i)
}
metadataSet[metadataStr] = true
// Deserialize and verify ChunkOffset
var metadata SSEKMSMetadata
if err := json.Unmarshal(chunk.SseMetadata, &metadata); err != nil {
t.Fatalf("Failed to deserialize chunk %d metadata: %v", i, err)
}
expectedOffset := chunk.Offset
if metadata.PartOffset != expectedOffset {
t.Errorf("Chunk %d: expected PartOffset=%d, got %d", i, expectedOffset, metadata.PartOffset)
}
t.Logf("✓ Chunk %d: PartOffset=%d (correct)", i, metadata.PartOffset)
}
// VERIFICATION 2: Verify metadata can be deserialized and has correct ChunkOffset
for i, chunk := range simulatedChunks {
// Deserialize chunk metadata
deserializedKey, err := DeserializeSSEKMSMetadata(chunk.SseMetadata)
if err != nil {
t.Fatalf("Failed to deserialize chunk %d metadata: %v", i, err)
}
// Verify the deserialized key has correct ChunkOffset
if deserializedKey.ChunkOffset != chunk.Offset {
t.Errorf("Chunk %d: deserialized ChunkOffset=%d, expected %d",
i, deserializedKey.ChunkOffset, chunk.Offset)
}
// Verify IV is set (should be inherited from base)
if len(deserializedKey.IV) != aes.BlockSize {
t.Errorf("Chunk %d: invalid IV length: %d", i, len(deserializedKey.IV))
}
// Verify KeyID matches
if deserializedKey.KeyID != sseKMSKey.KeyID {
t.Errorf("Chunk %d: KeyID mismatch", i)
}
t.Logf("✓ Chunk %d: metadata deserialized successfully (ChunkOffset=%d, KeyID=%s)",
i, deserializedKey.ChunkOffset, deserializedKey.KeyID)
}
// VERIFICATION 3: Ensure base metadata is NOT reused (the bug we're preventing)
var baseMetadataStruct SSEKMSMetadata
if err := json.Unmarshal(baseMetadata, &baseMetadataStruct); err != nil {
t.Fatalf("Failed to deserialize base metadata: %v", err)
}
// Base metadata should have ChunkOffset=0
if baseMetadataStruct.PartOffset != 0 {
t.Errorf("Base metadata should have PartOffset=0, got %d", baseMetadataStruct.PartOffset)
}
// Chunks 2 and 3 should NOT have the same metadata as base (proving we're not reusing)
for i := 1; i < len(simulatedChunks); i++ {
if bytes.Equal(simulatedChunks[i].SseMetadata, baseMetadata) {
t.Errorf("CRITICAL BUG: Chunk %d reuses base metadata (should have per-chunk metadata)", i)
}
}
t.Log("✓ All chunks have unique per-chunk metadata (bug prevented)")
}
// TestSSES3ChunkMetadataAssignment tests that SSE-S3 creates per-chunk metadata
// with offset-adjusted IVs for each chunk (matching the fix in putToFiler)
func TestSSES3ChunkMetadataAssignment(t *testing.T) {
// Initialize global SSE-S3 key manager
globalSSES3KeyManager = NewSSES3KeyManager()
defer func() {
globalSSES3KeyManager = NewSSES3KeyManager()
}()
keyManager := GetSSES3KeyManager()
keyManager.superKey = make([]byte, 32)
rand.Read(keyManager.superKey)
// Generate SSE-S3 key
sseS3Key, err := GenerateSSES3Key()
if err != nil {
t.Fatalf("Failed to generate SSE-S3 key: %v", err)
}
// Generate base IV
baseIV := make([]byte, aes.BlockSize)
rand.Read(baseIV)
sseS3Key.IV = baseIV
// Serialize base metadata (what putToFiler receives)
baseMetadata, err := SerializeSSES3Metadata(sseS3Key)
if err != nil {
t.Fatalf("Failed to serialize base SSE-S3 metadata: %v", err)
}
// Simulate multi-chunk upload scenario (what putToFiler does after UploadReaderInChunks)
simulatedChunks := []*filer_pb.FileChunk{
{FileId: "chunk1", Offset: 0, Size: 8 * 1024 * 1024}, // 8MB chunk at offset 0
{FileId: "chunk2", Offset: 8 * 1024 * 1024, Size: 8 * 1024 * 1024}, // 8MB chunk at offset 8MB
{FileId: "chunk3", Offset: 16 * 1024 * 1024, Size: 4 * 1024 * 1024}, // 4MB chunk at offset 16MB
}
// THIS IS THE CRITICAL FIX: Create per-chunk metadata (lines 444-468 in putToFiler)
for _, chunk := range simulatedChunks {
chunk.SseType = filer_pb.SSEType_SSE_S3
// Calculate chunk-specific IV using base IV and chunk offset
chunkIV, _ := calculateIVWithOffset(sseS3Key.IV, chunk.Offset)
// Create a copy of the SSE-S3 key with chunk-specific IV
chunkSSEKey := &SSES3Key{
Key: sseS3Key.Key,
KeyID: sseS3Key.KeyID,
Algorithm: sseS3Key.Algorithm,
IV: chunkIV, // Use chunk-specific IV
}
// Serialize per-chunk metadata
chunkMetadata, serErr := SerializeSSES3Metadata(chunkSSEKey)
if serErr != nil {
t.Fatalf("Failed to serialize SSE-S3 metadata for chunk at offset %d: %v", chunk.Offset, serErr)
}
chunk.SseMetadata = chunkMetadata
}
// VERIFICATION 1: Each chunk should have different metadata (due to different IVs)
metadataSet := make(map[string]bool)
for i, chunk := range simulatedChunks {
metadataStr := string(chunk.SseMetadata)
if metadataSet[metadataStr] {
t.Errorf("Chunk %d has duplicate metadata (should be unique per chunk)", i)
}
metadataSet[metadataStr] = true
// Deserialize and verify IV
deserializedKey, err := DeserializeSSES3Metadata(chunk.SseMetadata, keyManager)
if err != nil {
t.Fatalf("Failed to deserialize chunk %d metadata: %v", i, err)
}
// Calculate expected IV for this chunk
expectedIV, _ := calculateIVWithOffset(baseIV, chunk.Offset)
if !bytes.Equal(deserializedKey.IV, expectedIV) {
t.Errorf("Chunk %d: IV mismatch\nExpected: %x\nGot: %x",
i, expectedIV[:8], deserializedKey.IV[:8])
}
t.Logf("✓ Chunk %d: IV correctly adjusted for offset=%d", i, chunk.Offset)
}
// VERIFICATION 2: Verify decryption works with per-chunk IVs
for i, chunk := range simulatedChunks {
// Deserialize chunk metadata
deserializedKey, err := DeserializeSSES3Metadata(chunk.SseMetadata, keyManager)
if err != nil {
t.Fatalf("Failed to deserialize chunk %d metadata: %v", i, err)
}
// Simulate encryption/decryption with the chunk's IV
testData := []byte("Test data for SSE-S3 chunk decryption verification")
block, err := aes.NewCipher(deserializedKey.Key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
// Encrypt with chunk's IV
ciphertext := make([]byte, len(testData))
stream := cipher.NewCTR(block, deserializedKey.IV)
stream.XORKeyStream(ciphertext, testData)
// Decrypt with chunk's IV
plaintext := make([]byte, len(ciphertext))
block2, _ := aes.NewCipher(deserializedKey.Key)
stream2 := cipher.NewCTR(block2, deserializedKey.IV)
stream2.XORKeyStream(plaintext, ciphertext)
if !bytes.Equal(plaintext, testData) {
t.Errorf("Chunk %d: decryption failed", i)
}
t.Logf("✓ Chunk %d: encryption/decryption successful with chunk-specific IV", i)
}
// VERIFICATION 3: Ensure base IV is NOT reused for non-zero offset chunks (the bug we're preventing)
for i := 1; i < len(simulatedChunks); i++ {
if bytes.Equal(simulatedChunks[i].SseMetadata, baseMetadata) {
t.Errorf("CRITICAL BUG: Chunk %d reuses base metadata (should have per-chunk metadata)", i)
}
// Verify chunk metadata has different IV than base IV
deserializedKey, _ := DeserializeSSES3Metadata(simulatedChunks[i].SseMetadata, keyManager)
if bytes.Equal(deserializedKey.IV, baseIV) {
t.Errorf("CRITICAL BUG: Chunk %d uses base IV (should use offset-adjusted IV)", i)
}
}
t.Log("✓ All chunks have unique per-chunk IVs (bug prevented)")
}
// TestSSEChunkMetadataComparison tests that the bug (reusing same metadata for all chunks)
// would cause decryption failures, while the fix (per-chunk metadata) works correctly
func TestSSEChunkMetadataComparison(t *testing.T) {
// Generate test key and IV
key := make([]byte, 32)
rand.Read(key)
baseIV := make([]byte, aes.BlockSize)
rand.Read(baseIV)
// Create test data for 3 chunks
chunk0Data := []byte("Chunk 0 data at offset 0")
chunk1Data := []byte("Chunk 1 data at offset 8MB")
chunk2Data := []byte("Chunk 2 data at offset 16MB")
chunkOffsets := []int64{0, 8 * 1024 * 1024, 16 * 1024 * 1024}
chunkDataList := [][]byte{chunk0Data, chunk1Data, chunk2Data}
// Scenario 1: BUG - Using same IV for all chunks (what the old code did)
t.Run("Bug: Reusing base IV causes decryption failures", func(t *testing.T) {
var encryptedChunks [][]byte
// Encrypt each chunk with offset-adjusted IV (what encryption does)
for i, offset := range chunkOffsets {
adjustedIV, _ := calculateIVWithOffset(baseIV, offset)
block, _ := aes.NewCipher(key)
stream := cipher.NewCTR(block, adjustedIV)
ciphertext := make([]byte, len(chunkDataList[i]))
stream.XORKeyStream(ciphertext, chunkDataList[i])
encryptedChunks = append(encryptedChunks, ciphertext)
}
// Try to decrypt with base IV (THE BUG)
for i := range encryptedChunks {
block, _ := aes.NewCipher(key)
stream := cipher.NewCTR(block, baseIV) // BUG: Always using base IV
plaintext := make([]byte, len(encryptedChunks[i]))
stream.XORKeyStream(plaintext, encryptedChunks[i])
if i == 0 {
// Chunk 0 should work (offset 0 means base IV = adjusted IV)
if !bytes.Equal(plaintext, chunkDataList[i]) {
t.Errorf("Chunk 0 decryption failed (unexpected)")
}
} else {
// Chunks 1 and 2 should FAIL (wrong IV)
if bytes.Equal(plaintext, chunkDataList[i]) {
t.Errorf("BUG NOT REPRODUCED: Chunk %d decrypted correctly with base IV (should fail)", i)
} else {
t.Logf("✓ Chunk %d: Correctly failed to decrypt with base IV (bug reproduced)", i)
}
}
}
})
// Scenario 2: FIX - Using per-chunk offset-adjusted IVs (what the new code does)
t.Run("Fix: Per-chunk IVs enable correct decryption", func(t *testing.T) {
var encryptedChunks [][]byte
var chunkIVs [][]byte
// Encrypt each chunk with offset-adjusted IV
for i, offset := range chunkOffsets {
adjustedIV, _ := calculateIVWithOffset(baseIV, offset)
chunkIVs = append(chunkIVs, adjustedIV)
block, _ := aes.NewCipher(key)
stream := cipher.NewCTR(block, adjustedIV)
ciphertext := make([]byte, len(chunkDataList[i]))
stream.XORKeyStream(ciphertext, chunkDataList[i])
encryptedChunks = append(encryptedChunks, ciphertext)
}
// Decrypt with per-chunk IVs (THE FIX)
for i := range encryptedChunks {
block, _ := aes.NewCipher(key)
stream := cipher.NewCTR(block, chunkIVs[i]) // FIX: Using per-chunk IV
plaintext := make([]byte, len(encryptedChunks[i]))
stream.XORKeyStream(plaintext, encryptedChunks[i])
if !bytes.Equal(plaintext, chunkDataList[i]) {
t.Errorf("Chunk %d decryption failed with per-chunk IV (unexpected)", i)
} else {
t.Logf("✓ Chunk %d: Successfully decrypted with per-chunk IV", i)
}
}
})
}

View File

@@ -0,0 +1,189 @@
package s3api
import (
"bytes"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"io"
"testing"
)
// TestSSECDecryptChunkView_NoOffsetAdjustment verifies that SSE-C decryption
// does NOT apply calculateIVWithOffset, preventing the critical bug where
// offset adjustment would cause CTR stream misalignment and data corruption.
func TestSSECDecryptChunkView_NoOffsetAdjustment(t *testing.T) {
// Setup: Create test data
plaintext := []byte("This is a test message for SSE-C decryption without offset adjustment")
customerKey := &SSECustomerKey{
Key: make([]byte, 32), // 256-bit key
KeyMD5: "test-key-md5",
}
// Generate random AES key
if _, err := rand.Read(customerKey.Key); err != nil {
t.Fatalf("Failed to generate random key: %v", err)
}
// Generate random IV for this "part"
randomIV := make([]byte, aes.BlockSize)
if _, err := rand.Read(randomIV); err != nil {
t.Fatalf("Failed to generate random IV: %v", err)
}
// Encrypt the plaintext using the random IV (simulating SSE-C multipart upload)
// This is what CreateSSECEncryptedReader does - uses the IV directly without offset
block, err := aes.NewCipher(customerKey.Key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
ciphertext := make([]byte, len(plaintext))
stream := cipher.NewCTR(block, randomIV)
stream.XORKeyStream(ciphertext, plaintext)
partOffset := int64(1024) // Non-zero offset that should NOT be applied during SSE-C decryption
// TEST: Decrypt using stored IV directly (correct behavior)
decryptedReaderCorrect, err := CreateSSECDecryptedReader(
io.NopCloser(bytes.NewReader(ciphertext)),
customerKey,
randomIV, // Use stored IV directly - CORRECT
)
if err != nil {
t.Fatalf("Failed to create decrypted reader (correct): %v", err)
}
decryptedCorrect, err := io.ReadAll(decryptedReaderCorrect)
if err != nil {
t.Fatalf("Failed to read decrypted data (correct): %v", err)
}
// Verify correct decryption
if !bytes.Equal(decryptedCorrect, plaintext) {
t.Errorf("Correct decryption failed:\nExpected: %s\nGot: %s", plaintext, decryptedCorrect)
} else {
t.Logf("✓ Correct decryption (using stored IV directly) successful")
}
// ANTI-TEST: Decrypt using offset-adjusted IV (incorrect behavior - the bug)
adjustedIV, ivSkip := calculateIVWithOffset(randomIV, partOffset)
decryptedReaderWrong, err := CreateSSECDecryptedReader(
io.NopCloser(bytes.NewReader(ciphertext)),
customerKey,
adjustedIV, // Use adjusted IV - WRONG
)
if err != nil {
t.Fatalf("Failed to create decrypted reader (wrong): %v", err)
}
// Skip ivSkip bytes (as the buggy code would do)
if ivSkip > 0 {
io.CopyN(io.Discard, decryptedReaderWrong, int64(ivSkip))
}
decryptedWrong, err := io.ReadAll(decryptedReaderWrong)
if err != nil {
t.Fatalf("Failed to read decrypted data (wrong): %v", err)
}
// Verify that offset adjustment produces DIFFERENT (corrupted) output
if bytes.Equal(decryptedWrong, plaintext) {
t.Errorf("CRITICAL: Offset-adjusted IV produced correct plaintext! This shouldn't happen for SSE-C.")
} else {
t.Logf("✓ Verified: Offset-adjusted IV produces corrupted data (as expected for SSE-C)")
maxLen := 20
if len(plaintext) < maxLen {
maxLen = len(plaintext)
}
t.Logf(" Plaintext: %q", plaintext[:maxLen])
maxLen2 := 20
if len(decryptedWrong) < maxLen2 {
maxLen2 = len(decryptedWrong)
}
t.Logf(" Corrupted: %q", decryptedWrong[:maxLen2])
}
}
// TestSSEKMSDecryptChunkView_RequiresOffsetAdjustment verifies that SSE-KMS
// decryption DOES require calculateIVWithOffset, unlike SSE-C.
func TestSSEKMSDecryptChunkView_RequiresOffsetAdjustment(t *testing.T) {
// Setup: Create test data
plaintext := []byte("This is a test message for SSE-KMS decryption with offset adjustment")
// Generate base IV and key
baseIV := make([]byte, aes.BlockSize)
key := make([]byte, 32)
if _, err := rand.Read(baseIV); err != nil {
t.Fatalf("Failed to generate base IV: %v", err)
}
if _, err := rand.Read(key); err != nil {
t.Fatalf("Failed to generate key: %v", err)
}
chunkOffset := int64(2048) // Simulate chunk at offset 2048
// Encrypt using base IV + offset (simulating SSE-KMS multipart upload)
adjustedIV, ivSkip := calculateIVWithOffset(baseIV, chunkOffset)
block, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
ciphertext := make([]byte, len(plaintext))
stream := cipher.NewCTR(block, adjustedIV)
// Skip ivSkip bytes in the encryption stream if needed
if ivSkip > 0 {
dummy := make([]byte, ivSkip)
stream.XORKeyStream(dummy, dummy)
}
stream.XORKeyStream(ciphertext, plaintext)
// TEST: Decrypt using base IV + offset adjustment (correct for SSE-KMS)
adjustedIVDecrypt, ivSkipDecrypt := calculateIVWithOffset(baseIV, chunkOffset)
blockDecrypt, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create cipher for decryption: %v", err)
}
decrypted := make([]byte, len(ciphertext))
streamDecrypt := cipher.NewCTR(blockDecrypt, adjustedIVDecrypt)
// Skip ivSkip bytes in the decryption stream
if ivSkipDecrypt > 0 {
dummy := make([]byte, ivSkipDecrypt)
streamDecrypt.XORKeyStream(dummy, dummy)
}
streamDecrypt.XORKeyStream(decrypted, ciphertext)
// Verify correct decryption with offset adjustment
if !bytes.Equal(decrypted, plaintext) {
t.Errorf("SSE-KMS decryption with offset adjustment failed:\nExpected: %s\nGot: %s", plaintext, decrypted)
} else {
t.Logf("✓ SSE-KMS decryption with offset adjustment successful")
}
// ANTI-TEST: Decrypt using base IV directly (incorrect for SSE-KMS)
blockWrong, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("Failed to create cipher for wrong decryption: %v", err)
}
decryptedWrong := make([]byte, len(ciphertext))
streamWrong := cipher.NewCTR(blockWrong, baseIV) // Use base IV directly - WRONG for SSE-KMS
streamWrong.XORKeyStream(decryptedWrong, ciphertext)
// Verify that NOT using offset adjustment produces corrupted output
if bytes.Equal(decryptedWrong, plaintext) {
t.Errorf("CRITICAL: Base IV without offset produced correct plaintext! SSE-KMS requires offset adjustment.")
} else {
t.Logf("✓ Verified: Base IV without offset produces corrupted data (as expected for SSE-KMS)")
}
}
// TestSSEDecryptionDifferences documents the key differences between SSE types
func TestSSEDecryptionDifferences(t *testing.T) {
t.Log("SSE-C: Random IV per part → Use stored IV DIRECTLY (no offset)")
t.Log("SSE-KMS: Base IV + offset → MUST call calculateIVWithOffset(baseIV, offset)")
t.Log("SSE-S3: Base IV + offset → Stores ADJUSTED IV, use directly")
// This test documents the critical differences and serves as executable documentation
}

View File

@@ -0,0 +1,257 @@
package s3api
import (
"bytes"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"encoding/base64"
"io"
"testing"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
)
// TestSSES3MultipartUploadStoresDerivedIV verifies the critical fix where
// handleSSES3MultipartEncryption must store the DERIVED IV (not base IV)
// in the returned key so it gets serialized into chunk metadata.
//
// This test prevents the bug where the derived IV was discarded, causing
// decryption to use the wrong IV and produce corrupted plaintext.
func TestSSES3MultipartUploadStoresDerivedIV(t *testing.T) {
// Setup: Create a test key and base IV
keyManager := GetSSES3KeyManager()
sseS3Key, err := keyManager.GetOrCreateKey("")
if err != nil {
t.Fatalf("Failed to create SSE-S3 key: %v", err)
}
// Generate a random base IV
baseIV := make([]byte, aes.BlockSize)
if _, err := rand.Read(baseIV); err != nil {
t.Fatalf("Failed to generate base IV: %v", err)
}
// Test data for multipart upload parts
testCases := []struct {
name string
partOffset int64
data []byte
}{
{"Part 1 at offset 0", 0, []byte("First part of multipart upload")},
{"Part 2 at offset 1MB", 1024 * 1024, []byte("Second part of multipart upload")},
{"Part 3 at offset 5MB", 5 * 1024 * 1024, []byte("Third part at 5MB offset")},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
// Calculate the expected derived IV (what encryption will use)
expectedDerivedIV, ivSkip := calculateIVWithOffset(baseIV, tc.partOffset)
// Call CreateSSES3EncryptedReaderWithBaseIV to encrypt the data
dataReader := bytes.NewReader(tc.data)
encryptedReader, returnedDerivedIV, encErr := CreateSSES3EncryptedReaderWithBaseIV(
dataReader,
sseS3Key,
baseIV,
tc.partOffset,
)
if encErr != nil {
t.Fatalf("Failed to create encrypted reader: %v", encErr)
}
// Read the encrypted data
encryptedData, err := io.ReadAll(encryptedReader)
if err != nil {
t.Fatalf("Failed to read encrypted data: %v", err)
}
// CRITICAL VERIFICATION: The returned IV should be the DERIVED IV
if !bytes.Equal(returnedDerivedIV, expectedDerivedIV) {
t.Errorf("CreateSSES3EncryptedReaderWithBaseIV returned wrong IV:\nExpected: %x\nGot: %x",
expectedDerivedIV[:8], returnedDerivedIV[:8])
}
// CRITICAL TEST: Verify the key.IV field would be updated (simulating handleSSES3MultipartEncryption)
// This is what the fix does: key.IV = derivedIV
keyWithDerivedIV := &SSES3Key{
Key: sseS3Key.Key,
KeyID: sseS3Key.KeyID,
Algorithm: sseS3Key.Algorithm,
IV: returnedDerivedIV, // This simulates: key.IV = derivedIV
}
// TEST 1: Verify decryption with DERIVED IV produces correct plaintext (correct behavior)
decryptedWithDerivedIV := make([]byte, len(encryptedData))
block, err := aes.NewCipher(keyWithDerivedIV.Key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
stream := cipher.NewCTR(block, keyWithDerivedIV.IV)
// Handle ivSkip for non-block-aligned offsets
if ivSkip > 0 {
skipDummy := make([]byte, ivSkip)
stream.XORKeyStream(skipDummy, skipDummy)
}
stream.XORKeyStream(decryptedWithDerivedIV, encryptedData)
if !bytes.Equal(decryptedWithDerivedIV, tc.data) {
t.Errorf("Decryption with derived IV failed:\nExpected: %q\nGot: %q",
tc.data, decryptedWithDerivedIV)
} else {
t.Logf("✓ Derived IV decryption successful for offset %d", tc.partOffset)
}
// TEST 2: Verify decryption with BASE IV produces WRONG plaintext (bug behavior)
// This is what would happen if the bug wasn't fixed
if tc.partOffset > 0 { // Only test for non-zero offsets (where IVs differ)
keyWithBaseIV := &SSES3Key{
Key: sseS3Key.Key,
KeyID: sseS3Key.KeyID,
Algorithm: sseS3Key.Algorithm,
IV: baseIV, // BUG: Using base IV instead of derived IV
}
decryptedWithBaseIV := make([]byte, len(encryptedData))
blockWrong, err := aes.NewCipher(keyWithBaseIV.Key)
if err != nil {
t.Fatalf("Failed to create cipher for wrong decryption: %v", err)
}
streamWrong := cipher.NewCTR(blockWrong, keyWithBaseIV.IV)
streamWrong.XORKeyStream(decryptedWithBaseIV, encryptedData)
if bytes.Equal(decryptedWithBaseIV, tc.data) {
t.Errorf("CRITICAL BUG: Base IV produced correct plaintext at offset %d! Should produce corrupted data.", tc.partOffset)
} else {
t.Logf("✓ Verified: Base IV produces corrupted data at offset %d (bug would cause this)", tc.partOffset)
}
}
})
}
}
// TestHandleSSES3MultipartEncryptionFlow is an integration test that verifies
// the complete flow of handleSSES3MultipartEncryption, including that the
// returned key contains the derived IV (not base IV).
func TestHandleSSES3MultipartEncryptionFlow(t *testing.T) {
// This test simulates what happens in a real multipart upload request
// Generate test key manually (simulating a complete SSE-S3 key)
keyBytes := make([]byte, 32) // 256-bit key
if _, err := rand.Read(keyBytes); err != nil {
t.Fatalf("Failed to generate key: %v", err)
}
originalKey := &SSES3Key{
Key: keyBytes,
KeyID: "test-key-id",
Algorithm: SSES3Algorithm,
IV: nil, // Will be set later
}
baseIV := make([]byte, aes.BlockSize)
if _, err := rand.Read(baseIV); err != nil {
t.Fatalf("Failed to generate base IV: %v", err)
}
// For this test, we'll work directly with the key structure
// since SerializeSSES3Metadata requires KMS setup
// Test with a non-zero offset (where base IV != derived IV)
partOffset := int64(2 * 1024 * 1024) // 2MB offset
plaintext := []byte("Test data for part 2 of multipart upload")
// Calculate what the derived IV should be
expectedDerivedIV, ivSkip := calculateIVWithOffset(baseIV, partOffset)
// Simulate the upload by calling CreateSSES3EncryptedReaderWithBaseIV directly
// (This is what handleSSES3MultipartEncryption does internally)
dataReader := bytes.NewReader(plaintext)
// Encrypt with base IV and offset
encryptedReader, derivedIV, encErr := CreateSSES3EncryptedReaderWithBaseIV(
dataReader,
originalKey,
baseIV,
partOffset,
)
if encErr != nil {
t.Fatalf("Failed to create encrypted reader: %v", encErr)
}
// THE FIX: Update key.IV with derivedIV (this is what the bug fix does)
originalKey.IV = derivedIV
// Read encrypted data
encryptedData, err := io.ReadAll(encryptedReader)
if err != nil {
t.Fatalf("Failed to read encrypted data: %v", err)
}
// VERIFICATION 1: Derived IV should match expected
if !bytes.Equal(derivedIV, expectedDerivedIV) {
t.Errorf("Derived IV mismatch:\nExpected: %x\nGot: %x",
expectedDerivedIV[:8], derivedIV[:8])
}
// VERIFICATION 2: Key should now contain derived IV (the fix)
if !bytes.Equal(originalKey.IV, derivedIV) {
t.Errorf("Key.IV was not updated with derived IV!\nKey.IV: %x\nDerived IV: %x",
originalKey.IV[:8], derivedIV[:8])
} else {
t.Logf("✓ Key.IV correctly updated with derived IV")
}
// VERIFICATION 3: The IV stored in the key can be used for decryption
decryptedData := make([]byte, len(encryptedData))
block, err := aes.NewCipher(originalKey.Key)
if err != nil {
t.Fatalf("Failed to create cipher: %v", err)
}
stream := cipher.NewCTR(block, originalKey.IV)
// Handle ivSkip for non-block-aligned offsets
if ivSkip > 0 {
skipDummy := make([]byte, ivSkip)
stream.XORKeyStream(skipDummy, skipDummy)
}
stream.XORKeyStream(decryptedData, encryptedData)
if !bytes.Equal(decryptedData, plaintext) {
t.Errorf("Final decryption failed:\nExpected: %q\nGot: %q", plaintext, decryptedData)
} else {
t.Logf("✓ Full encrypt-update_key-decrypt cycle successful")
}
}
// TestSSES3HeaderEncoding tests that the header encoding/decoding works correctly
func TestSSES3HeaderEncoding(t *testing.T) {
// Generate test base IV
baseIV := make([]byte, aes.BlockSize)
if _, err := rand.Read(baseIV); err != nil {
t.Fatalf("Failed to generate base IV: %v", err)
}
// Encode as it would be in HTTP header
baseIVHeader := base64.StdEncoding.EncodeToString(baseIV)
// Decode (as handleSSES3MultipartEncryption does)
decodedBaseIV, err := base64.StdEncoding.DecodeString(baseIVHeader)
if err != nil {
t.Fatalf("Failed to decode base IV: %v", err)
}
// Verify round-trip
if !bytes.Equal(decodedBaseIV, baseIV) {
t.Errorf("Base IV encoding round-trip failed:\nOriginal: %x\nDecoded: %x",
baseIV, decodedBaseIV)
}
// Verify length
if len(decodedBaseIV) != s3_constants.AESBlockSize {
t.Errorf("Decoded base IV has wrong length: expected %d, got %d",
s3_constants.AESBlockSize, len(decodedBaseIV))
}
}

View File

@@ -121,7 +121,7 @@ func WriteResponse(w http.ResponseWriter, r *http.Request, statusCode int, respo
glog.V(4).Infof("status %d %s: %s", statusCode, mType, string(response))
_, err := w.Write(response)
if err != nil {
glog.V(0).Infof("write err: %v", err)
glog.V(1).Infof("write err: %v", err)
}
w.(http.Flusher).Flush()
}
@@ -129,6 +129,6 @@ func WriteResponse(w http.ResponseWriter, r *http.Request, statusCode int, respo
// If none of the http routes match respond with MethodNotAllowed
func NotFoundHandler(w http.ResponseWriter, r *http.Request) {
glog.V(0).Infof("unsupported %s %s", r.Method, r.RequestURI)
glog.V(2).Infof("unsupported %s %s", r.Method, r.RequestURI)
WriteErrorResponse(w, r, ErrMethodNotAllowed)
}