Test object lock and retention (#6997)
* fix GetObjectLockConfigurationHandler * cache and use bucket object lock config * subscribe to bucket configuration changes * increase bucket config cache TTL * refactor * Update weed/s3api/s3api_server.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * avoid duplidated work * rename variable * Update s3api_object_handlers_put.go * fix routing * admin ui and api handler are consistent now * use fields instead of xml * fix test * address comments * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/s3_retention_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/object_lock_utils.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * change error style * errorf * read entry once * add s3 tests for object lock and retention * use marker * install s3 tests * Update s3tests.yml * Update s3tests.yml * Update s3tests.conf * Update s3tests.conf * address test errors * address test errors With these fixes, the s3-tests should now: ✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets ✅ Return MalformedXML for invalid retention configurations ✅ Include VersionId in response headers when available ✅ Return proper HTTP status codes (403 Forbidden for retention mode changes) ✅ Handle all object lock validation errors consistently * fixes With these comprehensive fixes, the s3-tests should now: ✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets ✅ Return InvalidRetentionPeriod for invalid retention periods ✅ Return MalformedXML for malformed retention configurations ✅ Include VersionId in response headers when available ✅ Return proper HTTP status codes for all error conditions ✅ Handle all object lock validation errors consistently The workflow should now pass significantly more object lock tests, bringing SeaweedFS's S3 object lock implementation much closer to AWS S3 compatibility standards. * fixes With these final fixes, the s3-tests should now: ✅ Return MalformedXML for ObjectLockEnabled: 'Disabled' ✅ Return MalformedXML when both Days and Years are specified in retention configuration ✅ Return InvalidBucketState (409 Conflict) when trying to suspend versioning on buckets with object lock enabled ✅ Handle all object lock validation errors consistently with proper error codes * constants and fixes ✅ Return InvalidRetentionPeriod for invalid retention values (0 days, negative years) ✅ Return ObjectLockConfigurationNotFoundError when object lock configuration doesn't exist ✅ Handle all object lock validation errors consistently with proper error codes * fixes ✅ Return MalformedXML when both Days and Years are specified in the same retention configuration ✅ Return 400 (Bad Request) with InvalidRequest when object lock operations are attempted on buckets without object lock enabled ✅ Handle all object lock validation errors consistently with proper error codes * fixes ✅ Return 409 (Conflict) with InvalidBucketState for bucket-level object lock configuration operations on buckets without object lock enabled ✅ Allow increasing retention periods and overriding retention with same/later dates ✅ Only block decreasing retention periods without proper bypass permissions ✅ Handle all object lock validation errors consistently with proper error codes * fixes ✅ Include VersionId in multipart upload completion responses when versioning is enabled ✅ Block retention mode changes (GOVERNANCE ↔ COMPLIANCE) without bypass permissions ✅ Handle all object lock validation errors consistently with proper error codes ✅ Pass the remaining object lock tests * fix tests * fixes * pass tests * fix tests * fixes * add error mapping * Update s3tests.conf * fix test_object_lock_put_obj_lock_invalid_days * fixes * fix many issues * fix test_object_lock_delete_multipart_object_with_legal_hold_on * fix tests * refactor * fix test_object_lock_delete_object_with_retention_and_marker * fix tests * fix tests * fix tests * fix test itself * fix tests * fix test * Update weed/s3api/s3api_object_retention.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * reduce logs * address comments --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Chris Lu
@@ -23,16 +23,25 @@ import (
|
||||
|
||||
// Object lock validation errors
|
||||
var (
|
||||
ErrObjectLockVersioningRequired = errors.New("object lock headers can only be used on versioned buckets")
|
||||
ErrInvalidObjectLockMode = errors.New("invalid object lock mode")
|
||||
ErrInvalidLegalHoldStatus = errors.New("invalid legal hold status")
|
||||
ErrInvalidRetentionDateFormat = errors.New("invalid retention until date format")
|
||||
ErrRetentionDateMustBeFuture = errors.New("retention until date must be in the future")
|
||||
ErrObjectLockModeRequiresDate = errors.New("object lock mode requires retention until date")
|
||||
ErrRetentionDateRequiresMode = errors.New("retention until date requires object lock mode")
|
||||
ErrGovernanceBypassVersioningRequired = errors.New("governance bypass header can only be used on versioned buckets")
|
||||
ErrInvalidObjectLockDuration = errors.New("object lock duration must be greater than 0 days")
|
||||
ErrObjectLockDurationExceeded = errors.New("object lock duration exceeds maximum allowed days")
|
||||
ErrObjectLockVersioningRequired = errors.New("object lock headers can only be used on versioned buckets")
|
||||
ErrInvalidObjectLockMode = errors.New("invalid object lock mode")
|
||||
ErrInvalidLegalHoldStatus = errors.New("invalid legal hold status")
|
||||
ErrInvalidRetentionDateFormat = errors.New("invalid retention until date format")
|
||||
ErrRetentionDateMustBeFuture = errors.New("retain until date must be in the future")
|
||||
ErrObjectLockModeRequiresDate = errors.New("object lock mode requires retention until date")
|
||||
ErrRetentionDateRequiresMode = errors.New("retention until date requires object lock mode")
|
||||
ErrGovernanceBypassVersioningRequired = errors.New("governance bypass header can only be used on versioned buckets")
|
||||
ErrInvalidObjectLockDuration = errors.New("object lock duration must be greater than 0 days")
|
||||
ErrObjectLockDurationExceeded = errors.New("object lock duration exceeds maximum allowed days")
|
||||
ErrObjectLockConfigurationMissingEnabled = errors.New("object lock configuration must specify ObjectLockEnabled")
|
||||
ErrInvalidObjectLockEnabledValue = errors.New("invalid object lock enabled value")
|
||||
ErrRuleMissingDefaultRetention = errors.New("rule configuration must specify DefaultRetention")
|
||||
ErrDefaultRetentionMissingMode = errors.New("default retention must specify Mode")
|
||||
ErrInvalidDefaultRetentionMode = errors.New("invalid default retention mode")
|
||||
ErrDefaultRetentionMissingPeriod = errors.New("default retention must specify either Days or Years")
|
||||
ErrDefaultRetentionBothDaysAndYears = errors.New("default retention cannot specify both Days and Years")
|
||||
ErrDefaultRetentionDaysOutOfRange = errors.New("default retention days must be between 0 and 36500")
|
||||
ErrDefaultRetentionYearsOutOfRange = errors.New("default retention years must be between 0 and 100")
|
||||
)
|
||||
|
||||
func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -110,8 +119,8 @@ func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request)
|
||||
// For non-versioned buckets, check if existing object has object lock protections
|
||||
// that would prevent overwrite (PUT operations overwrite existing objects in non-versioned buckets)
|
||||
if !versioningEnabled {
|
||||
bypassGovernance := r.Header.Get("x-amz-bypass-governance-retention") == "true"
|
||||
if err := s3a.checkObjectLockPermissions(r, bucket, object, "", bypassGovernance); err != nil {
|
||||
governanceBypassAllowed := s3a.evaluateGovernanceBypassRequest(r, bucket, object)
|
||||
if err := s3a.enforceObjectLockProtections(r, bucket, object, "", governanceBypassAllowed); err != nil {
|
||||
glog.V(2).Infof("PutObjectHandler: object lock permissions check failed for %s/%s: %v", bucket, object, err)
|
||||
s3err.WriteErrorResponse(w, r, s3err.ErrAccessDenied)
|
||||
return
|
||||
@@ -460,7 +469,7 @@ func (s3a *S3ApiServer) applyBucketDefaultRetention(bucket string, entry *filer_
|
||||
return fmt.Errorf("default retention missing mode")
|
||||
}
|
||||
|
||||
if defaultRetention.Days == 0 && defaultRetention.Years == 0 {
|
||||
if !defaultRetention.DaysSet && !defaultRetention.YearsSet {
|
||||
return fmt.Errorf("default retention missing period")
|
||||
}
|
||||
|
||||
@@ -468,9 +477,9 @@ func (s3a *S3ApiServer) applyBucketDefaultRetention(bucket string, entry *filer_
|
||||
var retainUntilDate time.Time
|
||||
now := time.Now()
|
||||
|
||||
if defaultRetention.Days > 0 {
|
||||
if defaultRetention.DaysSet && defaultRetention.Days > 0 {
|
||||
retainUntilDate = now.AddDate(0, 0, defaultRetention.Days)
|
||||
} else if defaultRetention.Years > 0 {
|
||||
} else if defaultRetention.YearsSet && defaultRetention.Years > 0 {
|
||||
retainUntilDate = now.AddDate(defaultRetention.Years, 0, 0)
|
||||
}
|
||||
|
||||
@@ -553,26 +562,94 @@ func (s3a *S3ApiServer) validateObjectLockHeaders(r *http.Request, versioningEna
|
||||
|
||||
// mapValidationErrorToS3Error maps object lock validation errors to appropriate S3 error codes
|
||||
func mapValidationErrorToS3Error(err error) s3err.ErrorCode {
|
||||
// Check for sentinel errors first
|
||||
switch {
|
||||
case errors.Is(err, ErrObjectLockVersioningRequired):
|
||||
// For object lock operations on non-versioned buckets, return InvalidRequest
|
||||
// This matches the test expectations
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrInvalidObjectLockMode):
|
||||
// For invalid object lock mode, return InvalidRequest
|
||||
// This matches the test expectations
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrInvalidLegalHoldStatus):
|
||||
return s3err.ErrInvalidRequest
|
||||
// For invalid legal hold status in XML body, return MalformedXML
|
||||
// AWS S3 treats invalid status values in XML as malformed content
|
||||
return s3err.ErrMalformedXML
|
||||
case errors.Is(err, ErrInvalidRetentionDateFormat):
|
||||
// For malformed retention date format, return MalformedDate
|
||||
// This matches the test expectations
|
||||
return s3err.ErrMalformedDate
|
||||
case errors.Is(err, ErrRetentionDateMustBeFuture),
|
||||
errors.Is(err, ErrObjectLockModeRequiresDate),
|
||||
errors.Is(err, ErrRetentionDateRequiresMode):
|
||||
case errors.Is(err, ErrRetentionDateMustBeFuture):
|
||||
// For retention dates in the past, return InvalidRequest
|
||||
// This matches the test expectations
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrObjectLockModeRequiresDate):
|
||||
// For mode without retention date, return InvalidRequest
|
||||
// This matches the test expectations
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrRetentionDateRequiresMode):
|
||||
// For retention date without mode, return InvalidRequest
|
||||
// This matches the test expectations
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrGovernanceBypassVersioningRequired):
|
||||
// For governance bypass on non-versioned bucket, return InvalidRequest
|
||||
// This matches the test expectations
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrInvalidObjectLockDuration):
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrObjectLockDurationExceeded):
|
||||
return s3err.ErrInvalidRequest
|
||||
default:
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrMalformedXML):
|
||||
// For malformed XML in request body, return MalformedXML
|
||||
// This matches the test expectations for invalid retention mode and legal hold status
|
||||
return s3err.ErrMalformedXML
|
||||
case errors.Is(err, ErrInvalidRetentionPeriod):
|
||||
// For invalid retention period (e.g., Days <= 0), return InvalidRetentionPeriod
|
||||
// This matches the test expectations
|
||||
return s3err.ErrInvalidRetentionPeriod
|
||||
case errors.Is(err, ErrComplianceModeActive):
|
||||
// For compliance mode retention violations, return AccessDenied
|
||||
// This matches the test expectations
|
||||
return s3err.ErrAccessDenied
|
||||
case errors.Is(err, ErrGovernanceModeActive):
|
||||
// For governance mode retention violations, return AccessDenied
|
||||
// This matches the test expectations
|
||||
return s3err.ErrAccessDenied
|
||||
case errors.Is(err, ErrObjectUnderLegalHold):
|
||||
// For legal hold violations, return AccessDenied
|
||||
// This matches the test expectations
|
||||
return s3err.ErrAccessDenied
|
||||
case errors.Is(err, ErrGovernanceBypassNotPermitted):
|
||||
// For governance bypass permission violations, return AccessDenied
|
||||
// This matches the test expectations
|
||||
return s3err.ErrAccessDenied
|
||||
// Validation error constants
|
||||
case errors.Is(err, ErrObjectLockConfigurationMissingEnabled):
|
||||
return s3err.ErrMalformedXML
|
||||
case errors.Is(err, ErrInvalidObjectLockEnabledValue):
|
||||
return s3err.ErrMalformedXML
|
||||
case errors.Is(err, ErrRuleMissingDefaultRetention):
|
||||
return s3err.ErrMalformedXML
|
||||
case errors.Is(err, ErrDefaultRetentionMissingMode):
|
||||
return s3err.ErrMalformedXML
|
||||
case errors.Is(err, ErrInvalidDefaultRetentionMode):
|
||||
return s3err.ErrMalformedXML
|
||||
case errors.Is(err, ErrDefaultRetentionMissingPeriod):
|
||||
return s3err.ErrMalformedXML
|
||||
case errors.Is(err, ErrDefaultRetentionBothDaysAndYears):
|
||||
return s3err.ErrMalformedXML
|
||||
case errors.Is(err, ErrDefaultRetentionDaysOutOfRange):
|
||||
return s3err.ErrInvalidRetentionPeriod
|
||||
case errors.Is(err, ErrDefaultRetentionYearsOutOfRange):
|
||||
return s3err.ErrInvalidRetentionPeriod
|
||||
}
|
||||
|
||||
// Check for error constants from the updated validation functions
|
||||
switch {
|
||||
case errors.Is(err, ErrRetentionMissingMode):
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrRetentionMissingRetainUntilDate):
|
||||
return s3err.ErrInvalidRequest
|
||||
case errors.Is(err, ErrInvalidRetentionModeValue):
|
||||
return s3err.ErrMalformedXML
|
||||
}
|
||||
|
||||
return s3err.ErrInvalidRequest
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user