Files
seaweedFS/weed/s3api/s3_iam_simple_test.go
Chris Lu b9fa05153a Allow multipart upload operations when s3:PutObject is authorized (#8445)
* Allow multipart upload operations when s3:PutObject is authorized

Multipart upload is an implementation detail of putting objects, not a separate
permission. When a policy grants s3:PutObject, it should implicitly allow:
- s3:CreateMultipartUpload
- s3:UploadPart
- s3:CompleteMultipartUpload
- s3:AbortMultipartUpload
- s3:ListParts

This fixes a compatibility issue where clients like PyArrow that use multipart
uploads by default would fail even though the role had s3:PutObject permission.
The session policy intersection still applies - both the identity-based policy
AND session policy must allow s3:PutObject for multipart operations to work.

Implementation:
- Added constants for S3 multipart action strings
- Added multipartActionSet to efficiently check if action is multipart-related
- Updated MatchesAction method to implicitly grant multipart when PutObject allowed

* Update weed/s3api/policy_engine/types.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Add s3:ListMultipartUploads to multipart action set

Include s3:ListMultipartUploads in the multipartActionSet so that listing
multipart uploads is implicitly granted when s3:PutObject is authorized.
ListMultipartUploads is a critical part of the multipart upload workflow,
allowing clients to query in-progress uploads before completing them.

Changes:
- Added s3ListMultipartUploads constant definition
- Included s3ListMultipartUploads in multipartActionSet initialization
- Existing references to multipartActionSet automatically now cover ListMultipartUploads

All policy engine tests pass (0.351s execution time)

* Refactor: reuse multipart action constants from s3_constants package

Remove duplicate constant definitions from policy_engine/types.go and import
the canonical definitions from s3api/s3_constants/s3_actions.go instead.
This eliminates duplication and ensures a single source of truth for
multipart action strings:

- ACTION_CREATE_MULTIPART_UPLOAD
- ACTION_UPLOAD_PART
- ACTION_COMPLETE_MULTIPART
- ACTION_ABORT_MULTIPART
- ACTION_LIST_PARTS
- ACTION_LIST_MULTIPART_UPLOADS

All policy engine tests pass (0.350s execution time)

* Fix S3_ACTION_LIST_MULTIPART_UPLOADS constant value

Move S3_ACTION_LIST_MULTIPART_UPLOADS from bucket operations to multipart
operations section and change value from 's3:ListBucketMultipartUploads' to
's3:ListMultipartUploads' to match the action strings used in policy_engine
and s3_actions.go.

This ensures consistent action naming across all S3 constant definitions.

* refactor names

* Fix S3 action constant mismatches and MatchesAction early return bug

Fix two critical issues in policy engine:

1. S3Actions map had incorrect multipart action mappings:
   - 'ListMultipartUploads' was 's3:ListMultipartUploads' (should be 's3:ListBucketMultipartUploads')
   - 'ListParts' was 's3:ListParts' (should be 's3:ListMultipartUploadParts')
   These mismatches caused authorization checks to fail for list operations

2. CompiledStatement.MatchesAction() had early return bug:
   - Previously returned true immediately upon first direct action match
   - This prevented scanning remaining matchers for s3:PutObject permission
   - Now scans ALL matchers before returning, tracking both direct match and PutObject grant
   - Ensures multipart operations inherit s3:PutObject authorization even when
     explicitly requested action doesn't match (e.g., s3:ListMultipartUploadParts)

Changes:
- Track matchedAction flag to defer
Fix two critical issues in policy engine:

1. S3Actions map had incorrect multipart action mappings:
   - 'ListMultipartUploads' was 's3:ListMultipartUplPer
1. S3Actions map had incorrect multiparAll   - 'ListMultipartUploads(0.334s execution time)

* Refactor S3Actions map to use s3_constants

Replace hardcoded action strings in the S3Actions map with references to
canonical S3_ACTION_* constants from s3_constants/s3_action_strings.go.

Benefits:
- Single source of truth for S3 action values
- Eliminates string duplication across codebase
- Ensures consistency between policy engine and middleware
- Reduces maintenance burden when action strings need updates

All policy engine tests pass (0.334s execution time)

* Remove unused S3Actions map

The S3Actions map in types.go was never referenced anywhere in the codebase.
All action mappings are handled by GetActionMappings() in integration.go instead.
This removes 42 lines of dead code.

* Fix test: reload configuration function must also reload IAM state

TestEmbeddedIamAttachUserPolicyRefreshesIAM was failing because the test's
reloadConfigurationFunc only updated mockConfig but didn't reload the actual IAM
state. When AttachUserPolicy calls refreshIAMConfiguration(), it would use the
test's incomplete reload function instead of the real LoadS3ApiConfigurationFromCredentialManager().

Fixed by making the test's reloadConfigurationFunc also call
e.iam.LoadS3ApiConfigurationFromCredentialManager() so lookupByIdentityName()
sees the updated policy attachments.

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-25 12:31:04 -08:00

506 lines
14 KiB
Go

package s3api
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/iam/integration"
"github.com/seaweedfs/seaweedfs/weed/iam/policy"
"github.com/seaweedfs/seaweedfs/weed/iam/sts"
"github.com/seaweedfs/seaweedfs/weed/iam/utils"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestS3IAMMiddleware tests the basic S3 IAM middleware functionality
func TestS3IAMMiddleware(t *testing.T) {
// Create IAM manager
iamManager := integration.NewIAMManager()
// Initialize with test configuration
config := &integration.IAMConfig{
STS: &sts.STSConfig{
TokenDuration: sts.FlexibleDuration{Duration: time.Hour},
MaxSessionLength: sts.FlexibleDuration{Duration: time.Hour * 12},
Issuer: "test-sts",
SigningKey: []byte("test-signing-key-32-characters-long"),
},
Policy: &policy.PolicyEngineConfig{
DefaultEffect: "Deny",
StoreType: "memory",
},
Roles: &integration.RoleStoreConfig{
StoreType: "memory",
},
}
err := iamManager.Initialize(config, func() string {
return "localhost:8888" // Mock filer address for testing
})
require.NoError(t, err)
// Create S3 IAM integration
s3IAMIntegration := NewS3IAMIntegration(iamManager, "localhost:8888")
// Test that integration is created successfully
assert.NotNil(t, s3IAMIntegration)
assert.True(t, s3IAMIntegration.enabled)
}
// TestS3IAMMiddlewareJWTAuth tests JWT authentication
func TestS3IAMMiddlewareJWTAuth(t *testing.T) {
// Skip for now since it requires full setup
t.Skip("JWT authentication test requires full IAM setup")
// Create IAM integration
s3iam := NewS3IAMIntegration(nil, "localhost:8888") // Disabled integration
// Create test request with JWT token
req := httptest.NewRequest("GET", "/test-bucket/test-object", http.NoBody)
req.Header.Set("Authorization", "Bearer test-token")
// Test authentication (should return not implemented when disabled)
ctx := context.Background()
identity, errCode := s3iam.AuthenticateJWT(ctx, req)
assert.Nil(t, identity)
assert.NotEqual(t, errCode, 0) // Should return an error
}
// TestBuildS3ResourceArn tests resource ARN building
func TestBuildS3ResourceArn(t *testing.T) {
tests := []struct {
name string
bucket string
object string
expected string
}{
{
name: "empty bucket and object",
bucket: "",
object: "",
expected: "arn:aws:s3:::*",
},
{
name: "bucket only",
bucket: "test-bucket",
object: "",
expected: "arn:aws:s3:::test-bucket",
},
{
name: "bucket and object",
bucket: "test-bucket",
object: "test-object.txt",
expected: "arn:aws:s3:::test-bucket/test-object.txt",
},
{
name: "bucket and object with leading slash",
bucket: "test-bucket",
object: "/test-object.txt",
expected: "arn:aws:s3:::test-bucket/test-object.txt",
},
{
name: "bucket and nested object",
bucket: "test-bucket",
object: "folder/subfolder/test-object.txt",
expected: "arn:aws:s3:::test-bucket/folder/subfolder/test-object.txt",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := buildS3ResourceArn(tt.bucket, tt.object)
assert.Equal(t, tt.expected, result)
})
}
}
// TestDetermineGranularS3Action tests granular S3 action determination from HTTP requests
func TestDetermineGranularS3Action(t *testing.T) {
tests := []struct {
name string
method string
bucket string
objectKey string
queryParams map[string]string
fallbackAction Action
expected string
description string
}{
// Object-level operations
{
name: "get_object",
method: "GET",
bucket: "test-bucket",
objectKey: "test-object.txt",
queryParams: map[string]string{},
fallbackAction: s3_constants.ACTION_READ,
expected: "s3:GetObject",
description: "Basic object retrieval",
},
{
name: "get_object_acl",
method: "GET",
bucket: "test-bucket",
objectKey: "test-object.txt",
queryParams: map[string]string{"acl": ""},
fallbackAction: s3_constants.ACTION_READ_ACP,
expected: "s3:GetObjectAcl",
description: "Object ACL retrieval",
},
{
name: "get_object_tagging",
method: "GET",
bucket: "test-bucket",
objectKey: "test-object.txt",
queryParams: map[string]string{"tagging": ""},
fallbackAction: s3_constants.ACTION_TAGGING,
expected: "s3:GetObjectTagging",
description: "Object tagging retrieval",
},
{
name: "put_object",
method: "PUT",
bucket: "test-bucket",
objectKey: "test-object.txt",
queryParams: map[string]string{},
fallbackAction: s3_constants.ACTION_WRITE,
expected: "s3:PutObject",
description: "Basic object upload",
},
{
name: "put_object_acl",
method: "PUT",
bucket: "test-bucket",
objectKey: "test-object.txt",
queryParams: map[string]string{"acl": ""},
fallbackAction: s3_constants.ACTION_WRITE_ACP,
expected: "s3:PutObjectAcl",
description: "Object ACL modification",
},
{
name: "delete_object",
method: "DELETE",
bucket: "test-bucket",
objectKey: "test-object.txt",
queryParams: map[string]string{},
fallbackAction: s3_constants.ACTION_WRITE, // DELETE object uses WRITE fallback
expected: "s3:DeleteObject",
description: "Object deletion - correctly mapped to DeleteObject (not PutObject)",
},
{
name: "delete_object_tagging",
method: "DELETE",
bucket: "test-bucket",
objectKey: "test-object.txt",
queryParams: map[string]string{"tagging": ""},
fallbackAction: s3_constants.ACTION_TAGGING,
expected: "s3:DeleteObjectTagging",
description: "Object tag deletion",
},
// Multipart upload operations
{
name: "create_multipart_upload",
method: "POST",
bucket: "test-bucket",
objectKey: "large-file.txt",
queryParams: map[string]string{"uploads": ""},
fallbackAction: s3_constants.ACTION_WRITE,
expected: "s3:CreateMultipartUpload",
description: "Multipart upload initiation",
},
{
name: "upload_part",
method: "PUT",
bucket: "test-bucket",
objectKey: "large-file.txt",
queryParams: map[string]string{"uploadId": "12345", "partNumber": "1"},
fallbackAction: s3_constants.ACTION_WRITE,
expected: "s3:UploadPart",
description: "Multipart part upload",
},
{
name: "complete_multipart_upload",
method: "POST",
bucket: "test-bucket",
objectKey: "large-file.txt",
queryParams: map[string]string{"uploadId": "12345"},
fallbackAction: s3_constants.ACTION_WRITE,
expected: "s3:CompleteMultipartUpload",
description: "Multipart upload completion",
},
{
name: "abort_multipart_upload",
method: "DELETE",
bucket: "test-bucket",
objectKey: "large-file.txt",
queryParams: map[string]string{"uploadId": "12345"},
fallbackAction: s3_constants.ACTION_WRITE,
expected: "s3:AbortMultipartUpload",
description: "Multipart upload abort",
},
// Bucket-level operations
{
name: "list_bucket",
method: "GET",
bucket: "test-bucket",
objectKey: "",
queryParams: map[string]string{},
fallbackAction: s3_constants.ACTION_LIST,
expected: "s3:ListBucket",
description: "Bucket listing",
},
{
name: "get_bucket_acl",
method: "GET",
bucket: "test-bucket",
objectKey: "",
queryParams: map[string]string{"acl": ""},
fallbackAction: s3_constants.ACTION_READ_ACP,
expected: "s3:GetBucketAcl",
description: "Bucket ACL retrieval",
},
{
name: "put_bucket_policy",
method: "PUT",
bucket: "test-bucket",
objectKey: "",
queryParams: map[string]string{"policy": ""},
fallbackAction: s3_constants.ACTION_WRITE,
expected: "s3:PutBucketPolicy",
description: "Bucket policy modification",
},
{
name: "delete_bucket",
method: "DELETE",
bucket: "test-bucket",
objectKey: "",
queryParams: map[string]string{},
fallbackAction: s3_constants.ACTION_DELETE_BUCKET,
expected: "s3:DeleteBucket",
description: "Bucket deletion",
},
{
name: "list_multipart_uploads",
method: "GET",
bucket: "test-bucket",
objectKey: "",
queryParams: map[string]string{"uploads": ""},
fallbackAction: s3_constants.ACTION_LIST,
expected: "s3:ListBucketMultipartUploads",
description: "List multipart uploads in bucket",
},
// Fallback scenarios
{
name: "legacy_read_fallback",
method: "GET",
bucket: "",
objectKey: "",
queryParams: map[string]string{},
fallbackAction: s3_constants.ACTION_READ,
expected: "s3:GetObject",
description: "Legacy read action fallback",
},
{
name: "already_granular_action",
method: "GET",
bucket: "",
objectKey: "",
queryParams: map[string]string{},
fallbackAction: "s3:GetBucketLocation", // Already granular
expected: "s3:GetBucketLocation",
description: "Already granular action passed through",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Create HTTP request with query parameters
req := &http.Request{
Method: tt.method,
URL: &url.URL{Path: "/" + tt.bucket + "/" + tt.objectKey},
}
// Add query parameters
query := req.URL.Query()
for key, value := range tt.queryParams {
query.Set(key, value)
}
req.URL.RawQuery = query.Encode()
// Test the action determination
result := ResolveS3Action(req, string(tt.fallbackAction), tt.bucket, tt.objectKey)
assert.Equal(t, tt.expected, result,
"Test %s failed: %s. Expected %s but got %s",
tt.name, tt.description, tt.expected, result)
})
}
}
// TestMapLegacyActionToIAM tests the legacy action fallback mapping
func TestMapLegacyActionToIAM(t *testing.T) {
tests := []struct {
name string
legacyAction Action
expected string
}{
{
name: "read_action_fallback",
legacyAction: s3_constants.ACTION_READ,
expected: "s3:GetObject",
},
{
name: "write_action_fallback",
legacyAction: s3_constants.ACTION_WRITE,
expected: "s3:PutObject",
},
{
name: "admin_action_fallback",
legacyAction: s3_constants.ACTION_ADMIN,
expected: "s3:*",
},
{
name: "granular_multipart_action",
legacyAction: s3_constants.S3_ACTION_CREATE_MULTIPART,
expected: s3_constants.S3_ACTION_CREATE_MULTIPART,
},
{
name: "unknown_action_with_s3_prefix",
legacyAction: "s3:CustomAction",
expected: "s3:CustomAction",
},
{
name: "unknown_action_without_prefix",
legacyAction: "CustomAction",
expected: "s3:CustomAction",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := mapLegacyActionToIAM(tt.legacyAction)
assert.Equal(t, tt.expected, result)
})
}
}
// TestExtractSourceIP tests source IP extraction from requests
func TestExtractSourceIP(t *testing.T) {
tests := []struct {
name string
setupReq func() *http.Request
expectedIP string
}{
{
name: "X-Forwarded-For header",
setupReq: func() *http.Request {
req := httptest.NewRequest("GET", "/test", http.NoBody)
req.Header.Set("X-Forwarded-For", "192.168.1.100, 10.0.0.1")
// Set RemoteAddr to private IP to simulate trusted proxy
req.RemoteAddr = "127.0.0.1:12345"
return req
},
expectedIP: "192.168.1.100",
},
{
name: "X-Real-IP header",
setupReq: func() *http.Request {
req := httptest.NewRequest("GET", "/test", http.NoBody)
req.Header.Set("X-Real-IP", "192.168.1.200")
// Set RemoteAddr to private IP to simulate trusted proxy
req.RemoteAddr = "127.0.0.1:12345"
return req
},
expectedIP: "192.168.1.200",
},
{
name: "RemoteAddr fallback",
setupReq: func() *http.Request {
req := httptest.NewRequest("GET", "/test", http.NoBody)
req.RemoteAddr = "192.168.1.300:12345"
return req
},
expectedIP: "192.168.1.300",
},
{
name: "Untrusted proxy - public RemoteAddr ignores X-Forwarded-For",
setupReq: func() *http.Request {
req := httptest.NewRequest("GET", "/test", http.NoBody)
req.Header.Set("X-Forwarded-For", "192.168.1.100")
// Public IP - headers should NOT be trusted
req.RemoteAddr = "8.8.8.8:12345"
return req
},
expectedIP: "8.8.8.8", // Should use RemoteAddr, not X-Forwarded-For
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req := tt.setupReq()
result := extractSourceIP(req)
assert.Equal(t, tt.expectedIP, result)
})
}
}
// TestExtractRoleNameFromPrincipal tests role name extraction
func TestExtractRoleNameFromPrincipal(t *testing.T) {
tests := []struct {
name string
principal string
expected string
}{
{
name: "valid assumed role ARN",
principal: "arn:aws:sts::assumed-role/S3ReadOnlyRole/session-123",
expected: "S3ReadOnlyRole",
},
{
name: "invalid format",
principal: "invalid-principal",
expected: "", // Returns empty string to signal invalid format
},
{
name: "missing session name",
principal: "arn:aws:sts::assumed-role/TestRole",
expected: "TestRole", // Extracts role name even without session name
},
{
name: "empty principal",
principal: "",
expected: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := utils.ExtractRoleNameFromPrincipal(tt.principal)
assert.Equal(t, tt.expected, result)
})
}
}
// TestIAMIdentityIsAdmin tests the IsAdmin method
func TestIAMIdentityIsAdmin(t *testing.T) {
identity := &IAMIdentity{
Name: "test-identity",
Principal: "arn:aws:sts::assumed-role/TestRole/session",
SessionToken: "test-token",
}
// In our implementation, IsAdmin always returns false since admin status
// is determined by policies, not identity
result := identity.IsAdmin()
assert.False(t, result)
}