* s3: fix ListBuckets not showing buckets created by authenticated users Fixes #7647 ## Problem Users with proper Admin permissions could create buckets but couldn't list them. The issue occurred because ListBucketsHandler was not wrapped with the Auth middleware, so the authenticated identity was never set in the request context. ## Root Cause - PutBucketHandler uses iam.Auth() middleware which sets identity in context - ListBucketsHandler did NOT use iam.Auth() middleware - Without the middleware, GetIdentityNameFromContext() returned empty string - Bucket ownership checks failed because no identity was present ## Changes 1. Wrap ListBucketsHandler with iam.Auth() middleware (s3api_server.go) 2. Update ListBucketsHandler to get identity from context (s3api_bucket_handlers.go) 3. Add lookupByIdentityName() helper method (auth_credentials.go) 4. Add comprehensive test TestListBucketsIssue7647 (s3api_bucket_handlers_test.go) ## Testing - All existing tests pass (1348 tests in s3api package) - New test TestListBucketsIssue7647 validates the fix - Verified admin users can see their created buckets - Verified admin users can see all buckets - Verified backward compatibility maintained * s3: fix ListBuckets for JWT/Keycloak authentication The previous fix broke JWT/Keycloak authentication because JWT identities are created on-the-fly and not stored in the iam.identities list. The lookupByIdentityName() would return nil for JWT users. Solution: Store the full Identity object in the request context, not just the name. This allows ListBucketsHandler to retrieve the complete identity for all authentication types (SigV2, SigV4, JWT, Anonymous). Changes: - Add SetIdentityInContext/GetIdentityFromContext in s3_constants/header.go - Update Auth middleware to store full identity in context - Update ListBucketsHandler to retrieve identity from context first, with fallback to lookup for backward compatibility * s3: optimize lookupByIdentityName to O(1) using map Address code review feedback: Use a map for O(1) lookups instead of O(N) linear scan through identities list. Changes: - Add nameToIdentity map to IdentityAccessManagement struct - Populate map in loadS3ApiConfiguration (consistent with accessKeyIdent pattern) - Update lookupByIdentityName to use map lookup instead of loop This improves performance when many identities are configured and aligns with the existing pattern used for accessKeyIdent lookups. * s3: address code review feedback on nameToIdentity and logging Address two code review points: 1. Wire nameToIdentity into env-var fallback path - The AWS env-var fallback in NewIdentityAccessManagementWithStore now populates nameToIdentity map along with accessKeyIdent - Keeps all identity lookup maps in sync - Avoids potential issues if handlers rely on lookupByIdentityName 2. Improve access key lookup logging - Reduce log verbosity: V(1) -> V(2) for failed lookups - Truncate access keys in logs (show first 4 chars + ***) - Include key length for debugging - Prevents credential exposure in production logs - Reduces log noise from misconfigured clients * fmt * s3: refactor truncation logic and improve error handling Address additional code review feedback: 1. DRY principle: Extract key truncation logic into local function - Define truncate() helper at function start - Reuse throughout lookupByAccessKey - Eliminates code duplication 2. Enhanced security: Mask very short access keys - Keys <= 4 chars now show as '***' instead of full key - Prevents any credential exposure even for short keys - Consistent masking across all log statements 3. Improved robustness: Add warning log for type assertion failure - Log unexpected type when identity context object is wrong type - Helps debug potential middleware or context issues - Better production diagnostics 4. Documentation: Add comment about future optimization opportunity - Note potential for lightweight identity view in context - Suggests credential-free view for better data minimization - Documents design decision for future maintainers
218 lines
9.0 KiB
Go
218 lines
9.0 KiB
Go
/*
|
|
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
|
|
*
|
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
* you may not use this file except in compliance with the License.
|
|
* You may obtain a copy of the License at
|
|
*
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
*
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
* See the License for the specific language governing permissions and
|
|
* limitations under the License.
|
|
*/
|
|
|
|
package s3_constants
|
|
|
|
import (
|
|
"context"
|
|
"net/http"
|
|
"strings"
|
|
|
|
"github.com/gorilla/mux"
|
|
)
|
|
|
|
// S3 XML namespace
|
|
const (
|
|
S3Namespace = "http://s3.amazonaws.com/doc/2006-03-01/"
|
|
)
|
|
|
|
// Standard S3 HTTP request constants
|
|
const (
|
|
// S3 storage class
|
|
AmzStorageClass = "x-amz-storage-class"
|
|
|
|
// S3 user-defined metadata
|
|
AmzUserMetaPrefix = "X-Amz-Meta-"
|
|
AmzUserMetaDirective = "X-Amz-Metadata-Directive"
|
|
AmzUserMetaMtime = "X-Amz-Meta-Mtime"
|
|
|
|
// S3 object tagging
|
|
AmzObjectTagging = "X-Amz-Tagging"
|
|
AmzObjectTaggingPrefix = "X-Amz-Tagging-"
|
|
AmzObjectTaggingDirective = "X-Amz-Tagging-Directive"
|
|
AmzTagCount = "x-amz-tagging-count"
|
|
|
|
SeaweedFSUploadId = "X-Seaweedfs-Upload-Id"
|
|
SeaweedFSMultipartPartsCount = "X-Seaweedfs-Multipart-Parts-Count"
|
|
SeaweedFSMultipartPartBoundaries = "X-Seaweedfs-Multipart-Part-Boundaries" // JSON: [{part:1,start:0,end:2,etag:"abc"},{part:2,start:2,end:3,etag:"def"}]
|
|
SeaweedFSExpiresS3 = "X-Seaweedfs-Expires-S3"
|
|
AmzMpPartsCount = "x-amz-mp-parts-count"
|
|
|
|
// S3 ACL headers
|
|
AmzCannedAcl = "X-Amz-Acl"
|
|
AmzAclFullControl = "X-Amz-Grant-Full-Control"
|
|
AmzAclRead = "X-Amz-Grant-Read"
|
|
AmzAclWrite = "X-Amz-Grant-Write"
|
|
AmzAclReadAcp = "X-Amz-Grant-Read-Acp"
|
|
AmzAclWriteAcp = "X-Amz-Grant-Write-Acp"
|
|
|
|
// S3 Object Lock headers
|
|
AmzBucketObjectLockEnabled = "X-Amz-Bucket-Object-Lock-Enabled"
|
|
AmzObjectLockMode = "X-Amz-Object-Lock-Mode"
|
|
AmzObjectLockRetainUntilDate = "X-Amz-Object-Lock-Retain-Until-Date"
|
|
AmzObjectLockLegalHold = "X-Amz-Object-Lock-Legal-Hold"
|
|
|
|
// S3 conditional headers
|
|
IfMatch = "If-Match"
|
|
IfNoneMatch = "If-None-Match"
|
|
IfModifiedSince = "If-Modified-Since"
|
|
IfUnmodifiedSince = "If-Unmodified-Since"
|
|
|
|
// S3 conditional copy headers
|
|
AmzCopySourceIfMatch = "X-Amz-Copy-Source-If-Match"
|
|
AmzCopySourceIfNoneMatch = "X-Amz-Copy-Source-If-None-Match"
|
|
AmzCopySourceIfModifiedSince = "X-Amz-Copy-Source-If-Modified-Since"
|
|
AmzCopySourceIfUnmodifiedSince = "X-Amz-Copy-Source-If-Unmodified-Since"
|
|
|
|
// S3 Server-Side Encryption with Customer-provided Keys (SSE-C)
|
|
AmzServerSideEncryptionCustomerAlgorithm = "X-Amz-Server-Side-Encryption-Customer-Algorithm"
|
|
AmzServerSideEncryptionCustomerKey = "X-Amz-Server-Side-Encryption-Customer-Key"
|
|
AmzServerSideEncryptionCustomerKeyMD5 = "X-Amz-Server-Side-Encryption-Customer-Key-MD5"
|
|
AmzServerSideEncryptionContext = "X-Amz-Server-Side-Encryption-Context"
|
|
|
|
// S3 Server-Side Encryption with KMS (SSE-KMS)
|
|
AmzServerSideEncryption = "X-Amz-Server-Side-Encryption"
|
|
AmzServerSideEncryptionAwsKmsKeyId = "X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"
|
|
AmzServerSideEncryptionBucketKeyEnabled = "X-Amz-Server-Side-Encryption-Bucket-Key-Enabled"
|
|
|
|
// S3 SSE-C copy source headers
|
|
AmzCopySourceServerSideEncryptionCustomerAlgorithm = "X-Amz-Copy-Source-Server-Side-Encryption-Customer-Algorithm"
|
|
AmzCopySourceServerSideEncryptionCustomerKey = "X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key"
|
|
AmzCopySourceServerSideEncryptionCustomerKeyMD5 = "X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key-MD5"
|
|
)
|
|
|
|
// Metadata keys for internal storage
|
|
const (
|
|
// SSE-KMS metadata keys
|
|
AmzEncryptedDataKey = "x-amz-encrypted-data-key"
|
|
AmzEncryptionContextMeta = "x-amz-encryption-context"
|
|
|
|
// SeaweedFS internal metadata prefix (used to filter internal headers from client responses)
|
|
SeaweedFSInternalPrefix = "x-seaweedfs-"
|
|
|
|
// SeaweedFS internal metadata keys for encryption (prefixed to avoid automatic HTTP header conversion)
|
|
SeaweedFSSSEKMSKey = "x-seaweedfs-sse-kms-key" // Key for storing serialized SSE-KMS metadata
|
|
SeaweedFSSSES3Key = "x-seaweedfs-sse-s3-key" // Key for storing serialized SSE-S3 metadata
|
|
SeaweedFSSSEIV = "x-seaweedfs-sse-c-iv" // Key for storing SSE-C IV
|
|
|
|
// Multipart upload metadata keys for SSE-KMS (consistent with internal metadata key pattern)
|
|
SeaweedFSSSEKMSKeyID = "x-seaweedfs-sse-kms-key-id" // Key ID for multipart upload SSE-KMS inheritance
|
|
SeaweedFSSSEKMSEncryption = "x-seaweedfs-sse-kms-encryption" // Encryption type for multipart upload SSE-KMS inheritance
|
|
SeaweedFSSSEKMSBucketKeyEnabled = "x-seaweedfs-sse-kms-bucket-key-enabled" // Bucket key setting for multipart upload SSE-KMS inheritance
|
|
SeaweedFSSSEKMSEncryptionContext = "x-seaweedfs-sse-kms-encryption-context" // Encryption context for multipart upload SSE-KMS inheritance
|
|
SeaweedFSSSEKMSBaseIV = "x-seaweedfs-sse-kms-base-iv" // Base IV for multipart upload SSE-KMS (for IV offset calculation)
|
|
|
|
// Multipart upload metadata keys for SSE-S3
|
|
SeaweedFSSSES3Encryption = "x-seaweedfs-sse-s3-encryption" // Encryption type for multipart upload SSE-S3 inheritance
|
|
SeaweedFSSSES3BaseIV = "x-seaweedfs-sse-s3-base-iv" // Base IV for multipart upload SSE-S3 (for IV offset calculation)
|
|
SeaweedFSSSES3KeyData = "x-seaweedfs-sse-s3-key-data" // Encrypted key data for multipart upload SSE-S3 inheritance
|
|
)
|
|
|
|
// SeaweedFS internal headers for filer communication
|
|
const (
|
|
SeaweedFSSSEKMSKeyHeader = "X-SeaweedFS-SSE-KMS-Key" // Header for passing SSE-KMS metadata to filer
|
|
SeaweedFSSSEIVHeader = "X-SeaweedFS-SSE-IV" // Header for passing SSE-C IV to filer (SSE-C only)
|
|
SeaweedFSSSEKMSBaseIVHeader = "X-SeaweedFS-SSE-KMS-Base-IV" // Header for passing base IV for multipart SSE-KMS
|
|
SeaweedFSSSES3BaseIVHeader = "X-SeaweedFS-SSE-S3-Base-IV" // Header for passing base IV for multipart SSE-S3
|
|
SeaweedFSSSES3KeyDataHeader = "X-SeaweedFS-SSE-S3-Key-Data" // Header for passing key data for multipart SSE-S3
|
|
)
|
|
|
|
// Non-Standard S3 HTTP request constants
|
|
const (
|
|
AmzIdentityId = "s3-identity-id"
|
|
AmzAccountId = "s3-account-id"
|
|
AmzAuthType = "s3-auth-type"
|
|
)
|
|
|
|
func GetBucketAndObject(r *http.Request) (bucket, object string) {
|
|
vars := mux.Vars(r)
|
|
bucket = vars["bucket"]
|
|
object = vars["object"]
|
|
if !strings.HasPrefix(object, "/") {
|
|
object = "/" + object
|
|
}
|
|
|
|
return
|
|
}
|
|
|
|
func GetPrefix(r *http.Request) string {
|
|
query := r.URL.Query()
|
|
prefix := query.Get("prefix")
|
|
if !strings.HasPrefix(prefix, "/") {
|
|
prefix = "/" + prefix
|
|
}
|
|
|
|
return prefix
|
|
}
|
|
|
|
var PassThroughHeaders = map[string]string{
|
|
"response-cache-control": "Cache-Control",
|
|
"response-content-disposition": "Content-Disposition",
|
|
"response-content-encoding": "Content-Encoding",
|
|
"response-content-language": "Content-Language",
|
|
"response-content-type": "Content-Type",
|
|
"response-expires": "Expires",
|
|
}
|
|
|
|
// IsSeaweedFSInternalHeader checks if a header key is a SeaweedFS internal header
|
|
// that should be filtered from client responses.
|
|
// Header names are case-insensitive in HTTP, so this function normalizes to lowercase.
|
|
func IsSeaweedFSInternalHeader(headerKey string) bool {
|
|
return strings.HasPrefix(strings.ToLower(headerKey), SeaweedFSInternalPrefix)
|
|
}
|
|
|
|
// Context keys for storing authenticated identity information
|
|
type contextKey string
|
|
|
|
const (
|
|
contextKeyIdentityName contextKey = "s3-identity-name"
|
|
contextKeyIdentityObject contextKey = "s3-identity-object"
|
|
)
|
|
|
|
// SetIdentityNameInContext stores the authenticated identity name in the request context
|
|
// This is the secure way to propagate identity - headers can be spoofed, context cannot
|
|
func SetIdentityNameInContext(ctx context.Context, identityName string) context.Context {
|
|
if identityName != "" {
|
|
return context.WithValue(ctx, contextKeyIdentityName, identityName)
|
|
}
|
|
return ctx
|
|
}
|
|
|
|
// GetIdentityNameFromContext retrieves the authenticated identity name from the request context
|
|
// Returns empty string if no identity is set (unauthenticated request)
|
|
// This is the secure way to retrieve identity - never read from headers directly
|
|
func GetIdentityNameFromContext(r *http.Request) string {
|
|
if name, ok := r.Context().Value(contextKeyIdentityName).(string); ok {
|
|
return name
|
|
}
|
|
return ""
|
|
}
|
|
|
|
// SetIdentityInContext stores the full authenticated identity object in the request context
|
|
// This is used to pass the full identity (including for JWT users) to handlers
|
|
func SetIdentityInContext(ctx context.Context, identity interface{}) context.Context {
|
|
if identity != nil {
|
|
return context.WithValue(ctx, contextKeyIdentityObject, identity)
|
|
}
|
|
return ctx
|
|
}
|
|
|
|
// GetIdentityFromContext retrieves the full identity object from the request context
|
|
// Returns nil if no identity is set (unauthenticated request)
|
|
func GetIdentityFromContext(r *http.Request) interface{} {
|
|
return r.Context().Value(contextKeyIdentityObject)
|
|
}
|