Add TUS protocol support for resumable uploads (#7592)

* Add TUS protocol integration tests This commit adds integration tests for the TUS (resumable upload) protocol in preparation for implementing TUS support in the filer. Test coverage includes: - OPTIONS handler for capability discovery - Basic single-request upload - Chunked/resumable uploads - HEAD requests for offset tracking - DELETE for upload cancellation - Error handling (invalid offsets, missing uploads) - Creation-with-upload extension - Resume after interruption simulation Tests are skipped in short mode and require a running SeaweedFS cluster. * Add TUS session storage types and utilities Implements TUS upload session management: - TusSession struct for tracking upload state - Session creation with directory-based storage - Session persistence using filer entries - Session retrieval and offset updates - Session deletion with chunk cleanup - Upload completion with chunk assembly into final file Session data is stored in /.uploads.tus/{upload-id}/ directory, following the pattern used by S3 multipart uploads. * Add TUS HTTP handlers Implements TUS protocol HTTP handlers: - tusHandler: Main entry point routing requests - tusOptionsHandler: Capability discovery (OPTIONS) - tusCreateHandler: Create new upload (POST) - tusHeadHandler: Get upload offset (HEAD) - tusPatchHandler: Upload data at offset (PATCH) - tusDeleteHandler: Cancel upload (DELETE) - tusWriteData: Upload data to volume servers Features: - Supports creation-with-upload extension - Validates TUS protocol headers - Offset conflict detection - Automatic upload completion when size is reached - Metadata parsing from Upload-Metadata header * Wire up TUS protocol routes in filer server Add TUS handler route (/.tus/) to the filer HTTP server. The TUS route is registered before the catch-all route to ensure proper routing of TUS protocol requests. TUS protocol is now accessible at: - OPTIONS /.tus/ - Capability discovery - POST /.tus/{path} - Create upload - HEAD /.tus/.uploads/{id} - Get offset - PATCH /.tus/.uploads/{id} - Upload data - DELETE /.tus/.uploads/{id} - Cancel upload * Improve TUS integration test setup Add comprehensive Makefile for TUS tests with targets: - test-with-server: Run tests with automatic server management - test-basic/chunked/resume/errors: Specific test categories - manual-start/stop: For development testing - debug-logs/status: For debugging - ci-test: For CI/CD pipelines Update README.md with: - Detailed TUS protocol documentation - All endpoint descriptions with headers - Usage examples with curl commands - Architecture diagram - Comparison with S3 multipart uploads Follows the pattern established by other tests in test/ folder. * Fix TUS integration tests and creation-with-upload - Fix test URLs to use full URLs instead of relative paths - Fix creation-with-upload to refresh session before completing - Fix Makefile to properly handle test cleanup - Add FullURL helper function to TestCluster * Add TUS protocol tests to GitHub Actions CI - Add tus-tests.yml workflow that runs on PRs and pushes - Runs when TUS-related files are modified - Automatic server management for integration testing - Upload logs on failure for debugging * Make TUS base path configurable via CLI - Add -tus.path CLI flag to filer command - TUS is disabled by default (empty path) - Example: -tus.path=/.tus to enable at /.tus endpoint - Update test Makefile to use -tus.path flag - Update README with TUS enabling instructions * Rename -tus.path to -tusBasePath with default .tus - Rename CLI flag from -tus.path to -tusBasePath - Default to .tus (TUS enabled by default) - Add -filer.tusBasePath option to weed server command - Properly handle path prefix (prepend / if missing) * Address code review comments - Sort chunks by offset before assembling final file - Use chunk.Offset directly instead of recalculating - Return error on invalid file ID instead of skipping - Require Content-Length header for PATCH requests - Use fs.option.Cipher for encryption setting - Detect MIME type from data using http.DetectContentType - Fix concurrency group for push events in workflow - Use os.Interrupt instead of Kill for graceful shutdown in tests * fmt * Address remaining code review comments - Fix potential open redirect vulnerability by sanitizing uploadLocation path - Add language specifier to README code block - Handle os.Create errors in test setup - Use waitForHTTPServer instead of time.Sleep for master/volume readiness - Improve test reliability and debugging * Address critical and high-priority review comments - Add per-session locking to prevent race conditions in updateTusSessionOffset - Stream data directly to volume server instead of buffering entire chunk - Only buffer 512 bytes for MIME type detection, then stream remaining data - Clean up session locks when session is deleted * Fix race condition to work across multiple filer instances - Store each chunk as a separate file entry instead of updating session JSON - Chunk file names encode offset, size, and fileId for atomic storage - getTusSession loads chunks from directory listing (atomic read) - Eliminates read-modify-write race condition across multiple filers - Remove in-memory mutex that only worked for single filer instance * Address code review comments: fix variable shadowing, sniff size, and test stability - Rename path variable to reqPath to avoid shadowing path package - Make sniff buffer size respect contentLength (read at most contentLength bytes) - Handle Content-Length < 0 in creation-with-upload (return error for chunked encoding) - Fix test cluster: use temp directory for filer store, add startup delay * Fix test stability: increase cluster stabilization delay to 5 seconds The tests were intermittently failing because the volume server needed more time to create volumes and register with the master. Increasing the delay from 2 to 5 seconds fixes the flaky test behavior. * Address PR review comments for TUS protocol support - Fix strconv.Atoi error handling in test file (lines 386, 747) - Fix lossy fileId encoding: use base64 instead of underscore replacement - Add pagination support for ListDirectoryEntries in getTusSession - Batch delete chunks instead of one-by-one in deleteTusSession * Address additional PR review comments for TUS protocol - Fix UploadAt timestamp: use entry.Crtime instead of time.Now() - Remove redundant JSON content in chunk entry (metadata in filename) - Refactor tusWriteData to stream in 4MB chunks to avoid OOM on large uploads - Pass filer.Entry to parseTusChunkPath to preserve actual upload time * Address more PR review comments for TUS protocol - Normalize TUS path once in filer_server.go, store in option.TusPath - Remove redundant path normalization from TUS handlers - Remove goto statement in tusCreateHandler, simplify control flow * Remove unnecessary mutexes in tusWriteData The upload loop is sequential, so uploadErrLock and chunksLock are not needed. * Rename updateTusSessionOffset to saveTusChunk Remove unused newOffset parameter and rename function to better reflect its purpose. * Improve TUS upload performance and add path validation - Reuse operation.Uploader across sub-chunks for better connection reuse - Guard against TusPath='/' to prevent hijacking all filer routes * Address PR review comments for TUS protocol - Fix critical chunk filename parsing: use strings.Cut instead of SplitN to correctly handle base64-encoded fileIds that may contain underscores - Rename tusPath to tusBasePath for naming consistency across codebase - Add background garbage collection for expired TUS sessions (runs hourly) - Improve error messages with %w wrapping for better debuggability * Address additional TUS PR review comments - Fix tusBasePath default to use leading slash (/.tus) for consistency - Add chunk contiguity validation in completeTusUpload to detect gaps/overlaps - Fix offset calculation to find maximum contiguous range from 0, not just last chunk - Return 413 Request Entity Too Large instead of silently truncating content - Document tusChunkSize rationale (4MB balances memory vs request overhead) - Fix Makefile xargs portability by removing GNU-specific -r flag - Add explicit -tusBasePath flag to integration test for robustness - Fix README example to use /.uploads/tus path format * Revert log_buffer changes (moved to separate PR) * Minor style fixes from PR review - Simplify tusBasePath flag description to use example format - Add 'TUS upload' prefix to session not found error message - Remove duplicate tusChunkSize comment - Capitalize warning message for consistency - Add grep filter to Makefile xargs for better empty input handling
2025-12-14 21:56:07 -08:00
parent 221b352593
commit 1b1e5f69a2
12 changed files with 3058 additions and 13 deletions
--- a/weed/s3api/s3api_object_handlers_copy.go
+++ b/weed/s3api/s3api_object_handlers_copy.go
@@ -167,6 +167,14 @@ func (s3a *S3ApiServer) CopyObjectHandler(w http.ResponseWriter, r *http.Request
 	}

 	// Copy extended attributes from source, filtering out conflicting encryption metadata
+	// Pre-compute encryption state once for efficiency
+	srcHasSSEC := IsSSECEncrypted(entry.Extended)
+	srcHasSSEKMS := IsSSEKMSEncrypted(entry.Extended)
+	srcHasSSES3 := IsSSES3EncryptedInternal(entry.Extended)
+	dstWantsSSEC := IsSSECRequest(r)
+	dstWantsSSEKMS := IsSSEKMSRequest(r)
+	dstWantsSSES3 := IsSSES3RequestInternal(r)
+
 	for k, v := range entry.Extended {
 		// Skip encryption-specific headers that might conflict with destination encryption type
 		skipHeader := false
@@ -177,17 +185,9 @@ func (s3a *S3ApiServer) CopyObjectHandler(w http.ResponseWriter, r *http.Request
 			skipHeader = true
 		}

-		// If we're doing cross-encryption, skip conflicting headers
-		if !skipHeader && len(entry.GetChunks()) > 0 {
-			// Detect source and destination encryption types
-			srcHasSSEC := IsSSECEncrypted(entry.Extended)
-			srcHasSSEKMS := IsSSEKMSEncrypted(entry.Extended)
-			srcHasSSES3 := IsSSES3EncryptedInternal(entry.Extended)
-			dstWantsSSEC := IsSSECRequest(r)
-			dstWantsSSEKMS := IsSSEKMSRequest(r)
-			dstWantsSSES3 := IsSSES3RequestInternal(r)
-
-			// Use helper function to determine if header should be skipped
+		// Filter conflicting headers for cross-encryption or encrypted→unencrypted copies
+		// This applies to both inline files (no chunks) and chunked files - fixes GitHub #7562
+		if !skipHeader {
 			skipHeader = shouldSkipEncryptionHeader(k,
 				srcHasSSEC, srcHasSSEKMS, srcHasSSES3,
 				dstWantsSSEC, dstWantsSSEKMS, dstWantsSSES3)
@@ -212,10 +212,31 @@ func (s3a *S3ApiServer) CopyObjectHandler(w http.ResponseWriter, r *http.Request
 		dstEntry.Extended[k] = v
 	}

-	// For zero-size files or files without chunks, use the original approach
+	// For zero-size files or files without chunks, handle inline content
+	// This includes encrypted inline files that need decryption/re-encryption
 	if entry.Attributes.FileSize == 0 || len(entry.GetChunks()) == 0 {
-		// Just copy the entry structure without chunks for zero-size files
 		dstEntry.Chunks = nil
+
+		// Handle inline encrypted content - fixes GitHub #7562
+		if len(entry.Content) > 0 {
+			inlineContent, inlineMetadata, inlineErr := s3a.processInlineContentForCopy(
+				entry, r, dstBucket, dstObject,
+				srcHasSSEC, srcHasSSEKMS, srcHasSSES3,
+				dstWantsSSEC, dstWantsSSEKMS, dstWantsSSES3)
+			if inlineErr != nil {
+				glog.Errorf("CopyObjectHandler inline content error: %v", inlineErr)
+				s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
+				return
+			}
+			dstEntry.Content = inlineContent
+
+			// Apply inline destination metadata
+			if inlineMetadata != nil {
+				for k, v := range inlineMetadata {
+					dstEntry.Extended[k] = v
+				}
+			}
+		}
 	} else {
 		// Use unified copy strategy approach
 		dstChunks, dstMetadata, copyErr := s3a.executeUnifiedCopyStrategy(entry, r, dstBucket, srcObject, dstObject)
@@ -2508,3 +2529,233 @@ func shouldSkipEncryptionHeader(headerKey string,
 	// Default: don't skip the header
 	return false
 }
+
+// processInlineContentForCopy handles encryption/decryption for inline content during copy
+// This fixes GitHub #7562 where small files stored inline weren't properly decrypted/re-encrypted
+func (s3a *S3ApiServer) processInlineContentForCopy(
+	entry *filer_pb.Entry, r *http.Request, dstBucket, dstObject string,
+	srcSSEC, srcSSEKMS, srcSSES3 bool,
+	dstSSEC, dstSSEKMS, dstSSES3 bool) ([]byte, map[string][]byte, error) {
+
+	content := entry.Content
+	var dstMetadata map[string][]byte
+
+	// Check if source is encrypted and needs decryption
+	srcEncrypted := srcSSEC || srcSSEKMS || srcSSES3
+
+	// Check if destination needs encryption (explicit request or bucket default)
+	dstNeedsEncryption := dstSSEC || dstSSEKMS || dstSSES3
+	if !dstNeedsEncryption {
+		// Check bucket default encryption
+		bucketMetadata, err := s3a.getBucketMetadata(dstBucket)
+		if err == nil && bucketMetadata != nil && bucketMetadata.Encryption != nil {
+			switch bucketMetadata.Encryption.SseAlgorithm {
+			case "aws:kms":
+				dstSSEKMS = true
+				dstNeedsEncryption = true
+			case "AES256":
+				dstSSES3 = true
+				dstNeedsEncryption = true
+			}
+		}
+	}
+
+	// Decrypt source content if encrypted
+	if srcEncrypted {
+		decryptedContent, decErr := s3a.decryptInlineContent(entry, srcSSEC, srcSSEKMS, srcSSES3, r)
+		if decErr != nil {
+			return nil, nil, fmt.Errorf("failed to decrypt inline content: %w", decErr)
+		}
+		content = decryptedContent
+		glog.V(3).Infof("Decrypted inline content: %d bytes", len(content))
+	}
+
+	// Re-encrypt if destination needs encryption
+	if dstNeedsEncryption {
+		encryptedContent, encMetadata, encErr := s3a.encryptInlineContent(content, dstBucket, dstObject, dstSSEC, dstSSEKMS, dstSSES3, r)
+		if encErr != nil {
+			return nil, nil, fmt.Errorf("failed to encrypt inline content: %w", encErr)
+		}
+		content = encryptedContent
+		dstMetadata = encMetadata
+		glog.V(3).Infof("Encrypted inline content: %d bytes", len(content))
+	}
+
+	return content, dstMetadata, nil
+}
+
+// decryptInlineContent decrypts inline content from an encrypted source
+func (s3a *S3ApiServer) decryptInlineContent(entry *filer_pb.Entry, srcSSEC, srcSSEKMS, srcSSES3 bool, r *http.Request) ([]byte, error) {
+	content := entry.Content
+
+	if srcSSES3 {
+		// Get SSE-S3 key from metadata
+		keyData, exists := entry.Extended[s3_constants.SeaweedFSSSES3Key]
+		if !exists {
+			return nil, fmt.Errorf("SSE-S3 key not found in metadata")
+		}
+
+		keyManager := GetSSES3KeyManager()
+		sseKey, err := DeserializeSSES3Metadata(keyData, keyManager)
+		if err != nil {
+			return nil, fmt.Errorf("failed to deserialize SSE-S3 key: %w", err)
+		}
+
+		// Get IV
+		iv := sseKey.IV
+		if len(iv) == 0 {
+			return nil, fmt.Errorf("SSE-S3 IV not found")
+		}
+
+		// Decrypt content
+		decryptedReader, err := CreateSSES3DecryptedReader(bytes.NewReader(content), sseKey, iv)
+		if err != nil {
+			return nil, fmt.Errorf("failed to create SSE-S3 decrypted reader: %w", err)
+		}
+		return io.ReadAll(decryptedReader)
+
+	} else if srcSSEKMS {
+		// Get SSE-KMS key from metadata
+		keyData, exists := entry.Extended[s3_constants.SeaweedFSSSEKMSKey]
+		if !exists {
+			return nil, fmt.Errorf("SSE-KMS key not found in metadata")
+		}
+
+		sseKey, err := DeserializeSSEKMSMetadata(keyData)
+		if err != nil {
+			return nil, fmt.Errorf("failed to deserialize SSE-KMS key: %w", err)
+		}
+
+		// Decrypt content
+		decryptedReader, err := CreateSSEKMSDecryptedReader(bytes.NewReader(content), sseKey)
+		if err != nil {
+			return nil, fmt.Errorf("failed to create SSE-KMS decrypted reader: %w", err)
+		}
+		return io.ReadAll(decryptedReader)
+
+	} else if srcSSEC {
+		// Get SSE-C key from request headers
+		sourceKey, err := ParseSSECCopySourceHeaders(r)
+		if err != nil {
+			return nil, fmt.Errorf("failed to parse SSE-C copy source headers: %w", err)
+		}
+
+		// Get IV from metadata
+		iv, err := GetSSECIVFromMetadata(entry.Extended)
+		if err != nil {
+			return nil, fmt.Errorf("failed to get SSE-C IV: %w", err)
+		}
+
+		// Decrypt content
+		decryptedReader, err := CreateSSECDecryptedReader(bytes.NewReader(content), sourceKey, iv)
+		if err != nil {
+			return nil, fmt.Errorf("failed to create SSE-C decrypted reader: %w", err)
+		}
+		return io.ReadAll(decryptedReader)
+	}
+
+	// Source not encrypted, return as-is
+	return content, nil
+}
+
+// encryptInlineContent encrypts inline content for the destination
+func (s3a *S3ApiServer) encryptInlineContent(content []byte, dstBucket, dstObject string,
+	dstSSEC, dstSSEKMS, dstSSES3 bool, r *http.Request) ([]byte, map[string][]byte, error) {
+
+	dstMetadata := make(map[string][]byte)
+
+	if dstSSES3 {
+		// Generate SSE-S3 key
+		keyManager := GetSSES3KeyManager()
+		key, err := keyManager.GetOrCreateKey("")
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to generate SSE-S3 key: %w", err)
+		}
+
+		// Encrypt content
+		encryptedReader, iv, err := CreateSSES3EncryptedReader(bytes.NewReader(content), key)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to create SSE-S3 encrypted reader: %w", err)
+		}
+
+		encryptedContent, err := io.ReadAll(encryptedReader)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to read encrypted content: %w", err)
+		}
+
+		// Store IV on key and serialize metadata
+		key.IV = iv
+		keyData, err := SerializeSSES3Metadata(key)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to serialize SSE-S3 metadata: %w", err)
+		}
+
+		dstMetadata[s3_constants.SeaweedFSSSES3Key] = keyData
+		dstMetadata[s3_constants.AmzServerSideEncryption] = []byte("AES256")
+
+		return encryptedContent, dstMetadata, nil
+
+	} else if dstSSEKMS {
+		// Parse SSE-KMS headers
+		keyID, encryptionContext, bucketKeyEnabled, err := ParseSSEKMSCopyHeaders(r)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to parse SSE-KMS headers: %w", err)
+		}
+
+		// Build encryption context if needed
+		if encryptionContext == nil {
+			encryptionContext = BuildEncryptionContext(dstBucket, dstObject, bucketKeyEnabled)
+		}
+
+		// Encrypt content
+		encryptedReader, sseKey, err := CreateSSEKMSEncryptedReaderWithBucketKey(
+			bytes.NewReader(content), keyID, encryptionContext, bucketKeyEnabled)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to create SSE-KMS encrypted reader: %w", err)
+		}
+
+		encryptedContent, err := io.ReadAll(encryptedReader)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to read encrypted content: %w", err)
+		}
+
+		// Serialize metadata
+		keyData, err := SerializeSSEKMSMetadata(sseKey)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to serialize SSE-KMS metadata: %w", err)
+		}
+
+		dstMetadata[s3_constants.SeaweedFSSSEKMSKey] = keyData
+		dstMetadata[s3_constants.AmzServerSideEncryption] = []byte("aws:kms")
+
+		return encryptedContent, dstMetadata, nil
+
+	} else if dstSSEC {
+		// Parse SSE-C headers
+		destKey, err := ParseSSECHeaders(r)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to parse SSE-C headers: %w", err)
+		}
+
+		// Encrypt content
+		encryptedReader, iv, err := CreateSSECEncryptedReader(bytes.NewReader(content), destKey)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to create SSE-C encrypted reader: %w", err)
+		}
+
+		encryptedContent, err := io.ReadAll(encryptedReader)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to read encrypted content: %w", err)
+		}
+
+		// Store IV in metadata
+		StoreSSECIVInMetadata(dstMetadata, iv)
+		dstMetadata[s3_constants.AmzServerSideEncryptionCustomerAlgorithm] = []byte("AES256")
+		dstMetadata[s3_constants.AmzServerSideEncryptionCustomerKeyMD5] = []byte(destKey.KeyMD5)
+
+		return encryptedContent, dstMetadata, nil
+	}
+
+	// No encryption needed
+	return content, nil, nil
+}