gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add S3 volume encryption support with -s3.encryptVolumeData flag (#7890)

Browse Source 
* Add S3 volume encryption support with -s3.encryptVolumeData flag

This change adds volume-level encryption support for S3 uploads, similar
to the existing -filer.encryptVolumeData option. Each chunk is encrypted
with its own auto-generated CipherKey when the flag is enabled.

Changes:
- Add -s3.encryptVolumeData flag to weed s3, weed server, and weed mini
- Wire Cipher option through S3ApiServer and ChunkedUploadOption
- Add integration tests for multi-chunk range reads with encryption
- Tests verify encryption works across chunk boundaries

Usage:
  weed s3 -encryptVolumeData
  weed server -s3 -s3.encryptVolumeData
  weed mini -s3.encryptVolumeData

Integration tests:
  go test -v -tags=integration -timeout 5m ./test/s3/sse/...

* Add GitHub Actions CI for S3 volume encryption tests

- Add test-volume-encryption target to Makefile that starts server with -s3.encryptVolumeData
- Add s3-volume-encryption job to GitHub Actions workflow
- Tests run with integration build tag and 10m timeout
- Server logs uploaded on failure for debugging

* Fix S3 client credentials to use environment variables

The test was using hardcoded credentials "any"/"any" but the Makefile
sets AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY to "some_access_key1"/
"some_secret_key1". Updated getS3Client() to read from environment
variables with fallback to "any"/"any" for manual testing.

* Change bucket creation errors from skip to fatal

Tests should fail, not skip, when bucket creation fails. This ensures
that credential mismatches and other configuration issues are caught
rather than silently skipped.

* Make copy and multipart test jobs fail instead of succeed

Changed exit 0 to exit 1 for s3-sse-copy-operations and s3-sse-multipart
jobs. These jobs document known limitations but should fail to ensure
the issues are tracked and addressed, not silently ignored.

* Hardcode S3 credentials to match Makefile

Changed from environment variables to hardcoded credentials
"some_access_key1"/"some_secret_key1" to match the Makefile
configuration. This ensures tests work reliably.

* fix Double Encryption

* fix Chunk Size Mismatch

* Added IsCompressed

* is gzipped

* fix copying

* only perform HEAD request when len(cipherKey) > 0

* Revert "Make copy and multipart test jobs fail instead of succeed"

This reverts commit bc34a7eb3c103ae7ab2000da2a6c3925712eb226.

* fix security vulnerability

* fix security

* Update s3api_object_handlers_copy.go

* Update s3api_object_handlers_copy.go

* jwt to get content length
This commit is contained in:
Chris Lu
2025-12-27 00:09:14 -08:00
committed by GitHub
parent 935f41bff6
commit 8d6bcddf60
13 changed files with 593 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
weed/command/mini.go
View File

@@ -229,6 +229,7 @@ func initMiniS3Flags() {
miniS3Options.concurrentFileUploadLimit = cmdMini.Flag.Int("s3.concurrentFileUploadLimit", 0, "limit number of concurrent file uploads")
miniS3Options.enableIam = cmdMini.Flag.Bool("s3.iam", true, "enable embedded IAM API on the same port")
miniS3Options.dataCenter = cmdMini.Flag.String("s3.dataCenter", "", "prefer to read and write to volumes in this data center")
miniS3Options.cipher = cmdMini.Flag.Bool("s3.encryptVolumeData", false, "encrypt data on volume servers for S3 uploads")
miniS3Options.config = miniS3Config
miniS3Options.iamConfig = miniIamConfig
miniS3Options.auditLogConfig = cmdMini.Flag.String("s3.auditLogConfig", "", "path to the audit log config file")

3
weed/command/s3.go
View File

@@ -62,6 +62,7 @@ type S3Options struct {
enableIam *bool
debug *bool
debugPort *int
cipher *bool
}
func init() {
@@ -93,6 +94,7 @@ func init() {
s3StandaloneOptions.enableIam = cmdS3.Flag.Bool("iam", true, "enable embedded IAM API on the same port")
s3StandaloneOptions.debug = cmdS3.Flag.Bool("debug", false, "serves runtime profiling data via pprof on the port specified by -debug.port")
s3StandaloneOptions.debugPort = cmdS3.Flag.Int("debug.port", 6060, "http port for debugging")
s3StandaloneOptions.cipher = cmdS3.Flag.Bool("encryptVolumeData", false, "encrypt data on volume servers")
}
var cmdS3 = &Command{
@@ -290,6 +292,7 @@ func (s3opt *S3Options) startS3Server() bool {
ConcurrentUploadLimit: int64(*s3opt.concurrentUploadLimitMB) * 1024 * 1024,
ConcurrentFileUploadLimit: int64(*s3opt.concurrentFileUploadLimit),
EnableIam: *s3opt.enableIam, // Embedded IAM API (enabled by default)
Cipher: *s3opt.cipher, // encrypt data on volume servers
})
if s3ApiServer_err != nil {
glog.Fatalf("S3 API Server startup error: %v", s3ApiServer_err)

1
weed/command/server.go
View File

@@ -175,6 +175,7 @@ func init() {
s3Options.concurrentUploadLimitMB = cmdServer.Flag.Int("s3.concurrentUploadLimitMB", 0, "limit total concurrent upload size for S3, 0 means unlimited")
s3Options.concurrentFileUploadLimit = cmdServer.Flag.Int("s3.concurrentFileUploadLimit", 0, "limit number of concurrent file uploads for S3, 0 means unlimited")
s3Options.enableIam = cmdServer.Flag.Bool("s3.iam", true, "enable embedded IAM API on the same S3 port")
s3Options.cipher = cmdServer.Flag.Bool("s3.encryptVolumeData", false, "encrypt data on volume servers for S3 uploads")
sftpOptions.port = cmdServer.Flag.Int("sftp.port", 2022, "SFTP server listen port")
sftpOptions.sshPrivateKey = cmdServer.Flag.String("sftp.sshPrivateKey", "", "path to the SSH private key file for host authentication")

5
weed/operation/upload_chunked.go
View File

@@ -34,6 +34,7 @@ type ChunkedUploadOption struct {
SaveSmallInline bool
Jwt security.EncodedJwt
MimeType string
Cipher bool // encrypt data on volume servers
AssignFunc func(ctx context.Context, count int) (*VolumeAssignRequest, *AssignResult, error)
UploadFunc func(ctx context.Context, data []byte, option *UploadOption) (*UploadResult, error) // Optional: for testing
}
@@ -172,7 +173,7 @@ uploadLoop:
uploadOption := &UploadOption{
UploadUrl: uploadUrl,
Cipher: false,
Cipher: opt.Cipher,
IsInputCompressed: false,
MimeType: opt.MimeType,
PairMap: nil,
@@ -220,8 +221,8 @@ uploadLoop:
ETag: uploadResult.ContentMd5,
Fid: fid,
CipherKey: uploadResult.CipherKey,
IsCompressed: uploadResult.Gzip > 0,
}
fileChunksLock.Lock()
fileChunks = append(fileChunks, chunk)
glog.V(4).Infof("uploaded chunk %d to %s [%d,%d)", len(fileChunks), chunk.FileId, offset, offset+int64(chunk.Size))

8
weed/operation/upload_content.go
View File

@@ -249,8 +249,10 @@ func (uploader *Uploader) doUploadData(ctx context.Context, data []byte, option
compressed, compressErr := util.GzipData(data)
// fmt.Printf("data is compressed from %d ==> %d\n", len(data), len(compressed))
if compressErr == nil {
data = compressed
contentIsGzipped = true
if len(compressed) < len(data) {
data = compressed
contentIsGzipped = true
}
}
} else if option.IsInputCompressed {
// just to get the clear data length
@@ -290,7 +292,7 @@ func (uploader *Uploader) doUploadData(ctx context.Context, data []byte, option
uploadResult.Name = option.Filename
uploadResult.Mime = option.MimeType
uploadResult.CipherKey = cipherKey
uploadResult.Size = uint32(clearDataLen)
uploadResult.Size = uint32(len(data))
if contentIsGzipped {
uploadResult.Gzip = 1
}

4
weed/s3api/s3api_key_rotation.go
View File

@@ -182,7 +182,7 @@ func (s3a *S3ApiServer) rotateSSECChunk(chunk *filer_pb.FileChunk, sourceKey, de
}
// Download encrypted data
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size))
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size), chunk.CipherKey)
if err != nil {
return nil, fmt.Errorf("download chunk data: %w", err)
}
@@ -251,7 +251,7 @@ func (s3a *S3ApiServer) rotateSSEKMSChunk(chunk *filer_pb.FileChunk, srcKeyID, d
}
// Download data (this would be encrypted with the old KMS key)
chunkData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size))
chunkData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size), chunk.CipherKey)
if err != nil {
return nil, fmt.Errorf("download chunk data: %w", err)
}

2
weed/s3api/s3api_object_handlers.go
View File

@@ -1019,6 +1019,8 @@ func (s3a *S3ApiServer) streamFromVolumeServers(w http.ResponseWriter, r *http.R
if isRangeRequest {
w.Header().Set("Content-Range", fmt.Sprintf("bytes %d-%d/%d", offset, offset+size-1, totalSize))
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
} else {
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
}
headerSetTime = time.Since(tHeaderSet)

48
weed/s3api/s3api_object_handlers_copy.go
View File

@@ -846,7 +846,7 @@ func (s3a *S3ApiServer) copySingleChunk(chunk *filer_pb.FileChunk, dstPath strin
}
// Download and upload the chunk
chunkData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size))
chunkData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size), chunk.CipherKey)
if err != nil {
return nil, fmt.Errorf("download chunk data: %w", err)
}
@@ -881,7 +881,7 @@ func (s3a *S3ApiServer) copySingleChunkForRange(originalChunk, rangeChunk *filer
offsetInChunk := overlapStart - chunkStart
// Download and upload the chunk portion
chunkData, err := s3a.downloadChunkData(srcUrl, fileId, offsetInChunk, int64(rangeChunk.Size))
chunkData, err := s3a.downloadChunkData(srcUrl, fileId, offsetInChunk, int64(rangeChunk.Size), originalChunk.CipherKey)
if err != nil {
return nil, fmt.Errorf("download chunk range data: %w", err)
}
@@ -1199,10 +1199,40 @@ func (s3a *S3ApiServer) uploadChunkData(chunkData []byte, assignResult *filer_pb
}
// downloadChunkData downloads chunk data from the source URL
func (s3a *S3ApiServer) downloadChunkData(srcUrl, fileId string, offset, size int64) ([]byte, error) {
func (s3a *S3ApiServer) downloadChunkData(srcUrl, fileId string, offset, size int64, cipherKey []byte) ([]byte, error) {
jwt := filer.JwtForVolumeServer(fileId)
// Only perform HEAD request for encrypted chunks to get physical size
if offset == 0 && len(cipherKey) > 0 {
req, err := http.NewRequest(http.MethodHead, srcUrl, nil)
if err == nil {
if jwt != "" {
req.Header.Set("Authorization", "BEARER "+string(jwt))
}
resp, err := util_http.GetGlobalHttpClient().Do(req)
if err == nil {
defer util_http.CloseResponse(resp)
if resp.StatusCode == http.StatusOK {
contentLengthStr := resp.Header.Get("Content-Length")
if contentLength, err := strconv.ParseInt(contentLengthStr, 10, 64); err == nil {
// Validate contentLength fits in int32 range before comparison
if contentLength > int64(2147483647) { // math.MaxInt32
return nil, fmt.Errorf("content length %d exceeds maximum int32 size", contentLength)
}
if contentLength > size {
size = contentLength
}
}
}
}
}
}
// Validate size fits in int32 range before conversion to int
if size > int64(2147483647) { // math.MaxInt32
return nil, fmt.Errorf("chunk size %d exceeds maximum int32 size", size)
}
sizeInt := int(size)
var chunkData []byte
shouldRetry, err := util_http.ReadUrlAsStream(context.Background(), srcUrl, jwt, nil, false, false, offset, int(size), func(data []byte) {
shouldRetry, err := util_http.ReadUrlAsStream(context.Background(), srcUrl, jwt, nil, false, false, offset, sizeInt, func(data []byte) {
chunkData = append(chunkData, data...)
})
if err != nil {
@@ -1334,7 +1364,7 @@ func (s3a *S3ApiServer) copyMultipartSSEKMSChunk(chunk *filer_pb.FileChunk, dest
}
// Download encrypted chunk data
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size))
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size), chunk.CipherKey)
if err != nil {
return nil, fmt.Errorf("download encrypted chunk data: %w", err)
}
@@ -1433,7 +1463,7 @@ func (s3a *S3ApiServer) copyMultipartSSECChunk(chunk *filer_pb.FileChunk, copySo
}
// Download encrypted chunk data
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size))
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size), chunk.CipherKey)
if err != nil {
return nil, nil, fmt.Errorf("download encrypted chunk data: %w", err)
}
@@ -1714,7 +1744,7 @@ func (s3a *S3ApiServer) copyCrossEncryptionChunk(chunk *filer_pb.FileChunk, sour
}
// Download encrypted chunk data
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size))
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size), chunk.CipherKey)
if err != nil {
return nil, fmt.Errorf("download encrypted chunk data: %w", err)
}
@@ -2076,7 +2106,7 @@ func (s3a *S3ApiServer) copyChunkWithReencryption(chunk *filer_pb.FileChunk, cop
}
// Download encrypted chunk data
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size))
encryptedData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size), chunk.CipherKey)
if err != nil {
return nil, fmt.Errorf("download encrypted chunk data: %w", err)
}
@@ -2295,7 +2325,7 @@ func (s3a *S3ApiServer) copyChunkWithSSEKMSReencryption(chunk *filer_pb.FileChun
}
// Download chunk data
chunkData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size))
chunkData, err := s3a.downloadChunkData(srcUrl, fileId, 0, int64(chunk.Size), chunk.CipherKey)
if err != nil {
return nil, fmt.Errorf("download chunk data: %w", err)
}

3
weed/s3api/s3api_object_handlers_put.go
View File

@@ -299,7 +299,7 @@ func (s3a *S3ApiServer) putToFiler(r *http.Request, filePath string, dataReader
// Apply bucket default encryption if no explicit encryption was provided
// This implements AWS S3 behavior where bucket default encryption automatically applies
if !hasExplicitEncryption(customerKey, sseKMSKey, sseS3Key) {
if !hasExplicitEncryption(customerKey, sseKMSKey, sseS3Key) && !s3a.cipher {
glog.V(4).Infof("putToFiler: no explicit encryption detected, checking for bucket default encryption")
// Apply bucket default encryption and get the result
@@ -392,6 +392,7 @@ func (s3a *S3ApiServer) putToFiler(r *http.Request, filePath string, dataReader
DataCenter: s3a.option.DataCenter,
SaveSmallInline: false, // S3 API always creates chunks, never stores inline
MimeType: r.Header.Get("Content-Type"),
Cipher: s3a.cipher, // encrypt data on volume servers
AssignFunc: assignFunc,
})
if err != nil {

5
weed/s3api/s3api_server.go
View File

@@ -51,6 +51,7 @@ type S3ApiServerOption struct {
ConcurrentUploadLimit int64
ConcurrentFileUploadLimit int64
EnableIam bool // Enable embedded IAM API on the same port
Cipher bool // encrypt data on volume servers
}
type S3ApiServer struct {
@@ -70,7 +71,8 @@ type S3ApiServer struct {
inFlightDataSize int64
inFlightUploads int64
inFlightDataLimitCond *sync.Cond
embeddedIam *EmbeddedIamApi // Embedded IAM API server (when enabled)
embeddedIam *EmbeddedIamApi // Embedded IAM API server (when enabled)
cipher bool // encrypt data on volume servers
}
func NewS3ApiServer(router *mux.Router, option *S3ApiServerOption) (s3ApiServer *S3ApiServer, err error) {
@@ -154,6 +156,7 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
bucketConfigCache: NewBucketConfigCache(60 * time.Minute), // Increased TTL since cache is now event-driven
policyEngine: policyEngine, // Initialize bucket policy engine
inFlightDataLimitCond: sync.NewCond(new(sync.Mutex)),
cipher: option.Cipher,
}
// Set s3a reference in circuit breaker for upload limiting