Files
seaweedFS/weed/s3api/s3api_sosapi_test.go
Chris Lu 2b529e310d s3: Add SOSAPI support for Veeam integration (#7899)
* s3api: Add SOSAPI core implementation and tests

Implement Smart Object Storage API (SOSAPI) support for Veeam integration.

- Add s3api_sosapi.go with XML structures and handlers for system.xml and capacity.xml
- Implement virtual object detection and dynamic XML generation
- Add capacity retrieval via gRPC (to be optimized in follow-up)
- Include comprehensive unit tests covering detection, XML generation, and edge cases

This enables Veeam Backup & Replication to discover SeaweedFS capabilities and capacity.

* s3api: Integrate SOSAPI handlers into GetObject and HeadObject

Add early interception for SOSAPI virtual objects in GetObjectHandler and HeadObjectHandler.

- Check for SOSAPI objects (.system-*/system.xml, .system-*/capacity.xml) before normal processing
- Delegate to handleSOSAPIGetObject and handleSOSAPIHeadObject when detected
- Ensures virtual objects are served without hitting storage layer

* s3api: Allow anonymous access to SOSAPI virtual objects

Enable discovery of SOSAPI capabilities without requiring credentials.

- Modify AuthWithPublicRead to bypass auth for SOSAPI objects if bucket exists
- Supports Veeam's initial discovery phase before full IAM setup
- Validates bucket existence to prevent information disclosure

* s3api: Fix SOSAPI capacity retrieval to use proper master connection

Fix gRPC error by connecting directly to master servers instead of through filer.

- Use pb.WithOneOfGrpcMasterClients with s3a.option.Masters
- Matches pattern used in bucket_size_metrics.go
- Resolves "unknown service master_pb.Seaweed" error
- Gracefully handles missing master configuration

* Merge origin/master and implement robust SOSAPI capacity logic

- Resolved merge conflict in s3api_sosapi.go
- Replaced global Statistics RPC with VolumeList (topology) for accurate bucket-specific 'Used' calculation
- Added bucket quota support (report quota as Capacity if set)
- Implemented cluster-wide capacity calculation from topology when no quota
- Added unit tests for topology capacity and usage calculations

* s3api: Remove anonymous access to SOSAPI virtual objects

Reverts the implicit public access for system.xml and capacity.xml.
Requests to these objects now require standard S3 authentication,
unless the bucket has a public-read policy.

* s3api: Refactor SOSAPI handlers to use http.ServeContent

- Consolidate handleSOSAPIGetObject and handleSOSAPIHeadObject into serveSOSAPI
- Use http.ServeContent for standard Range, HEAD, and ETag handling
- Remove manual range request handler and reduce code duplication

* s3api: Unify SOSAPI request handling

- Replaced handleSOSAPIGetObject and handleSOSAPIHeadObject with single HandleSOSAPI function
- Updated call sites in s3api_object_handlers.go
- Simplifies logic and ensures consistent handling for both GET and HEAD requests via http.ServeContent

* s3api: Restore distinct SOSAPI GET/HEAD handlers

- Reverted unified handler to enforce distinct behavior for GET and HEAD
- GET: Supports Range requests via http.ServeContent
- HEAD: Explicitly ignores Range requests (matches MinIO behavior) and writes headers only

* s3api: Refactor SOSAPI handlers to eliminate duplication

- Extracted shared content generation logic into generateSOSAPIContent helper
- handleSOSAPIGetObject: Uses http.ServeContent (supports Range requests)
- handleSOSAPIHeadObject: Manually sets headers (no Range, no body)
- Maintains distinct behavior while following DRY principle

* s3api: Remove low-value SOSAPI tests

Removed tests that validate standard library behavior or trivial constant checks:
- TestSOSAPIConstants (string prefix/suffix checks)
- TestSystemInfoXMLRootElement (redundant with TestGenerateSystemXML)
- TestSOSAPIXMLContentType (tests httptest, not our code)
- TestHTTPTimeFormat (tests standard library)
- TestCapacityInfoXMLStruct (tests Go's XML marshaling)

Kept tests that validate actual business logic and edge cases.

* s3api: Use consistent S3-compliant error responses in SOSAPI

Replaced http.Error() with s3err.WriteErrorResponse() for internal errors
to ensure all SOSAPI errors return S3-compliant XML instead of plain text.

* s3api: Return error when no masters configured for SOSAPI capacity

Changed getCapacityInfo to return an error instead of silently returning
zero capacity when no master servers are configured. This helps surface
configuration issues rather than masking them.

* s3api: Use collection name with FilerGroup prefix for SOSAPI capacity

Fixed collectBucketUsageFromTopology to use s3a.getCollectionName(bucket)
instead of raw bucket name. This ensures collection comparisons match actual
volume collection names when FilerGroup prefix is configured.

* s3api: Apply PR review feedback for SOSAPI

- Renamed `bucket` parameter to `collectionName` in collectBucketUsageFromTopology for clarity
- Changed error checks from `==` to `errors.Is()` for better wrapped error handling
- Added `errors` import

* s3api: Avoid variable shadowing in SOSAPI capacity retrieval

Refactored `getCapacityInfo` to use distinct variable names for errors
to improve code clarity and avoid unintentional shadowing of the
return parameter.
2025-12-28 14:07:58 -08:00

249 lines
6.3 KiB
Go

package s3api
import (
"encoding/xml"
"net/http/httptest"
"strings"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
)
func TestIsSOSAPIObject(t *testing.T) {
tests := []struct {
name string
object string
expected bool
}{
{
name: "system.xml should be detected",
object: ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/system.xml",
expected: true,
},
{
name: "capacity.xml should be detected",
object: ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/capacity.xml",
expected: true,
},
{
name: "regular object should not be detected",
object: "myfile.txt",
expected: false,
},
{
name: "similar but different path should not be detected",
object: ".system-other-uuid/system.xml",
expected: false,
},
{
name: "nested path should not be detected",
object: "prefix/.system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/system.xml",
expected: false,
},
{
name: "empty string should not be detected",
object: "",
expected: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := isSOSAPIObject(tt.object)
if result != tt.expected {
t.Errorf("isSOSAPIObject(%q) = %v, want %v", tt.object, result, tt.expected)
}
})
}
}
func TestIsSOSAPIClient(t *testing.T) {
tests := []struct {
name string
userAgent string
expected bool
}{
{
name: "Veeam backup client should be detected",
userAgent: "APN/1.0 Veeam/1.0 Backup/10.0",
expected: true,
},
{
name: "exact match should be detected",
userAgent: "APN/1.0 Veeam/1.0",
expected: true,
},
{
name: "AWS CLI should not be detected",
userAgent: "aws-cli/2.0.0 Python/3.8",
expected: false,
},
{
name: "empty user agent should not be detected",
userAgent: "",
expected: false,
},
{
name: "partial match should not be detected",
userAgent: "Veeam/1.0",
expected: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req := httptest.NewRequest("GET", "/bucket/object", nil)
req.Header.Set("User-Agent", tt.userAgent)
result := isSOSAPIClient(req)
if result != tt.expected {
t.Errorf("isSOSAPIClient() with User-Agent %q = %v, want %v", tt.userAgent, result, tt.expected)
}
})
}
}
func TestGenerateSystemXML(t *testing.T) {
xmlData, err := generateSystemXML()
if err != nil {
t.Fatalf("generateSystemXML() failed: %v", err)
}
// Verify it's valid XML
var si SystemInfo
if err := xml.Unmarshal(xmlData, &si); err != nil {
t.Fatalf("generated XML is invalid: %v", err)
}
// Verify required fields
if si.ProtocolVersion != sosAPIProtocolVersion {
t.Errorf("ProtocolVersion = %q, want %q", si.ProtocolVersion, sosAPIProtocolVersion)
}
if !strings.Contains(si.ModelName, "SeaweedFS") {
t.Errorf("ModelName = %q, should contain 'SeaweedFS'", si.ModelName)
}
if !si.ProtocolCapabilities.CapacityInfo {
t.Error("ProtocolCapabilities.CapacityInfo should be true")
}
if si.SystemRecommendations == nil {
t.Fatal("SystemRecommendations should not be nil")
}
if si.SystemRecommendations.KBBlockSize != sosAPIDefaultBlockSizeKB {
t.Errorf("KBBlockSize = %d, want %d", si.SystemRecommendations.KBBlockSize, sosAPIDefaultBlockSizeKB)
}
}
func TestSOSAPIObjectDetectionEdgeCases(t *testing.T) {
edgeCases := []struct {
object string
expected bool
}{
// With leading slash
{"/.system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/system.xml", false},
// URL encoded
{".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c%2Fsystem.xml", false},
// Mixed case
{".System-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/system.xml", false},
// Extra slashes
{".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c//system.xml", false},
// Correct paths
{".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/system.xml", true},
{".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/capacity.xml", true},
}
for _, tc := range edgeCases {
result := isSOSAPIObject(tc.object)
if result != tc.expected {
t.Errorf("isSOSAPIObject(%q) = %v, want %v", tc.object, result, tc.expected)
}
}
}
func TestCollectBucketUsageFromTopology(t *testing.T) {
topo := &master_pb.TopologyInfo{
DataCenterInfos: []*master_pb.DataCenterInfo{
{
RackInfos: []*master_pb.RackInfo{
{
DataNodeInfos: []*master_pb.DataNodeInfo{
{
DiskInfos: map[string]*master_pb.DiskInfo{
"hdd": {
VolumeInfos: []*master_pb.VolumeInformationMessage{
{Id: 1, Size: 100, Collection: "bucket1"},
{Id: 2, Size: 200, Collection: "bucket2"},
{Id: 3, Size: 300, Collection: "bucket1"},
{Id: 1, Size: 100, Collection: "bucket1"}, // Duplicate (replica), should be ignored
},
},
},
},
},
},
},
},
},
}
usage := collectBucketUsageFromTopology(topo, "bucket1")
expected := int64(400) // 100 + 300
if usage != expected {
t.Errorf("collectBucketUsageFromTopology = %d, want %d", usage, expected)
}
usage2 := collectBucketUsageFromTopology(topo, "bucket2")
expected2 := int64(200)
if usage2 != expected2 {
t.Errorf("collectBucketUsageFromTopology = %d, want %d", usage2, expected2)
}
}
func TestCalculateClusterCapacity(t *testing.T) {
topo := &master_pb.TopologyInfo{
DataCenterInfos: []*master_pb.DataCenterInfo{
{
RackInfos: []*master_pb.RackInfo{
{
DataNodeInfos: []*master_pb.DataNodeInfo{
{
DiskInfos: map[string]*master_pb.DiskInfo{
"hdd": {
MaxVolumeCount: 100,
FreeVolumeCount: 40,
},
},
},
{
DiskInfos: map[string]*master_pb.DiskInfo{
"hdd": {
MaxVolumeCount: 200,
FreeVolumeCount: 160,
},
},
},
},
},
},
},
},
}
volumeSizeLimitMb := uint64(1000) // 1GB
volumeSizeBytes := int64(1000) * 1024 * 1024
total, available := calculateClusterCapacity(topo, volumeSizeLimitMb)
expectedTotal := int64(300) * volumeSizeBytes
expectedAvailable := int64(200) * volumeSizeBytes
if total != expectedTotal {
t.Errorf("calculateClusterCapacity total = %d, want %d", total, expectedTotal)
}
if available != expectedAvailable {
t.Errorf("calculateClusterCapacity available = %d, want %d", available, expectedAvailable)
}
}