* Add TUS protocol integration tests
This commit adds integration tests for the TUS (resumable upload) protocol
in preparation for implementing TUS support in the filer.
Test coverage includes:
- OPTIONS handler for capability discovery
- Basic single-request upload
- Chunked/resumable uploads
- HEAD requests for offset tracking
- DELETE for upload cancellation
- Error handling (invalid offsets, missing uploads)
- Creation-with-upload extension
- Resume after interruption simulation
Tests are skipped in short mode and require a running SeaweedFS cluster.
* Add TUS session storage types and utilities
Implements TUS upload session management:
- TusSession struct for tracking upload state
- Session creation with directory-based storage
- Session persistence using filer entries
- Session retrieval and offset updates
- Session deletion with chunk cleanup
- Upload completion with chunk assembly into final file
Session data is stored in /.uploads.tus/{upload-id}/ directory,
following the pattern used by S3 multipart uploads.
* Add TUS HTTP handlers
Implements TUS protocol HTTP handlers:
- tusHandler: Main entry point routing requests
- tusOptionsHandler: Capability discovery (OPTIONS)
- tusCreateHandler: Create new upload (POST)
- tusHeadHandler: Get upload offset (HEAD)
- tusPatchHandler: Upload data at offset (PATCH)
- tusDeleteHandler: Cancel upload (DELETE)
- tusWriteData: Upload data to volume servers
Features:
- Supports creation-with-upload extension
- Validates TUS protocol headers
- Offset conflict detection
- Automatic upload completion when size is reached
- Metadata parsing from Upload-Metadata header
* Wire up TUS protocol routes in filer server
Add TUS handler route (/.tus/) to the filer HTTP server.
The TUS route is registered before the catch-all route to ensure
proper routing of TUS protocol requests.
TUS protocol is now accessible at:
- OPTIONS /.tus/ - Capability discovery
- POST /.tus/{path} - Create upload
- HEAD /.tus/.uploads/{id} - Get offset
- PATCH /.tus/.uploads/{id} - Upload data
- DELETE /.tus/.uploads/{id} - Cancel upload
* Improve TUS integration test setup
Add comprehensive Makefile for TUS tests with targets:
- test-with-server: Run tests with automatic server management
- test-basic/chunked/resume/errors: Specific test categories
- manual-start/stop: For development testing
- debug-logs/status: For debugging
- ci-test: For CI/CD pipelines
Update README.md with:
- Detailed TUS protocol documentation
- All endpoint descriptions with headers
- Usage examples with curl commands
- Architecture diagram
- Comparison with S3 multipart uploads
Follows the pattern established by other tests in test/ folder.
* Fix TUS integration tests and creation-with-upload
- Fix test URLs to use full URLs instead of relative paths
- Fix creation-with-upload to refresh session before completing
- Fix Makefile to properly handle test cleanup
- Add FullURL helper function to TestCluster
* Add TUS protocol tests to GitHub Actions CI
- Add tus-tests.yml workflow that runs on PRs and pushes
- Runs when TUS-related files are modified
- Automatic server management for integration testing
- Upload logs on failure for debugging
* Make TUS base path configurable via CLI
- Add -tus.path CLI flag to filer command
- TUS is disabled by default (empty path)
- Example: -tus.path=/.tus to enable at /.tus endpoint
- Update test Makefile to use -tus.path flag
- Update README with TUS enabling instructions
* Rename -tus.path to -tusBasePath with default .tus
- Rename CLI flag from -tus.path to -tusBasePath
- Default to .tus (TUS enabled by default)
- Add -filer.tusBasePath option to weed server command
- Properly handle path prefix (prepend / if missing)
* Address code review comments
- Sort chunks by offset before assembling final file
- Use chunk.Offset directly instead of recalculating
- Return error on invalid file ID instead of skipping
- Require Content-Length header for PATCH requests
- Use fs.option.Cipher for encryption setting
- Detect MIME type from data using http.DetectContentType
- Fix concurrency group for push events in workflow
- Use os.Interrupt instead of Kill for graceful shutdown in tests
* fmt
* Address remaining code review comments
- Fix potential open redirect vulnerability by sanitizing uploadLocation path
- Add language specifier to README code block
- Handle os.Create errors in test setup
- Use waitForHTTPServer instead of time.Sleep for master/volume readiness
- Improve test reliability and debugging
* Address critical and high-priority review comments
- Add per-session locking to prevent race conditions in updateTusSessionOffset
- Stream data directly to volume server instead of buffering entire chunk
- Only buffer 512 bytes for MIME type detection, then stream remaining data
- Clean up session locks when session is deleted
* Fix race condition to work across multiple filer instances
- Store each chunk as a separate file entry instead of updating session JSON
- Chunk file names encode offset, size, and fileId for atomic storage
- getTusSession loads chunks from directory listing (atomic read)
- Eliminates read-modify-write race condition across multiple filers
- Remove in-memory mutex that only worked for single filer instance
* Address code review comments: fix variable shadowing, sniff size, and test stability
- Rename path variable to reqPath to avoid shadowing path package
- Make sniff buffer size respect contentLength (read at most contentLength bytes)
- Handle Content-Length < 0 in creation-with-upload (return error for chunked encoding)
- Fix test cluster: use temp directory for filer store, add startup delay
* Fix test stability: increase cluster stabilization delay to 5 seconds
The tests were intermittently failing because the volume server needed more
time to create volumes and register with the master. Increasing the delay
from 2 to 5 seconds fixes the flaky test behavior.
* Address PR review comments for TUS protocol support
- Fix strconv.Atoi error handling in test file (lines 386, 747)
- Fix lossy fileId encoding: use base64 instead of underscore replacement
- Add pagination support for ListDirectoryEntries in getTusSession
- Batch delete chunks instead of one-by-one in deleteTusSession
* Address additional PR review comments for TUS protocol
- Fix UploadAt timestamp: use entry.Crtime instead of time.Now()
- Remove redundant JSON content in chunk entry (metadata in filename)
- Refactor tusWriteData to stream in 4MB chunks to avoid OOM on large uploads
- Pass filer.Entry to parseTusChunkPath to preserve actual upload time
* Address more PR review comments for TUS protocol
- Normalize TUS path once in filer_server.go, store in option.TusPath
- Remove redundant path normalization from TUS handlers
- Remove goto statement in tusCreateHandler, simplify control flow
* Remove unnecessary mutexes in tusWriteData
The upload loop is sequential, so uploadErrLock and chunksLock are not needed.
* Rename updateTusSessionOffset to saveTusChunk
Remove unused newOffset parameter and rename function to better reflect its purpose.
* Improve TUS upload performance and add path validation
- Reuse operation.Uploader across sub-chunks for better connection reuse
- Guard against TusPath='/' to prevent hijacking all filer routes
* Address PR review comments for TUS protocol
- Fix critical chunk filename parsing: use strings.Cut instead of SplitN
to correctly handle base64-encoded fileIds that may contain underscores
- Rename tusPath to tusBasePath for naming consistency across codebase
- Add background garbage collection for expired TUS sessions (runs hourly)
- Improve error messages with %w wrapping for better debuggability
* Address additional TUS PR review comments
- Fix tusBasePath default to use leading slash (/.tus) for consistency
- Add chunk contiguity validation in completeTusUpload to detect gaps/overlaps
- Fix offset calculation to find maximum contiguous range from 0, not just last chunk
- Return 413 Request Entity Too Large instead of silently truncating content
- Document tusChunkSize rationale (4MB balances memory vs request overhead)
- Fix Makefile xargs portability by removing GNU-specific -r flag
- Add explicit -tusBasePath flag to integration test for robustness
- Fix README example to use /.uploads/tus path format
* Revert log_buffer changes (moved to separate PR)
* Minor style fixes from PR review
- Simplify tusBasePath flag description to use example format
- Add 'TUS upload' prefix to session not found error message
- Remove duplicate tusChunkSize comment
- Capitalize warning message for consistency
- Add grep filter to Makefile xargs for better empty input handling
227 lines
8.2 KiB
Makefile
227 lines
8.2 KiB
Makefile
# Makefile for TUS Protocol Integration Tests
|
|
# This Makefile provides targets for running TUS (resumable upload) integration tests
|
|
|
|
# Default values
|
|
SEAWEEDFS_BINARY ?= weed
|
|
FILER_PORT ?= 18888
|
|
VOLUME_PORT ?= 18080
|
|
MASTER_PORT ?= 19333
|
|
TEST_TIMEOUT ?= 10m
|
|
VOLUME_MAX_SIZE_MB ?= 50
|
|
VOLUME_MAX_COUNT ?= 100
|
|
|
|
# Test directory
|
|
TEST_DIR := $(shell pwd)
|
|
SEAWEEDFS_ROOT := $(shell cd ../.. && pwd)
|
|
|
|
# Colors for output
|
|
RED := \033[0;31m
|
|
GREEN := \033[0;32m
|
|
YELLOW := \033[1;33m
|
|
NC := \033[0m # No Color
|
|
|
|
.PHONY: all test clean start-seaweedfs stop-seaweedfs check-binary build-weed help test-basic test-chunked test-resume test-errors test-with-server
|
|
|
|
all: test
|
|
|
|
# Build SeaweedFS binary
|
|
build-weed:
|
|
@echo "Building SeaweedFS binary..."
|
|
@cd $(SEAWEEDFS_ROOT)/weed && go build -o weed
|
|
@echo "$(GREEN)SeaweedFS binary built successfully$(NC)"
|
|
|
|
help:
|
|
@echo "SeaweedFS TUS Protocol Integration Tests"
|
|
@echo ""
|
|
@echo "Available targets:"
|
|
@echo " test - Run all TUS integration tests"
|
|
@echo " test-basic - Run basic TUS upload tests"
|
|
@echo " test-chunked - Run chunked upload tests"
|
|
@echo " test-resume - Run upload resume tests"
|
|
@echo " test-errors - Run error handling tests"
|
|
@echo " test-with-server - Run tests with automatic server management"
|
|
@echo " start-seaweedfs - Start SeaweedFS server for testing"
|
|
@echo " stop-seaweedfs - Stop SeaweedFS server"
|
|
@echo " clean - Clean up test artifacts"
|
|
@echo " check-binary - Check if SeaweedFS binary exists"
|
|
@echo " build-weed - Build SeaweedFS binary"
|
|
@echo ""
|
|
@echo "Configuration:"
|
|
@echo " SEAWEEDFS_BINARY=$(SEAWEEDFS_BINARY)"
|
|
@echo " FILER_PORT=$(FILER_PORT)"
|
|
@echo " VOLUME_PORT=$(VOLUME_PORT)"
|
|
@echo " MASTER_PORT=$(MASTER_PORT)"
|
|
@echo " TEST_TIMEOUT=$(TEST_TIMEOUT)"
|
|
|
|
check-binary:
|
|
@if ! command -v $(SEAWEEDFS_BINARY) > /dev/null 2>&1 && [ ! -f "$(SEAWEEDFS_ROOT)/weed/weed" ]; then \
|
|
echo "$(RED)Error: SeaweedFS binary not found$(NC)"; \
|
|
echo "Please build SeaweedFS first: make build-weed"; \
|
|
exit 1; \
|
|
fi
|
|
@echo "$(GREEN)SeaweedFS binary found$(NC)"
|
|
|
|
start-seaweedfs: check-binary
|
|
@echo "$(YELLOW)Starting SeaweedFS server for TUS testing...$(NC)"
|
|
@# Clean up any existing processes on our test ports
|
|
@lsof -ti :$(MASTER_PORT) | grep -v '^$$' | xargs kill -TERM 2>/dev/null || true
|
|
@lsof -ti :$(VOLUME_PORT) | grep -v '^$$' | xargs kill -TERM 2>/dev/null || true
|
|
@lsof -ti :$(FILER_PORT) | grep -v '^$$' | xargs kill -TERM 2>/dev/null || true
|
|
@sleep 2
|
|
|
|
# Create necessary directories
|
|
@mkdir -p /tmp/seaweedfs-test-tus-master
|
|
@mkdir -p /tmp/seaweedfs-test-tus-volume
|
|
@mkdir -p /tmp/seaweedfs-test-tus-filer
|
|
|
|
# Start master server (use freshly built binary)
|
|
@echo "Starting master server..."
|
|
@nohup $(SEAWEEDFS_ROOT)/weed/weed master \
|
|
-port=$(MASTER_PORT) \
|
|
-mdir=/tmp/seaweedfs-test-tus-master \
|
|
-volumeSizeLimitMB=$(VOLUME_MAX_SIZE_MB) \
|
|
-ip=127.0.0.1 \
|
|
> /tmp/seaweedfs-tus-master.log 2>&1 &
|
|
@sleep 3
|
|
|
|
# Start volume server
|
|
@echo "Starting volume server..."
|
|
@nohup $(SEAWEEDFS_ROOT)/weed/weed volume \
|
|
-port=$(VOLUME_PORT) \
|
|
-mserver=127.0.0.1:$(MASTER_PORT) \
|
|
-dir=/tmp/seaweedfs-test-tus-volume \
|
|
-max=$(VOLUME_MAX_COUNT) \
|
|
-ip=127.0.0.1 \
|
|
> /tmp/seaweedfs-tus-volume.log 2>&1 &
|
|
@sleep 3
|
|
|
|
# Start filer server with TUS enabled (default tusBasePath is .tus)
|
|
@echo "Starting filer server..."
|
|
@nohup $(SEAWEEDFS_ROOT)/weed/weed filer \
|
|
-port=$(FILER_PORT) \
|
|
-master=127.0.0.1:$(MASTER_PORT) \
|
|
-ip=127.0.0.1 \
|
|
> /tmp/seaweedfs-tus-filer.log 2>&1 &
|
|
@sleep 5
|
|
|
|
# Wait for filer to be ready
|
|
@echo "$(YELLOW)Waiting for filer to be ready...$(NC)"
|
|
@for i in $$(seq 1 30); do \
|
|
if curl -s -f http://127.0.0.1:$(FILER_PORT)/ > /dev/null 2>&1; then \
|
|
echo "$(GREEN)Filer is ready$(NC)"; \
|
|
break; \
|
|
fi; \
|
|
if [ $$i -eq 30 ]; then \
|
|
echo "$(RED)Filer failed to start within 30 seconds$(NC)"; \
|
|
$(MAKE) debug-logs; \
|
|
exit 1; \
|
|
fi; \
|
|
echo "Waiting for filer... ($$i/30)"; \
|
|
sleep 1; \
|
|
done
|
|
|
|
@echo "$(GREEN)SeaweedFS server started successfully for TUS testing$(NC)"
|
|
@echo "Master: http://localhost:$(MASTER_PORT)"
|
|
@echo "Volume: http://localhost:$(VOLUME_PORT)"
|
|
@echo "Filer: http://localhost:$(FILER_PORT)"
|
|
@echo "TUS Endpoint: http://localhost:$(FILER_PORT)/.tus/"
|
|
|
|
stop-seaweedfs:
|
|
@echo "$(YELLOW)Stopping SeaweedFS server...$(NC)"
|
|
@lsof -ti :$(MASTER_PORT) | grep -v '^$$' | xargs kill -TERM 2>/dev/null || true
|
|
@lsof -ti :$(VOLUME_PORT) | grep -v '^$$' | xargs kill -TERM 2>/dev/null || true
|
|
@lsof -ti :$(FILER_PORT) | grep -v '^$$' | xargs kill -TERM 2>/dev/null || true
|
|
@sleep 2
|
|
@echo "$(GREEN)SeaweedFS server stopped$(NC)"
|
|
|
|
clean:
|
|
@echo "$(YELLOW)Cleaning up TUS test artifacts...$(NC)"
|
|
@rm -rf /tmp/seaweedfs-test-tus-*
|
|
@rm -f /tmp/seaweedfs-tus-*.log
|
|
@echo "$(GREEN)TUS test cleanup completed$(NC)"
|
|
|
|
# Run all tests
|
|
test: check-binary
|
|
@echo "$(YELLOW)Running all TUS integration tests...$(NC)"
|
|
@cd $(SEAWEEDFS_ROOT) && go test -v -timeout=$(TEST_TIMEOUT) ./test/tus/...
|
|
@echo "$(GREEN)All TUS tests completed$(NC)"
|
|
|
|
# Run basic upload tests
|
|
test-basic: check-binary
|
|
@echo "$(YELLOW)Running basic TUS upload tests...$(NC)"
|
|
@cd $(SEAWEEDFS_ROOT) && go test -v -timeout=$(TEST_TIMEOUT) -run "TestTusBasicUpload|TestTusOptionsHandler" ./test/tus/...
|
|
@echo "$(GREEN)Basic TUS tests completed$(NC)"
|
|
|
|
# Run chunked upload tests
|
|
test-chunked: check-binary
|
|
@echo "$(YELLOW)Running chunked TUS upload tests...$(NC)"
|
|
@cd $(SEAWEEDFS_ROOT) && go test -v -timeout=$(TEST_TIMEOUT) -run "TestTusChunkedUpload" ./test/tus/...
|
|
@echo "$(GREEN)Chunked TUS tests completed$(NC)"
|
|
|
|
# Run resume tests
|
|
test-resume: check-binary
|
|
@echo "$(YELLOW)Running TUS upload resume tests...$(NC)"
|
|
@cd $(SEAWEEDFS_ROOT) && go test -v -timeout=$(TEST_TIMEOUT) -run "TestTusResumeAfterInterruption|TestTusHeadRequest" ./test/tus/...
|
|
@echo "$(GREEN)TUS resume tests completed$(NC)"
|
|
|
|
# Run error handling tests
|
|
test-errors: check-binary
|
|
@echo "$(YELLOW)Running TUS error handling tests...$(NC)"
|
|
@cd $(SEAWEEDFS_ROOT) && go test -v -timeout=$(TEST_TIMEOUT) -run "TestTusInvalidOffset|TestTusUploadNotFound|TestTusDeleteUpload" ./test/tus/...
|
|
@echo "$(GREEN)TUS error tests completed$(NC)"
|
|
|
|
# Run tests with automatic server management
|
|
test-with-server: build-weed
|
|
@echo "$(YELLOW)Running TUS tests with automatic server management...$(NC)"
|
|
@$(MAKE) -C $(TEST_DIR) start-seaweedfs && \
|
|
sleep 3 && \
|
|
cd $(SEAWEEDFS_ROOT) && go test -v -timeout=$(TEST_TIMEOUT) ./test/tus/...; \
|
|
TEST_RESULT=$$?; \
|
|
$(MAKE) -C $(TEST_DIR) stop-seaweedfs; \
|
|
$(MAKE) -C $(TEST_DIR) clean; \
|
|
if [ $$TEST_RESULT -eq 0 ]; then echo "$(GREEN)All TUS tests passed!$(NC)"; fi; \
|
|
exit $$TEST_RESULT
|
|
|
|
# Debug targets
|
|
debug-logs:
|
|
@echo "$(YELLOW)=== Master Log ===$(NC)"
|
|
@tail -n 50 /tmp/seaweedfs-tus-master.log 2>/dev/null || echo "No master log found"
|
|
@echo "$(YELLOW)=== Volume Log ===$(NC)"
|
|
@tail -n 50 /tmp/seaweedfs-tus-volume.log 2>/dev/null || echo "No volume log found"
|
|
@echo "$(YELLOW)=== Filer Log ===$(NC)"
|
|
@tail -n 50 /tmp/seaweedfs-tus-filer.log 2>/dev/null || echo "No filer log found"
|
|
|
|
debug-status:
|
|
@echo "$(YELLOW)=== Process Status ===$(NC)"
|
|
@ps aux | grep -E "(weed|seaweedfs)" | grep -v grep || echo "No SeaweedFS processes found"
|
|
@echo "$(YELLOW)=== Port Status ===$(NC)"
|
|
@lsof -i :$(MASTER_PORT) -i :$(VOLUME_PORT) -i :$(FILER_PORT) 2>/dev/null || echo "No ports in use"
|
|
|
|
# Manual testing targets
|
|
manual-start: start-seaweedfs
|
|
@echo "$(GREEN)SeaweedFS is now running for manual TUS testing$(NC)"
|
|
@echo ""
|
|
@echo "TUS Endpoints:"
|
|
@echo " OPTIONS /.tus/ - Capability discovery"
|
|
@echo " POST /.tus/{path} - Create upload"
|
|
@echo " HEAD /.tus/.uploads/{id} - Get offset"
|
|
@echo " PATCH /.tus/.uploads/{id} - Upload data"
|
|
@echo " DELETE /.tus/.uploads/{id} - Cancel upload"
|
|
@echo ""
|
|
@echo "Example curl commands:"
|
|
@echo " curl -X OPTIONS http://localhost:$(FILER_PORT)/.tus/ -H 'Tus-Resumable: 1.0.0'"
|
|
@echo ""
|
|
@echo "Run 'make manual-stop' when finished"
|
|
|
|
manual-stop: stop-seaweedfs clean
|
|
|
|
# CI targets
|
|
ci-test: test-with-server
|
|
|
|
# Skip integration tests (short mode)
|
|
test-short:
|
|
@echo "$(YELLOW)Running TUS tests in short mode (skipping integration tests)...$(NC)"
|
|
@cd $(SEAWEEDFS_ROOT) && go test -v -short ./test/tus/...
|
|
@echo "$(GREEN)Short tests completed$(NC)"
|
|
|