fix: SFTP HomeDir path translation for user operations (#7611)

* fix: SFTP HomeDir path translation for user operations When users have a non-root HomeDir (e.g., '/sftp/user'), their SFTP operations should be relative to that directory. Previously, when a user uploaded to '/' via SFTP, the path was not translated to their home directory, causing 'permission denied for / for permission write'. This fix adds a toAbsolutePath() method that implements chroot-like behavior where the user's HomeDir becomes their root. All file and directory operations now translate paths through this method. Example: User with HomeDir='/sftp/user' uploading to '/' now correctly maps to '/sftp/user'. Fixes: https://github.com/seaweedfs/seaweedfs/issues/7470 * test: add SFTP integration tests Add comprehensive integration tests for the SFTP server including: - HomeDir path translation tests (verifies fix for issue #7470) - Basic file upload/download operations - Directory operations (mkdir, rmdir, list) - Large file handling (1MB test) - File rename operations - Stat/Lstat operations - Path edge cases (trailing slashes, .., unicode filenames) - Admin root access verification The test framework starts a complete SeaweedFS cluster with: - Master server - Volume server - Filer server - SFTP server with test user credentials Test users are configured in testdata/userstore.json: - admin: HomeDir=/ with full access - testuser: HomeDir=/sftp/testuser with access to home - readonly: HomeDir=/public with read-only access * fix: correct SFTP HomeDir path translation and add CI Fix path.Join issue where paths starting with '/' weren't joined correctly. path.Join('/sftp/user', '/file') returns '/file' instead of '/sftp/user/file'. Now we strip the leading '/' before joining. Test improvements: - Update go.mod to Go 1.24 - Fix weed binary discovery to prefer local build over PATH - Add stabilization delay after service startup - All 8 SFTP integration tests pass locally Add GitHub Actions workflow for SFTP tests: - Runs on push/PR affecting sftpd code or tests - Tests HomeDir path translation, file ops, directory ops - Covers issue #7470 fix verification * security: update golang.org/x/crypto to v0.45.0 Addresses security vulnerability in golang.org/x/crypto < 0.45.0 * security: use proper SSH host key verification in tests Replace ssh.InsecureIgnoreHostKey() with ssh.FixedHostKey() that verifies the server's host key matches the known test key we generated. This addresses CodeQL warning go/insecure-hostkeycallback. Also updates go.mod to specify go 1.24.0 explicitly. * security: fix path traversal vulnerability in SFTP toAbsolutePath The previous implementation had a critical security vulnerability: - Path traversal via '../..' could escape the HomeDir chroot jail - Absolute paths were not correctly prefixed with HomeDir The fix: 1. Concatenate HomeDir with userPath directly, then clean 2. Add security check to ensure final path stays within HomeDir 3. If traversal detected, safely return HomeDir instead Also adds path traversal prevention tests to verify the fix. * fix: address PR review comments 1. Fix SkipCleanup check to use actual test config instead of default - Added skipCleanup field to SftpTestFramework struct - Store config.SkipCleanup during Setup() - Use f.skipCleanup in Cleanup() instead of DefaultTestConfig() 2. Fix path prefix check false positive in mkdir - Changed from strings.HasPrefix(absPath, fs.user.HomeDir) - To: absPath == fs.user.HomeDir || strings.HasPrefix(absPath, fs.user.HomeDir+"/") - Prevents matching partial directory names (e.g., /sftp/username when HomeDir is /sftp/user) * fix: check write permission on parent dir for mkdir Aligns makeDir's permission check with newFileWriter for consistency. To create a directory, a user needs write permission on the parent directory, not mkdir permission on the new directory path. * fix: refine SFTP path traversal logic and tests 1. Refine toAbsolutePath: - Use path.Join with strings.TrimPrefix for idiomatic path construction - Return explicit error on path traversal attempt instead of clamping - Updated all call sites to handle the error 2. Add Unit Tests: - Added sftp_server_test.go to verify toAbsolutePath logic - Covers normal paths, root path, and various traversal attempts 3. Update Integration Tests: - Updated PathTraversalPrevention test to reflect that standard SFTP clients sanitize paths before sending. The test now verifies successful containment within the jail rather than blocking (since the server receives a clean path). - The server-side blocking is verified by the new unit tests. 4. Makefile: - Removed -v from default test target * fix: address PR comments on tests and makefile 1. Enhanced Unit Tests: - Added edge cases (empty path, multiple slashes, trailing slash) to sftp_server_test.go 2. Makefile Improvements: - Added 'all' target as default entry point 3. Code Clarity: - Added comment to mkdir permission check explaining defensive nature of HomeDir check * fix: address PR review comments on permissions and tests 1. Security: - Added write permission check on target directory in renameEntry 2. Logging: - Changed dispatch log verbosity from V(0) to V(1) 3. Testing: - Updated Makefile .PHONY targets - Added unit test cases for empty/root HomeDir behavior in toAbsolutePath * fix: set SFTP starting directory to virtual root 1. Critical Fix: - Changed sftp.WithStartDirectory from fs.user.HomeDir to '/' - Prevents double-prefixing when toAbsolutePath translates paths - Users now correctly start at their virtual root which maps to HomeDir 2. Test Improvements: - Use pointer for homeDir in tests for clearer nil vs empty distinction * fix: clean HomeDir at config load time Clean HomeDir path when loading users from JSON config. This handles trailing slashes and other path anomalies at the source, ensuring consistency throughout the codebase and avoiding repeated cleaning on every toAbsolutePath call. * test: strengthen assertions and add error checking in SFTP tests 1. Add error checking for cleanup operations in TestWalk 2. Strengthen cwd assertion to expect '/' explicitly in TestCurrentWorkingDirectory 3. Add error checking for cleanup in PathTraversalPrevention test
2025-12-03 13:42:05 -08:00
parent d59cc1b09f
commit e361daa754
14 changed files with 1607 additions and 32 deletions
--- a/test/sftp/framework.go
+++ b/test/sftp/framework.go
@@ -0,0 +1,423 @@
+package sftp
+
+import (
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"syscall"
+	"testing"
+	"time"
+
+	"github.com/pkg/sftp"
+	"github.com/stretchr/testify/require"
+	"golang.org/x/crypto/ssh"
+)
+
+// SftpTestFramework provides utilities for SFTP integration testing
+type SftpTestFramework struct {
+	t             *testing.T
+	tempDir       string
+	dataDir       string
+	masterProcess *os.Process
+	volumeProcess *os.Process
+	filerProcess  *os.Process
+	sftpProcess   *os.Process
+	masterAddr    string
+	volumeAddr    string
+	filerAddr     string
+	sftpAddr      string
+	weedBinary    string
+	userStoreFile string
+	hostKeyFile   string
+	isSetup       bool
+	skipCleanup   bool
+}
+
+// TestConfig holds configuration for SFTP tests
+type TestConfig struct {
+	NumVolumes    int
+	EnableDebug   bool
+	SkipCleanup   bool // for debugging failed tests
+	UserStoreFile string
+}
+
+// DefaultTestConfig returns a default configuration for SFTP tests
+func DefaultTestConfig() *TestConfig {
+	return &TestConfig{
+		NumVolumes:    3,
+		EnableDebug:   false,
+		SkipCleanup:   false,
+		UserStoreFile: "",
+	}
+}
+
+// NewSftpTestFramework creates a new SFTP testing framework
+func NewSftpTestFramework(t *testing.T, config *TestConfig) *SftpTestFramework {
+	if config == nil {
+		config = DefaultTestConfig()
+	}
+
+	tempDir, err := os.MkdirTemp("", "seaweedfs_sftp_test_")
+	require.NoError(t, err)
+
+	// Generate SSH host key for SFTP server
+	hostKeyFile := filepath.Join(tempDir, "ssh_host_key")
+	cmd := exec.Command("ssh-keygen", "-t", "ed25519", "-f", hostKeyFile, "-N", "")
+	err = cmd.Run()
+	require.NoError(t, err, "failed to generate SSH host key")
+
+	// Use provided userstore or copy the test one
+	userStoreFile := config.UserStoreFile
+	if userStoreFile == "" {
+		// Copy test userstore to temp dir
+		userStoreFile = filepath.Join(tempDir, "userstore.json")
+		testDataPath := findTestDataPath()
+		input, err := os.ReadFile(filepath.Join(testDataPath, "userstore.json"))
+		require.NoError(t, err, "failed to read test userstore.json")
+		err = os.WriteFile(userStoreFile, input, 0644)
+		require.NoError(t, err, "failed to write userstore.json")
+	}
+
+	return &SftpTestFramework{
+		t:             t,
+		tempDir:       tempDir,
+		dataDir:       filepath.Join(tempDir, "data"),
+		masterAddr:    "127.0.0.1:19333",
+		volumeAddr:    "127.0.0.1:18080",
+		filerAddr:     "127.0.0.1:18888",
+		sftpAddr:      "127.0.0.1:12022",
+		weedBinary:    findWeedBinary(),
+		userStoreFile: userStoreFile,
+		hostKeyFile:   hostKeyFile,
+		isSetup:       false,
+	}
+}
+
+// Setup starts SeaweedFS cluster with SFTP server
+func (f *SftpTestFramework) Setup(config *TestConfig) error {
+	if f.isSetup {
+		return fmt.Errorf("framework already setup")
+	}
+
+	// Create all data directories
+	dirs := []string{
+		f.dataDir,
+		filepath.Join(f.dataDir, "master"),
+		filepath.Join(f.dataDir, "volume"),
+	}
+	for _, dir := range dirs {
+		if err := os.MkdirAll(dir, 0755); err != nil {
+			return fmt.Errorf("failed to create directory %s: %v", dir, err)
+		}
+	}
+
+	// Start master
+	if err := f.startMaster(config); err != nil {
+		return fmt.Errorf("failed to start master: %v", err)
+	}
+
+	// Wait for master to be ready
+	if err := f.waitForService(f.masterAddr, 30*time.Second); err != nil {
+		return fmt.Errorf("master not ready: %v", err)
+	}
+
+	// Start volume server
+	if err := f.startVolumeServer(config); err != nil {
+		return fmt.Errorf("failed to start volume server: %v", err)
+	}
+
+	// Wait for volume server to be ready
+	if err := f.waitForService(f.volumeAddr, 30*time.Second); err != nil {
+		return fmt.Errorf("volume server not ready: %v", err)
+	}
+
+	// Start filer
+	if err := f.startFiler(config); err != nil {
+		return fmt.Errorf("failed to start filer: %v", err)
+	}
+
+	// Wait for filer to be ready
+	if err := f.waitForService(f.filerAddr, 30*time.Second); err != nil {
+		return fmt.Errorf("filer not ready: %v", err)
+	}
+
+	// Start SFTP server
+	if err := f.startSftpServer(config); err != nil {
+		return fmt.Errorf("failed to start SFTP server: %v", err)
+	}
+
+	// Wait for SFTP server to be ready
+	if err := f.waitForService(f.sftpAddr, 30*time.Second); err != nil {
+		return fmt.Errorf("SFTP server not ready: %v", err)
+	}
+
+	// Additional wait for all services to stabilize (gRPC endpoints)
+	time.Sleep(500 * time.Millisecond)
+
+	f.skipCleanup = config.SkipCleanup
+	f.isSetup = true
+	return nil
+}
+
+// Cleanup stops all processes and removes temporary files
+func (f *SftpTestFramework) Cleanup() {
+	// Stop processes in reverse order
+	processes := []*os.Process{f.sftpProcess, f.filerProcess, f.volumeProcess, f.masterProcess}
+	for _, proc := range processes {
+		if proc != nil {
+			proc.Signal(syscall.SIGTERM)
+			proc.Wait()
+		}
+	}
+
+	// Remove temp directory
+	if !f.skipCleanup {
+		os.RemoveAll(f.tempDir)
+	}
+}
+
+// GetSftpAddr returns the SFTP server address
+func (f *SftpTestFramework) GetSftpAddr() string {
+	return f.sftpAddr
+}
+
+// GetFilerAddr returns the filer address
+func (f *SftpTestFramework) GetFilerAddr() string {
+	return f.filerAddr
+}
+
+// ConnectSFTP creates an SFTP client connection with the given credentials
+func (f *SftpTestFramework) ConnectSFTP(username, password string) (*sftp.Client, *ssh.Client, error) {
+	// Load the known host public key for verification
+	hostKeyCallback, err := f.getHostKeyCallback()
+	if err != nil {
+		return nil, nil, fmt.Errorf("failed to get host key callback: %v", err)
+	}
+
+	config := &ssh.ClientConfig{
+		User: username,
+		Auth: []ssh.AuthMethod{
+			ssh.Password(password),
+		},
+		HostKeyCallback: hostKeyCallback,
+		Timeout:         5 * time.Second,
+	}
+
+	sshConn, err := ssh.Dial("tcp", f.sftpAddr, config)
+	if err != nil {
+		return nil, nil, fmt.Errorf("failed to connect SSH: %v", err)
+	}
+
+	sftpClient, err := sftp.NewClient(sshConn)
+	if err != nil {
+		sshConn.Close()
+		return nil, nil, fmt.Errorf("failed to create SFTP client: %v", err)
+	}
+
+	return sftpClient, sshConn, nil
+}
+
+// getHostKeyCallback returns a callback that verifies the server's host key
+// matches the known test server key we generated
+func (f *SftpTestFramework) getHostKeyCallback() (ssh.HostKeyCallback, error) {
+	// Read the public key file generated alongside the private key
+	pubKeyFile := f.hostKeyFile + ".pub"
+	pubKeyBytes, err := os.ReadFile(pubKeyFile)
+	if err != nil {
+		return nil, fmt.Errorf("failed to read host public key: %v", err)
+	}
+
+	// Parse the public key
+	pubKey, _, _, _, err := ssh.ParseAuthorizedKey(pubKeyBytes)
+	if err != nil {
+		return nil, fmt.Errorf("failed to parse host public key: %v", err)
+	}
+
+	// Return a callback that verifies the server key matches our known key
+	return ssh.FixedHostKey(pubKey), nil
+}
+
+// startMaster starts the SeaweedFS master server
+func (f *SftpTestFramework) startMaster(config *TestConfig) error {
+	args := []string{
+		"master",
+		"-ip=127.0.0.1",
+		"-port=19333",
+		"-mdir=" + filepath.Join(f.dataDir, "master"),
+		"-raftBootstrap",
+		"-peers=none",
+	}
+
+	cmd := exec.Command(f.weedBinary, args...)
+	cmd.Dir = f.tempDir
+	if config.EnableDebug {
+		cmd.Stdout = os.Stdout
+		cmd.Stderr = os.Stderr
+	}
+	if err := cmd.Start(); err != nil {
+		return err
+	}
+	f.masterProcess = cmd.Process
+	return nil
+}
+
+// startVolumeServer starts SeaweedFS volume server
+func (f *SftpTestFramework) startVolumeServer(config *TestConfig) error {
+	args := []string{
+		"volume",
+		"-mserver=" + f.masterAddr,
+		"-ip=127.0.0.1",
+		"-port=18080",
+		"-dir=" + filepath.Join(f.dataDir, "volume"),
+		fmt.Sprintf("-max=%d", config.NumVolumes),
+	}
+
+	cmd := exec.Command(f.weedBinary, args...)
+	cmd.Dir = f.tempDir
+	if config.EnableDebug {
+		cmd.Stdout = os.Stdout
+		cmd.Stderr = os.Stderr
+	}
+	if err := cmd.Start(); err != nil {
+		return err
+	}
+	f.volumeProcess = cmd.Process
+	return nil
+}
+
+// startFiler starts the SeaweedFS filer server
+func (f *SftpTestFramework) startFiler(config *TestConfig) error {
+	args := []string{
+		"filer",
+		"-master=" + f.masterAddr,
+		"-ip=127.0.0.1",
+		"-port=18888",
+	}
+
+	cmd := exec.Command(f.weedBinary, args...)
+	cmd.Dir = f.tempDir
+	if config.EnableDebug {
+		cmd.Stdout = os.Stdout
+		cmd.Stderr = os.Stderr
+	}
+	if err := cmd.Start(); err != nil {
+		return err
+	}
+	f.filerProcess = cmd.Process
+	return nil
+}
+
+// startSftpServer starts the SeaweedFS SFTP server
+func (f *SftpTestFramework) startSftpServer(config *TestConfig) error {
+	args := []string{
+		"sftp",
+		"-filer=" + f.filerAddr,
+		"-ip.bind=127.0.0.1",
+		"-port=12022",
+		"-sshPrivateKey=" + f.hostKeyFile,
+		"-userStoreFile=" + f.userStoreFile,
+	}
+
+	cmd := exec.Command(f.weedBinary, args...)
+	cmd.Dir = f.tempDir
+	if config.EnableDebug {
+		cmd.Stdout = os.Stdout
+		cmd.Stderr = os.Stderr
+	}
+	if err := cmd.Start(); err != nil {
+		return err
+	}
+	f.sftpProcess = cmd.Process
+	return nil
+}
+
+// waitForService waits for a service to be available
+func (f *SftpTestFramework) waitForService(addr string, timeout time.Duration) error {
+	deadline := time.Now().Add(timeout)
+	for time.Now().Before(deadline) {
+		conn, err := net.DialTimeout("tcp", addr, 1*time.Second)
+		if err == nil {
+			conn.Close()
+			return nil
+		}
+		time.Sleep(100 * time.Millisecond)
+	}
+	return fmt.Errorf("service at %s not ready within timeout", addr)
+}
+
+// findWeedBinary locates the weed binary
+// Prefers local build over system-installed weed to ensure we test the latest code
+func findWeedBinary() string {
+	// Get the directory where this source file is located
+	// This ensures we find the locally built weed binary first
+	_, thisFile, _, ok := runtime.Caller(0)
+	if ok {
+		thisDir := filepath.Dir(thisFile)
+		// From test/sftp/, the weed binary should be at ../../weed/weed
+		candidates := []string{
+			filepath.Join(thisDir, "../../weed/weed"),
+			filepath.Join(thisDir, "../weed/weed"),
+		}
+		for _, candidate := range candidates {
+			if _, err := os.Stat(candidate); err == nil {
+				abs, _ := filepath.Abs(candidate)
+				return abs
+			}
+		}
+	}
+
+	// Try relative paths from current working directory
+	cwd, _ := os.Getwd()
+	candidates := []string{
+		filepath.Join(cwd, "../../weed/weed"),
+		filepath.Join(cwd, "../weed/weed"),
+		filepath.Join(cwd, "./weed"),
+	}
+	for _, candidate := range candidates {
+		if _, err := os.Stat(candidate); err == nil {
+			abs, _ := filepath.Abs(candidate)
+			return abs
+		}
+	}
+
+	// Fallback to PATH only if local build not found
+	if path, err := exec.LookPath("weed"); err == nil {
+		return path
+	}
+
+	// Default fallback
+	return "weed"
+}
+
+// findTestDataPath locates the testdata directory
+func findTestDataPath() string {
+	// Get the directory where this source file is located
+	_, thisFile, _, ok := runtime.Caller(0)
+	if ok {
+		thisDir := filepath.Dir(thisFile)
+		testDataPath := filepath.Join(thisDir, "testdata")
+		if _, err := os.Stat(testDataPath); err == nil {
+			return testDataPath
+		}
+	}
+
+	// Try relative paths from current working directory
+	cwd, _ := os.Getwd()
+	candidates := []string{
+		filepath.Join(cwd, "testdata"),
+		filepath.Join(cwd, "../sftp/testdata"),
+		filepath.Join(cwd, "test/sftp/testdata"),
+	}
+
+	for _, candidate := range candidates {
+		if _, err := os.Stat(candidate); err == nil {
+			return candidate
+		}
+	}
+
+	return "./testdata"
+}
+