Files
seaweedFS/weed/shell/command_cluster_raft_leader_transfer_test.go
Chris Lu f4cdfcc5fd Add cluster.raft.leader.transfer command for graceful leader change (#7819)
* proto: add RaftLeadershipTransfer RPC for forced leader change

Add new gRPC RPC and messages for leadership transfer:
- RaftLeadershipTransferRequest: optional target_id and target_address
- RaftLeadershipTransferResponse: previous_leader and new_leader

This enables graceful leadership transfer before master maintenance,
reducing errors in filers during planned maintenance windows.

Ref: https://github.com/seaweedfs/seaweedfs/issues/7527

* proto: regenerate Go files for RaftLeadershipTransfer

Generated from master.proto changes.

* master: implement RaftLeadershipTransfer gRPC handler

Add gRPC handler for leadership transfer with support for:
- Transfer to any eligible follower (when target_id is empty)
- Transfer to a specific server (when target_id and target_address are provided)

Uses hashicorp/raft LeadershipTransfer() and LeadershipTransferToServer() APIs.

Returns the previous and new leader in the response.

* shell: add cluster.raft.leader.transfer command

Add weed shell command for graceful leadership transfer:
- Displays current cluster status before transfer
- Supports auto-selection of target (any eligible follower)
- Supports targeted transfer with -id and -address flags
- Provides clear feedback on success/failure with troubleshooting tips

Usage:
  cluster.raft.leader.transfer
  cluster.raft.leader.transfer -id <server_id> -address <grpc_address>

* master: add unit tests for raft gRPC handlers

Add tests covering:
- RaftLeadershipTransfer with no raft initialized
- RaftLeadershipTransfer with target_id but no address
- RaftListClusterServers with no raft initialized
- RaftAddServer with no raft initialized
- RaftRemoveServer with no raft initialized

These tests verify error handling when raft is not configured.

* shell: add tests for cluster.raft.leader.transfer command

Add tests covering:
- Command name and help text validation
- HasTag returns false for ResourceHeavy
- Validation of -id without -address
- Argument parsing with unknown flags

* master: clarify that leadership transfer requires -raftHashicorp

The default raft implementation (seaweedfs/raft, a goraft fork) does not
support graceful leadership transfer. This feature is only available when
using hashicorp raft (-raftHashicorp=true).

Update error messages and help text to make this requirement clear:
- gRPC handler returns specific error for goraft users
- Shell command help text notes the requirement
- Added test for goraft case

* test: use strings.Contains instead of custom helper

Replace custom contains/containsHelper functions with the standard
library strings.Contains for better maintainability.

* shell: return flag parsing errors instead of swallowing them

- Return the error from flag.Parse() instead of returning nil
- Update test to explicitly assert error for unknown flags

* test: document integration test scenarios for Raft leadership transfer

Add comments explaining:
- Why these unit tests only cover 'Raft not initialized' scenarios
- What integration tests should cover (with multi-master cluster)
- hashicorp/raft uses concrete types that cannot be easily mocked

* fix: address reviewer feedback on tests and leader routing

- Remove misleading tests that couldn't properly validate their
  documented behavior without a real Raft cluster:
  - TestRaftLeadershipTransfer_GoraftNotSupported
  - TestRaftLeadershipTransfer_ValidationTargetIdWithoutAddress

- Change WithClient(false) to WithClient(true) for RaftLeadershipTransfer
  RPC to ensure the request is routed to the current leader

* Improve cluster.raft.transferLeader command

- Rename command from cluster.raft.leader.transfer to cluster.raft.transferLeader
- Add symmetric validation: -id and -address must be specified together
- Handle case where same leader is re-elected after transfer
- Add test for -address without -id validation
- Add docker compose file for 5-master raft cluster testing
2025-12-19 00:15:39 -08:00

90 lines
2.4 KiB
Go

package shell
import (
"bytes"
"strings"
"testing"
)
func TestRaftLeaderTransfer_Name(t *testing.T) {
cmd := &commandRaftLeaderTransfer{}
expected := "cluster.raft.transferLeader"
if cmd.Name() != expected {
t.Errorf("expected name %q, got %q", expected, cmd.Name())
}
}
func TestRaftLeaderTransfer_Help(t *testing.T) {
cmd := &commandRaftLeaderTransfer{}
help := cmd.Help()
// Verify help text contains key information
expectedPhrases := []string{
"transfer raft leadership",
"cluster.raft.transferLeader",
"-id",
"-address",
"cluster.raft.ps",
"-raftHashicorp",
}
for _, phrase := range expectedPhrases {
if !strings.Contains(help, phrase) {
t.Errorf("help text should contain %q", phrase)
}
}
}
func TestRaftLeaderTransfer_HasTag(t *testing.T) {
cmd := &commandRaftLeaderTransfer{}
// The command should not have any special tags
if cmd.HasTag(ResourceHeavy) {
t.Error("expected HasTag to return false for ResourceHeavy")
}
}
func TestRaftLeaderTransfer_ValidateTargetIdWithoutAddress(t *testing.T) {
cmd := &commandRaftLeaderTransfer{}
var buf bytes.Buffer
// Create a mock command environment - this will fail because no master client
// but we can verify argument parsing
err := cmd.Do([]string{"-id", "test-server"}, nil, &buf)
// Should fail because -address is required when -id is specified
if err == nil {
t.Error("expected error when -id is specified without -address")
}
if err != nil && !strings.Contains(err.Error(), "-address is required") {
t.Errorf("expected error about missing -address, got: %v", err)
}
}
func TestRaftLeaderTransfer_ValidateTargetAddressWithoutId(t *testing.T) {
cmd := &commandRaftLeaderTransfer{}
var buf bytes.Buffer
// Verify argument parsing - address without id should fail
err := cmd.Do([]string{"-address", "localhost:19333"}, nil, &buf)
// Should fail because -id is required when -address is specified
if err == nil {
t.Error("expected error when -address is specified without -id")
}
if err != nil && !strings.Contains(err.Error(), "-id is required") {
t.Errorf("expected error about missing -id, got: %v", err)
}
}
func TestRaftLeaderTransfer_UnknownFlag(t *testing.T) {
cmd := &commandRaftLeaderTransfer{}
var buf bytes.Buffer
// Unknown flag should return an error
err := cmd.Do([]string{"-unknown-flag"}, nil, &buf)
if err == nil {
t.Error("expected error for unknown flag")
}
}