gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add design for mq balancer

Browse Source
This commit is contained in:
chrislu
2023-09-30 13:19:05 -07:00
parent d7c211fe8c
commit 597ab1e4bf
1 changed files with 17 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
weed/mq/balancer/balancer.go
View File

@@ -12,6 +12,23 @@ const (
LockBrokerBalancer = "broker_balancer" LockBrokerBalancer = "broker_balancer"
) )
// Balancer collects stats from all brokers.
//
// When publishers wants to create topics, it picks brokers to assign the topic partitions.
// When consumers wants to subscribe topics, it tells which brokers are serving the topic partitions.
//
// When a partition needs to be split or merged, or a partition needs to be moved to another broker,
// the balancer will let the broker tell the consumer instance to stop processing the partition.
// The existing consumer instance will flush the internal state, and then stop processing.
// Then the balancer will tell the brokers to start sending new messages in the new/moved partition to the consumer instances.
//
// Failover to standby consumer instances:
//
// A consumer group can have min and max number of consumer instances.
// For consumer instances joined after the max number, they will be in standby mode.
//
// When a consumer instance is down, the broker will notice this and inform the balancer.
// The balancer will then tell the broker to send the partition to another standby consumer instance.
type Balancer struct { type Balancer struct {
Brokers cmap.ConcurrentMap[string, *BrokerStats] Brokers cmap.ConcurrentMap[string, *BrokerStats]
} }