Commit Graph

53 Commits

Author SHA1 Message Date
Chris Lu
5ed0b00fb9 Support separate volume server ID independent of RPC bind address (#7609)
* pb: add id field to Heartbeat message for stable volume server identification

This adds an 'id' field to the Heartbeat protobuf message that allows
volume servers to identify themselves independently of their IP:port address.

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487

* storage: add Id field to Store struct

Add Id field to Store struct and include it in CollectHeartbeat().
The Id field provides a stable volume server identity independent of IP:port.

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487

* topology: support id-based DataNode identification

Update GetOrCreateDataNode to accept an id parameter for stable node
identification. When id is provided, the DataNode can maintain its identity
even when its IP address changes (e.g., in Kubernetes pod reschedules).

For backward compatibility:
- If id is provided, use it as the node ID
- If id is empty, fall back to ip:port

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487

* volume: add -id flag for stable volume server identity

Add -id command line flag to volume server that allows specifying a stable
identifier independent of the IP address. This is useful for Kubernetes
deployments with hostPath volumes where pods can be rescheduled to different
nodes while the persisted data remains on the original node.

Usage: weed volume -id=node-1 -ip=10.0.0.1 ...

If -id is not specified, it defaults to ip:port for backward compatibility.

Fixes https://github.com/seaweedfs/seaweedfs/issues/7487

* server: add -volume.id flag to weed server command

Support the -volume.id flag in the all-in-one 'weed server' command,
consistent with the standalone 'weed volume' command.

Usage: weed server -volume.id=node-1 ...

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487

* topology: add test for id-based DataNode identification

Test the key scenarios:
1. Create DataNode with explicit id
2. Same id with different IP returns same DataNode (K8s reschedule)
3. IP/PublicUrl are updated when node reconnects with new address
4. Different id creates new DataNode
5. Empty id falls back to ip:port (backward compatibility)

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487

* pb: add address field to DataNodeInfo for proper node addressing

Previously, DataNodeInfo.Id was used as the node address, which worked
when Id was always ip:port. Now that Id can be an explicit string,
we need a separate Address field for connection purposes.

Changes:
- Add 'address' field to DataNodeInfo protobuf message
- Update ToDataNodeInfo() to populate the address field
- Update NewServerAddressFromDataNode() to use Address (with Id fallback)
- Fix LookupEcVolume to use dn.Url() instead of dn.Id()

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487

* fix: trim whitespace from volume server id and fix test

- Trim whitespace from -id flag to treat ' ' as empty
- Fix store_load_balancing_test.go to include id parameter in NewStore call

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487

* refactor: extract GetVolumeServerId to util package

Move the volume server ID determination logic to a shared utility function
to avoid code duplication between volume.go and rack.go.

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487

* fix: improve transition logic for legacy nodes

- Use exact ip:port match instead of net.SplitHostPort heuristic
- Update GrpcPort and PublicUrl during transition for consistency
- Remove unused net import

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487

* fix: add id normalization and address change logging

- Normalize id parameter at function boundary (trim whitespace)
- Log when DataNode IP:Port changes (helps debug K8s pod rescheduling)

Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
2025-12-02 22:08:11 -08:00
Chris Lu
7acebf11ea Master: volume assignment concurrency (#7159)
* volume assginment concurrency

* accurate tests

* ensure uniqness

* reserve atomically

* address comments

* atomic

* ReserveOneVolumeForReservation

* duplicated

* Update weed/topology/node.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/topology/node.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* atomic counter

* dedup

* select the appropriate functions based on the useReservations flag

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-08-23 21:02:30 -07:00
chrislu
35fd1e1c9a optimize memory usage for large number of volumes
1. unwrap the map to avoid extra map object creation
2. fix ec shard counting in UpdateEcShards
2024-10-10 10:00:30 -07:00
wusong
6f58ab7e8b [master] fix master panic (#5893) 2024-08-15 06:20:51 -07:00
小羽
e8537d7172 Different disk labels should not use the same DiskUsages instance while master received volume heatbeat (#5770) 2024-07-12 08:09:51 -07:00
chrislu
ca7cc61319 reduce the window size between unregistering a volume server and creating volumes on that server
fix https://github.com/seaweedfs/seaweedfs/issues/4467
2023-05-30 08:20:00 -07:00
chrislu
6690236754 rename variables 2022-09-11 19:18:42 -07:00
Konstantin Lebedev
695e3a909c avoid data race on currentDiskUsage.maxVolumeCount (#3592)
https://github.com/seaweedfs/seaweedfs/issues/3590
2022-09-07 09:51:15 -07:00
Konstantin Lebedev
4d08393b7c filer prefer volume server in same data center (#3405)
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
2022-08-04 17:35:00 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
ningfd
58f2dd6740 change topology mapping definition, improve spelling 2022-07-22 17:18:56 +08:00
ningfd
6f882eb354 optimiz: master ui will render data in order 2022-07-22 14:37:49 +08:00
Chris Lu
8a66306064 calculate disk usage in case of race condition
related to https://github.com/chrislusf/seaweedfs/issues/2357
2021-10-04 23:32:07 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
574485ec69 better IP v6 support 2021-09-07 19:29:42 -07:00
Chris Lu
65af3cf4df master: disconnect only the phantom volume server
fix https://github.com/chrislusf/seaweedfs/issues/2311
2021-09-05 15:20:03 -07:00
Chris Lu
87a32bfef4 avoid possible nil when node is disconnected from its parent
fix https://github.com/chrislusf/seaweedfs/issues/2073
2021-05-19 10:02:01 -07:00
Chris Lu
565f7a6e72 Update data_node.go 2021-02-19 14:22:36 -08:00
Chris Lu
a37473ae60 add back volume ids
address https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-782339576
2021-02-19 14:22:12 -08:00
Chris Lu
c576ad04ac fix volume server display for volumes 2021-02-19 01:38:56 -08:00
Chris Lu
53ca7e66ef avoid dead lock 2021-02-16 10:48:16 -08:00
Chris Lu
cb9cc29518 volume.list display; fix updating maxVolumeCount for disk 2021-02-16 03:55:24 -08:00
Chris Lu
3fe628f04e use hdd instead of empty string 2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu
9c9ba3c209 nil related
related to https://github.com/chrislusf/seaweedfs/issues/1676
2021-01-03 12:25:58 -08:00
Chris Lu
d9e8479c06 adjust UI max count 2020-12-17 13:47:51 -08:00
Chris Lu
3cdf5945a2 adjust UI 2020-12-17 13:37:00 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu
289e62a305 master: better locking of in memory volume data
related to https://github.com/chrislusf/seaweedfs/issues/1436#issuecomment-695880135
2020-09-20 23:07:55 -07:00
cheng.li01
dad1161c70 fix dn.volumes Iterate when write issue 2020-07-08 19:57:19 +08:00
Evgenii Kozlov
0e0db70f55 Set volumes ReadOnly if low free disk space 2020-06-05 18:18:15 +03:00
bingoohuang
1f8782a1ed try showing the first 100 volume ids and an extra ... 2020-05-29 16:15:33 +08:00
bingoohuang
1a642b9876 add Volume Ids column only for max 100 volumes for convenience in the master ui. 2020-05-29 15:37:58 +08:00
Chris Lu
e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu
eaa76f11b7 free volume slots factor in ec shard counts 2019-06-04 23:41:56 -07:00
Chris Lu
2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu
6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu
4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu
7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Chris Lu
6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
Chris Lu
a12c7b86b0 broadcast messages of new and deleted volumes 2018-07-27 23:09:55 -07:00
Chris Lu
e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00
Chris Lu
0559aa9673 use Lock instead of RLock
fix https://github.com/chrislusf/seaweedfs/issues/364
2016-09-07 18:21:05 -07:00
霍晓栋
f04d8fcbcc if replicated volume has one copy in readonly mode at one node,it should be removed from writable list 2016-08-09 20:12:39 +08:00
Chris Lu
5ce6bbf076 directory structure change to work with glide
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
2016-06-02 18:09:14 -07:00