gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6b98b52accb1d0737ebc648a2c6a4ac80bf1e65b
seaweedFS/weed/server
History
Lisandro Pin 6b98b52acc Fix reporting of EC shard sizes from nodes to masters. (#7835) 
SeaweedFS tracks EC shard sizes on topology data stuctures, but this information is never
relayed to master servers :( The end result is that commands reporting disk usage, such
as `volume.list` and `cluster.status`, yield incorrect figures when EC shards are present.

As an example for a simple 5-node test cluster, before...

```
> volume.list
Topology volumeSizeLimit:30000 MB hdd(volume:6/40 active:6 free:33 remote:0)
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9001 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:3  size:88967096  file_count:172  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[1 5]
        Disk hdd total size:88967096 file_count:172
      DataNode 192.168.10.111:9001 total size:88967096 file_count:172
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9002 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:77267536  file_count:166  replica_placement:2  version:3  modified_at_second:1766349617
          volume id:3  size:88967096  file_count:172  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[0 4]
        Disk hdd total size:166234632 file_count:338
      DataNode 192.168.10.111:9002 total size:166234632 file_count:338
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9003 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:2  size:77267536  file_count:166  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[2 6]
        Disk hdd total size:77267536 file_count:166
      DataNode 192.168.10.111:9003 total size:77267536 file_count:166
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9004 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:77267536  file_count:166  replica_placement:2  version:3  modified_at_second:1766349617
          volume id:3  size:88967096  file_count:172  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[3 7]
        Disk hdd total size:166234632 file_count:338
      DataNode 192.168.10.111:9004 total size:166234632 file_count:338
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9005 hdd(volume:0/8 active:0 free:8 remote:0)
        Disk hdd(volume:0/8 active:0 free:8 remote:0) id:0
          ec volume id:1 collection: shards:[8 9 10 11 12 13]
        Disk hdd total size:0 file_count:0
    Rack DefaultRack total size:498703896 file_count:1014
  DataCenter DefaultDataCenter total size:498703896 file_count:1014
total size:498703896 file_count:1014
```

...and after:

```
> volume.list
Topology volumeSizeLimit:30000 MB hdd(volume:6/40 active:6 free:33 remote:0)
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9001 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:2  size:81761800  file_count:161  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[1 5 9] sizes:[1:8.00 MiB 5:8.00 MiB 9:8.00 MiB] total:24.00 MiB
        Disk hdd total size:81761800 file_count:161
      DataNode 192.168.10.111:9001 total size:81761800 file_count:161
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9002 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:3  size:88678712  file_count:170  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[11 12 13] sizes:[11:8.00 MiB 12:8.00 MiB 13:8.00 MiB] total:24.00 MiB
        Disk hdd total size:88678712 file_count:170
      DataNode 192.168.10.111:9002 total size:88678712 file_count:170
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9003 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:81761800  file_count:161  replica_placement:2  version:3  modified_at_second:1766349495
          volume id:3  size:88678712  file_count:170  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[0 4 8] sizes:[0:8.00 MiB 4:8.00 MiB 8:8.00 MiB] total:24.00 MiB
        Disk hdd total size:170440512 file_count:331
      DataNode 192.168.10.111:9003 total size:170440512 file_count:331
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9004 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:81761800  file_count:161  replica_placement:2  version:3  modified_at_second:1766349495
          volume id:3  size:88678712  file_count:170  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[2 6 10] sizes:[2:8.00 MiB 6:8.00 MiB 10:8.00 MiB] total:24.00 MiB
        Disk hdd total size:170440512 file_count:331
      DataNode 192.168.10.111:9004 total size:170440512 file_count:331
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9005 hdd(volume:0/8 active:0 free:8 remote:0)
        Disk hdd(volume:0/8 active:0 free:8 remote:0) id:0
          ec volume id:1 collection: shards:[3 7] sizes:[3:8.00 MiB 7:8.00 MiB] total:16.00 MiB
        Disk hdd total size:0 file_count:0
    Rack DefaultRack total size:511321536 file_count:993
  DataCenter DefaultDataCenter total size:511321536 file_count:993
total size:511321536 file_count:993
```
2025-12-28 19:30:42 -08:00
..
constants
Nit: use time.Durations instead of constants in seconds. (#7438)
2025-11-04 13:02:22 -08:00
filer_ui
fix(filer-ui): support folder creation with JWT token in URL (#7271)
2025-11-26 12:35:08 -08:00
master_ui
Fix/parse upload filename (#6241)
2024-11-15 11:25:18 -08:00
postgres
Message Queue: Add sql querying (#7185)
2025-09-09 01:01:03 -07:00
volume_server_ui
Admin: misc improvements on admin server and workers. EC now works. (#7055)
2025-07-30 12:38:03 -07:00
common_test.go
jwt check the base file id
2019-03-03 10:17:44 -08:00
common.go
fix: Use mime.FormatMediaType for RFC 6266 compliant Content-Disposition (#7635)
2025-12-05 15:59:12 -08:00
filer_grpc_server_admin.go
Changes logging function (#6919)
2025-06-24 08:44:06 -07:00
filer_grpc_server_dlm.go
reduce logs
2025-12-26 13:26:25 -08:00
filer_grpc_server_kv.go
fix(filer): add missing return after KvDelete in KvPut with empty value (#6846)
2025-06-06 05:58:03 -07:00
filer_grpc_server_remote.go
s3: fix remote object not caching (#7790)
2025-12-16 12:41:04 -08:00
filer_grpc_server_rename.go
conditional delete
2024-08-01 13:00:54 -07:00
filer_grpc_server_sub_meta.go
avoid repeated reading disk (#7369)
2025-10-24 01:26:42 -07:00
filer_grpc_server_traverse_meta_test.go
bootstrap filer from one peer
2024-06-28 14:57:20 -07:00
filer_grpc_server_traverse_meta.go
Add error list each entry func (#7485)
2025-11-25 19:35:19 -08:00
filer_grpc_server.go
Add error list each entry func (#7485)
2025-11-25 19:35:19 -08:00
filer_server_handlers_copy.go
use one http client
2025-08-11 16:45:13 -07:00
filer_server_handlers_proxy.go
Changes logging function (#6919)
2025-06-24 08:44:06 -07:00
filer_server_handlers_read_dir.go
Changes logging function (#6919)
2025-06-24 08:44:06 -07:00
filer_server_handlers_read.go
S3: set identity to request context, and remove obsolete code (#7523)
2025-11-21 14:46:32 -08:00
filer_server_handlers_tagging.go
Changes logging function (#6919)
2025-06-24 08:44:06 -07:00
filer_server_handlers_write_autochunk.go
filer write request use context without cancellation (#7567)
2025-11-28 11:52:57 -08:00
filer_server_handlers_write_cipher.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
filer_server_handlers_write_merge.go
S3 API: Add SSE-KMS (#7144)
2025-08-21 08:28:07 -07:00
filer_server_handlers_write_upload.go
filer write request use context without cancellation (#7567)
2025-11-28 11:52:57 -08:00
filer_server_handlers_write.go
Reduce memory allocations in hot paths (#7725)
2025-12-12 12:51:48 -08:00
filer_server_handlers.go
Metrics: Add Prometheus metrics for concurrent upload tracking (#7555)
2025-11-26 15:51:38 -08:00
filer_server_rocksdb.go
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
filer_server_tus_handlers.go
Add TUS protocol support for resumable uploads (#7592)
2025-12-14 21:56:07 -08:00
filer_server_tus_session.go
Add TUS protocol support for resumable uploads (#7592)
2025-12-14 21:56:07 -08:00
filer_server.go
s3: fix remote object not caching (#7790)
2025-12-16 12:41:04 -08:00
master_grpc_server_admin.go
math/rand => math/rand/v2
2024-08-29 09:52:21 -07:00
master_grpc_server_assign.go
add disable volume_growth flag (#7196)
2025-09-04 05:39:56 -07:00
master_grpc_server_cluster.go
math/rand => math/rand/v2
2024-08-29 09:52:21 -07:00
master_grpc_server_collection.go
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
master_grpc_server_raft_test.go
Add cluster.raft.leader.transfer command for graceful leader change (#7819)
2025-12-19 00:15:39 -08:00
master_grpc_server_raft.go
Add cluster.raft.leader.transfer command for graceful leader change (#7819)
2025-12-19 00:15:39 -08:00
master_grpc_server_volume.go
Support separate volume server ID independent of RPC bind address (#7609)
2025-12-02 22:08:11 -08:00
master_grpc_server.go
Add 'weed mini' command for S3 beginners and small/dev use cases (#7831)
2025-12-21 11:10:01 -08:00
master_server_handlers_admin.go
follow grow volume option version
2025-06-19 13:54:54 -07:00
master_server_handlers_ui.go
hide millseconds in up time (#7553)
2025-11-26 08:01:19 -08:00
master_server_handlers.go
add disable volume_growth flag (#7196)
2025-09-04 05:39:56 -07:00
master_server.go
master: Fix raft url (#7255)
2025-09-18 14:46:53 -07:00
raft_hashicorp.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
raft_server_handlers.go
avoid logging nil error (#4659)
2023-07-10 23:38:52 -07:00
raft_server.go
master:fix empty target in Build() (#6069)
2024-09-27 00:45:18 -07:00
volume_grpc_admin.go
fix: restore volume mount when VolumeConfigure fails (#7669)
2025-12-08 16:43:35 -08:00
volume_grpc_batch_delete.go
fix has changes false if deleted result size eq zero (#4909)
2023-10-12 08:15:20 -07:00
volume_grpc_client_to_master.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
volume_grpc_copy_incremental.go
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
volume_grpc_copy.go
fix: correctly detect missing source file during volume copy (#7784)
2025-12-15 22:38:28 -08:00
volume_grpc_erasure_coding.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
volume_grpc_query.go
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
volume_grpc_read_all.go
weed/server: remove unneeded err from streamReadOneVolume() signature (#5143)
2023-12-28 11:45:13 -08:00
volume_grpc_read_write.go
ADHOC: volume fsck using append at ns (#3906)
2022-10-24 22:09:38 -07:00
volume_grpc_remote.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
volume_grpc_tail.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
volume_grpc_tier_download.go
avoid load volume file with BytesOffset mismatch (#3841)
2022-10-14 00:18:09 -07:00
volume_grpc_tier_upload.go
avoid load volume file with BytesOffset mismatch (#3841)
2022-10-14 00:18:09 -07:00
volume_grpc_vacuum.go
Fix no more writable volumes by delay judgment (#4548)
2023-06-05 10:17:21 -07:00
volume_server_handlers_admin.go
fix: volume server healthz now checks local conditions only (#7610)
2025-12-02 23:19:14 -08:00
volume_server_handlers_helper.go
directory structure change to work with glide
2016-06-02 18:09:14 -07:00
volume_server_handlers_read.go
fix: Use mime.FormatMediaType for RFC 6266 compliant Content-Disposition (#7635)
2025-12-05 15:59:12 -08:00
volume_server_handlers_ui.go
hide millseconds in up time (#7553)
2025-11-26 08:01:19 -08:00
volume_server_handlers_write.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
volume_server_handlers.go
fix: JWT validation failures during replication (#7788) (#7795)
2025-12-16 13:42:18 -08:00
volume_server.go
Separate vacuum speed from replication speed (#7632)
2025-12-05 12:24:38 -08:00
webdav_server.go
mount: improve read throughput with parallel chunk fetching (#7569)
2025-11-29 10:06:11 -08:00
wrapped_webdav_fs.go
fix: return etag with md5 in webdav responses (#5158)
2024-01-03 08:50:32 -08:00