gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6b98b52accb1d0737ebc648a2c6a4ac80bf1e65b
seaweedFS/weed/shell
History
Lisandro Pin 6b98b52acc Fix reporting of EC shard sizes from nodes to masters. (#7835) 
SeaweedFS tracks EC shard sizes on topology data stuctures, but this information is never
relayed to master servers :( The end result is that commands reporting disk usage, such
as `volume.list` and `cluster.status`, yield incorrect figures when EC shards are present.

As an example for a simple 5-node test cluster, before...

```
> volume.list
Topology volumeSizeLimit:30000 MB hdd(volume:6/40 active:6 free:33 remote:0)
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9001 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:3  size:88967096  file_count:172  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[1 5]
        Disk hdd total size:88967096 file_count:172
      DataNode 192.168.10.111:9001 total size:88967096 file_count:172
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9002 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:77267536  file_count:166  replica_placement:2  version:3  modified_at_second:1766349617
          volume id:3  size:88967096  file_count:172  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[0 4]
        Disk hdd total size:166234632 file_count:338
      DataNode 192.168.10.111:9002 total size:166234632 file_count:338
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9003 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:2  size:77267536  file_count:166  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[2 6]
        Disk hdd total size:77267536 file_count:166
      DataNode 192.168.10.111:9003 total size:77267536 file_count:166
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9004 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:77267536  file_count:166  replica_placement:2  version:3  modified_at_second:1766349617
          volume id:3  size:88967096  file_count:172  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[3 7]
        Disk hdd total size:166234632 file_count:338
      DataNode 192.168.10.111:9004 total size:166234632 file_count:338
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9005 hdd(volume:0/8 active:0 free:8 remote:0)
        Disk hdd(volume:0/8 active:0 free:8 remote:0) id:0
          ec volume id:1 collection: shards:[8 9 10 11 12 13]
        Disk hdd total size:0 file_count:0
    Rack DefaultRack total size:498703896 file_count:1014
  DataCenter DefaultDataCenter total size:498703896 file_count:1014
total size:498703896 file_count:1014
```

...and after:

```
> volume.list
Topology volumeSizeLimit:30000 MB hdd(volume:6/40 active:6 free:33 remote:0)
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9001 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:2  size:81761800  file_count:161  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[1 5 9] sizes:[1:8.00 MiB 5:8.00 MiB 9:8.00 MiB] total:24.00 MiB
        Disk hdd total size:81761800 file_count:161
      DataNode 192.168.10.111:9001 total size:81761800 file_count:161
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9002 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:3  size:88678712  file_count:170  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[11 12 13] sizes:[11:8.00 MiB 12:8.00 MiB 13:8.00 MiB] total:24.00 MiB
        Disk hdd total size:88678712 file_count:170
      DataNode 192.168.10.111:9002 total size:88678712 file_count:170
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9003 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:81761800  file_count:161  replica_placement:2  version:3  modified_at_second:1766349495
          volume id:3  size:88678712  file_count:170  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[0 4 8] sizes:[0:8.00 MiB 4:8.00 MiB 8:8.00 MiB] total:24.00 MiB
        Disk hdd total size:170440512 file_count:331
      DataNode 192.168.10.111:9003 total size:170440512 file_count:331
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9004 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:81761800  file_count:161  replica_placement:2  version:3  modified_at_second:1766349495
          volume id:3  size:88678712  file_count:170  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[2 6 10] sizes:[2:8.00 MiB 6:8.00 MiB 10:8.00 MiB] total:24.00 MiB
        Disk hdd total size:170440512 file_count:331
      DataNode 192.168.10.111:9004 total size:170440512 file_count:331
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9005 hdd(volume:0/8 active:0 free:8 remote:0)
        Disk hdd(volume:0/8 active:0 free:8 remote:0) id:0
          ec volume id:1 collection: shards:[3 7] sizes:[3:8.00 MiB 7:8.00 MiB] total:16.00 MiB
        Disk hdd total size:0 file_count:0
    Rack DefaultRack total size:511321536 file_count:993
  DataCenter DefaultDataCenter total size:511321536 file_count:993
total size:511321536 file_count:993
```
2025-12-28 19:30:42 -08:00
..
command_cluster_check.go
[shell] use constant for hdd of type (#6337)
2024-12-10 08:43:59 -08:00
command_cluster_ps.go
refactor
2024-09-29 10:38:22 -07:00
command_cluster_raft_add.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_cluster_raft_leader_transfer_test.go
Add cluster.raft.leader.transfer command for graceful leader change (#7819)
2025-12-19 00:15:39 -08:00
command_cluster_raft_leader_transfer.go
Add cluster.raft.leader.transfer command for graceful leader change (#7819)
2025-12-19 00:15:39 -08:00
command_cluster_raft_ps.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_cluster_raft_remove.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_cluster_status_test.go
Give cluster.status detailed file metrics for regular volumes (#7791)
2025-12-17 16:40:27 -08:00
command_cluster_status.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_collection_delete.go
Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). (#7450)
2025-11-09 19:58:38 -08:00
command_collection_list.go
refactor
2024-09-29 10:38:22 -07:00
command_ec_balance.go
ec: add -diskType flag to EC commands for SSD support (#7607)
2025-12-10 22:42:52 -08:00
command_ec_common_test.go
fix: EC rebalance fails with replica placement 000 (#7812)
2025-12-19 13:29:12 -08:00
command_ec_common.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_ec_decode.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_ec_encode.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_ec_rebuild_test.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_ec_rebuild.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_ec_test.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_fs_cat.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_cd.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_configure.go
Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). (#7450)
2025-11-09 19:58:38 -08:00
command_fs_du.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_log.go
added context to filer_client method calls (#6808)
2025-05-22 09:46:49 -07:00
command_fs_ls.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_merge_volumes.go
Add context with request (#6824)
2025-05-28 11:34:02 -07:00
command_fs_meta_cat.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_meta_change_volume_id.go
Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). (#7450)
2025-11-09 19:58:38 -08:00
command_fs_meta_load.go
de/compress the fs meta file if filename ends with gz/gzip (#7500)
2025-11-18 07:45:12 -08:00
command_fs_meta_notify.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_meta_save.go
fs.meta.save: fix directory entry parent path in FullEntry construction (#7886)
2025-12-26 12:30:30 -08:00
command_fs_mkdir.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_mv.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_pwd.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_rm.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_tree.go
Shell: Added a helper function isHelpRequest() (#7380)
2025-10-24 20:21:35 -07:00
command_fs_verify.go
Add context with request (#6824)
2025-05-28 11:34:02 -07:00
command_lock_unlock.go
refactor
2024-09-29 10:38:22 -07:00
command_mount_configure.go
Clean up logs and deprecated functions (#7339)
2025-10-17 22:11:50 -07:00
command_mq_balance.go
refactor
2024-09-29 10:38:22 -07:00
command_mq_topic_compact.go
Add Kafka Gateway (#7231)
2025-10-13 18:05:17 -07:00
command_mq_topic_configure.go
Add message queue agent (#6463)
2025-01-20 22:19:27 -08:00
command_mq_topic_desc.go
Add message queue agent (#6463)
2025-01-20 22:19:27 -08:00
command_mq_topic_list.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_mq_topic_truncate.go
Message Queue: Add sql querying (#7185)
2025-09-09 01:01:03 -07:00
command_remote_cache.go
s3: fix remote object not caching (#7790)
2025-12-16 12:41:04 -08:00
command_remote_configure.go
add one example
2025-12-28 11:39:06 -08:00
command_remote_meta_sync.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_remote_mount_buckets.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_remote_mount.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_remote_uncache.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_remote_unmount.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_s3_bucket_create.go
shell: add -owner flag to s3.bucket.create command (#7728)
2025-12-12 18:06:13 -08:00
command_s3_bucket_delete.go
fix: admin UI bucket delete now properly deletes collection and checks Object Lock (#7734)
2025-12-13 13:41:25 -08:00
command_s3_bucket_list.go
shell: add -owner flag to s3.bucket.create command (#7728)
2025-12-12 18:06:13 -08:00
command_s3_bucket_owner.go
shell: add -owner flag to s3.bucket.create command (#7728)
2025-12-12 18:06:13 -08:00
command_s3_bucket_quota_check.go
Reduce memory allocations in hot paths (#7725)
2025-12-12 12:51:48 -08:00
command_s3_bucket_quota.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_s3_circuitbreaker_test.go
refactor(shell): readability improvements (#3704)
2022-09-16 02:43:49 -07:00
command_s3_circuitbreaker.go
chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)
2025-03-31 21:42:54 -07:00
command_s3_clean_uploads.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_s3_configure.go
Account Info (#7507)
2025-11-19 20:34:38 -08:00
command_volume_balance_test.go
Shell: support regular expression for collection selection (#7158)
2025-08-23 11:04:24 -07:00
command_volume_balance.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_volume_check_disk_test.go
Mutex command output writes for volume.check.disk. (#7605)
2025-12-02 10:14:24 -08:00
command_volume_check_disk.go
Fix handling of fixed read-only volumes for volume.check.disk. (#7612)
2025-12-03 11:33:35 -08:00
command_volume_configure_replication.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
command_volume_copy.go
[shell] volume copy add param noLock (#6871)
2025-06-16 07:39:19 -07:00
command_volume_delete_empty.go
Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). (#7450)
2025-11-09 19:58:38 -08:00
command_volume_delete.go
refactor
2024-09-29 10:38:22 -07:00
command_volume_fix_replication_test.go
Fix/copy before delete replication (#6064)
2024-09-26 08:34:13 -07:00
command_volume_fix_replication.go
verbose mode
2025-12-26 12:42:00 -08:00
command_volume_fsck.go
volume.fsck: increase default cutoffTimeAgo from 5 minutes to 5 hours (#7730)
2025-12-12 23:42:27 -08:00
command_volume_grow.go
refactor
2024-09-29 10:38:22 -07:00
command_volume_list_test.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_volume_list.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_volume_mark.go
refactor
2024-09-29 10:38:22 -07:00
command_volume_mount.go
refactor
2024-09-29 10:38:22 -07:00
command_volume_move.go
Parallelize volume replica operations within ec.encode. (#6374)
2024-12-18 11:59:48 -08:00
command_volume_replica_check_test.go
fix: sync replica entries before ec.encode and volume.tier.move (#7798)
2025-12-16 23:16:07 -08:00
command_volume_replica_check.go
fix: sync replica entries before ec.encode and volume.tier.move (#7798)
2025-12-16 23:16:07 -08:00
command_volume_server_evacuate_test.go
fix tests
2023-03-13 10:00:27 -07:00
command_volume_server_evacuate.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
command_volume_server_leave.go
Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). (#7450)
2025-11-09 19:58:38 -08:00
command_volume_tier_download.go
Shell: support regular expression for collection selection (#7158)
2025-08-23 11:04:24 -07:00
command_volume_tier_move.go
fix: sync replica entries before ec.encode and volume.tier.move (#7798)
2025-12-16 23:16:07 -08:00
command_volume_tier_upload.go
Shell: support regular expression for collection selection (#7158)
2025-08-23 11:04:24 -07:00
command_volume_unmount.go
refactor
2024-09-29 10:38:22 -07:00
command_volume_vacuum_disable.go
refactor
2024-09-29 10:38:22 -07:00
command_volume_vacuum_enable.go
refactor
2024-09-29 10:38:22 -07:00
command_volume_vacuum.go
refactor
2024-09-29 10:38:22 -07:00
command.go
refactor
2024-09-29 10:38:22 -07:00
commands.go
Fix masterclient vidmap race condition (#7412)
2025-10-30 23:36:06 -07:00
common_test.go
Humanize output for weed.server by default (#7758)
2025-12-15 11:18:45 -08:00
common.go
Parallelize read-only volume check pass for volume.check.disk. (#7602)
2025-12-02 09:29:27 -08:00
ec_proportional_rebalance_test.go
fix: EC rebalance fails with replica placement 000 (#7812)
2025-12-19 13:29:12 -08:00
ec_proportional_rebalance.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
ec_rebalance_slots_test.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
shell_liner.go
Store full shell command in shell history (#7378)
2025-10-24 19:02:44 -07:00
volume.ecshards.txt
Humanize output for weed.server by default (#7758)
2025-12-15 11:18:45 -08:00
volume.list2.txt
Humanize output for weed.server by default (#7758)
2025-12-15 11:18:45 -08:00
volume.list.txt
Humanize output for weed.server by default (#7758)
2025-12-15 11:18:45 -08:00