gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
288ba5fec8364ccf4e1d3d9323b9e12828458e21
seaweedFS/weed/storage
History
Lisandro Pin 6b98b52acc Fix reporting of EC shard sizes from nodes to masters. (#7835) 
SeaweedFS tracks EC shard sizes on topology data stuctures, but this information is never
relayed to master servers :( The end result is that commands reporting disk usage, such
as `volume.list` and `cluster.status`, yield incorrect figures when EC shards are present.

As an example for a simple 5-node test cluster, before...

```
> volume.list
Topology volumeSizeLimit:30000 MB hdd(volume:6/40 active:6 free:33 remote:0)
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9001 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:3  size:88967096  file_count:172  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[1 5]
        Disk hdd total size:88967096 file_count:172
      DataNode 192.168.10.111:9001 total size:88967096 file_count:172
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9002 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:77267536  file_count:166  replica_placement:2  version:3  modified_at_second:1766349617
          volume id:3  size:88967096  file_count:172  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[0 4]
        Disk hdd total size:166234632 file_count:338
      DataNode 192.168.10.111:9002 total size:166234632 file_count:338
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9003 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:2  size:77267536  file_count:166  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[2 6]
        Disk hdd total size:77267536 file_count:166
      DataNode 192.168.10.111:9003 total size:77267536 file_count:166
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9004 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:77267536  file_count:166  replica_placement:2  version:3  modified_at_second:1766349617
          volume id:3  size:88967096  file_count:172  replica_placement:2  version:3  modified_at_second:1766349617
          ec volume id:1 collection: shards:[3 7]
        Disk hdd total size:166234632 file_count:338
      DataNode 192.168.10.111:9004 total size:166234632 file_count:338
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9005 hdd(volume:0/8 active:0 free:8 remote:0)
        Disk hdd(volume:0/8 active:0 free:8 remote:0) id:0
          ec volume id:1 collection: shards:[8 9 10 11 12 13]
        Disk hdd total size:0 file_count:0
    Rack DefaultRack total size:498703896 file_count:1014
  DataCenter DefaultDataCenter total size:498703896 file_count:1014
total size:498703896 file_count:1014
```

...and after:

```
> volume.list
Topology volumeSizeLimit:30000 MB hdd(volume:6/40 active:6 free:33 remote:0)
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9001 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:2  size:81761800  file_count:161  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[1 5 9] sizes:[1:8.00 MiB 5:8.00 MiB 9:8.00 MiB] total:24.00 MiB
        Disk hdd total size:81761800 file_count:161
      DataNode 192.168.10.111:9001 total size:81761800 file_count:161
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9002 hdd(volume:1/8 active:1 free:7 remote:0)
        Disk hdd(volume:1/8 active:1 free:7 remote:0) id:0
          volume id:3  size:88678712  file_count:170  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[11 12 13] sizes:[11:8.00 MiB 12:8.00 MiB 13:8.00 MiB] total:24.00 MiB
        Disk hdd total size:88678712 file_count:170
      DataNode 192.168.10.111:9002 total size:88678712 file_count:170
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9003 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:81761800  file_count:161  replica_placement:2  version:3  modified_at_second:1766349495
          volume id:3  size:88678712  file_count:170  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[0 4 8] sizes:[0:8.00 MiB 4:8.00 MiB 8:8.00 MiB] total:24.00 MiB
        Disk hdd total size:170440512 file_count:331
      DataNode 192.168.10.111:9003 total size:170440512 file_count:331
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9004 hdd(volume:2/8 active:2 free:6 remote:0)
        Disk hdd(volume:2/8 active:2 free:6 remote:0) id:0
          volume id:2  size:81761800  file_count:161  replica_placement:2  version:3  modified_at_second:1766349495
          volume id:3  size:88678712  file_count:170  replica_placement:2  version:3  modified_at_second:1766349495
          ec volume id:1 collection: shards:[2 6 10] sizes:[2:8.00 MiB 6:8.00 MiB 10:8.00 MiB] total:24.00 MiB
        Disk hdd total size:170440512 file_count:331
      DataNode 192.168.10.111:9004 total size:170440512 file_count:331
  DataCenter DefaultDataCenter hdd(volume:6/40 active:6 free:33 remote:0)
    Rack DefaultRack hdd(volume:6/40 active:6 free:33 remote:0)
      DataNode 192.168.10.111:9005 hdd(volume:0/8 active:0 free:8 remote:0)
        Disk hdd(volume:0/8 active:0 free:8 remote:0) id:0
          ec volume id:1 collection: shards:[3 7] sizes:[3:8.00 MiB 7:8.00 MiB] total:16.00 MiB
        Disk hdd total size:0 file_count:0
    Rack DefaultRack total size:511321536 file_count:993
  DataCenter DefaultDataCenter total size:511321536 file_count:993
total size:511321536 file_count:993
```
2025-12-28 19:30:42 -08:00
..
backend
Refactor: Replace removeDuplicateSlashes with NormalizeObjectKey (#7873)
2025-12-24 19:07:08 -08:00
erasure_coding
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
idx
chore: make function comment match function name (#6607)
2025-03-07 00:45:47 -08:00
needle
fix: return error on size mismatch in ReadNeedleMeta for consistency (#7687)
2025-12-09 13:17:18 -08:00
needle_map
Fix uncleanable size=0 orphans with volume.fsck -forcePurging (#7783)
2025-12-15 21:39:27 -08:00
super_block
refactoring
2025-06-16 22:25:22 -07:00
types
Fix uncleanable size=0 orphans with volume.fsck -forcePurging (#7783)
2025-12-15 21:39:27 -08:00
volume_info
fix: prevent empty .vif files from ec.decode causing parse errors (#7686)
2025-12-09 12:34:39 -08:00
disk_location_ec_realworld_test.go
Volume Server: handle incomplete ec encoding (#7384)
2025-10-26 22:48:58 -07:00
disk_location_ec_shard_size_test.go
Volume Server: handle incomplete ec encoding (#7384)
2025-10-26 22:48:58 -07:00
disk_location_ec_test.go
Volume Server: handle incomplete ec encoding (#7384)
2025-10-26 22:48:58 -07:00
disk_location_ec.go
Erasure Coding: Ec refactoring (#7396)
2025-10-27 22:13:31 -07:00
disk_location_test.go
[shell] use constant for hdd of type (#6337)
2024-12-10 08:43:59 -08:00
disk_location.go
Add 'weed mini' command for S3 beginners and small/dev use cases (#7831)
2025-12-21 11:10:01 -08:00
idx_binary_search_test.go
pass volume version when creating a volume
2025-06-19 01:15:25 -07:00
needle_map_leveldb.go
Fix uncleanable size=0 orphans with volume.fsck -forcePurging (#7783)
2025-12-15 21:39:27 -08:00
needle_map_memory.go
Fix uncleanable size=0 orphans with volume.fsck -forcePurging (#7783)
2025-12-15 21:39:27 -08:00
needle_map_metric_test.go
fix tests
2023-03-08 01:48:28 -08:00
needle_map_metric.go
ReadAt may return io.EOF t end of file
2024-11-21 00:37:38 -08:00
needle_map_sorted_file.go
Fix sorted needle file's write mode for (*SortedFileNeedleMap).Delete() (#5633)
2024-06-02 14:12:25 -07:00
needle_map.go
Lazy loading (#3958)
2022-11-14 00:19:27 -08:00
store_disk_space_test.go
Volume Server: avoid aggressive volume assignment (#7501)
2025-11-17 21:19:55 -08:00
store_ec_delete.go
fix: skip cookie validation for EC volume deletion when SkipCookieCheck is set (#7608)
2025-12-02 17:00:05 -08:00
store_ec.go
Fix reporting of EC shard sizes from nodes to masters. (#7835)
2025-12-28 19:30:42 -08:00
store_load_balancing_simple_test.go
create new volumes on less occupied disk locations (#7349)
2025-10-20 16:11:29 -07:00
store_load_balancing_test.go
Support separate volume server ID independent of RPC bind address (#7609)
2025-12-02 22:08:11 -08:00
store_vacuum_test.go
fix tests
2025-06-30 13:57:28 -07:00
store_vacuum.go
fix disk space calculation
2025-06-30 10:11:30 -07:00
store.go
fix: volume server healthz now checks local conditions only (#7610)
2025-12-02 23:19:14 -08:00
volume_backup_test.go
refactoring
2019-04-17 22:04:49 -07:00
volume_backup.go
add version to volume proto
2025-06-16 22:05:06 -07:00
volume_checking.go
convert error fromating to %w everywhere (#6995)
2025-07-16 23:39:27 -07:00
volume_info_test.go
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
volume_info.go
Admin: misc improvements on admin server and workers. EC now works. (#7055)
2025-07-30 12:38:03 -07:00
volume_loading.go
Add 'weed mini' command for S3 beginners and small/dev use cases (#7831)
2025-12-21 11:10:01 -08:00
volume_read_all.go
Include name/mime in ReadAllNeedles (#4005)
2022-11-23 15:59:38 -08:00
volume_read_test.go
pass volume version when creating a volume
2025-06-19 01:15:25 -07:00
volume_read.go
S3: Perf related (#7463)
2025-11-10 20:30:21 -08:00
volume_super_block.go
pass volume version when creating a volume
2025-06-19 01:15:25 -07:00
volume_tier.go
refactoring
2025-06-16 22:25:22 -07:00
volume_vacuum_test.go
fix: add missing backslash for volume extraArgs in helm chart (#7676)
2025-12-08 23:21:02 -08:00
volume_vacuum.go
[master] vaccum fix warn (#7312)
2025-10-28 17:57:51 -07:00
volume_write_test.go
pass volume version when creating a volume
2025-06-19 01:15:25 -07:00
volume_write.go
Fix uncleanable orphans issue with volume.fsck -forcePurging (#7332)
2025-10-16 12:21:51 -07:00
volume.go
Admin: misc improvements on admin server and workers. EC now works. (#7055)
2025-07-30 12:38:03 -07:00