Lisandro Pin 848d1f7c34 Improve safety for weed shell's ec.encode. (#6773) ...

Improve safety for weed shells `ec.encode`.

The current process for `ec.encode` is:

1. EC shards for a volume are generated and added to a single server
2. The original volume is deleted
3. EC shards get re-balanced across the entire topology

It is then possible to lose data between #2 and #3, if the underlying volume storage/server/rack/DC
happens to fail, for whatever reason. As a fix, this MR reworks `ec.encode` so:

  * Newly created EC shards are spread across all locations for the source volume.
  * Source volumes are deleted only after EC shards are converted and balanced.

2025-05-09 09:01:32 -07:00

..

command_cluster_check.go

[shell] use constant for hdd of type (#6337)

2024-12-10 08:43:59 -08:00

command_cluster_ps.go

refactor

2024-09-29 10:38:22 -07:00

command_cluster_raft_add.go

refactor

2024-09-29 10:38:22 -07:00

command_cluster_raft_ps.go

refactor

2024-09-29 10:38:22 -07:00

command_cluster_raft_remove.go

refactor

2024-09-29 10:38:22 -07:00

command_collection_delete.go

refactor

2024-09-29 10:38:22 -07:00

command_collection_list.go

refactor

2024-09-29 10:38:22 -07:00

command_ec_balance.go

Allow configuring the maximum number of concurrent tasks for EC parallelization. (#6376)

2024-12-18 13:26:26 -08:00

command_ec_common_test.go

ec.encode: Fix resolution of target collections. (#6585)

2025-02-28 11:42:19 -08:00

command_ec_common.go

Improve safety for weed shell's ec.encode. (#6773)

2025-05-09 09:01:32 -07:00

command_ec_decode.go

Unify the re-balancing logic for ec.encode with ec.balance. (#6339)

2024-12-10 13:30:13 -08:00

command_ec_encode.go

Improve safety for weed shell's ec.encode. (#6773)

2025-05-09 09:01:32 -07:00

command_ec_rebuild.go

Unify the re-balancing logic for ec.encode with ec.balance. (#6339)

2024-12-10 13:30:13 -08:00

command_ec_test.go

Delete legacy balancing code for ec.encode. (#6344)

2024-12-12 07:42:03 -08:00

command_fs_cat.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_cd.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_configure.go

chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)

2025-03-31 21:42:54 -07:00

command_fs_du.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_log.go

fix help message

2024-09-29 21:53:07 -07:00

command_fs_ls.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_merge_volumes.go

fix: error info size bug in command_fs_merge_volumes.go (#6567)

2025-02-23 06:36:47 -08:00

command_fs_meta_cat.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_meta_change_volume_id.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_meta_load.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_meta_notify.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_meta_save.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_mkdir.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_mv.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_pwd.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_rm.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_tree.go

refactor

2024-09-29 10:38:22 -07:00

command_fs_verify.go

"golang.org/x/exp/slices" => "slices" and go fmt

2024-12-19 19:25:06 -08:00

command_lock_unlock.go

refactor

2024-09-29 10:38:22 -07:00

command_mount_configure.go

refactor

2024-09-29 10:38:22 -07:00

command_mq_balance.go

refactor

2024-09-29 10:38:22 -07:00

command_mq_topic_compact.go

merge current message queue code changes (#6201)

2024-11-04 12:08:25 -08:00

command_mq_topic_configure.go

Add message queue agent (#6463)

2025-01-20 22:19:27 -08:00

command_mq_topic_desc.go

Add message queue agent (#6463)

2025-01-20 22:19:27 -08:00

command_mq_topic_list.go

refactor

2024-09-29 10:38:22 -07:00

command_remote_cache.go

refactor

2024-09-29 10:38:22 -07:00

command_remote_configure.go

refactor

2024-09-29 10:38:22 -07:00

command_remote_meta_sync.go

refactor

2024-09-29 10:38:22 -07:00

command_remote_mount_buckets.go

refactor

2024-09-29 10:38:22 -07:00

command_remote_mount.go

refactor

2024-09-29 10:38:22 -07:00

command_remote_uncache.go

refactor

2024-09-29 10:38:22 -07:00

command_remote_unmount.go

refactor

2024-09-29 10:38:22 -07:00

command_s3_bucket_create.go

refactor

2024-09-29 10:38:22 -07:00

command_s3_bucket_delete.go

refactor

2024-09-29 10:38:22 -07:00

command_s3_bucket_list.go

refactor

2024-09-29 10:38:22 -07:00

command_s3_bucket_quota_check.go

refactor

2024-09-29 10:38:22 -07:00

command_s3_bucket_quota.go

refactor

2024-09-29 10:38:22 -07:00

command_s3_circuitbreaker_test.go

refactor(shell): readability improvements (#3704)

2022-09-16 02:43:49 -07:00

command_s3_circuitbreaker.go

chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)

2025-03-31 21:42:54 -07:00

command_s3_clean_uploads.go

chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)

2025-03-31 21:42:54 -07:00

command_s3_configure.go

chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)

2025-03-31 21:42:54 -07:00

command_volume_balance_test.go

[shell] only apply the balancing for writable volumes (#6346)

2024-12-13 01:10:00 -08:00

command_volume_balance.go

fix force arg dropped during volume balance command (#6432)

2025-01-12 23:30:18 -08:00

command_volume_check_disk_test.go

fix: skipping checking active volumes with the same number of files at the moment (#4893)

2023-10-09 09:57:26 -07:00

command_volume_check_disk.go

"golang.org/x/exp/slices" => "slices" and go fmt

2024-12-19 19:25:06 -08:00

command_volume_configure_replication.go

Unify usage of shell.EcNode.dc as DataCenterId. (#6258)

2024-11-19 06:33:18 -08:00

command_volume_copy.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_delete_empty.go

Unify usage of shell.EcNode.dc as DataCenterId. (#6258)

2024-11-19 06:33:18 -08:00

command_volume_delete.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_fix_replication_test.go

Fix/copy before delete replication (#6064)

2024-09-26 08:34:13 -07:00

command_volume_fix_replication.go

[weed] change -n to -force (#6421)

2025-01-08 09:57:18 -08:00

command_volume_fsck.go

Unify usage of shell.EcNode.dc as DataCenterId. (#6258)

2024-11-19 06:33:18 -08:00

command_volume_grow.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_list_test.go

Improve EC shards rebalancing logic across racks (#6270)

2024-11-21 08:46:24 -08:00

command_volume_list.go

fix: volume.list volume info output not in order (#6737)

2025-04-27 08:52:49 -07:00

command_volume_mark.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_mount.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_move.go

Parallelize volume replica operations within ec.encode. (#6374)

2024-12-18 11:59:48 -08:00

command_volume_server_evacuate_test.go

fix tests

2023-03-13 10:00:27 -07:00

command_volume_server_evacuate.go

"golang.org/x/exp/slices" => "slices" and go fmt

2024-12-19 19:25:06 -08:00

command_volume_server_leave.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_tier_download.go

Unify usage of shell.EcNode.dc as DataCenterId. (#6258)

2024-11-19 06:33:18 -08:00

command_volume_tier_move.go

Unify usage of shell.EcNode.dc as DataCenterId. (#6258)

2024-11-19 06:33:18 -08:00

command_volume_tier_upload.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_unmount.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_vacuum_disable.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_vacuum_enable.go

refactor

2024-09-29 10:38:22 -07:00

command_volume_vacuum.go

refactor

2024-09-29 10:38:22 -07:00

command.go

refactor

2024-09-29 10:38:22 -07:00

commands.go

[shell] add noLock param for volume.move (#6261)

2024-11-20 08:35:24 -08:00

shell_liner.go

add more help message

2025-03-13 09:11:16 -07:00

volume.ecshards.txt

Improve EC shards rebalancing logic across racks (#6270)

2024-11-21 08:46:24 -08:00

volume.list2.txt

tests add parsing ec shard info

2022-10-09 22:47:55 -07:00

volume.list.txt

Revert "a new test case"

2022-09-10 15:35:42 -07:00