Commit Graph

27 Commits

Author SHA1 Message Date
Chris Lu
935f41bff6 filer.backup: ignore missing volume/lookup errors when -ignore404Error is set (#7889)
* filer.backup: ignore missing volume/lookup errors when -ignore404Error is set (#7888)

* simplify
2025-12-26 15:44:30 -08:00
Chris Lu
1261e93ef2 fix: comprehensive go vet error fixes and add CI enforcement (#7861)
* fix: use keyed fields in struct literals

- Replace unsafe reflect.StringHeader/SliceHeader with safe unsafe.String/Slice (weed/query/sqltypes/unsafe.go)
- Add field names to Type_ScalarType struct literals (weed/mq/schema/schema_builder.go)
- Add Duration field name to FlexibleDuration struct literals across test files
- Add field names to bson.D struct literals (weed/filer/mongodb/mongodb_store_kv.go)

Fixes go vet warnings about unkeyed struct literals.

* fix: remove unreachable code

- Remove unreachable return statements after infinite for loops
- Remove unreachable code after if/else blocks where all paths return
- Simplify recursive logic by removing unnecessary for loop (inode_to_path.go)
- Fix Type_ScalarType literal to use enum value directly (schema_builder.go)
- Call onCompletionFn on stream error (subscribe_session.go)

Files fixed:
- weed/query/sqltypes/unsafe.go
- weed/mq/schema/schema_builder.go
- weed/mq/client/sub_client/connect_to_sub_coordinator.go
- weed/filer/redis3/ItemList.go
- weed/mq/client/agent_client/subscribe_session.go
- weed/mq/broker/broker_grpc_pub_balancer.go
- weed/mount/inode_to_path.go
- weed/util/skiplist/name_list.go

* fix: avoid copying lock values in protobuf messages

- Use proto.Merge() instead of direct assignment to avoid copying sync.Mutex in S3ApiConfiguration (iamapi_server.go)
- Add explicit comments noting that channel-received values are already copies before taking addresses (volume_grpc_client_to_master.go)

The protobuf messages contain sync.Mutex fields from the message state, which should not be copied.
Using proto.Merge() properly merges messages without copying the embedded mutex.

* fix: correct byte array size for uint32 bit shift operations

The generateAccountId() function only needs 4 bytes to create a uint32 value.
Changed from allocating 8 bytes to 4 bytes to match the actual usage.

This fixes go vet warning about shifting 8-bit values (bytes) by more than 8 bits.

* fix: ensure context cancellation on all error paths

In broker_client_subscribe.go, ensure subscriberCancel() is called on all error return paths:
- When stream creation fails
- When partition assignment fails
- When sending initialization message fails

This prevents context leaks when an error occurs during subscriber creation.

* fix: ensure subscriberCancel called for CreateFreshSubscriber stream.Send error

Ensure subscriberCancel() is called when stream.Send fails in CreateFreshSubscriber.

* ci: add go vet step to prevent future lint regressions

- Add go vet step to GitHub Actions workflow
- Filter known protobuf lock warnings (MessageState sync.Mutex)
  These are expected in generated protobuf code and are safe
- Prevents accumulation of go vet errors in future PRs
- Step runs before build to catch issues early

* fix: resolve remaining syntax and logic errors in vet fixes

- Fixed syntax errors in filer_sync.go caused by missing closing braces
- Added missing closing brace for if block and function
- Synchronized fixes to match previous commits on branch

* fix: add missing return statements to daemon functions

- Add 'return false' after infinite loops in filer_backup.go and filer_meta_backup.go
- Satisfies declared bool return type signatures
- Maintains consistency with other daemon functions (runMaster, runFilerSynchronize, runWorker)
- While unreachable, explicitly declares the return satisfies function signature contract

* fix: add nil check for onCompletionFn in SubscribeMessageRecord

- Check if onCompletionFn is not nil before calling it
- Prevents potential panic if nil function is passed
- Matches pattern used in other callback functions

* docs: clarify unreachable return statements in daemon functions

- Add comments documenting that return statements satisfy function signature
- Explains that these returns follow infinite loops and are unreachable
- Improves code clarity for future maintainers
2025-12-23 14:48:50 -08:00
Max Denushev
a5fe6e21bc feat(filer.backup): add ignore errors option (#6235)
* feat(filer.backup): add ignore errors option

* feat(filer.backup): fix 404 error wrap

* feat(filer.backup): fix wrapping function

* feat(filer.backup): fix wrapping errors in genProcessFunction

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

---------

Co-authored-by: Max Denushev <denushev@tochka.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-11-14 08:40:55 -08:00
vadimartynov
b796c21fa9 Added loadSecurityConfigOnce (#5792) 2024-07-16 09:15:55 -07:00
chrislu
3a82f5ffad ensure metadata follow a specific folder
fix https://github.com/seaweedfs/seaweedfs/issues/5774
2024-07-12 11:17:30 -07:00
chrislu
a829f36d17 adjust error handling
fix https://github.com/seaweedfs/seaweedfs/issues/5677
2024-06-17 10:59:41 -07:00
Konstantin Lebedev
170b63d6f8 [filer.backup] add param uploader_part_size for S3sink (#5352)
* fix: install cronie

* chore: refactor configure S3Sink

* chore: refactor cinfig

* add filer-backup compose file

* fix: X-Amz-Meta-Mtime and resolve with comments

* fix: attr mtime

* fix: MaxUploadPartst is reduced to the maximum allowable

* fix: env and force set max MaxUploadParts

* fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB
2024-03-07 08:35:51 -08:00
Konstantin Lebedev
2b229e98ce fix: add doDeleteFile option for filer backup 2023-11-17 07:37:28 -08:00
Konstantin Lebedev
e55fa68848 fix: excludeFileName is empty 2023-11-13 06:23:46 -08:00
Konstantin Lebedev
3c5295a1a6 filer backup add option for exclude file names that match the regexp to sync on filer 2023-11-13 06:23:46 -08:00
chrislu
5db9fcccd4 refactoring 2023-03-21 23:01:49 -07:00
zemul
cec55feeb0 [filer.backup] add retention_duration (#3941) 2022-11-04 00:34:40 -07:00
chrislu
67814a5c79 refactor and fix strings.Split 2022-08-07 01:34:32 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev
7e09a548a6 exclude directories to sync on filer 2022-07-27 19:22:57 +05:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu
5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
8e125339d5 line wrap 2021-08-31 23:30:28 -07:00
Chris Lu
68ea99190f add additionalPathPrefixes 2021-08-31 23:26:26 -07:00
Chris Lu
c5f38c365d go fmt 2021-08-08 22:30:36 -07:00
Chris Lu
6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu
3afbf04007 add TLS grpc support for filer meta clients 2021-08-03 01:30:35 -07:00
Chris Lu
540441fd38 go fmt 2021-02-28 20:34:14 -08:00
Chris Lu
e52c94640e filer.backup: added to replace filer.replicate 2021-02-28 16:22:27 -08:00