Commit Graph

35 Commits

Author SHA1 Message Date
Chris Lu
995dfc4d5d chore: remove ~50k lines of unreachable dead code (#8913)
* chore: remove unreachable dead code across the codebase

Remove ~50,000 lines of unreachable code identified by static analysis.

Major removals:
- weed/filer/redis_lua: entire unused Redis Lua filer store implementation
- weed/wdclient/net2, resource_pool: unused connection/resource pool packages
- weed/plugin/worker/lifecycle: unused lifecycle plugin worker
- weed/s3api: unused S3 policy templates, presigned URL IAM, streaming copy,
  multipart IAM, key rotation, and various SSE helper functions
- weed/mq/kafka: unused partition mapping, compression, schema, and protocol functions
- weed/mq/offset: unused SQL storage and migration code
- weed/worker: unused registry, task, and monitoring functions
- weed/query: unused SQL engine, parquet scanner, and type functions
- weed/shell: unused EC proportional rebalance functions
- weed/storage/erasure_coding/distribution: unused distribution analysis functions
- Individual unreachable functions removed from 150+ files across admin,
  credential, filer, iam, kms, mount, mq, operation, pb, s3api, server,
  shell, storage, topology, and util packages

* fix(s3): reset shared memory store in IAM test to prevent flaky failure

TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey was flaky because
the MemoryStore credential backend is a singleton registered via init().
Earlier tests that create anonymous identities pollute the shared store,
causing LookupAnonymous() to unexpectedly return true.

Fix by calling Reset() on the memory store before the test runs.

* style: run gofmt on changed files

* fix: restore KMS functions used by integration tests

* fix(plugin): prevent panic on send to closed worker session channel

The Plugin.sendToWorker method could panic with "send on closed channel"
when a worker disconnected while a message was being sent. The race was
between streamSession.close() closing the outgoing channel and sendToWorker
writing to it concurrently.

Add a done channel to streamSession that is closed before the outgoing
channel, and check it in sendToWorker's select to safely detect closed
sessions without panicking.
2026-04-03 16:04:27 -07:00
promalert
9012069bd7 chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code

Signed-off-by: promalert <promalert@outlook.com>

* goimports -w .

---------

Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-07 13:06:08 -08:00
Chris Lu
1261e93ef2 fix: comprehensive go vet error fixes and add CI enforcement (#7861)
* fix: use keyed fields in struct literals

- Replace unsafe reflect.StringHeader/SliceHeader with safe unsafe.String/Slice (weed/query/sqltypes/unsafe.go)
- Add field names to Type_ScalarType struct literals (weed/mq/schema/schema_builder.go)
- Add Duration field name to FlexibleDuration struct literals across test files
- Add field names to bson.D struct literals (weed/filer/mongodb/mongodb_store_kv.go)

Fixes go vet warnings about unkeyed struct literals.

* fix: remove unreachable code

- Remove unreachable return statements after infinite for loops
- Remove unreachable code after if/else blocks where all paths return
- Simplify recursive logic by removing unnecessary for loop (inode_to_path.go)
- Fix Type_ScalarType literal to use enum value directly (schema_builder.go)
- Call onCompletionFn on stream error (subscribe_session.go)

Files fixed:
- weed/query/sqltypes/unsafe.go
- weed/mq/schema/schema_builder.go
- weed/mq/client/sub_client/connect_to_sub_coordinator.go
- weed/filer/redis3/ItemList.go
- weed/mq/client/agent_client/subscribe_session.go
- weed/mq/broker/broker_grpc_pub_balancer.go
- weed/mount/inode_to_path.go
- weed/util/skiplist/name_list.go

* fix: avoid copying lock values in protobuf messages

- Use proto.Merge() instead of direct assignment to avoid copying sync.Mutex in S3ApiConfiguration (iamapi_server.go)
- Add explicit comments noting that channel-received values are already copies before taking addresses (volume_grpc_client_to_master.go)

The protobuf messages contain sync.Mutex fields from the message state, which should not be copied.
Using proto.Merge() properly merges messages without copying the embedded mutex.

* fix: correct byte array size for uint32 bit shift operations

The generateAccountId() function only needs 4 bytes to create a uint32 value.
Changed from allocating 8 bytes to 4 bytes to match the actual usage.

This fixes go vet warning about shifting 8-bit values (bytes) by more than 8 bits.

* fix: ensure context cancellation on all error paths

In broker_client_subscribe.go, ensure subscriberCancel() is called on all error return paths:
- When stream creation fails
- When partition assignment fails
- When sending initialization message fails

This prevents context leaks when an error occurs during subscriber creation.

* fix: ensure subscriberCancel called for CreateFreshSubscriber stream.Send error

Ensure subscriberCancel() is called when stream.Send fails in CreateFreshSubscriber.

* ci: add go vet step to prevent future lint regressions

- Add go vet step to GitHub Actions workflow
- Filter known protobuf lock warnings (MessageState sync.Mutex)
  These are expected in generated protobuf code and are safe
- Prevents accumulation of go vet errors in future PRs
- Step runs before build to catch issues early

* fix: resolve remaining syntax and logic errors in vet fixes

- Fixed syntax errors in filer_sync.go caused by missing closing braces
- Added missing closing brace for if block and function
- Synchronized fixes to match previous commits on branch

* fix: add missing return statements to daemon functions

- Add 'return false' after infinite loops in filer_backup.go and filer_meta_backup.go
- Satisfies declared bool return type signatures
- Maintains consistency with other daemon functions (runMaster, runFilerSynchronize, runWorker)
- While unreachable, explicitly declares the return satisfies function signature contract

* fix: add nil check for onCompletionFn in SubscribeMessageRecord

- Check if onCompletionFn is not nil before calling it
- Prevents potential panic if nil function is passed
- Matches pattern used in other callback functions

* docs: clarify unreachable return statements in daemon functions

- Add comments documenting that return statements satisfy function signature
- Explains that these returns follow infinite loops and are unreachable
- Improves code clarity for future maintainers
2025-12-23 14:48:50 -08:00
tam-i13
b669607fcd Add error list each entry func (#7485)
* added error return in type ListEachEntryFunc

* return error if errClose

* fix fmt.Errorf

* fix return errClose

* use %w fmt.Errorf

* added entry in messege error

* add callbackErr in ListDirectoryEntries

* fix error

* add log

* clear err when the scanner stops on io.EOF, so returning err doesn’t surface EOF as a failure.

* more info in error

* add ctx to logs, error handling

* fix return eachEntryFunc

* fix

* fix log

* fix return

* fix foundationdb test s

* fix eachEntryFunc

* fix return resEachEntryFuncErr

* Update weed/filer/filer.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/filer/elastic/v7/elastic_store.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/filer/hbase/hbase_store.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/filer/foundationdb/foundationdb_store.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/filer/ydb/ydb_store.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* fix

* add scanErr

---------

Co-authored-by: Roman Tamarov <r.tamarov@kryptonite.ru>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-11-25 19:35:19 -08:00
Chris Lu
69553e5ba6 convert error fromating to %w everywhere (#6995) 2025-07-16 23:39:27 -07:00
Aleksey Kosov
4511c2cc1f Changes logging function (#6919)
* updated logging methods for stores

* updated logging methods for stores

* updated logging methods for filer

* updated logging methods for uploader and http_util

* updated logging methods for weed server

---------

Co-authored-by: akosov <a.kosov@kryptonite.ru>
2025-06-24 08:44:06 -07:00
Jade Devin Cabatlao
21514e1fea feat(redis): add mTLS support for Redis connection initialization (#6738)
* feat(redis): add mTLS support for Redis connection initialization

- Enhanced the Redis2Store initialization to support mutual TLS (mTLS) by adding configuration options for CA certificate, client certificate, and client key paths.
- Updated the Redis client setup to use TLS configuration when mTLS is enabled, ensuring secure connections to the Redis server.

* feat(redis): extend Redis3Store initialization to support mTLS

- Added configuration options for enabling mutual TLS (mTLS) in Redis3Store.
- Implemented logic to load client certificates and CA certificates for secure Redis connections.
- Updated the Redis client setup to utilize TLS configuration when mTLS is enabled.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2025-05-04 05:31:27 -07:00
chrislu
9bd4c47299 update redis support status 2025-02-26 17:48:44 -08:00
chrislu
ec155022e7 "golang.org/x/exp/slices" => "slices" and go fmt 2024-12-19 19:25:06 -08:00
chrislu
2addcd1623 fix test 2024-07-11 07:53:02 -07:00
chrislu
c59517e284 upgrade to github.com/redis/go-redis/v9
fix https://github.com/seaweedfs/seaweedfs/issues/4532
2024-07-11 00:15:19 -07:00
chrislu
3fbf4f6189 Revert "Initial patch for redis version 7 support (#4572)"
This reverts commit a25bca0692.
2023-06-12 22:27:38 -07:00
Aaron Gipson
a25bca0692 Initial patch for redis version 7 support (#4572) 2023-06-12 13:30:03 -07:00
chrislu
70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu
f5d4952d73 filer: redis store reduce from 2 redis operations to 1 for updates. 2022-10-12 23:50:09 -07:00
chrislu
21c0587900 go fmt 2022-09-14 23:06:44 -07:00
Ryan Russell
f2dde2b1da refactor(ItemList): Incluseive -> Inclusive (#3673)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:00:16 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev
21033ff4c3 refactor use const CountEntryChunksForGzip 2022-05-01 22:28:55 +05:00
Chris Lu
689f5513a9 redis3 supports sentinel 2021-11-29 01:09:51 -08:00
Chris Lu
f4676824a7 fix test code compilation 2021-10-10 22:14:13 -07:00
Chris Lu
8205166009 disable testing with redis server
--- FAIL: TestNameList (0.00s)
panic: exec: "redis-server": executable file not found in $PATH [recovered]
	panic: exec: "redis-server": executable file not found in $PATH

goroutine 37 [running]:
testing.tRunner.func1.2({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1209 +0x24e
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1212 +0x218
panic({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/panic.go:1038 +0x215
github.com/chrislusf/seaweedfs/weed/filer/redis3.TestNameList(0x407c59)
	/home/runner/work/seaweedfs/seaweedfs/weed/filer/redis3/kv_directory_children_test.go:45 +0x376
testing.tRunner(0xc0003e2680, 0xf57f48)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x35a
2021-10-10 21:57:55 -07:00
Chris Lu
17e08a16f4 fix redis3 deletion 2021-10-09 04:54:48 -07:00
Chris Lu
e4830bd93d go fmt 2021-10-07 21:13:31 -07:00
Chris Lu
f3d8232e14 reduce one redis lookup on hot path 2021-10-06 22:01:19 -07:00
Chris Lu
2336a397dc use pipeline to save some time 2021-10-06 20:40:18 -07:00
Chris Lu
371fead8a5 redis3 using redis native sorted set 2021-10-06 18:18:24 -07:00
Chris Lu
8668d49c9d test with real redis
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkNameList-12    	    1789	    760599 ns/op
BenchmarkRedis-12       	   17539	     64122 ns/op
PASS
2021-10-06 01:25:37 -07:00
Chris Lu
6b31f3c97a add benchmark test 2021-10-06 00:37:57 -07:00
Chris Lu
893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu
947add39e6 clean up *SkipListElementReference loaded from Redis 2021-10-04 02:31:38 -07:00
Chris Lu
2b9aab3442 use 1000 per batch 2021-10-04 01:03:40 -07:00
Chris Lu
366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu
8f2e4be074 wip 2021-09-18 14:04:30 -07:00