504b258258
* s3: fix remote object not caching * s3: address review comments for remote object caching - Fix leading slash in object name by using strings.TrimPrefix - Return cached entry from CacheRemoteObjectToLocalCluster to get updated local chunk locations - Reuse existing helper function instead of inline gRPC call * s3/filer: add singleflight deduplication for remote object caching - Add singleflight.Group to FilerServer to deduplicate concurrent cache operations - Wrap CacheRemoteObjectToLocalCluster with singleflight to ensure only one caching operation runs per object when multiple clients request the same file - Add early-return check for already-cached objects - S3 API calls filer gRPC with timeout and graceful fallback on error - Clear negative bucket cache when bucket is created via weed shell - Add integration tests for remote cache with singleflight deduplication This benefits all clients (S3, HTTP, Hadoop) accessing remote-mounted objects by preventing redundant cache operations and improving concurrent access performance. Fixes: https://github.com/seaweedfs/seaweedfs/discussions/7599 * fix: data race in concurrent remote object caching - Add mutex to protect chunks slice from concurrent append - Add mutex to protect fetchAndWriteErr from concurrent read/write - Fix incorrect error check (was checking assignResult.Error instead of parseErr) - Rename inner variable to avoid shadowing fetchAndWriteErr * fix: address code review comments - Remove duplicate remote caching block in GetObjectHandler, keep only singleflight version - Add mutex protection for concurrent chunk slice and error access (data race fix) - Use lazy initialization for S3 client in tests to avoid panic during package load - Fix markdown linting: add language specifier to code fence, blank lines around tables - Add 'all' target to Makefile as alias for test-with-server - Remove unused 'util' import * style: remove emojis from test files * fix: add defensive checks and sort chunks by offset - Add nil check and type assertion check for singleflight result - Sort chunks by offset after concurrent fetching to maintain file order * fix: improve test diagnostics and path normalization - runWeedShell now returns error for better test diagnostics - Add all targets to .PHONY in Makefile (logs-primary, logs-remote, health) - Strip leading slash from normalizedObject to avoid double slashes in path --------- Co-authored-by: chrislu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
133 lines
2.4 KiB
Plaintext
133 lines
2.4 KiB
Plaintext
.goxc*
|
|
vendor
|
|
tags
|
|
*.swp
|
|
### OSX template
|
|
.DS_Store
|
|
.AppleDouble
|
|
.LSOverride
|
|
|
|
# Icon must end with two \r
|
|
Icon
|
|
|
|
# Thumbnails
|
|
._*
|
|
|
|
# Files that might appear in the root of a volume
|
|
.DocumentRevisions-V100
|
|
.fseventsd
|
|
.Spotlight-V100
|
|
.TemporaryItems
|
|
.Trashes
|
|
.VolumeIcon.icns
|
|
|
|
# Directories potentially created on remote AFP share
|
|
.AppleDB
|
|
.AppleDesktop
|
|
Network Trash Folder
|
|
Temporary Items
|
|
.apdisk
|
|
### JetBrains template
|
|
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio
|
|
|
|
*.iml
|
|
|
|
## Directory-based project format:
|
|
.idea/
|
|
# if you remove the above rule, at least ignore the following:
|
|
|
|
# User-specific stuff:
|
|
# .idea/workspace.xml
|
|
# .idea/tasks.xml
|
|
# .idea/dictionaries
|
|
|
|
# Sensitive or high-churn files:
|
|
# .idea/dataSources.ids
|
|
# .idea/dataSources.xml
|
|
# .idea/sqlDataSources.xml
|
|
# .idea/dynamic.xml
|
|
# .idea/uiDesigner.xml
|
|
|
|
# Gradle:
|
|
# .idea/gradle.xml
|
|
# .idea/libraries
|
|
|
|
# Mongo Explorer plugin:
|
|
# .idea/mongoSettings.xml
|
|
|
|
## vscode
|
|
.vscode
|
|
## File-based project format:
|
|
*.ipr
|
|
*.iws
|
|
|
|
## Plugin-specific files:
|
|
|
|
# IntelliJ
|
|
/out/
|
|
|
|
# mpeltonen/sbt-idea plugin
|
|
.idea_modules/
|
|
|
|
# JIRA plugin
|
|
atlassian-ide-plugin.xml
|
|
|
|
# Crashlytics plugin (for Android Studio and IntelliJ)
|
|
com_crashlytics_export_strings.xml
|
|
crashlytics.properties
|
|
crashlytics-build.properties
|
|
|
|
workspace/
|
|
|
|
test_data
|
|
build
|
|
target
|
|
*.class
|
|
other/java/hdfs/dependency-reduced-pom.xml
|
|
|
|
# binary file
|
|
weed/weed
|
|
docker/weed
|
|
|
|
# test generated files
|
|
weed/*/*.jpg
|
|
docker/weed_sub
|
|
docker/weed_pub
|
|
weed/mq/schema/example.parquet
|
|
docker/agent_sub_record
|
|
test/mq/bin/consumer
|
|
test/mq/bin/producer
|
|
test/producer
|
|
bin/weed
|
|
weed_binary
|
|
/test/s3/copying/filerldb2
|
|
/filerldb2
|
|
/test/s3/retention/test-volume-data
|
|
test/s3/cors/weed-test.log
|
|
test/s3/cors/weed-server.pid
|
|
/test/s3/cors/test-volume-data
|
|
test/s3/cors/cors.test
|
|
/test/s3/retention/filerldb2
|
|
test/s3/retention/weed-server.pid
|
|
test/s3/retention/weed-test.log
|
|
/test/s3/versioning/test-volume-data
|
|
test/s3/versioning/weed-test.log
|
|
/docker/admin_integration/data
|
|
docker/agent_pub_record
|
|
docker/admin_integration/weed-local
|
|
/seaweedfs-rdma-sidecar/bin
|
|
/test/s3/encryption/filerldb2
|
|
/test/s3/sse/filerldb2
|
|
test/s3/sse/weed-test.log
|
|
ADVANCED_IAM_DEVELOPMENT_PLAN.md
|
|
/test/s3/iam/test-volume-data
|
|
*.log
|
|
weed-iam
|
|
test/kafka/kafka-client-loadtest/weed-linux-arm64
|
|
/test/tus/filerldb2
|
|
coverage.out
|
|
/test/s3/remote_cache/test-primary-data
|
|
/test/s3/remote_cache/test-remote-data
|
|
test/s3/remote_cache/remote-server.pid
|
|
test/s3/remote_cache/primary-server.pid
|