Commit Graph

37 Commits

Author SHA1 Message Date
Peter Dodd
0910252e31 feat: add statfile remote storage (#8443)
* feat: add statfile; add error for remote storage misses

* feat: statfile implementations for storage providers

* test: add unit tests for StatFile method across providers

Add comprehensive unit tests for the StatFile implementation covering:
- S3: interface compliance and error constant accessibility
- Azure: interface compliance, error constants, and field population
- GCS: interface compliance, error constants, error detection, and field population

Also fix variable shadowing issue in S3 and Azure StatFile implementations where
named return parameters were being shadowed by local variable declarations.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: address StatFile review feedback

- Use errors.New for ErrRemoteObjectNotFound sentinel
- Fix S3 HeadObject 404 detection to use awserr.Error code check
- Remove hollow field-population tests that tested nothing
- Remove redundant stdlib error detection tests
- Trim verbose doc comment on ErrRemoteObjectNotFound

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: address second round of StatFile review feedback

- Rename interface assertion tests to TestXxxRemoteStorageClientImplementsInterface
- Delegate readFileRemoteEntry to StatFile in all three providers
- Revert S3 404 detection to RequestFailure.StatusCode() check
- Fix double-slash in GCS error message format string
- Add storage type prefix to S3 error message for consistency

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: comments

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-25 10:24:06 -08:00
Chris Lu
ccf35459be Explicitly disable signing for public buckets. (#8263) 2026-02-09 11:28:07 -08:00
promalert
9012069bd7 chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code

Signed-off-by: promalert <promalert@outlook.com>

* goimports -w .

---------

Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-07 13:06:08 -08:00
Chris Lu
69553e5ba6 convert error fromating to %w everywhere (#6995) 2025-07-16 23:39:27 -07:00
chrislu
bd4891a117 change version directory 2025-06-03 22:46:10 -07:00
mervynzhang
df400e6c71 Concurrency works better (#4663)
Co-authored-by: mervyn.zhang <mervyn.zhang@sap.com>
2023-07-12 23:04:54 -07:00
mervynzhang
1ebb549f77 support swift (#4480) 2023-05-19 06:39:25 -07:00
chrislu
9920d65bc0 gateway to remote object store: adjust upload concurrency 2022-08-26 23:47:37 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
61b8c9c361 remote object store gateway: disable tagging for backblaze 2022-06-11 09:50:59 -07:00
Chris Lu
04663c3611 remote.mount: print out metadata sync errors 2021-11-06 11:29:50 -07:00
Chris Lu
2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu
53b9b521c9 adjust error message 2021-09-04 13:46:06 -07:00
Chris Lu
0652805236 cloud drive: add createBucket() deleteBucket() 2021-09-03 22:30:55 -07:00
Chris Lu
83cd0fc739 cloud drive: add list buckets 2021-09-03 20:42:02 -07:00
Chris Lu
bca4a9de78 simplify 2021-09-02 23:09:24 -07:00
Chris Lu
958125bd02 conforming to http user agent common practice 2021-09-02 22:55:35 -07:00
Chris Lu
701cc133f3 cloud drive: s3 add option for v4 signature 2021-08-30 17:28:33 -07:00
Chris Lu
a31f2907f0 cloud drive: filer.remote.sync supports remove folder 2021-08-29 18:46:28 -07:00
Chris Lu
001a472057 cloud mount: remote storage support hdfs 2021-08-29 18:41:29 -07:00
Chris Lu
05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
Chris Lu
c08ac536ed cloud drive: add support for Wasabi
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
2021-08-25 17:34:29 -07:00
Chris Lu
00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu
f0cc130849 do not force path style for better compatibility 2021-08-23 03:09:41 -07:00
Chris Lu
2836a58d87 cloud drive: S3 supports storage class 2021-08-23 02:18:59 -07:00
Chris Lu
c45c12aedc metadata updates 2021-08-15 19:27:30 -07:00
Chris Lu
507411bdf0 fix https://github.com/chrislusf/seaweedfs/issues/2260
fix https://github.com/chrislusf/seaweedfs/issues/2260
2021-08-15 19:16:00 -07:00
Chris Lu
9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu
713c035a6e shell: remote.cache remote.uncache 2021-08-09 14:35:18 -07:00
Chris Lu
c5f38c365d go fmt 2021-08-08 22:30:36 -07:00
Chris Lu
7412ccdf88 write back remote entry to local entry after uploading to remote 2021-08-08 17:55:03 -07:00
Chris Lu
13e45e1605 filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
Chris Lu
9df7d16791 read <- remote_storage 2021-07-31 22:39:38 -07:00
Chris Lu
899963ac20 remote storage location changed to struct 2021-07-29 02:08:55 -07:00
Chris Lu
035b0bae29 refactor 2021-07-27 03:32:24 -07:00
Chris Lu
1752eeb538 remote.mount saves the mapping 2021-07-27 03:26:35 -07:00
Chris Lu
99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00