Commit Graph

38 Commits

Author SHA1 Message Date
chrislu
5167bbd2a9 Remove deprecated allowEmptyFolder CLI option
The allowEmptyFolder option is no longer functional because:
1. The code that used it was already commented out
2. Empty folder cleanup is now handled asynchronously by EmptyFolderCleaner

The CLI flags are kept for backward compatibility but marked as deprecated
and ignored. This removes:
- S3ApiServerOption.AllowEmptyFolder field
- The actual usage in s3api_object_handlers_list.go
- Helm chart values and template references
- References in test Makefiles and docker-compose files
2025-12-06 21:54:12 -08:00
Chris Lu
12f50d37fa test versioning also (#7000)
* test versioning also

* fix some versioning tests

* fall back

* fixes

Never-versioned buckets: No VersionId headers, no Status field
Pre-versioning objects: Regular files, VersionId="null", included in all operations
Post-versioning objects: Stored in .versions directories with real version IDs
Suspended versioning: Proper status handling and null version IDs

* fixes

Bucket Versioning Status Compliance
Fixed: New buckets now return no Status field (AWS S3 compliant)
Before: Always returned "Suspended" 
After: Returns empty VersioningConfiguration for unconfigured buckets 
2. Multi-Object Delete Versioning Support
Fixed: DeleteMultipleObjectsHandler now fully versioning-aware
Before: Always deleted physical files, breaking versioning 
After: Creates delete markers or deletes specific versions properly 
Added: DeleteMarker field in response structure for AWS compatibility
3. Copy Operations Versioning Support
Fixed: CopyObjectHandler and CopyObjectPartHandler now versioning-aware
Before: Only copied regular files, couldn't handle versioned sources 
After: Parses version IDs from copy source, creates versions in destination 
Added: pathToBucketObjectAndVersion() function for version ID parsing
4. Pre-versioning Object Handling
Fixed: getLatestObjectVersion() now has proper fallback logic
Before: Failed when .versions directory didn't exist 
After: Falls back to regular objects for pre-versioning scenarios 
5. Enhanced Object Version Listings
Fixed: listObjectVersions() includes both versioned AND pre-versioning objects
Before: Only showed .versions directories, ignored pre-versioning objects 
After: Shows complete version history with VersionId="null" for pre-versioning 
6. Null Version ID Handling
Fixed: getSpecificObjectVersion() properly handles versionId="null"
Before: Couldn't retrieve pre-versioning objects by version ID 
After: Returns regular object files for "null" version requests 
7. Version ID Response Headers
Fixed: PUT operations only return x-amz-version-id when appropriate
Before: Returned version IDs for non-versioned buckets 
After: Only returns version IDs for explicitly configured versioning 

* more fixes

* fix copying with versioning, multipart upload

* more fixes

* reduce volume size for easier dev test

* fix

* fix version id

* fix versioning

* Update filer_multipart.go

* fix multipart versioned upload

* more fixes

* more fixes

* fix versioning on suspended

* fixes

* fixing test_versioning_obj_suspended_copy

* Update s3api_object_versioning.go

* fix versions

* skipping test_versioning_obj_suspend_versions

* > If the versioning state has never been set on a bucket, it has no versioning state; a GetBucketVersioning request does not return a versioning state value.

* fix tests, avoid duplicated bucket creation, skip tests

* only run s3tests_boto3/functional/test_s3.py

* fix checking filer_pb.ErrNotFound

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_handlers_copy.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_bucket_config.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update test/s3/versioning/s3_versioning_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-19 21:43:34 -07:00
Chris Lu
d892538d32 More efficient copy object (#6665)
* it compiles

* refactored

* reduce to 4 concurrent chunk upload

* CopyObjectPartHandler

* copy a range of the chunk data, fix offset size in copied chunks

* Update s3api_object_handlers_copy.go

What the PR Accomplishes:
CopyObjectHandler - Now copies entire objects by copying chunks individually instead of downloading/uploading the entire file
CopyObjectPartHandler - Handles copying parts of objects for multipart uploads by copying only the relevant chunk portions
Efficient Chunk Copying - Uses direct chunk-to-chunk copying with proper volume assignment and concurrent processing (limited to 4 concurrent operations)
Range Support - Properly handles range-based copying for partial object copies

* fix compilation

* fix part destination

* handling small objects

* use mkFile

* copy to existing file or part

* add testing tools

* adjust tests

* fix chunk lookup

* refactoring

* fix TestObjectCopyRetainingMetadata

* ensure bucket name not conflicting

* fix conditional copying tests

* remove debug messages

* add custom s3 copy tests
2025-07-11 18:51:32 -07:00
Chris Lu
1defee3d68 Add admin component (#6928)
* init version

* relocate

* add s3 bucket link

* refactor handlers into weed/admin folder

* fix login logout

* adding favicon

* remove fall back to http get topology

* grpc dial option, disk total capacity

* show filer count

* fix each volume disk usage

* add filers to dashboard

* adding hosts, volumes, collections

* refactor code and menu

* remove "refresh" button

* fix data for collections

* rename cluster hosts into volume servers

* add masters, filers

* reorder

* adding file browser

* create folder and upload files

* add filer version, created at time

* remove mock data

* remove fields

* fix submenu item highlighting

* fix bucket creation

* purge files

* delete multiple

* fix bucket creation

* remove region from buckets

* add object store with buckets and users

* rendering permission

* refactor

* get bucket objects and size

* link to file browser

* add file size and count for collections page

* paginate the volumes

* fix possible SSRF

https://github.com/seaweedfs/seaweedfs/pull/6928/checks?check_run_id=45108469801

* Update weed/command/admin.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/command/admin.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix build

* import

* remove filer CLI option

* remove filer option

* remove CLI options

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-01 01:28:09 -07:00
MaratKarimov
ba3afd1803 Tarantool filer store (#6669)
Co-authored-by: Marat Karimov <m.karimov@digitalms.ru>
2025-03-29 21:12:06 -07:00
Konstantin Lebedev
5bddf0c085 [shell] volume.balance collect volume servers by dc rack node (#6191)
* chore: balance by rack

* fix: rm check lock

* fix: selected racks

* fix: selected nodes

* fix: containts

* fix: one collectVolumeServersByDcRackNode

* fix: revert lock and add lock

* fix: panic test

* revert noLock
2024-11-03 11:08:45 -08:00
Konstantin Lebedev
33537ae29f [s3] fix s3 test_multipart_get_part (#5476)
* try fix s3  test_multipart_get_part

* add passed s3 tests

* fix SeaweedFSUploadId

* rm spaces

* convert part request to range

* add passed s3 tests of multipart
2024-04-14 10:41:32 -07:00
Konstantin Lebedev
35cba720a5 [s3] add s3 pass test_multipart_upload (#5474)
add s3 pass test_multipart_upload
2024-04-07 11:51:22 -07:00
Konstantin Lebedev
a7fc723ae0 chore: add status code for request_total metrics (#5188) 2024-01-10 10:05:27 -08:00
chrislu
7a50762b9e reduce binary size by skipping rclone 2023-11-23 00:48:27 -08:00
Konstantin Lebedev
6175102ccf add make benchmark with WARP (#4914)
add make benchmark

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-15 23:02:27 -07:00
Konstantin Lebedev
f8b94cac0e [s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859)
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface

https://github.com/seaweedfs/seaweedfs/issues/4519

* fix: test bucket acl default and
adjust the variable names

* fix: s3 api config test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-25 08:34:12 -07:00
Konstantin Lebedev
7e09a548a6 exclude directories to sync on filer 2022-07-27 19:22:57 +05:00
yulai.li
824cbe32be Make tikv filer enable/disable by build tags 2022-06-27 10:57:24 +08:00
Konstantin Lebedev
5b388ed6c1 initial advancedtls 2022-06-23 23:32:15 +05:00
chrislu
1d0c53ea56 remote storage: stop supporting hdfs as a remote storage 2022-06-20 14:15:59 -07:00
Konstantin Lebedev
f58adaab25 build tag ydb since the growth of the binary is 5 mb 2022-05-03 23:20:09 +05:00
chrislu
ad01c63b84 conditionally skip hdfs related code 2022-04-21 01:43:01 -07:00
chrislu
4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
Chris Lu
93741a6752 remove unused travis related files
fix https://github.com/chrislusf/seaweedfs/issues/2277
2021-08-23 13:33:33 -07:00
Konstantin Lebedev
1f301e4ba4 gccgo build 2021-08-17 20:27:47 +05:00
Chris Lu
6e8c2bfdd0 fix README and makefile
fix https://github.com/chrislusf/seaweedfs/issues/1548
2020-10-20 18:24:10 -07:00
mwpeterson
80fb139a10 add 5_byte_arm63 to release 2020-07-29 14:07:04 -04:00
mwpeterson
9f6ecfbd42 add 5-byte large disk builds for arm and arm64 2020-07-28 15:57:49 -04:00
Chris Lu
bc2ec6774d inject git version into build 2020-06-02 00:10:38 -07:00
Chris Lu
d9a997cb9e work with travis 2019-09-03 11:26:11 -07:00
Chris Lu
6361e2b2fc fix travis 2019-09-03 10:20:16 -07:00
Chris Lu
e33f5f5b33 travis build 2019-09-03 09:24:35 -07:00
Chris Lu
88ba08cb9e add large disk support 2019-04-08 22:32:42 -07:00
Michael Peick
57e159b369 Build statically linked release executables. 2018-11-29 13:26:48 +01:00
Chris Lu
19159ec479 adjust makefile and travis 2017-01-03 22:20:15 -08:00
Chris Lu
9cb034d49e adjust makefile 2017-01-03 22:11:33 -08:00
Chris Lu
d1562e48e1 default to release 2017-01-03 22:08:53 -08:00
Chris Lu
2d13382c68 add releasing configs 2017-01-03 21:14:46 -08:00
Chris Lu
bb839ba268 fix 2016-06-05 10:57:14 -07:00
tnextday
662915e691 Delete all chunks when delete a ChunkManifest
LoadChunkManifest can uncompress buffer
move compress.go from storage to operation because of import cycle
MakeFile add cross complete command
2015-12-02 21:27:29 +08:00
euforia
94757a209b Added basic makefile 2015-11-17 18:03:23 -08:00