gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
12,870 Commits 1 Branch 0 Tags
ac242d04eeb5e092e869eafe9ae6f71105bb2450
Commit Graph

12870 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Chris Lu
ac242d04ee one time manual run 2026-02-11 13:20:52 -08:00
Chris Lu
21543134c8 fix manual build process 2026-02-11 13:20:41 -08:00
Chris Lu
8b5d31e5eb s3api/policy_engine: use forwarded client IP for aws:SourceIp (#8304) 
* s3api: honor forwarded source IP for policy conditions

Prefer X-Forwarded-For/X-Real-Ip before RemoteAddr when populating aws:SourceIp in policy condition evaluation. Also avoid noisy parsing behavior for unix socket markers and add coverage for precedence/fallback paths.\n\nFixes #8301.

* s3api: simplify remote addr parsing

* s3api: guard aws:SourceIp against DNS hosts

* s3api: simplify remote addr fallback

* s3api: simplify remote addr parsing

* Update weed/s3api/policy_engine/engine.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix TestExtractConditionValuesFromRequestSourceIPPrecedence using trusted private IP

* Refactor extractSourceIP to use R-to-L XFF parsing and net.IP.IsPrivate

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-02-11 12:47:03 -08:00
Chris Lu
7151181d54 fix flaky tests 2026-02-11 12:23:35 -08:00
Lisandro Pin
e657e7d827 Implement local scrubbing for EC volumes. (#8283) 2026-02-11 11:04:08 -08:00
Lisandro Pin
2a73219397 Add weed shell command volumeServer.state to query/update volume server state settings. (#8271) 
Add weed shell command `volumeServer.state` to query/update volume server states.
 2026-02-11 11:02:37 -08:00
Chris Lu
7fcbffed7f filer.sync: support manifest chunks (#8299) 
* filer.sync support manifest chunks

* filersink: address manifest sync review feedback
 2026-02-10 20:18:35 -08:00
Chris Lu
be0379f6fd Fix filer.sync retry on stale chunk (#8298) 
* Fix filer.sync stale chunk uploads

* Tweak filersink stale logging
 2026-02-10 19:06:35 -08:00
Chris Lu
b57429ef2e Switch empty-folder cleanup to bucket policy (#8292) 
* Fix Spark _temporary cleanup and add issue #8285 regression test

* Generalize empty folder cleanup for Spark temp artifacts

* Revert synchronous folder pruning and add cleanup diagnostics

* Add actionable empty-folder cleanup diagnostics

* Fix Spark temp marker cleanup in async folder cleaner

* Fix Spark temp cleanup with implicit directory markers

* Keep explicit directory markers non-implicit

* logging

* more logs

* Switch empty-folder cleanup to bucket policy

* Seaweed-X-Amz-Allow-Empty-Folders

* less logs

* go vet

* less logs

* refactoring
 2026-02-10 18:38:38 -08:00
Chris Lu
5c365e7090 s3api: return 400 for invalid namespace query in REST table routes (#8296) 
* s3api: reject invalid namespace query in REST table routes

* s3api: expand namespace validation REST tests
 2026-02-10 17:57:08 -08:00
Chris Lu
822dbed552 s3api: fix ListObjectsV2 NextContinuationToken duplication for nested prefix (#8294) 
* s3api: fix duplicate ListObjectsV2 continuation token for nested prefix

* s3api: include prefix in common-prefix continuation token
 2026-02-10 14:17:41 -08:00
Chris Lu
2d97685390 ci: fix container_release_unified manual dispatch and workflow parsing (#8293) 
ci: fix unified release workflow dispatch matrix filtering
 2026-02-10 13:15:52 -08:00
Chris Lu
1b2f719d7c admin: fix file browser items-per-page selector (#8291) 
* admin: fix file browser page size selector

Fix file browser pagination page-size selectors to use explicit select IDs instead of this.value in templ-generated handlers, which could resolve to undefined and produce limit=undefined in requests.

Add a focused template render regression test to prevent this from recurring.

Fixes #8284

* revert file browser template regression test
 2026-02-10 12:56:34 -08:00
Chris Lu
b73bd08470 ci: move manual container builds to unified release workflow (#8290) 
* ci: move manual dev container build into unified release workflow

* ci: make unified manual container build release-tag based
 2026-02-10 12:47:39 -08:00
Chris Lu
17f85361e9 Remove unsupported iceberg rest signing-region from tests and docs (#8289) 2026-02-10 12:23:31 -08:00
Chris Lu
b261c89675 Fix RocksDB container build compatibility and add manual rocksdb dispatch (#8288) 
* Fix RocksDB build compatibility and add manual rocksdb trigger

* Upgrade RocksDB defaults and keep grocksdb v1.10.7

* Add manual latest-image trigger inputs for ref and variant

* Allow manual latest build to set image tag and source ref

* Fix manual variant selection using setup job matrix output
 2026-02-10 11:48:42 -08:00
Chris Lu
0385acba02 s3tables: fix shared table-location bucket mapping collisions (#8286) 
* s3tables: prevent shared table-location bucket mapping overwrite

* Update weed/s3api/bucket_paths.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-02-10 11:28:29 -08:00
Chris Lu
d6825ffce2 Iceberg: implement stage-create finalize flow (phase 1) (#8279) 
* iceberg: implement stage-create and create-on-commit finalize

* iceberg: add create validation error typing and stage-create integration test

* tests: merge stage-create integration check into catalog suite

* tests: cover stage-create finalize lifecycle in catalog integration

* iceberg: persist and cleanup stage-create markers

* iceberg: add stage-create rollout flag and marker pruning

* docs: add stage-create support design and rollout plan

* docs: drop stage-create design draft from PR

* iceberg: use conservative 72h stage-marker retention

* iceberg: address review comments on create-on-commit and tests

* iceberg: keep stage-create metadata out of table location

* refactor(iceberg): split iceberg.go into focused files
 2026-02-10 09:46:09 -08:00
Chris Lu
d88f6ed0af Iceberg commit reliability: preserve statistics updates and return 409 conflicts (#8277) 
* iceberg: harden table commit updates and conflict handling

* iceberg: refine commit retry and statistics patching

* iceberg: cleanup metadata on non-conflict commit errors
 2026-02-09 23:00:03 -08:00
Chris Lu
5ae3be44d1 iceberg: persist namespace properties for create/get (#8276) 
* iceberg: persist namespace properties via s3tables metadata

* iceberg: simplify namespace properties normalization

* s3tables: broaden namespace properties round-trip test

* adjust logs

* adjust logs
 2026-02-09 22:20:45 -08:00
Chris Lu
1c62808c0e iceberg: wire pagination for list namespaces/tables REST APIs (#8275) 
* s3api/iceberg: wire list pagination tokens and page size

* fmt

* Update weed/s3api/iceberg/iceberg.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-02-09 21:46:55 -08:00
Chris Lu
db76eb26e7 compile 2026-02-09 21:06:07 -08:00
Chris Lu
4ccc7668ce admin: resolve merge conflicts 2026-02-09 20:56:20 -08:00
Chris Lu
aef2de3109 s3tables: support multi-level namespaces in parser/admin paths (#8273) 
* s3tables: support multi-level namespace normalization

* admin: handle namespace parsing errors centrally

* admin: clean namespace validation duplication
 2026-02-09 20:20:05 -08:00
Chris Lu
be26ce74ce s3tables: support multi-level namespace normalization 2026-02-09 19:42:31 -08:00
Chris Lu
0b80f055c2 Merge branch 'fix/8270-leader-not-elected' 2026-02-09 18:15:59 -08:00
Chris Lu
af8273386d 4.12 2026-02-09 18:15:19 -08:00
Chris Lu
ba8e2aaae9 Fix master leader election when grpc ports change (#8272) 
* Fix master leader detection when grpc ports change

* Canonicalize self peer entry to avoid raft self-alias panic

* Normalize and deduplicate master peer addresses
 2026-02-09 18:13:02 -08:00
Chris Lu
15d0a46679 Normalize and deduplicate master peer addresses 2026-02-09 18:10:54 -08:00
Chris Lu
ae27e17e6f Canonicalize self peer entry to avoid raft self-alias panic 2026-02-09 18:07:01 -08:00
Chris Lu
02dac23119 Fix master leader detection when grpc ports change 2026-02-09 18:00:09 -08:00
Lisandro Pin
f400fb44a0 Update cluster.status to resolve file details on EC volumes. (#8268) 
Also parallelizes queries for file metrics collections when the `--files`
flag is specified, and improves the command's output for readability:

```
> cluster.status --files
collecting file stats: 100%

cluster:
	id:       topo
	status:   LOCKED
	nodes:    10
	topology: 1 DC, 10 disks on 1 rack

volumes:
	total:    3 volumes, 1 collection
	max size: 32 GB
	regular:  1/80 volume on 3 replicas, 3 writable (100%), 0 read-only (0%)
	EC:       2 EC volumes on 28 shards (14 shards/volume)

storage:
	total:           269 MB (522 MB raw, 193.95%)
	regular volumes: 91 MB (272 MB raw, 300%)
	EC volumes:      178 MB (250 MB raw, 140%)

files:
	total:   363 files, 300 readable (82.64%), 63 deleted (17.35%), avg 522 kB per file
	regular: 168 files, 105 readable (62.5%), 63 deleted (37.5%), avg 540 kB per file
	EC:      195 files, 195 readable (100%), 0 deleted (0%), avg 506 kB per file
```
 2026-02-09 17:52:43 -08:00
Chris Lu
30812b85f3 fix ec.encode skipping volumes when one replica is on a full disk (#8227) 
* fix ec.encode skipping volumes when one replica is on a full disk

This fixes issue #8218. Previously, ec.encode would skip a volume if ANY
of its replicas resided on a disk with low free volume count. Now it
accepts the volume if AT LEAST ONE replica is on a healthy disk.

* refine noFreeDisk counter logic in ec.encode

Ensure noFreeDisk is decremented if a volume initially marked as bad
is later found to have a healthy replica. This ensures accurate
summary statistics.

* defer noFreeDisk counting and refine logging in ec.encode

Updated logging to be replica-scoped and deferred noFreeDisk counting to
the final pass over vidMap. This ensures that the counter only reflects
volumes that are definitively excluded because all replicas are on full
disks.

* filter replicas by free space during ec.encode

Updated doEcEncode to filter out replicas on disks with
FreeVolumeCount < 2 before selecting the best replica for encoding.
This ensures that EC shards are not generated on healthy source
replicas that happen to be on disks with low free space.
 2026-02-09 14:23:11 -08:00
Chris Lu
6a61037333 fix issue #8230: volume.fsck deletion logic to respect purgeAbsent flag (#8266) 
* fix issue #8230: volume.fsck deletion logic to respect purgeAbsent flag

This commit fixes two issues in volume.fsck:
1. Missing chunks in existing volumes are now deleted if -reallyDeleteFilerEntries is set.
2. Missing volumes are now properly handled when a -volumeId filter is specified, allowing deletion of filer entries for those volumes.

* address PR feedback for issue #8230

- Ensure volume filter is applied before reporting missing volumes
- Fix potential nil-pointer dereferences in httpDelete method
- Use proper error checking throughout httpDelete

* address second round PR feedback for issue #8230

- Use fmt.Fprintf(c.writer, ...) instead of fmt.Printf
- Add missing newline in "deleting path" log message
 2026-02-09 13:23:17 -08:00
Chris Lu
839028b2e0 Fix EC rebuild shard detection (#8265) 
Fix EC rebuild shard counting
 2026-02-09 12:34:38 -08:00
Lisandro Pin
1a5679a5eb Implement a VolumeEcStatus() RPC for volume servers. (#8006) 
Just like `VolumeStatus()`, this call allows inspecting details for
a given EC volume - including number of files and their total size.
 2026-02-09 11:52:08 -08:00
dependabot[bot]
818a1ff8b1 build(deps): bump golang.org/x/crypto from 0.47.0 to 0.48.0 (#8258) 
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.47.0 to 0.48.0.
- [Commits](https://github.com/golang/crypto/compare/v0.47.0...v0.48.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.48.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-02-09 11:49:23 -08:00
Chris Lu
59b02e0cba s3api: fix multipart Complete ETag matching and lower empty-upload log noise (#8264) 
s3api: fix multipart part etag validation and reduce empty upload warning noise
 2026-02-09 11:45:57 -08:00
dependabot[bot]
d9987669cb build(deps): bump github.com/jhump/protoreflect from 1.17.0 to 1.18.0 (#8261) 
Bumps [github.com/jhump/protoreflect](https://github.com/jhump/protoreflect) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/jhump/protoreflect/releases)
- [Commits](https://github.com/jhump/protoreflect/compare/v1.17.0...v1.18.0)

---
updated-dependencies:
- dependency-name: github.com/jhump/protoreflect
  dependency-version: 1.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-02-09 11:32:55 -08:00
dependabot[bot]
0c16663014 build(deps): bump golang.org/x/sys from 0.40.0 to 0.41.0 (#8260) 
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.40.0 to 0.41.0.
- [Commits](https://github.com/golang/sys/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-version: 0.41.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-02-09 11:31:01 -08:00
Chris Lu
ccf35459be Explicitly disable signing for public buckets. (#8263) 2026-02-09 11:28:07 -08:00
dependabot[bot]
88b1768814 build(deps): bump go.etcd.io/etcd/client/v3 from 3.6.6 to 3.6.7 (#8257) 
Bumps [go.etcd.io/etcd/client/v3](https://github.com/etcd-io/etcd) from 3.6.6 to 3.6.7.
- [Release notes](https://github.com/etcd-io/etcd/releases)
- [Commits](https://github.com/etcd-io/etcd/compare/v3.6.6...v3.6.7)

---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/v3
  dependency-version: 3.6.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-02-09 11:23:12 -08:00
dependabot[bot]
62063de1ca build(deps): bump github.com/linxGnu/grocksdb from 1.10.3 to 1.10.7 (#8259) 
Bumps [github.com/linxGnu/grocksdb](https://github.com/linxGnu/grocksdb) from 1.10.3 to 1.10.7.
- [Release notes](https://github.com/linxGnu/grocksdb/releases)
- [Commits](https://github.com/linxGnu/grocksdb/compare/v1.10.3...v1.10.7)

---
updated-dependencies:
- dependency-name: github.com/linxGnu/grocksdb
  dependency-version: 1.10.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-02-09 11:00:34 -08:00
Lisandro Pin
63b846b73b Parallelize operations for the volume.scrub and ec.scrub commands (#8247) 
Parallelize operations for the `volume.scrub` and `ec.scrub` commands.
 2026-02-09 09:07:06 -08:00
Chris Lu
c9428c2c0f Modify AI review comments checklist in PR template 
Updated AI code review checklist to include potential additional reviews.
 2026-02-09 08:58:35 -08:00
Chris Lu
cb9e21cdc5 Normalize hashicorp raft peer ids (#8253) 
* Normalize raft voter ids

* 4.11

* Update raft_hashicorp.go
 2026-02-09 07:46:34 -08:00
Chris Lu
2ed5a8f65c add tests 2026-02-09 01:37:56 -08:00
Chris Lu
5a279c4d2f fmt 2026-02-08 21:19:00 -08:00
Chris Lu
0c89185291 4.10 2026-02-08 21:16:58 -08:00
Chris Lu
458c12fb99 test: add Spark S3 integration regression for issue #8234 (#8249) 
* test: add Spark S3 regression integration test for issue 8234

* Update test/s3/spark/setup_test.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-02-08 21:13:31 -08:00