Commit Graph

12918 Commits

Author SHA1 Message Date
Chris Lu
8b5d31e5eb s3api/policy_engine: use forwarded client IP for aws:SourceIp (#8304)
* s3api: honor forwarded source IP for policy conditions

Prefer X-Forwarded-For/X-Real-Ip before RemoteAddr when populating aws:SourceIp in policy condition evaluation. Also avoid noisy parsing behavior for unix socket markers and add coverage for precedence/fallback paths.\n\nFixes #8301.

* s3api: simplify remote addr parsing

* s3api: guard aws:SourceIp against DNS hosts

* s3api: simplify remote addr fallback

* s3api: simplify remote addr parsing

* Update weed/s3api/policy_engine/engine.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix TestExtractConditionValuesFromRequestSourceIPPrecedence using trusted private IP

* Refactor extractSourceIP to use R-to-L XFF parsing and net.IP.IsPrivate

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-11 12:47:03 -08:00
Chris Lu
7151181d54 fix flaky tests 2026-02-11 12:23:35 -08:00
Lisandro Pin
e657e7d827 Implement local scrubbing for EC volumes. (#8283) 2026-02-11 11:04:08 -08:00
Lisandro Pin
2a73219397 Add weed shell command volumeServer.state to query/update volume server state settings. (#8271)
Add weed shell command `volumeServer.state` to query/update volume server states.
2026-02-11 11:02:37 -08:00
Chris Lu
7fcbffed7f filer.sync: support manifest chunks (#8299)
* filer.sync support manifest chunks

* filersink: address manifest sync review feedback
2026-02-10 20:18:35 -08:00
Chris Lu
be0379f6fd Fix filer.sync retry on stale chunk (#8298)
* Fix filer.sync stale chunk uploads

* Tweak filersink stale logging
2026-02-10 19:06:35 -08:00
Chris Lu
b57429ef2e Switch empty-folder cleanup to bucket policy (#8292)
* Fix Spark _temporary cleanup and add issue #8285 regression test

* Generalize empty folder cleanup for Spark temp artifacts

* Revert synchronous folder pruning and add cleanup diagnostics

* Add actionable empty-folder cleanup diagnostics

* Fix Spark temp marker cleanup in async folder cleaner

* Fix Spark temp cleanup with implicit directory markers

* Keep explicit directory markers non-implicit

* logging

* more logs

* Switch empty-folder cleanup to bucket policy

* Seaweed-X-Amz-Allow-Empty-Folders

* less logs

* go vet

* less logs

* refactoring
2026-02-10 18:38:38 -08:00
Chris Lu
5c365e7090 s3api: return 400 for invalid namespace query in REST table routes (#8296)
* s3api: reject invalid namespace query in REST table routes

* s3api: expand namespace validation REST tests
2026-02-10 17:57:08 -08:00
Chris Lu
822dbed552 s3api: fix ListObjectsV2 NextContinuationToken duplication for nested prefix (#8294)
* s3api: fix duplicate ListObjectsV2 continuation token for nested prefix

* s3api: include prefix in common-prefix continuation token
2026-02-10 14:17:41 -08:00
Chris Lu
2d97685390 ci: fix container_release_unified manual dispatch and workflow parsing (#8293)
ci: fix unified release workflow dispatch matrix filtering
2026-02-10 13:15:52 -08:00
Chris Lu
1b2f719d7c admin: fix file browser items-per-page selector (#8291)
* admin: fix file browser page size selector

Fix file browser pagination page-size selectors to use explicit select IDs instead of this.value in templ-generated handlers, which could resolve to undefined and produce limit=undefined in requests.

Add a focused template render regression test to prevent this from recurring.

Fixes #8284

* revert file browser template regression test
2026-02-10 12:56:34 -08:00
Chris Lu
b73bd08470 ci: move manual container builds to unified release workflow (#8290)
* ci: move manual dev container build into unified release workflow

* ci: make unified manual container build release-tag based
2026-02-10 12:47:39 -08:00
Chris Lu
17f85361e9 Remove unsupported iceberg rest signing-region from tests and docs (#8289) 2026-02-10 12:23:31 -08:00
Chris Lu
b261c89675 Fix RocksDB container build compatibility and add manual rocksdb dispatch (#8288)
* Fix RocksDB build compatibility and add manual rocksdb trigger

* Upgrade RocksDB defaults and keep grocksdb v1.10.7

* Add manual latest-image trigger inputs for ref and variant

* Allow manual latest build to set image tag and source ref

* Fix manual variant selection using setup job matrix output
2026-02-10 11:48:42 -08:00
Chris Lu
0385acba02 s3tables: fix shared table-location bucket mapping collisions (#8286)
* s3tables: prevent shared table-location bucket mapping overwrite

* Update weed/s3api/bucket_paths.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-10 11:28:29 -08:00
Chris Lu
d6825ffce2 Iceberg: implement stage-create finalize flow (phase 1) (#8279)
* iceberg: implement stage-create and create-on-commit finalize

* iceberg: add create validation error typing and stage-create integration test

* tests: merge stage-create integration check into catalog suite

* tests: cover stage-create finalize lifecycle in catalog integration

* iceberg: persist and cleanup stage-create markers

* iceberg: add stage-create rollout flag and marker pruning

* docs: add stage-create support design and rollout plan

* docs: drop stage-create design draft from PR

* iceberg: use conservative 72h stage-marker retention

* iceberg: address review comments on create-on-commit and tests

* iceberg: keep stage-create metadata out of table location

* refactor(iceberg): split iceberg.go into focused files
2026-02-10 09:46:09 -08:00
Chris Lu
d88f6ed0af Iceberg commit reliability: preserve statistics updates and return 409 conflicts (#8277)
* iceberg: harden table commit updates and conflict handling

* iceberg: refine commit retry and statistics patching

* iceberg: cleanup metadata on non-conflict commit errors
2026-02-09 23:00:03 -08:00
Chris Lu
5ae3be44d1 iceberg: persist namespace properties for create/get (#8276)
* iceberg: persist namespace properties via s3tables metadata

* iceberg: simplify namespace properties normalization

* s3tables: broaden namespace properties round-trip test

* adjust logs

* adjust logs
2026-02-09 22:20:45 -08:00
Chris Lu
1c62808c0e iceberg: wire pagination for list namespaces/tables REST APIs (#8275)
* s3api/iceberg: wire list pagination tokens and page size

* fmt

* Update weed/s3api/iceberg/iceberg.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-09 21:46:55 -08:00
Chris Lu
db76eb26e7 compile 2026-02-09 21:06:07 -08:00
Chris Lu
4ccc7668ce admin: resolve merge conflicts 2026-02-09 20:56:20 -08:00
Chris Lu
aef2de3109 s3tables: support multi-level namespaces in parser/admin paths (#8273)
* s3tables: support multi-level namespace normalization

* admin: handle namespace parsing errors centrally

* admin: clean namespace validation duplication
2026-02-09 20:20:05 -08:00
Chris Lu
be26ce74ce s3tables: support multi-level namespace normalization 2026-02-09 19:42:31 -08:00
Chris Lu
0b80f055c2 Merge branch 'fix/8270-leader-not-elected' 2026-02-09 18:15:59 -08:00
Chris Lu
af8273386d 4.12 2026-02-09 18:15:19 -08:00
Chris Lu
ba8e2aaae9 Fix master leader election when grpc ports change (#8272)
* Fix master leader detection when grpc ports change

* Canonicalize self peer entry to avoid raft self-alias panic

* Normalize and deduplicate master peer addresses
2026-02-09 18:13:02 -08:00
Chris Lu
15d0a46679 Normalize and deduplicate master peer addresses 2026-02-09 18:10:54 -08:00
Chris Lu
ae27e17e6f Canonicalize self peer entry to avoid raft self-alias panic 2026-02-09 18:07:01 -08:00
Chris Lu
02dac23119 Fix master leader detection when grpc ports change 2026-02-09 18:00:09 -08:00
Lisandro Pin
f400fb44a0 Update cluster.status to resolve file details on EC volumes. (#8268)
Also parallelizes queries for file metrics collections when the `--files`
flag is specified, and improves the command's output for readability:

```
> cluster.status --files
collecting file stats: 100%

cluster:
	id:       topo
	status:   LOCKED
	nodes:    10
	topology: 1 DC, 10 disks on 1 rack

volumes:
	total:    3 volumes, 1 collection
	max size: 32 GB
	regular:  1/80 volume on 3 replicas, 3 writable (100%), 0 read-only (0%)
	EC:       2 EC volumes on 28 shards (14 shards/volume)

storage:
	total:           269 MB (522 MB raw, 193.95%)
	regular volumes: 91 MB (272 MB raw, 300%)
	EC volumes:      178 MB (250 MB raw, 140%)

files:
	total:   363 files, 300 readable (82.64%), 63 deleted (17.35%), avg 522 kB per file
	regular: 168 files, 105 readable (62.5%), 63 deleted (37.5%), avg 540 kB per file
	EC:      195 files, 195 readable (100%), 0 deleted (0%), avg 506 kB per file
```
2026-02-09 17:52:43 -08:00
Chris Lu
30812b85f3 fix ec.encode skipping volumes when one replica is on a full disk (#8227)
* fix ec.encode skipping volumes when one replica is on a full disk

This fixes issue #8218. Previously, ec.encode would skip a volume if ANY
of its replicas resided on a disk with low free volume count. Now it
accepts the volume if AT LEAST ONE replica is on a healthy disk.

* refine noFreeDisk counter logic in ec.encode

Ensure noFreeDisk is decremented if a volume initially marked as bad
is later found to have a healthy replica. This ensures accurate
summary statistics.

* defer noFreeDisk counting and refine logging in ec.encode

Updated logging to be replica-scoped and deferred noFreeDisk counting to
the final pass over vidMap. This ensures that the counter only reflects
volumes that are definitively excluded because all replicas are on full
disks.

* filter replicas by free space during ec.encode

Updated doEcEncode to filter out replicas on disks with
FreeVolumeCount < 2 before selecting the best replica for encoding.
This ensures that EC shards are not generated on healthy source
replicas that happen to be on disks with low free space.
2026-02-09 14:23:11 -08:00
Chris Lu
6a61037333 fix issue #8230: volume.fsck deletion logic to respect purgeAbsent flag (#8266)
* fix issue #8230: volume.fsck deletion logic to respect purgeAbsent flag

This commit fixes two issues in volume.fsck:
1. Missing chunks in existing volumes are now deleted if -reallyDeleteFilerEntries is set.
2. Missing volumes are now properly handled when a -volumeId filter is specified, allowing deletion of filer entries for those volumes.

* address PR feedback for issue #8230

- Ensure volume filter is applied before reporting missing volumes
- Fix potential nil-pointer dereferences in httpDelete method
- Use proper error checking throughout httpDelete

* address second round PR feedback for issue #8230

- Use fmt.Fprintf(c.writer, ...) instead of fmt.Printf
- Add missing newline in "deleting path" log message
2026-02-09 13:23:17 -08:00
Chris Lu
839028b2e0 Fix EC rebuild shard detection (#8265)
Fix EC rebuild shard counting
2026-02-09 12:34:38 -08:00
Lisandro Pin
1a5679a5eb Implement a VolumeEcStatus() RPC for volume servers. (#8006)
Just like `VolumeStatus()`, this call allows inspecting details for
a given EC volume - including number of files and their total size.
2026-02-09 11:52:08 -08:00
dependabot[bot]
818a1ff8b1 build(deps): bump golang.org/x/crypto from 0.47.0 to 0.48.0 (#8258)
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.47.0 to 0.48.0.
- [Commits](https://github.com/golang/crypto/compare/v0.47.0...v0.48.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.48.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-09 11:49:23 -08:00
Chris Lu
59b02e0cba s3api: fix multipart Complete ETag matching and lower empty-upload log noise (#8264)
s3api: fix multipart part etag validation and reduce empty upload warning noise
2026-02-09 11:45:57 -08:00
dependabot[bot]
d9987669cb build(deps): bump github.com/jhump/protoreflect from 1.17.0 to 1.18.0 (#8261)
Bumps [github.com/jhump/protoreflect](https://github.com/jhump/protoreflect) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/jhump/protoreflect/releases)
- [Commits](https://github.com/jhump/protoreflect/compare/v1.17.0...v1.18.0)

---
updated-dependencies:
- dependency-name: github.com/jhump/protoreflect
  dependency-version: 1.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-09 11:32:55 -08:00
dependabot[bot]
0c16663014 build(deps): bump golang.org/x/sys from 0.40.0 to 0.41.0 (#8260)
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.40.0 to 0.41.0.
- [Commits](https://github.com/golang/sys/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-version: 0.41.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-09 11:31:01 -08:00
Chris Lu
ccf35459be Explicitly disable signing for public buckets. (#8263) 2026-02-09 11:28:07 -08:00
dependabot[bot]
88b1768814 build(deps): bump go.etcd.io/etcd/client/v3 from 3.6.6 to 3.6.7 (#8257)
Bumps [go.etcd.io/etcd/client/v3](https://github.com/etcd-io/etcd) from 3.6.6 to 3.6.7.
- [Release notes](https://github.com/etcd-io/etcd/releases)
- [Commits](https://github.com/etcd-io/etcd/compare/v3.6.6...v3.6.7)

---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/v3
  dependency-version: 3.6.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-09 11:23:12 -08:00
dependabot[bot]
62063de1ca build(deps): bump github.com/linxGnu/grocksdb from 1.10.3 to 1.10.7 (#8259)
Bumps [github.com/linxGnu/grocksdb](https://github.com/linxGnu/grocksdb) from 1.10.3 to 1.10.7.
- [Release notes](https://github.com/linxGnu/grocksdb/releases)
- [Commits](https://github.com/linxGnu/grocksdb/compare/v1.10.3...v1.10.7)

---
updated-dependencies:
- dependency-name: github.com/linxGnu/grocksdb
  dependency-version: 1.10.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-09 11:00:34 -08:00
Lisandro Pin
63b846b73b Parallelize operations for the volume.scrub and ec.scrub commands (#8247)
Parallelize operations for the `volume.scrub` and `ec.scrub` commands.
2026-02-09 09:07:06 -08:00
Chris Lu
c9428c2c0f Modify AI review comments checklist in PR template
Updated AI code review checklist to include potential additional reviews.
2026-02-09 08:58:35 -08:00
Chris Lu
cb9e21cdc5 Normalize hashicorp raft peer ids (#8253)
* Normalize raft voter ids

* 4.11

* Update raft_hashicorp.go
2026-02-09 07:46:34 -08:00
Chris Lu
2ed5a8f65c add tests 2026-02-09 01:37:56 -08:00
Chris Lu
5a279c4d2f fmt 2026-02-08 21:19:00 -08:00
Chris Lu
0c89185291 4.10 2026-02-08 21:16:58 -08:00
Chris Lu
458c12fb99 test: add Spark S3 integration regression for issue #8234 (#8249)
* test: add Spark S3 regression integration test for issue 8234

* Update test/s3/spark/setup_test.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-08 21:13:31 -08:00
Chris Lu
5a0204310c Add Iceberg admin UI (#8246)
* Add Iceberg table details view

* Enhance Iceberg catalog browsing UI

* Fix Iceberg UI security and logic issues

- Fix selectSchema() and partitionFieldsFromFullMetadata() to always search for matching IDs instead of checking != 0
- Fix snapshotsFromFullMetadata() to defensive-copy before sorting to prevent mutating caller's slice
- Fix XSS vulnerabilities in s3tables.js: replace innerHTML with textContent/createElement for user-controlled data
- Fix deleteIcebergTable() to redirect to namespace tables list on details page instead of reloading
- Fix data-bs-target in iceberg_namespaces.templ: remove templ.SafeURL for CSS selector
- Add catalogName to delete modal data attributes for proper redirect
- Remove unused hidden inputs from create table form (icebergTableBucketArn, icebergTableNamespace)

* Regenerate templ files for Iceberg UI updates

* Support complex Iceberg type objects in schema

Change Type field from string to json.RawMessage in both IcebergSchemaFieldInfo
and internal icebergSchemaField to properly handle Iceberg spec's complex type
objects (e.g. {"type": "struct", "fields": [...]}). Currently test data
only shows primitive string types, but this change makes the implementation
defensively robust for future complex types by preserving the exact JSON
representation. Add typeToString() helper and update schema extraction
functions to marshal string types as JSON. Update template to convert
json.RawMessage to string for display.

* Regenerate templ files for Type field changes

* templ

* Fix additional Iceberg UI issues from code review

- Fix lazy-load flag that was set before async operation completed, preventing retries
  on error; now sets loaded flag only after successful load and throws error to caller
  for proper error handling and UI updates
- Add zero-time guards for CreatedAt and ModifiedAt fields in table details to avoid
  displaying Go zero-time values; render dash when time is zero
- Add URL path escaping for all catalog/namespace/table names in URLs to prevent
  malformed URLs when names contain special characters like /, ?, or #
- Remove redundant innerHTML clear in loadIcebergNamespaceTables that cleared twice
  before appending the table list
- Fix selectSnapshotForMetrics to remove != 0 guard for consistency with selectSchema
  fix; now always searches for CurrentSnapshotID without zero-value gate
- Enhance typeToString() helper to display '(complex)' for non-primitive JSON types

* Regenerate templ files for Phase 3 updates

* Fix template generation to use correct file paths

Run templ generate from repo root instead of weed/admin directory to ensure
generated _templ.go files have correct absolute paths in error messages
(e.g., 'weed/admin/view/app/iceberg_table_details.templ' instead of
'app/iceberg_table_details.templ'). This ensures both 'make admin-generate'
at repo root and 'make generate' in weed/admin directory produce identical
output with consistent file path references.

* Regenerate template files with correct path references

* Validate S3 Tables names in UI

- Add client-side validation for table bucket and namespace names to surface
  errors for invalid characters (dots/underscores) before submission
- Use HTML validity messages with reportValidity for immediate feedback
- Update namespace helper text to reflect actual constraints (single-level,
  lowercase letters, numbers, and underscores)

* Regenerate templ files for namespace helper text

* Fix Iceberg catalog REST link and actions

* Disallow S3 object access on table buckets

* Validate Iceberg layout for table bucket objects

* Fix REST API link to /v1/config

* merge iceberg page with table bucket page

* Allowed Trino/Iceberg stats files in metadata validation

* fixes

  - Backend/data handling:
      - Normalized Iceberg type display and fallback handling in weed/admin/dash/s3tables_management.go.
      - Fixed snapshot fallback pointer semantics in weed/admin/dash/s3tables_management.go.
      - Added CSRF token generation/propagation/validation for namespace create/delete in:
          - weed/admin/dash/csrf.go
          - weed/admin/dash/auth_middleware.go
          - weed/admin/dash/middleware.go
          - weed/admin/dash/s3tables_management.go
          - weed/admin/view/layout/layout.templ
          - weed/admin/static/js/s3tables.js
  - UI/template fixes:
      - Zero-time guards for CreatedAt fields in:
          - weed/admin/view/app/iceberg_namespaces.templ
          - weed/admin/view/app/iceberg_tables.templ
      - Fixed invalid templ-in-script interpolation and host/port rendering in:
          - weed/admin/view/app/iceberg_catalog.templ
          - weed/admin/view/app/s3tables_buckets.templ
      - Added data-catalog-name consistency on Iceberg delete action in weed/admin/view/app/iceberg_tables.templ.
      - Updated retry wording in weed/admin/static/js/s3tables.js.
      - Regenerated all affected _templ.go files.
  - S3 API/comment follow-ups:
      - Reused cached table-bucket validator in weed/s3api/bucket_paths.go.
      - Added validation-failure debug logging in weed/s3api/s3api_object_handlers_tagging.go.
      - Added multipart path-validation design comment in weed/s3api/s3api_object_handlers_multipart.go.
  - Build tooling:
      - Fixed templ generate working directory issues in weed/admin/Makefile (watch + pattern rule).

* populate data

* test/s3tables: harden populate service checks

* admin: skip table buckets in object-store bucket list

* admin sidebar: move object store to top-level links

* admin iceberg catalog: guard zero times and escape links

* admin forms: add csrf/error handling and client-side name validation

* admin s3tables: fix namespace delete modal redeclaration

* admin: replace native confirm dialogs with modal helpers

* admin modal-alerts: remove noisy confirm usage console log

* reduce logs

* test/s3tables: use partitioned tables in trino and spark populate

* admin file browser: normalize filer ServerAddress for HTTP parsing
2026-02-08 20:06:32 -08:00
Chris Lu
bbcb8b7590 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-02-08 20:05:24 -08:00