gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2bb21ea2766388b8a1e76f590fea6d534fec9b91
seaweedFS/weed/admin/view/app/iceberg_catalog.templ
Chris Lu 2bb21ea276 feat: Add Iceberg REST Catalog server and admin UI (#8175) 
* feat: Add Iceberg REST Catalog server

Implement Iceberg REST Catalog API on a separate port (default 8181)
that exposes S3 Tables metadata through the Apache Iceberg REST protocol.

- Add new weed/s3api/iceberg package with REST handlers
- Implement /v1/config endpoint returning catalog configuration
- Implement namespace endpoints (list/create/get/head/delete)
- Implement table endpoints (list/create/load/head/delete/update)
- Add -port.iceberg flag to S3 standalone server (s3.go)
- Add -s3.port.iceberg flag to combined server mode (server.go)
- Add -s3.port.iceberg flag to mini cluster mode (mini.go)
- Support prefix-based routing for multiple catalogs

The Iceberg REST server reuses S3 Tables metadata storage under
/table-buckets and enables DuckDB, Spark, and other Iceberg clients
to connect to SeaweedFS as a catalog.

* feat: Add Iceberg Catalog pages to admin UI

Add admin UI pages to browse Iceberg catalogs, namespaces, and tables.

- Add Iceberg Catalog menu item under Object Store navigation
- Create iceberg_catalog.templ showing catalog overview with REST info
- Create iceberg_namespaces.templ listing namespaces in a catalog
- Create iceberg_tables.templ listing tables in a namespace
- Add handlers and routes in admin_handlers.go
- Add Iceberg data provider methods in s3tables_management.go
- Add Iceberg data types in types.go

The Iceberg Catalog pages provide visibility into the same S3 Tables
data through an Iceberg-centric lens, including REST endpoint examples
for DuckDB and PyIceberg.

* test: Add Iceberg catalog integration tests and reorg s3tables tests

- Reorganize existing s3tables tests to test/s3tables/table-buckets/
- Add new test/s3tables/catalog/ for Iceberg REST catalog tests
- Add TestIcebergConfig to verify /v1/config endpoint
- Add TestIcebergNamespaces to verify namespace listing
- Add TestDuckDBIntegration for DuckDB connectivity (requires Docker)
- Update CI workflow to use new test paths

* fix: Generate proper random UUIDs for Iceberg tables

Address code review feedback:
- Replace placeholder UUID with crypto/rand-based UUID v4 generation
- Add detailed TODO comments for handleUpdateTable stub explaining
  the required atomic metadata swap implementation

* fix: Serve Iceberg on localhost listener when binding to different interface

Address code review feedback: properly serve the localhost listener
when the Iceberg server is bound to a non-localhost interface.

* ci: Add Iceberg catalog integration tests to CI

Add new job to run Iceberg catalog tests in CI, along with:
- Iceberg package build verification
- Iceberg unit tests
- Iceberg go vet checks
- Iceberg format checks

* fix: Address code review feedback for Iceberg implementation

- fix: Replace hardcoded account ID with s3_constants.AccountAdminId in buildTableBucketARN()
- fix: Improve UUID generation error handling with deterministic fallback (timestamp + PID + counter)
- fix: Update handleUpdateTable to return HTTP 501 Not Implemented instead of fake success
- fix: Better error handling in handleNamespaceExists to distinguish 404 from 500 errors
- fix: Use relative URL in template instead of hardcoded localhost:8181
- fix: Add HTTP timeout to test's waitForService function to avoid hangs
- fix: Use dynamic ephemeral ports in integration tests to avoid flaky parallel failures
- fix: Add Iceberg port to final port configuration logging in mini.go

* fix: Address critical issues in Iceberg implementation

- fix: Cache table UUIDs to ensure persistence across LoadTable calls
  The UUID now remains stable for the lifetime of the server session.
  TODO: For production, UUIDs should be persisted in S3 Tables metadata.

- fix: Remove redundant URL-encoded namespace parsing
  mux router already decodes %1F to \x1F before passing to handlers.
  Redundant ReplaceAll call could cause bugs with literal %1F in namespace.

* fix: Improve test robustness and reduce code duplication

- fix: Make DuckDB test more robust by failing on unexpected errors
  Instead of silently logging errors, now explicitly check for expected
  conditions (extension not available) and skip the test appropriately.

- fix: Extract username helper method to reduce duplication
  Created getUsername() helper in AdminHandlers to avoid duplicating
  the username retrieval logic across Iceberg page handlers.

* fix: Add mutex protection to table UUID cache

Protects concurrent access to the tableUUIDs map with sync.RWMutex.
Uses read-lock for fast path when UUID already cached, and write-lock
for generating new UUIDs. Includes double-check pattern to handle race
condition between read-unlock and write-lock.

* style: fix go fmt errors

* feat(iceberg): persist table UUID in S3 Tables metadata

* feat(admin): configure Iceberg port in Admin UI and commands

* refactor: address review comments (flags, tests, handlers)

- command/mini: fix tracking of explicit s3.port.iceberg flag
- command/admin: add explicit -iceberg.port flag
- admin/handlers: reuse getUsername helper
- tests: use 127.0.0.1 for ephemeral ports and os.Stat for file size check

* test: check error from FileStat in verify_gc_empty_test
2026-02-02 23:12:13 -08:00

216 lines
6.7 KiB
Plaintext
Raw Blame History

package app
import (
"fmt"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
templ IcebergCatalog(data dash.IcebergCatalogData) {
<div class="d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom">
<h1 class="h2">
<i class="fas fa-snowflake me-2"></i>Iceberg Catalog
</h1>
<div class="btn-toolbar mb-2 mb-md-0">
<div class="btn-group me-2">
<a href={ templ.SafeURL("/v1/config") } target="_blank" class="btn btn-sm btn-outline-secondary">
<i class="fas fa-external-link-alt me-1"></i>REST API
</a>
</div>
</div>
</div>
<div id="iceberg-catalog-content">
<!-- Info Alert about Iceberg REST -->
<div class="alert alert-info mb-4">
<div class="d-flex align-items-center">
<i class="fas fa-info-circle fa-2x me-3"></i>
<div>
<strong>Iceberg REST Catalog</strong>
<p class="mb-0 mt-1">
Connect your Iceberg clients (DuckDB, Spark, etc.) to:
<code>http://<span id="iceberg-host">localhost</span>:{fmt.Sprintf("%d", data.IcebergPort)}/v1</code>
</p>
<script>
document.getElementById('iceberg-host').innerText = window.location.hostname;
</script>
</div>
</div>
</div>
<!-- Stats Cards -->
<div class="row mb-4">
<div class="col-xl-4 col-md-6 mb-4">
<div class="card border-left-primary shadow h-100 py-2">
<div class="card-body">
<div class="row no-gutters align-items-center">
<div class="col mr-2">
<div class="text-xs font-weight-bold text-primary text-uppercase mb-1">
Catalogs (Table Buckets)
</div>
<div class="h5 mb-0 font-weight-bold text-gray-800">
{ fmt.Sprintf("%d", data.TotalCatalogs) }
</div>
</div>
<div class="col-auto">
<i class="fas fa-database fa-2x text-gray-300"></i>
</div>
</div>
</div>
</div>
</div>
<div class="col-xl-4 col-md-6 mb-4">
<div class="card border-left-success shadow h-100 py-2">
<div class="card-body">
<div class="row no-gutters align-items-center">
<div class="col mr-2">
<div class="text-xs font-weight-bold text-success text-uppercase mb-1">
REST Port
</div>
<div class="h5 mb-0 font-weight-bold text-gray-800">
{ fmt.Sprintf("%d", data.IcebergPort) }
</div>
</div>
<div class="col-auto">
<i class="fas fa-plug fa-2x text-gray-300"></i>
</div>
</div>
</div>
</div>
</div>
<div class="col-xl-4 col-md-6 mb-4">
<div class="card border-left-info shadow h-100 py-2">
<div class="card-body">
<div class="row no-gutters align-items-center">
<div class="col mr-2">
<div class="text-xs font-weight-bold text-info text-uppercase mb-1">
Last Updated
</div>
<div class="h6 mb-0 font-weight-bold text-gray-800">
{ data.LastUpdated.Format("15:04") }
</div>
</div>
<div class="col-auto">
<i class="fas fa-clock fa-2x text-gray-300"></i>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Catalog List -->
<div class="row">
<div class="col-12">
<div class="card shadow mb-4">
<div class="card-header py-3 d-flex flex-row align-items-center justify-content-between">
<h6 class="m-0 font-weight-bold text-primary">
<i class="fas fa-snowflake me-2"></i>Available Catalogs
</h6>
</div>
<div class="card-body">
<p class="text-muted mb-3">
Each S3 Table Bucket acts as an Iceberg catalog. Use the bucket name as the catalog prefix in your REST API calls.
</p>
<div class="table-responsive">
<table class="table table-hover" width="100%" cellspacing="0" id="icebergCatalogsTable">
<thead>
<tr>
<th>Catalog Name</th>
<th>Owner</th>
<th>REST Endpoint</th>
<th>Created</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
for _, catalog := range data.Catalogs {
<tr>
<td>
<i class="fas fa-snowflake text-info me-2"></i>
<strong>{ catalog.Name }</strong>
</td>
<td>{ catalog.OwnerAccountID }</td>
<td>
<code class="small">/v1/{ catalog.Name }/namespaces</code>
</td>
<td>{ catalog.CreatedAt.Format("2006-01-02 15:04") }</td>
<td>
<div class="btn-group btn-group-sm" role="group">
{{ bucketName, parseErr := s3tables.ParseBucketNameFromARN(catalog.ARN) }}
if parseErr == nil {
<a class="btn btn-outline-primary btn-sm" href={ templ.SafeURL(fmt.Sprintf("/object-store/iceberg/%s/namespaces", bucketName)) } title="Browse Namespaces">
<i class="fas fa-folder-open"></i>
</a>
}
</div>
</td>
</tr>
}
if len(data.Catalogs) == 0 {
<tr>
<td colspan="5" class="text-center text-muted py-4">
<i class="fas fa-snowflake fa-3x mb-3 text-muted"></i>
<div>
<h5>No catalogs available</h5>
<p>Create an S3 Table Bucket first to use as an Iceberg catalog.</p>
<a href="/object-store/s3tables/buckets" class="btn btn-primary">
<i class="fas fa-plus me-1"></i>Create Table Bucket
</a>
</div>
</td>
</tr>
}
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
<!-- Example Usage Card -->
<div class="row">
<div class="col-12">
<div class="card shadow mb-4">
<div class="card-header py-3">
<h6 class="m-0 font-weight-bold text-primary">
<i class="fas fa-code me-2"></i>Example Usage
</h6>
</div>
<div class="card-body">
<h6>DuckDB</h6>
<pre class="bg-light p-3 border rounded"><code>-- Install and load Iceberg extension
INSTALL iceberg;
LOAD iceberg;
-- Create a catalog connection
CREATE SECRET (
TYPE ICEBERG,
ENDPOINT 'http://localhost:{ fmt.Sprintf("%d", data.IcebergPort) }',
SCOPE 's3://my-table-bucket/'
);
-- Query tables
SELECT * FROM iceberg_scan('s3://my-table-bucket/my-namespace/my-table');</code></pre>
<h6 class="mt-4">Python (PyIceberg)</h6>
<pre class="bg-light p-3 border rounded"><code>from pyiceberg.catalog import load_catalog
catalog = load_catalog(
name="seaweedfs",
**{"{"}
"type": "rest",
"uri": "http://localhost:{ fmt.Sprintf("%d", data.IcebergPort) }",
{"}"}
)
# List namespaces
namespaces = catalog.list_namespaces()</code></pre>
</div>
</div>
</div>
</div>
</div>
}
Reference in New Issue View Git Blame Copy Permalink