2bb21ea276
* feat: Add Iceberg REST Catalog server Implement Iceberg REST Catalog API on a separate port (default 8181) that exposes S3 Tables metadata through the Apache Iceberg REST protocol. - Add new weed/s3api/iceberg package with REST handlers - Implement /v1/config endpoint returning catalog configuration - Implement namespace endpoints (list/create/get/head/delete) - Implement table endpoints (list/create/load/head/delete/update) - Add -port.iceberg flag to S3 standalone server (s3.go) - Add -s3.port.iceberg flag to combined server mode (server.go) - Add -s3.port.iceberg flag to mini cluster mode (mini.go) - Support prefix-based routing for multiple catalogs The Iceberg REST server reuses S3 Tables metadata storage under /table-buckets and enables DuckDB, Spark, and other Iceberg clients to connect to SeaweedFS as a catalog. * feat: Add Iceberg Catalog pages to admin UI Add admin UI pages to browse Iceberg catalogs, namespaces, and tables. - Add Iceberg Catalog menu item under Object Store navigation - Create iceberg_catalog.templ showing catalog overview with REST info - Create iceberg_namespaces.templ listing namespaces in a catalog - Create iceberg_tables.templ listing tables in a namespace - Add handlers and routes in admin_handlers.go - Add Iceberg data provider methods in s3tables_management.go - Add Iceberg data types in types.go The Iceberg Catalog pages provide visibility into the same S3 Tables data through an Iceberg-centric lens, including REST endpoint examples for DuckDB and PyIceberg. * test: Add Iceberg catalog integration tests and reorg s3tables tests - Reorganize existing s3tables tests to test/s3tables/table-buckets/ - Add new test/s3tables/catalog/ for Iceberg REST catalog tests - Add TestIcebergConfig to verify /v1/config endpoint - Add TestIcebergNamespaces to verify namespace listing - Add TestDuckDBIntegration for DuckDB connectivity (requires Docker) - Update CI workflow to use new test paths * fix: Generate proper random UUIDs for Iceberg tables Address code review feedback: - Replace placeholder UUID with crypto/rand-based UUID v4 generation - Add detailed TODO comments for handleUpdateTable stub explaining the required atomic metadata swap implementation * fix: Serve Iceberg on localhost listener when binding to different interface Address code review feedback: properly serve the localhost listener when the Iceberg server is bound to a non-localhost interface. * ci: Add Iceberg catalog integration tests to CI Add new job to run Iceberg catalog tests in CI, along with: - Iceberg package build verification - Iceberg unit tests - Iceberg go vet checks - Iceberg format checks * fix: Address code review feedback for Iceberg implementation - fix: Replace hardcoded account ID with s3_constants.AccountAdminId in buildTableBucketARN() - fix: Improve UUID generation error handling with deterministic fallback (timestamp + PID + counter) - fix: Update handleUpdateTable to return HTTP 501 Not Implemented instead of fake success - fix: Better error handling in handleNamespaceExists to distinguish 404 from 500 errors - fix: Use relative URL in template instead of hardcoded localhost:8181 - fix: Add HTTP timeout to test's waitForService function to avoid hangs - fix: Use dynamic ephemeral ports in integration tests to avoid flaky parallel failures - fix: Add Iceberg port to final port configuration logging in mini.go * fix: Address critical issues in Iceberg implementation - fix: Cache table UUIDs to ensure persistence across LoadTable calls The UUID now remains stable for the lifetime of the server session. TODO: For production, UUIDs should be persisted in S3 Tables metadata. - fix: Remove redundant URL-encoded namespace parsing mux router already decodes %1F to \x1F before passing to handlers. Redundant ReplaceAll call could cause bugs with literal %1F in namespace. * fix: Improve test robustness and reduce code duplication - fix: Make DuckDB test more robust by failing on unexpected errors Instead of silently logging errors, now explicitly check for expected conditions (extension not available) and skip the test appropriately. - fix: Extract username helper method to reduce duplication Created getUsername() helper in AdminHandlers to avoid duplicating the username retrieval logic across Iceberg page handlers. * fix: Add mutex protection to table UUID cache Protects concurrent access to the tableUUIDs map with sync.RWMutex. Uses read-lock for fast path when UUID already cached, and write-lock for generating new UUIDs. Includes double-check pattern to handle race condition between read-unlock and write-lock. * style: fix go fmt errors * feat(iceberg): persist table UUID in S3 Tables metadata * feat(admin): configure Iceberg port in Admin UI and commands * refactor: address review comments (flags, tests, handlers) - command/mini: fix tracking of explicit s3.port.iceberg flag - command/admin: add explicit -iceberg.port flag - admin/handlers: reuse getUsername helper - tests: use 127.0.0.1 for ephemeral ports and os.Stat for file size check * test: check error from FileStat in verify_gc_empty_test
127 lines
3.8 KiB
Plaintext
127 lines
3.8 KiB
Plaintext
package app
|
|
|
|
import (
|
|
"fmt"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
|
|
)
|
|
|
|
templ IcebergNamespaces(data dash.IcebergNamespacesData) {
|
|
<div class="d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom">
|
|
<h1 class="h2">
|
|
<nav aria-label="breadcrumb">
|
|
<ol class="breadcrumb mb-0">
|
|
<li class="breadcrumb-item">
|
|
<a href="/object-store/iceberg">
|
|
<i class="fas fa-snowflake me-1"></i>Iceberg Catalog
|
|
</a>
|
|
</li>
|
|
<li class="breadcrumb-item active">{ data.CatalogName }</li>
|
|
</ol>
|
|
</nav>
|
|
</h1>
|
|
</div>
|
|
<div id="iceberg-namespaces-content">
|
|
<!-- Stats Cards -->
|
|
<div class="row mb-4">
|
|
<div class="col-xl-4 col-md-6 mb-4">
|
|
<div class="card border-left-primary shadow h-100 py-2">
|
|
<div class="card-body">
|
|
<div class="row no-gutters align-items-center">
|
|
<div class="col mr-2">
|
|
<div class="text-xs font-weight-bold text-primary text-uppercase mb-1">
|
|
Catalog
|
|
</div>
|
|
<div class="h5 mb-0 font-weight-bold text-gray-800">
|
|
{ data.CatalogName }
|
|
</div>
|
|
</div>
|
|
<div class="col-auto">
|
|
<i class="fas fa-snowflake fa-2x text-gray-300"></i>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div class="col-xl-4 col-md-6 mb-4">
|
|
<div class="card border-left-success shadow h-100 py-2">
|
|
<div class="card-body">
|
|
<div class="row no-gutters align-items-center">
|
|
<div class="col mr-2">
|
|
<div class="text-xs font-weight-bold text-success text-uppercase mb-1">
|
|
Total Namespaces
|
|
</div>
|
|
<div class="h5 mb-0 font-weight-bold text-gray-800">
|
|
{ fmt.Sprintf("%d", data.TotalNamespaces) }
|
|
</div>
|
|
</div>
|
|
<div class="col-auto">
|
|
<i class="fas fa-folder fa-2x text-gray-300"></i>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<!-- Namespace List -->
|
|
<div class="row">
|
|
<div class="col-12">
|
|
<div class="card shadow mb-4">
|
|
<div class="card-header py-3 d-flex flex-row align-items-center justify-content-between">
|
|
<h6 class="m-0 font-weight-bold text-primary">
|
|
<i class="fas fa-folder me-2"></i>Namespaces
|
|
</h6>
|
|
</div>
|
|
<div class="card-body">
|
|
<div class="table-responsive">
|
|
<table class="table table-hover" width="100%" cellspacing="0">
|
|
<thead>
|
|
<tr>
|
|
<th>Namespace</th>
|
|
<th>REST Endpoint</th>
|
|
<th>Created</th>
|
|
<th>Actions</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
for _, ns := range data.Namespaces {
|
|
<tr>
|
|
<td>
|
|
<i class="fas fa-folder text-warning me-2"></i>
|
|
<strong>{ ns.Name }</strong>
|
|
</td>
|
|
<td>
|
|
<code class="small">/v1/{ data.CatalogName }/namespaces/{ ns.Name }/tables</code>
|
|
</td>
|
|
<td>{ ns.CreatedAt.Format("2006-01-02 15:04") }</td>
|
|
<td>
|
|
<div class="btn-group btn-group-sm" role="group">
|
|
<a class="btn btn-outline-primary btn-sm" href={ templ.SafeURL(fmt.Sprintf("/object-store/iceberg/%s/namespaces/%s/tables", data.CatalogName, ns.Name)) } title="Browse Tables">
|
|
<i class="fas fa-table"></i>
|
|
</a>
|
|
</div>
|
|
</td>
|
|
</tr>
|
|
}
|
|
if len(data.Namespaces) == 0 {
|
|
<tr>
|
|
<td colspan="4" class="text-center text-muted py-4">
|
|
<i class="fas fa-folder-open fa-3x mb-3 text-muted"></i>
|
|
<div>
|
|
<h5>No namespaces found</h5>
|
|
<p>Create namespaces via the Iceberg REST API or S3 Tables API.</p>
|
|
</div>
|
|
</td>
|
|
</tr>
|
|
}
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
}
|