feat: Add Iceberg REST Catalog server and admin UI (#8175)
* feat: Add Iceberg REST Catalog server Implement Iceberg REST Catalog API on a separate port (default 8181) that exposes S3 Tables metadata through the Apache Iceberg REST protocol. - Add new weed/s3api/iceberg package with REST handlers - Implement /v1/config endpoint returning catalog configuration - Implement namespace endpoints (list/create/get/head/delete) - Implement table endpoints (list/create/load/head/delete/update) - Add -port.iceberg flag to S3 standalone server (s3.go) - Add -s3.port.iceberg flag to combined server mode (server.go) - Add -s3.port.iceberg flag to mini cluster mode (mini.go) - Support prefix-based routing for multiple catalogs The Iceberg REST server reuses S3 Tables metadata storage under /table-buckets and enables DuckDB, Spark, and other Iceberg clients to connect to SeaweedFS as a catalog. * feat: Add Iceberg Catalog pages to admin UI Add admin UI pages to browse Iceberg catalogs, namespaces, and tables. - Add Iceberg Catalog menu item under Object Store navigation - Create iceberg_catalog.templ showing catalog overview with REST info - Create iceberg_namespaces.templ listing namespaces in a catalog - Create iceberg_tables.templ listing tables in a namespace - Add handlers and routes in admin_handlers.go - Add Iceberg data provider methods in s3tables_management.go - Add Iceberg data types in types.go The Iceberg Catalog pages provide visibility into the same S3 Tables data through an Iceberg-centric lens, including REST endpoint examples for DuckDB and PyIceberg. * test: Add Iceberg catalog integration tests and reorg s3tables tests - Reorganize existing s3tables tests to test/s3tables/table-buckets/ - Add new test/s3tables/catalog/ for Iceberg REST catalog tests - Add TestIcebergConfig to verify /v1/config endpoint - Add TestIcebergNamespaces to verify namespace listing - Add TestDuckDBIntegration for DuckDB connectivity (requires Docker) - Update CI workflow to use new test paths * fix: Generate proper random UUIDs for Iceberg tables Address code review feedback: - Replace placeholder UUID with crypto/rand-based UUID v4 generation - Add detailed TODO comments for handleUpdateTable stub explaining the required atomic metadata swap implementation * fix: Serve Iceberg on localhost listener when binding to different interface Address code review feedback: properly serve the localhost listener when the Iceberg server is bound to a non-localhost interface. * ci: Add Iceberg catalog integration tests to CI Add new job to run Iceberg catalog tests in CI, along with: - Iceberg package build verification - Iceberg unit tests - Iceberg go vet checks - Iceberg format checks * fix: Address code review feedback for Iceberg implementation - fix: Replace hardcoded account ID with s3_constants.AccountAdminId in buildTableBucketARN() - fix: Improve UUID generation error handling with deterministic fallback (timestamp + PID + counter) - fix: Update handleUpdateTable to return HTTP 501 Not Implemented instead of fake success - fix: Better error handling in handleNamespaceExists to distinguish 404 from 500 errors - fix: Use relative URL in template instead of hardcoded localhost:8181 - fix: Add HTTP timeout to test's waitForService function to avoid hangs - fix: Use dynamic ephemeral ports in integration tests to avoid flaky parallel failures - fix: Add Iceberg port to final port configuration logging in mini.go * fix: Address critical issues in Iceberg implementation - fix: Cache table UUIDs to ensure persistence across LoadTable calls The UUID now remains stable for the lifetime of the server session. TODO: For production, UUIDs should be persisted in S3 Tables metadata. - fix: Remove redundant URL-encoded namespace parsing mux router already decodes %1F to \x1F before passing to handlers. Redundant ReplaceAll call could cause bugs with literal %1F in namespace. * fix: Improve test robustness and reduce code duplication - fix: Make DuckDB test more robust by failing on unexpected errors Instead of silently logging errors, now explicitly check for expected conditions (extension not available) and skip the test appropriately. - fix: Extract username helper method to reduce duplication Created getUsername() helper in AdminHandlers to avoid duplicating the username retrieval logic across Iceberg page handlers. * fix: Add mutex protection to table UUID cache Protects concurrent access to the tableUUIDs map with sync.RWMutex. Uses read-lock for fast path when UUID already cached, and write-lock for generating new UUIDs. Includes double-check pattern to handle race condition between read-unlock and write-lock. * style: fix go fmt errors * feat(iceberg): persist table UUID in S3 Tables metadata * feat(admin): configure Iceberg port in Admin UI and commands * refactor: address review comments (flags, tests, handlers) - command/mini: fix tracking of explicit s3.port.iceberg flag - command/admin: add explicit -iceberg.port flag - admin/handlers: reuse getUsername helper - tests: use 127.0.0.1 for ephemeral ports and os.Stat for file size check * test: check error from FileStat in verify_gc_empty_test
This commit is contained in:
600
weed/s3api/iceberg/iceberg.go
Normal file
600
weed/s3api/iceberg/iceberg.go
Normal file
@@ -0,0 +1,600 @@
|
||||
// Package iceberg provides Iceberg REST Catalog API support.
|
||||
// It implements the Apache Iceberg REST Catalog specification
|
||||
// backed by S3 Tables metadata storage.
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// FilerClient provides access to the filer for storage operations.
|
||||
type FilerClient interface {
|
||||
WithFilerClient(streamingMode bool, fn func(client filer_pb.SeaweedFilerClient) error) error
|
||||
}
|
||||
|
||||
// Server implements the Iceberg REST Catalog API.
|
||||
type Server struct {
|
||||
filerClient FilerClient
|
||||
tablesManager *s3tables.Manager
|
||||
prefix string // optional prefix for routes
|
||||
}
|
||||
|
||||
// NewServer creates a new Iceberg REST Catalog server.
|
||||
func NewServer(filerClient FilerClient) *Server {
|
||||
manager := s3tables.NewManager()
|
||||
return &Server{
|
||||
filerClient: filerClient,
|
||||
tablesManager: manager,
|
||||
prefix: "",
|
||||
}
|
||||
}
|
||||
|
||||
// RegisterRoutes registers Iceberg REST API routes on the provided router.
|
||||
func (s *Server) RegisterRoutes(router *mux.Router) {
|
||||
// Configuration endpoint
|
||||
router.HandleFunc("/v1/config", s.handleConfig).Methods(http.MethodGet)
|
||||
|
||||
// Namespace endpoints
|
||||
router.HandleFunc("/v1/namespaces", s.handleListNamespaces).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces", s.handleCreateNamespace).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}", s.handleGetNamespace).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}", s.handleNamespaceExists).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}", s.handleDropNamespace).Methods(http.MethodDelete)
|
||||
|
||||
// Table endpoints
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables", s.handleListTables).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables", s.handleCreateTable).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.handleLoadTable).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.handleTableExists).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.handleDropTable).Methods(http.MethodDelete)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.handleUpdateTable).Methods(http.MethodPost)
|
||||
|
||||
// With prefix support
|
||||
router.HandleFunc("/v1/{prefix}/namespaces", s.handleListNamespaces).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces", s.handleCreateNamespace).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", s.handleGetNamespace).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", s.handleNamespaceExists).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", s.handleDropNamespace).Methods(http.MethodDelete)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables", s.handleListTables).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables", s.handleCreateTable).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.handleLoadTable).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.handleTableExists).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.handleDropTable).Methods(http.MethodDelete)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.handleUpdateTable).Methods(http.MethodPost)
|
||||
|
||||
glog.V(0).Infof("Registered Iceberg REST Catalog routes")
|
||||
}
|
||||
|
||||
// parseNamespace parses the namespace from path parameter.
|
||||
// Iceberg uses unit separator (0x1F) for multi-level namespaces.
|
||||
// Note: mux already decodes URL-encoded path parameters, so we only split by unit separator.
|
||||
func parseNamespace(encoded string) []string {
|
||||
if encoded == "" {
|
||||
return nil
|
||||
}
|
||||
parts := strings.Split(encoded, "\x1F")
|
||||
// Filter empty parts
|
||||
result := make([]string, 0, len(parts))
|
||||
for _, p := range parts {
|
||||
if p != "" {
|
||||
result = append(result, p)
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// encodeNamespace encodes namespace parts for response.
|
||||
func encodeNamespace(parts []string) string {
|
||||
return strings.Join(parts, "\x1F")
|
||||
}
|
||||
|
||||
// writeJSON writes a JSON response.
|
||||
func writeJSON(w http.ResponseWriter, status int, v interface{}) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
if v != nil {
|
||||
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||
glog.Errorf("Iceberg: failed to encode response: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// writeError writes an Iceberg error response.
|
||||
func writeError(w http.ResponseWriter, status int, errType, message string) {
|
||||
resp := ErrorResponse{
|
||||
Error: ErrorModel{
|
||||
Message: message,
|
||||
Type: errType,
|
||||
Code: status,
|
||||
},
|
||||
}
|
||||
writeJSON(w, status, resp)
|
||||
}
|
||||
|
||||
// getBucketFromPrefix extracts table bucket name from prefix parameter.
|
||||
// For now, we use the prefix as the table bucket name.
|
||||
func getBucketFromPrefix(r *http.Request) string {
|
||||
vars := mux.Vars(r)
|
||||
if prefix := vars["prefix"]; prefix != "" {
|
||||
return prefix
|
||||
}
|
||||
// Default bucket if no prefix
|
||||
return "default"
|
||||
}
|
||||
|
||||
// buildTableBucketARN builds an ARN for a table bucket.
|
||||
func buildTableBucketARN(bucketName string) string {
|
||||
arn, _ := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3_constants.AccountAdminId, bucketName)
|
||||
return arn
|
||||
}
|
||||
|
||||
// handleConfig returns catalog configuration.
|
||||
func (s *Server) handleConfig(w http.ResponseWriter, r *http.Request) {
|
||||
config := CatalogConfig{
|
||||
Defaults: map[string]string{},
|
||||
Overrides: map[string]string{},
|
||||
}
|
||||
writeJSON(w, http.StatusOK, config)
|
||||
}
|
||||
|
||||
// handleListNamespaces lists namespaces in a catalog.
|
||||
func (s *Server) handleListNamespaces(w http.ResponseWriter, r *http.Request) {
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
// Use S3 Tables manager to list namespaces
|
||||
var resp s3tables.ListNamespacesResponse
|
||||
req := &s3tables.ListNamespacesRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
MaxNamespaces: 1000,
|
||||
}
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "ListNamespaces", req, &resp, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
glog.V(1).Infof("Iceberg: ListNamespaces error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Convert to Iceberg format
|
||||
namespaces := make([]Namespace, 0, len(resp.Namespaces))
|
||||
for _, ns := range resp.Namespaces {
|
||||
namespaces = append(namespaces, Namespace(ns.Namespace))
|
||||
}
|
||||
|
||||
result := ListNamespacesResponse{
|
||||
Namespaces: namespaces,
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleCreateNamespace creates a new namespace.
|
||||
func (s *Server) handleCreateNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
var req CreateNamespaceRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
if len(req.Namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
// Use S3 Tables manager to create namespace
|
||||
createReq := &s3tables.CreateNamespaceRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: req.Namespace,
|
||||
}
|
||||
var createResp s3tables.CreateNamespaceResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "CreateNamespace", createReq, &createResp, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "already exists") {
|
||||
writeError(w, http.StatusConflict, "NamespaceAlreadyExistsException", err.Error())
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: CreateNamespace error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
result := CreateNamespaceResponse{
|
||||
Namespace: req.Namespace,
|
||||
Properties: req.Properties,
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleGetNamespace gets namespace metadata.
|
||||
func (s *Server) handleGetNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
// Use S3 Tables manager to get namespace
|
||||
getReq := &s3tables.GetNamespaceRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
}
|
||||
var getResp s3tables.GetNamespaceResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "GetNamespace", getReq, &getResp, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: GetNamespace error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
result := GetNamespaceResponse{
|
||||
Namespace: namespace,
|
||||
Properties: map[string]string{},
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleNamespaceExists checks if a namespace exists.
|
||||
func (s *Server) handleNamespaceExists(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
getReq := &s3tables.GetNamespaceRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
}
|
||||
var getResp s3tables.GetNamespaceResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "GetNamespace", getReq, &getResp, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: NamespaceExists error: %v", err)
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// handleDropNamespace deletes a namespace.
|
||||
func (s *Server) handleDropNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
deleteReq := &s3tables.DeleteNamespaceRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
}
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "DeleteNamespace", deleteReq, nil, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
|
||||
return
|
||||
}
|
||||
if strings.Contains(err.Error(), "not empty") {
|
||||
writeError(w, http.StatusConflict, "NamespaceNotEmptyException", "Namespace is not empty")
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: DropNamespace error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// handleListTables lists tables in a namespace.
|
||||
func (s *Server) handleListTables(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
listReq := &s3tables.ListTablesRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
MaxTables: 1000,
|
||||
}
|
||||
var listResp s3tables.ListTablesResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "ListTables", listReq, &listResp, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: ListTables error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Convert to Iceberg format
|
||||
identifiers := make([]TableIdentifier, 0, len(listResp.Tables))
|
||||
for _, t := range listResp.Tables {
|
||||
identifiers = append(identifiers, TableIdentifier{
|
||||
Namespace: namespace,
|
||||
Name: t.Name,
|
||||
})
|
||||
}
|
||||
|
||||
result := ListTablesResponse{
|
||||
Identifiers: identifiers,
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleCreateTable creates a new table.
|
||||
func (s *Server) handleCreateTable(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
var req CreateTableRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
if req.Name == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Table name is required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
// Generate UUID for the new table
|
||||
tableUUID := uuid.New().String()
|
||||
location := fmt.Sprintf("s3://%s/%s/%s", bucketName, encodeNamespace(namespace), req.Name)
|
||||
|
||||
metadata := TableMetadata{
|
||||
FormatVersion: 2,
|
||||
TableUUID: tableUUID,
|
||||
Location: location,
|
||||
}
|
||||
|
||||
// Use S3 Tables manager to create table
|
||||
createReq := &s3tables.CreateTableRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
Name: req.Name,
|
||||
Format: "ICEBERG",
|
||||
Metadata: &s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{
|
||||
TableUUID: tableUUID,
|
||||
},
|
||||
},
|
||||
}
|
||||
var createResp s3tables.CreateTableResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "CreateTable", createReq, &createResp, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "already exists") {
|
||||
writeError(w, http.StatusConflict, "TableAlreadyExistsException", err.Error())
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: CreateTable error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
result := LoadTableResult{
|
||||
MetadataLocation: createResp.MetadataLocation,
|
||||
Metadata: metadata,
|
||||
Config: map[string]string{},
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleLoadTable loads table metadata.
|
||||
func (s *Server) handleLoadTable(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
tableName := vars["table"]
|
||||
|
||||
if len(namespace) == 0 || tableName == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Namespace and table name are required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
getReq := &s3tables.GetTableRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
Name: tableName,
|
||||
}
|
||||
var getResp s3tables.GetTableResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "GetTable", getReq, &getResp, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
writeError(w, http.StatusNotFound, "NoSuchTableException", fmt.Sprintf("Table does not exist: %s", tableName))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: LoadTable error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Build table metadata
|
||||
location := fmt.Sprintf("s3://%s/%s/%s", bucketName, encodeNamespace(namespace), tableName)
|
||||
tableUUID := ""
|
||||
if getResp.Metadata != nil && getResp.Metadata.Iceberg != nil {
|
||||
tableUUID = getResp.Metadata.Iceberg.TableUUID
|
||||
}
|
||||
// Fallback if UUID is not found (e.g. for tables created before UUID persistence)
|
||||
if tableUUID == "" {
|
||||
tableUUID = uuid.New().String()
|
||||
}
|
||||
|
||||
metadata := TableMetadata{
|
||||
FormatVersion: 2,
|
||||
TableUUID: tableUUID,
|
||||
Location: location,
|
||||
}
|
||||
|
||||
result := LoadTableResult{
|
||||
MetadataLocation: getResp.MetadataLocation,
|
||||
Metadata: metadata,
|
||||
Config: map[string]string{},
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleTableExists checks if a table exists.
|
||||
func (s *Server) handleTableExists(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
tableName := vars["table"]
|
||||
|
||||
if len(namespace) == 0 || tableName == "" {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
getReq := &s3tables.GetTableRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
Name: tableName,
|
||||
}
|
||||
var getResp s3tables.GetTableResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "GetTable", getReq, &getResp, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// handleDropTable deletes a table.
|
||||
func (s *Server) handleDropTable(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
tableName := vars["table"]
|
||||
|
||||
if len(namespace) == 0 || tableName == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "Namespace and table name are required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
deleteReq := &s3tables.DeleteTableRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
Name: tableName,
|
||||
}
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "DeleteTable", deleteReq, nil, "")
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
writeError(w, http.StatusNotFound, "NoSuchTableException", fmt.Sprintf("Table does not exist: %s", tableName))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: DropTable error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// handleUpdateTable commits updates to a table.
|
||||
func (s *Server) handleUpdateTable(w http.ResponseWriter, r *http.Request) {
|
||||
// Return 501 Not Implemented
|
||||
writeError(w, http.StatusNotImplemented, "UnsupportedOperationException", "Table update/commit not implemented")
|
||||
}
|
||||
174
weed/s3api/iceberg/types.go
Normal file
174
weed/s3api/iceberg/types.go
Normal file
@@ -0,0 +1,174 @@
|
||||
// Package iceberg defines types for the Iceberg REST Catalog API.
|
||||
package iceberg
|
||||
|
||||
// CatalogConfig is returned by GET /v1/config.
|
||||
type CatalogConfig struct {
|
||||
Defaults map[string]string `json:"defaults"`
|
||||
Overrides map[string]string `json:"overrides"`
|
||||
Endpoints []string `json:"endpoints,omitempty"`
|
||||
}
|
||||
|
||||
// ErrorModel represents an Iceberg error.
|
||||
type ErrorModel struct {
|
||||
Message string `json:"message"`
|
||||
Type string `json:"type"`
|
||||
Code int `json:"code"`
|
||||
Stack []string `json:"stack,omitempty"`
|
||||
}
|
||||
|
||||
// ErrorResponse wraps an error in the standard Iceberg format.
|
||||
type ErrorResponse struct {
|
||||
Error ErrorModel `json:"error"`
|
||||
}
|
||||
|
||||
// Namespace is a reference to one or more levels of a namespace.
|
||||
type Namespace []string
|
||||
|
||||
// TableIdentifier identifies a table within a namespace.
|
||||
type TableIdentifier struct {
|
||||
Namespace Namespace `json:"namespace"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// ListNamespacesResponse is returned by GET /v1/namespaces.
|
||||
type ListNamespacesResponse struct {
|
||||
NextPageToken string `json:"next-page-token,omitempty"`
|
||||
Namespaces []Namespace `json:"namespaces"`
|
||||
}
|
||||
|
||||
// CreateNamespaceRequest is sent to POST /v1/namespaces.
|
||||
type CreateNamespaceRequest struct {
|
||||
Namespace Namespace `json:"namespace"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
// CreateNamespaceResponse is returned by POST /v1/namespaces.
|
||||
type CreateNamespaceResponse struct {
|
||||
Namespace Namespace `json:"namespace"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
// GetNamespaceResponse is returned by GET /v1/namespaces/{namespace}.
|
||||
type GetNamespaceResponse struct {
|
||||
Namespace Namespace `json:"namespace"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
// ListTablesResponse is returned by GET /v1/namespaces/{namespace}/tables.
|
||||
type ListTablesResponse struct {
|
||||
NextPageToken string `json:"next-page-token,omitempty"`
|
||||
Identifiers []TableIdentifier `json:"identifiers"`
|
||||
}
|
||||
|
||||
// Schema represents an Iceberg table schema.
|
||||
type Schema struct {
|
||||
Type string `json:"type"`
|
||||
SchemaID int `json:"schema-id"`
|
||||
Fields []SchemaField `json:"fields,omitempty"`
|
||||
}
|
||||
|
||||
// SchemaField represents a field in a schema.
|
||||
type SchemaField struct {
|
||||
ID int `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Required bool `json:"required"`
|
||||
Doc string `json:"doc,omitempty"`
|
||||
}
|
||||
|
||||
// PartitionSpec represents partition specification.
|
||||
type PartitionSpec struct {
|
||||
SpecID int `json:"spec-id"`
|
||||
Fields []PartitionSpecField `json:"fields,omitempty"`
|
||||
}
|
||||
|
||||
// PartitionSpecField represents a partition field.
|
||||
type PartitionSpecField struct {
|
||||
FieldID int `json:"field-id"`
|
||||
SourceID int `json:"source-id"`
|
||||
Name string `json:"name"`
|
||||
Transform string `json:"transform"`
|
||||
}
|
||||
|
||||
// SortOrder represents sort order specification.
|
||||
type SortOrder struct {
|
||||
OrderID int `json:"order-id"`
|
||||
Fields []SortOrderField `json:"fields,omitempty"`
|
||||
}
|
||||
|
||||
// SortOrderField represents a sort field.
|
||||
type SortOrderField struct {
|
||||
SourceID int `json:"source-id"`
|
||||
Transform string `json:"transform"`
|
||||
Direction string `json:"direction"`
|
||||
NullOrder string `json:"null-order"`
|
||||
}
|
||||
|
||||
// Snapshot represents an Iceberg table snapshot.
|
||||
type Snapshot struct {
|
||||
SnapshotID int64 `json:"snapshot-id"`
|
||||
ParentSnapshotID *int64 `json:"parent-snapshot-id,omitempty"`
|
||||
SequenceNumber int64 `json:"sequence-number,omitempty"`
|
||||
TimestampMs int64 `json:"timestamp-ms"`
|
||||
ManifestList string `json:"manifest-list,omitempty"`
|
||||
Summary map[string]string `json:"summary,omitempty"`
|
||||
SchemaID *int `json:"schema-id,omitempty"`
|
||||
}
|
||||
|
||||
// TableMetadata represents Iceberg table metadata.
|
||||
type TableMetadata struct {
|
||||
FormatVersion int `json:"format-version"`
|
||||
TableUUID string `json:"table-uuid"`
|
||||
Location string `json:"location,omitempty"`
|
||||
LastUpdatedMs int64 `json:"last-updated-ms,omitempty"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
Schemas []Schema `json:"schemas,omitempty"`
|
||||
CurrentSchemaID int `json:"current-schema-id,omitempty"`
|
||||
LastColumnID int `json:"last-column-id,omitempty"`
|
||||
PartitionSpecs []PartitionSpec `json:"partition-specs,omitempty"`
|
||||
DefaultSpecID int `json:"default-spec-id,omitempty"`
|
||||
LastPartitionID int `json:"last-partition-id,omitempty"`
|
||||
SortOrders []SortOrder `json:"sort-orders,omitempty"`
|
||||
DefaultSortOrderID int `json:"default-sort-order-id,omitempty"`
|
||||
Snapshots []Snapshot `json:"snapshots,omitempty"`
|
||||
CurrentSnapshotID *int64 `json:"current-snapshot-id,omitempty"`
|
||||
LastSequenceNumber int64 `json:"last-sequence-number,omitempty"`
|
||||
}
|
||||
|
||||
// CreateTableRequest is sent to POST /v1/namespaces/{namespace}/tables.
|
||||
type CreateTableRequest struct {
|
||||
Name string `json:"name"`
|
||||
Location string `json:"location,omitempty"`
|
||||
Schema *Schema `json:"schema,omitempty"`
|
||||
PartitionSpec *PartitionSpec `json:"partition-spec,omitempty"`
|
||||
WriteOrder *SortOrder `json:"write-order,omitempty"`
|
||||
StageCreate bool `json:"stage-create,omitempty"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
// LoadTableResult is returned by GET/POST table endpoints.
|
||||
type LoadTableResult struct {
|
||||
MetadataLocation string `json:"metadata-location,omitempty"`
|
||||
Metadata TableMetadata `json:"metadata"`
|
||||
Config map[string]string `json:"config,omitempty"`
|
||||
}
|
||||
|
||||
// CommitTableRequest is sent to POST /v1/namespaces/{namespace}/tables/{table}.
|
||||
type CommitTableRequest struct {
|
||||
Identifier *TableIdentifier `json:"identifier,omitempty"`
|
||||
Requirements []TableRequirement `json:"requirements"`
|
||||
Updates []TableUpdate `json:"updates"`
|
||||
}
|
||||
|
||||
// TableRequirement represents a requirement for table commit.
|
||||
type TableRequirement struct {
|
||||
Type string `json:"type"`
|
||||
Ref string `json:"ref,omitempty"`
|
||||
SnapshotID *int64 `json:"snapshot-id,omitempty"`
|
||||
}
|
||||
|
||||
// TableUpdate represents an update to table metadata.
|
||||
type TableUpdate struct {
|
||||
Action string `json:"action"`
|
||||
// Additional fields depend on the action type
|
||||
}
|
||||
@@ -396,6 +396,7 @@ func (h *S3TablesHandler) handleGetTable(w http.ResponseWriter, r *http.Request,
|
||||
OwnerAccountID: metadata.OwnerAccountID,
|
||||
MetadataLocation: metadata.MetadataLocation,
|
||||
VersionToken: metadata.VersionToken,
|
||||
Metadata: metadata.Metadata,
|
||||
}
|
||||
|
||||
h.writeJSON(w, http.StatusOK, resp)
|
||||
|
||||
@@ -135,7 +135,8 @@ type IcebergSchema struct {
|
||||
}
|
||||
|
||||
type IcebergMetadata struct {
|
||||
Schema IcebergSchema `json:"schema"`
|
||||
Schema IcebergSchema `json:"schema"`
|
||||
TableUUID string `json:"tableUuid,omitempty"`
|
||||
}
|
||||
|
||||
type TableMetadata struct {
|
||||
@@ -177,15 +178,16 @@ type GetTableRequest struct {
|
||||
}
|
||||
|
||||
type GetTableResponse struct {
|
||||
Name string `json:"name"`
|
||||
TableARN string `json:"tableARN"`
|
||||
Namespace []string `json:"namespace"`
|
||||
Format string `json:"format"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
ModifiedAt time.Time `json:"modifiedAt"`
|
||||
OwnerAccountID string `json:"ownerAccountId"`
|
||||
MetadataLocation string `json:"metadataLocation,omitempty"`
|
||||
VersionToken string `json:"versionToken"`
|
||||
Name string `json:"name"`
|
||||
TableARN string `json:"tableARN"`
|
||||
Namespace []string `json:"namespace"`
|
||||
Format string `json:"format"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
ModifiedAt time.Time `json:"modifiedAt"`
|
||||
OwnerAccountID string `json:"ownerAccountId"`
|
||||
MetadataLocation string `json:"metadataLocation,omitempty"`
|
||||
VersionToken string `json:"versionToken"`
|
||||
Metadata *TableMetadata `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type ListTablesRequest struct {
|
||||
|
||||
Reference in New Issue
Block a user