master
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Chris Lu
|
e6ee293c17 |
Add table operations test (#8241)
* Add Trino blog operations test * Update test/s3tables/catalog_trino/trino_blog_operations_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * feat: add table bucket path helpers and filer operations - Add table object root and table location mapping directories - Implement ensureDirectory, upsertFile, deleteEntryIfExists helpers - Support table location bucket mapping for S3 access * feat: manage table bucket object roots on creation/deletion - Create .objects directory for table buckets on creation - Clean up table object bucket paths on deletion - Enable S3 operations on table bucket object roots * feat: add table location mapping for Iceberg REST - Track table location bucket mappings when tables are created/updated/deleted - Enable location-based routing for S3 operations on table data * feat: route S3 operations to table bucket object roots - Route table-s3 bucket names to mapped table paths - Route table buckets to object root directories - Support table location bucket mapping lookup * feat: emit table-s3 locations from Iceberg REST - Generate unique table-s3 bucket names with UUID suffix - Store table metadata under table bucket paths - Return table-s3 locations for Trino compatibility * fix: handle missing directories in S3 list operations - Propagate ErrNotFound from ListEntries for non-existent directories - Treat missing directories as empty results for list operations - Fixes Trino non-empty location checks on table creation * test: improve Trino CSV parsing for single-value results - Sanitize Trino output to skip jline warnings - Handle single-value CSV results without header rows - Strip quotes from numeric values in tests * refactor: use bucket path helpers throughout S3 API - Replace direct bucket path operations with helper functions - Leverage centralized table bucket routing logic - Improve maintainability with consistent path resolution * fix: add table bucket cache and improve filer error handling - Cache table bucket lookups to reduce filer overhead on repeated checks - Use filer_pb.CreateEntry and filer_pb.UpdateEntry helpers to check resp.Error - Fix delete order in handler_bucket_get_list_delete: delete table object before directory - Make location mapping errors best-effort: log and continue, don't fail API - Update table location mappings to delete stale prior bucket mappings on update - Add 1-second sleep before timestamp time travel query to ensure timestamps are in past - Fix CSV parsing: examine all lines, not skip first; handle single-value rows * fix: properly handle stale metadata location mapping cleanup - Capture oldMetadataLocation before mutation in handleUpdateTable - Update updateTableLocationMapping to accept both old and new locations - Use passed-in oldMetadataLocation to detect location changes - Delete stale mapping only when location actually changes - Pass empty string for oldLocation in handleCreateTable (new tables have no prior mapping) - Improve logging to show old -> new location transitions * refactor: cleanup imports and cache design - Remove unused 'sync' import from bucket_paths.go - Use filer_pb.UpdateEntry helper in setExtendedAttribute and deleteExtendedAttribute for consistent error handling - Add dedicated tableBucketCache map[string]bool to BucketRegistry instead of mixing concerns with metadataCache - Improve cache separation: table buckets cache is now separate from bucket metadata cache * fix: improve cache invalidation and add transient error handling Cache invalidation (critical fix): - Add tableLocationCache to BucketRegistry for location mapping lookups - Clear tableBucketCache and tableLocationCache in RemoveBucketMetadata - Prevents stale cache entries when buckets are deleted/recreated Transient error handling: - Only cache table bucket lookups when conclusive (found or ErrNotFound) - Skip caching on transient errors (network, permission, etc) - Prevents marking real table buckets as non-table due to transient failures Performance optimization: - Cache tableLocationDir results to avoid repeated filer RPCs on hot paths - tableLocationDir now checks cache before making expensive filer lookups - Cache stores empty string for 'not found' to avoid redundant lookups Code clarity: - Add comment to deleteDirectory explaining DeleteEntry response lacks Error field * go fmt * fix: mirror transient error handling in tableLocationDir and optimize bucketDir Transient error handling: - tableLocationDir now only caches definitive results - Mirrors isTableBucket behavior to prevent treating transient errors as permanent misses - Improves reliability on flaky systems or during recovery Performance optimization: - bucketDir avoids redundant isTableBucket call via bucketRoot - Directly use s3a.option.BucketsPath for regular buckets - Saves one cache lookup for every non-table bucket operation * fix: revert bucketDir optimization to preserve bucketRoot logic The optimization to directly use BucketsPath bypassed bucketRoot's logic and caused issues with S3 list operations on delimiter+prefix cases. Revert to using path.Join(s3a.bucketRoot(bucket), bucket) which properly handles all bucket types and ensures consistent path resolution across the codebase. The slight performance cost of an extra cache lookup is worth the correctness and consistency benefits. * feat: move table buckets under /buckets Add a table-bucket marker attribute, reuse bucket metadata cache for table bucket detection, and update list/validation/UI/test paths to treat table buckets as /buckets entries. * Fix S3 Tables code review issues - handler_bucket_create.go: Fix bucket existence check to properly validate entryResp.Entry before setting s3BucketExists flag (nil Entry should not indicate existing bucket) - bucket_paths.go: Add clarifying comment to bucketRoot() explaining unified buckets root path for all bucket types - file_browser_data.go: Optimize by extracting table bucket check early to avoid redundant WithFilerClient call * Fix list prefix delimiter handling * Handle list errors conservatively * Fix Trino FOR TIMESTAMP query - use past timestamp Iceberg requires the timestamp to be strictly in the past. Use current_timestamp - interval '1' second instead of current_timestamp. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
2ee6e4f391 |
mount: refresh and evict hot dir cache (#8174)
* mount: refresh and evict hot dir cache * mount: guard dir update window and extend TTL * mount: reuse timestamp for cache mark * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * mount: make dir cache tuning configurable * mount: dedupe dir update notices * mount: restore invalidate-all cache helper * mount: keep hot dir tuning constants * mount: centralize cache state reset * mount: mark refresh completion time * mount: allow disabling idle eviction --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
79722bcf30 |
Add s3tables shell and admin UI (#8172)
* Add shared s3tables manager * Add s3tables shell commands * Add s3tables admin API * Add s3tables admin UI * Fix admin s3tables namespace create * Rename table buckets menu * Centralize s3tables tag validation * Reuse s3tables manager in admin * Extract s3tables list limit * Add s3tables bucket ARN helper * Remove write middleware from s3tables APIs * Fix bucket link and policy hint * Fix table tag parsing and nav link * Disable namespace table link on invalid ARN * Improve s3tables error decode * Return flag parse errors for s3tables tag * Accept query params for namespace create * Bind namespace create form data * Read s3tables JS data from DOM * s3tables: allow empty region ARN * shell: pass s3tables account id * shell: require account for table buckets * shell: use bucket name for namespaces * shell: use bucket name for tables * shell: use bucket name for tags * admin: add table buckets links in file browser * s3api: reuse s3tables tag validation * admin: harden s3tables UI handlers * fix admin list table buckets * allow admin s3tables access * validate s3tables bucket tags * log s3tables bucket metadata errors * rollback table bucket on owner failure * show s3tables bucket owner * add s3tables iam conditions * Add s3tables user permissions UI * Authorize s3tables using identity actions * Add s3tables permissions to user modal * Disambiguate bucket scope in user permissions * Block table bucket names that match S3 buckets * Pretty-print IAM identity JSON * Include tags in s3tables permission context * admin: refactor S3 Tables inline JavaScript into a separate file * s3tables: extend IAM policy condition operators support * shell: use LookupEntry wrapper for s3tables bucket conflict check * admin: handle buildBucketPermissions validation in create/update flows |
||
|
Chris Lu
|
6de6061ce9 |
admin: add cursor-based pagination to file browser (#7891)
* adjust menu items * admin: add cursor-based pagination to file browser - Implement cursor-based pagination using lastFileName parameter - Add customizable page size selector (20/50/100/200 entries) - Add compact pagination controls in header and footer - Remove summary cards for cleaner UI - Make directory names clickable to return to first page - Support forward-only navigation (Next button) - Preserve cursor position when changing page size - Remove sorting to align with filer's storage order approach * Update file_browser_templ.go * admin: remove directory icons from breadcrumbs * Update file_browser_templ.go * admin: address PR comments - Fix fragile EOF check: use io.EOF instead of string comparison - Cap page size at 200 to prevent potential DoS - Remove unused helper functions from template - Use safer templ script for page size selector to prevent XSS * admin: cleanup redundant first button * Update file_browser_templ.go * admin: remove entry counting logic * admin: remove unused variables in file browser data * admin: remove unused logic for FirstFileName and HasPrevPage * admin: remove unused TotalEntries and TotalSize fields * Update file_browser_data.go |
||
|
undefined
|
9c784cf9e2 |
fix: use path to handle urls in weed admin file browser (#7858)
* fix: use path instead of filepath to handle urls in weed admin file browser * test: add comprehensive tests for file browser path handling - Test breadcrumb generation for various path scenarios - Test path handling with forward slashes (URL compatibility) - Test parent path calculation for Windows compatibility - Test file extension handling using path.Ext - Test bucket path detection logic These tests verify that the switch from filepath to path package works correctly and handles URLs properly across all platforms. * refactor: simplify fullPath construction using path.Join Replace verbose manual path construction with path.Join which: - Handles trailing slashes automatically - Is more concise and readable - Is more robust for edge cases * fix: normalize path in ShowFileBrowser and rename generateBreadcrumbs parameter Critical fix: - Add util.CleanWindowsPath() normalization to path parameter in ShowFileBrowser handler, matching the pattern used in other file operation handlers (lines 273, 464) - This ensures Windows-style backslashes are converted to forward slashes before processing, fixing path handling issues on Windows Consistency improvement: - Rename path parameter to dir in generateBreadcrumbs function - Aligns with parameter rename in GetFileBrowser for consistent naming throughout the file * test: improve coverage for Windows path handling and production code behavior Address reviewer feedback by enhancing test quality: 1. Improved test documentation: - Added clear comments explaining what each test validates - Clarified that some tests validate expected behavior vs production code - Documented the Windows path normalization flow 2. Enhanced actual production code testing: - TestGenerateBreadcrumbs: Calls actual production function - TestBreadcrumbPathFormatting: Validates production output format - TestDirectoryNavigation: Integration-style test for complete flow 3. Added new test functions for better coverage: - TestPathJoinHandlesEdgeCases: Verifies path.Join behavior - TestWindowsPathNormalizationBehavior: Documents expected normalization - TestDirectoryNavigation: Complete navigation flow test 4. Improved test organization: - Fixed duplicate field naming issues - Better test names for clarity - More comprehensive edge case coverage These improvements ensure the fix for issue #7628 (Windows path handling) is properly validated across the complete flow from handler to path logic. * test: use actual util.CleanWindowsPath function in Windows path normalization test Address reviewer feedback by testing the actual production function: - Import util package for CleanWindowsPath - Call the real util.CleanWindowsPath() instead of reimplementing logic - Ensures test validates actual implementation, not just expected behavior - Added more test cases for edge cases (simple path, deep nesting) This change validates that the Windows path normalization in the ShowFileBrowser handler (handlers/file_browser_handlers.go:64) works correctly with the actual util.CleanWindowsPath function. * style: fix indentation in TestPathJoinHandlesEdgeCases Align t.Errorf statement inside the if block with proper indentation. The error message now correctly aligns with the if block body, maintaining consistent indentation throughout the function. * test: restore backslash validation check in TestPathJoinHandlesEdgeCases --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
687a6a6c1d |
Admin UI: Add policies (#6968)
* add policies to UI, accessing filer directly * view, edit policies * add back buttons for "users" page * remove unused * fix ui dark mode when modal is closed * bucket view details button * fix browser buttons * filer action button works * clean up masters page * fix volume servers action buttons * fix collections page action button * fix properties page * more obvious * fix directory creation file mode * Update file_browser_handlers.go * directory permission |
||
|
chrislu
|
6b706f9ccd |
rename files
*_server.go - main server files *_management.go - business logic *_data.go - data structures and types *_middleware.go - middleware logic |