Files
seaweedFS/weed/admin/view/app/task_config_schema.templ
Chris Lu 5a7c74feac migrate IAM policies to multi-file storage (#8114)
* Add IAM gRPC service definition

- Add GetConfiguration/PutConfiguration for config management
- Add CreateUser/GetUser/UpdateUser/DeleteUser/ListUsers for user management
- Add CreateAccessKey/DeleteAccessKey/GetUserByAccessKey for access key management
- Methods mirror existing IAM HTTP API functionality

* Add IAM gRPC handlers on filer server

- Implement IamGrpcServer with CredentialManager integration
- Handle configuration get/put operations
- Handle user CRUD operations
- Handle access key create/delete operations
- All methods delegate to CredentialManager for actual storage

* Wire IAM gRPC service to filer server

- Add CredentialManager field to FilerOption and FilerServer
- Import credential store implementations in filer command
- Initialize CredentialManager from credential.toml if available
- Register IAM gRPC service on filer gRPC server
- Enable credential management via gRPC alongside existing filer services

* Regenerate IAM protobuf with gRPC service methods

* fix: compilation error in DeleteUser

* fix: address code review comments for IAM migration

* feat: migrate policies to multi-file layout and fix identity duplicated content

* refactor: remove configuration.json and migrate Service Accounts to multi-file layout

* refactor: standardize Service Accounts as distinct store entities and fix Admin Server persistence

* config: set ServiceAccountsDirectory to /etc/iam/service_accounts

* Fix Chrome dialog auto-dismiss with Bootstrap modals

- Add modal-alerts.js library with Bootstrap modal replacements
- Replace all 15 confirm() calls with showConfirm/showDeleteConfirm
- Auto-override window.alert() for all alert() calls
- Fixes Chrome 132+ aggressively blocking native dialogs

* Upgrade Bootstrap from 5.3.2 to 5.3.8

* Fix syntax error in object_store_users.templ - remove duplicate closing braces

* create policy

* display errors

* migrate to multi-file policies

* address PR feedback: use showDeleteConfirm and showErrorMessage in policies.templ, refine migration check

* Update policies_templ.go

* add service account to iam grpc

* iam: fix potential path traversal in policy names by validating name pattern

* iam: add GetServiceAccountByAccessKey to CredentialStore interface

* iam: implement service account support for PostgresStore

Includes full CRUD operations and efficient lookup by access key.

* iam: implement GetServiceAccountByAccessKey for filer_etc, grpc, and memory stores

Provides efficient lookup of service accounts by access key where possible,
with linear scan fallbacks for file-based stores.

* iam: remove filer_multiple support

Deleted its implementation and references in imports, scaffold config,
and core interface constants. Redundant with filer_etc.

* clear comment

* dash: robustify service account construction

- Guard against nil sa.Credential when constructing responses
- Fix Expiration logic to only set if > 0, avoiding Unix epoch 1970
- Ensure consistency across Get, Create, and Update handlers

* credential/filer_etc: improve error propagation in configuration handlers

- Return error from loadServiceAccountsFromMultiFile to callers
- Ensure listEntries errors in SaveConfiguration (cleanup logic) are
  propagated unless they are "not found" failures.
- Fixes potential silent failures during IAM configuration sync.

* credential/filer_etc: add existence check to CreateServiceAccount

Ensures consistency with other stores by preventing accidental overwrite
of existing service accounts during creation.

* credential/memory: improve store robustness and Reset logic

- Enforce ID immutability in UpdateServiceAccount to prevent orphans
- Update Reset() to also clear the policies map, ensuring full state
  cleanup for tests.

* dash: improve service account robustness and policy docs

- Wrap parent user lookup errors to preserve context
- Strictly validate Status field in UpdateServiceAccount
- Add deprecation comments to legacy policy management methods

* credential/filer_etc: protect against path traversal in service accounts

Implemented ID validation (alphanumeric, underscores, hyphens) and applied
it to Get, Save, and Delete operations to ensure no directory traversal
via saId.json filenames.

* credential/postgres: improve robustness and cleanup comments

- Removed brainstorming comments in GetServiceAccountByAccessKey
- Added missing rows.Err() check during iteration
- Properly propagate Scan and Unmarshal errors instead of swallowing them

* admin: unify UI alerts and confirmations using Bootstrap modals

- Updated modal-alerts.js with improved automated alert type detection
- Replaced native alert() and confirm() with showAlert(), showConfirm(),
  and showDeleteConfirm() across various Templ components
- Improved UX for delete operations by providing better context and styling
- Ensured consistent error reporting across IAM and Maintenance views

* admin: additional UI consistency fixes for alerts and confirmations

- Replaced native alert() and confirm() with Bootstrap modals in:
  - EC volumes (repair flow)
  - Collection details (repair flow)
  - File browser (properties and delete)
  - Maintenance config schema (save and reset)
- Improved delete confirmation in file browser with item context
- Ensured consistent success/error/info styling for all feedbacks

* make

* iam: add GetServiceAccountByAccessKey RPC and update GetConfiguration

* iam: implement GetServiceAccountByAccessKey on server and client

* iam: centralize policy and service account validation

* iam: optimize MemoryStore service account lookups with indexing

* iam: fix postgres service_accounts table and optimize lookups

* admin: refactor modal alerts and clean up dashboard logic

* admin: fix EC shards table layout mismatch

* admin: URL-encode IAM path parameters for safety

* admin: implement pauseWorker logic in maintenance view

* iam: add rows.Err() check to postgres ListServiceAccounts

* iam: standardize ErrServiceAccountNotFound across credential stores

* iam: map ErrServiceAccountNotFound to codes.NotFound in DeleteServiceAccount

* iam: refine service account store logic, errors and schema

* iam: add validation to GetServiceAccountByAccessKey

* admin: refine modal titles and ensure URL safety

* admin: address bot review comments for alerts and async usage

* iam: fix syntax error by restoring missing function declaration

* [FilerEtcStore] improve error handling in CreateServiceAccount

Refine error handling to provide clearer messages when checking for
existing service accounts.

* [PostgresStore] add nil guards and validation to service account methods

Ensure input parameters are not nil and required IDs are present
to prevent runtime panics and ensure data integrity.

* [JS] add shared IAM utility script

Consolidate common IAM operations like deleteUser and deleteAccessKey
into a shared utility script for better maintainability.

* [View] include shared IAM utilities in layout

Include iam-utils.js in the main layout to make IAM functions
available across all administrative pages.

* [View] refactor IAM logic and restore async in EC Shards view

Remove redundant local IAM functions and ensure that delete
confirmation callbacks are properly marked as async.

* [View] consolidate IAM logic in Object Store Users view

Remove redundant local definitions of deleteUser and deleteAccessKey,
relying on the shared utilities instead.

* [View] update generated templ files for UI consistency

* credential/postgres: remove redundant name column from service_accounts table

The id is already used as the unique identifier and was being copied to the name column.
This removes the name column from the schema and updates the INSERT/UPDATE queries.

* credential/filer_etc: improve logging for policy migration failures

Added Errorf log if AtomicRenameEntry fails during migration to ensure visibility of common failure points.

* credential: allow uppercase characters in service account ID username

Updated ServiceAccountIdPattern to allow [A-Za-z0-9_-]+ for the username component,
matching the actual service account creation logic which uses the parent user name directly.

* Update object_store_users_templ.go

* admin: fix ec_shards pagination to handle numeric page arguments

Updated goToPage in cluster_ec_shards.templ to accept either an Event
or a numeric page argument. This prevents errors when goToPage(1)
is called directly. Corrected both the .templ source and generated Go code.

* credential/filer_etc: improve service account storage robustness

Added nil guard to saveServiceAccount, updated GetServiceAccount
to return ErrServiceAccountNotFound for empty data, and improved
deleteServiceAccount to handle response-level Filer errors.
2026-01-26 11:28:23 -08:00

487 lines
18 KiB
Plaintext

package app
import (
"encoding/base64"
"encoding/json"
"fmt"
"reflect"
"strings"
"github.com/seaweedfs/seaweedfs/weed/admin/maintenance"
"github.com/seaweedfs/seaweedfs/weed/worker/tasks"
"github.com/seaweedfs/seaweedfs/weed/admin/config"
"github.com/seaweedfs/seaweedfs/weed/admin/view/components"
"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
)
// Helper function to convert task schema to JSON string
func taskSchemaToJSON(schema *tasks.TaskConfigSchema) string {
if schema == nil {
return "{}"
}
data := map[string]interface{}{
"fields": schema.Fields,
}
jsonBytes, err := json.Marshal(data)
if err != nil {
return "{}"
}
return string(jsonBytes)
}
// Helper function to base64 encode the JSON to avoid HTML escaping issues
func taskSchemaToBase64JSON(schema *tasks.TaskConfigSchema) string {
jsonStr := taskSchemaToJSON(schema)
return base64.StdEncoding.EncodeToString([]byte(jsonStr))
}
templ TaskConfigSchema(data *maintenance.TaskConfigData, schema *tasks.TaskConfigSchema, config interface{}) {
<div class="container-fluid">
<div class="row mb-4">
<div class="col-12">
<div class="d-flex justify-content-between align-items-center">
<h2 class="mb-0">
<i class={schema.Icon + " me-2"}></i>
{schema.DisplayName} Configuration
</h2>
<div class="btn-group">
<a href="/maintenance/config" class="btn btn-outline-secondary">
<i class="fas fa-arrow-left me-1"></i>
Back to System Config
</a>
</div>
</div>
</div>
</div>
<!-- Configuration Card -->
<div class="row">
<div class="col-12">
<div class="card">
<div class="card-header">
<h5 class="mb-0">
<i class="fas fa-cogs me-2"></i>
Task Configuration
</h5>
<p class="mb-0 text-muted small">{schema.Description}</p>
</div>
<div class="card-body">
<form id="taskConfigForm" method="POST">
<!-- Dynamically render all schema fields in defined order -->
for _, field := range schema.Fields {
@TaskConfigField(field, config)
}
<div class="d-flex gap-2">
<button type="submit" class="btn btn-primary">
<i class="fas fa-save me-1"></i>
Save Configuration
</button>
<button type="button" class="btn btn-secondary" onclick="resetToDefaults()">
<i class="fas fa-undo me-1"></i>
Reset to Defaults
</button>
</div>
</form>
</div>
</div>
</div>
</div>
<!-- Performance Notes Card -->
<div class="row mt-4">
<div class="col-12">
<div class="card">
<div class="card-header">
<h5 class="mb-0">
<i class="fas fa-info-circle me-2"></i>
Important Notes
</h5>
</div>
<div class="card-body">
<div class="alert alert-info" role="alert">
if schema.TaskName == "vacuum" {
<h6 class="alert-heading">Vacuum Operations:</h6>
<p class="mb-2"><strong>Performance:</strong> Vacuum operations are I/O intensive and may impact cluster performance.</p>
<p class="mb-2"><strong>Safety:</strong> Only volumes meeting age and garbage thresholds will be processed.</p>
<p class="mb-0"><strong>Recommendation:</strong> Monitor cluster load and adjust concurrent limits accordingly.</p>
} else if schema.TaskName == "balance" {
<h6 class="alert-heading">Balance Operations:</h6>
<p class="mb-2"><strong>Performance:</strong> Volume balancing involves data movement and can impact cluster performance.</p>
<p class="mb-2"><strong>Safety:</strong> Requires adequate server count to ensure data safety during moves.</p>
<p class="mb-0"><strong>Recommendation:</strong> Run during off-peak hours to minimize impact on production workloads.</p>
} else if schema.TaskName == "erasure_coding" {
<h6 class="alert-heading">Erasure Coding Operations:</h6>
<p class="mb-2"><strong>Performance:</strong> Erasure coding is CPU and I/O intensive. Consider running during off-peak hours.</p>
<p class="mb-2"><strong>Durability:</strong> With { fmt.Sprintf("%d+%d", erasure_coding.DataShardsCount, erasure_coding.ParityShardsCount) } configuration, can tolerate up to { fmt.Sprintf("%d", erasure_coding.ParityShardsCount) } shard failures.</p>
<p class="mb-0"><strong>Configuration:</strong> Fullness ratio should be between 0.5 and 1.0 (e.g., 0.90 for 90%).</p>
}
</div>
</div>
</div>
</div>
</div>
</div>
<script>
function resetToDefaults() {
showConfirm('Are you sure you want to reset to default configuration? This will overwrite your current settings.', function() {
// Reset form fields to their default values
const form = document.getElementById('taskConfigForm');
const schemaFields = window.taskConfigSchema ? window.taskConfigSchema.fields : {};
Object.keys(schemaFields).forEach(fieldName => {
const field = schemaFields[fieldName];
const element = document.getElementById(fieldName);
if (element && field.default_value !== undefined) {
if (field.input_type === 'checkbox') {
element.checked = field.default_value;
} else if (field.input_type === 'interval') {
// Handle interval fields with value and unit
const valueElement = document.getElementById(fieldName + '_value');
const unitElement = document.getElementById(fieldName + '_unit');
if (valueElement && unitElement && field.default_value) {
const defaultSeconds = field.default_value;
const { value, unit } = convertSecondsToTaskIntervalValueUnit(defaultSeconds);
valueElement.value = value;
unitElement.value = unit;
}
} else {
element.value = field.default_value;
}
}
});
});
}
function convertSecondsToTaskIntervalValueUnit(totalSeconds) {
if (totalSeconds === 0) {
return { value: 0, unit: 'minutes' };
}
// Check if it's evenly divisible by days
if (totalSeconds % (24 * 3600) === 0) {
return { value: totalSeconds / (24 * 3600), unit: 'days' };
}
// Check if it's evenly divisible by hours
if (totalSeconds % 3600 === 0) {
return { value: totalSeconds / 3600, unit: 'hours' };
}
// Default to minutes
return { value: totalSeconds / 60, unit: 'minutes' };
}
// Store schema data for JavaScript access (moved to after div is created)
</script>
<!-- Hidden element to store schema data -->
<div data-task-schema={ taskSchemaToBase64JSON(schema) } style="display: none;"></div>
<script>
// Load schema data now that the div exists
const base64Data = document.querySelector('[data-task-schema]').getAttribute('data-task-schema');
const jsonStr = atob(base64Data);
window.taskConfigSchema = JSON.parse(jsonStr);
</script>
}
// TaskConfigField renders a single task configuration field based on schema with typed field lookup
templ TaskConfigField(field *config.Field, config interface{}) {
if field.InputType == "interval" {
<!-- Interval field with number input + unit dropdown -->
<div class="mb-3">
<label for={ field.JSONName } class="form-label">
{ field.DisplayName }
if field.Required {
<span class="text-danger">*</span>
}
</label>
<div class="input-group">
<input
type="number"
class="form-control"
id={ field.JSONName + "_value" }
name={ field.JSONName + "_value" }
value={ fmt.Sprintf("%.0f", components.ConvertInt32SecondsToDisplayValue(getTaskConfigInt32Field(config, field.JSONName))) }
step="1"
min="1"
if field.Required {
required
}
/>
<select
class="form-select"
id={ field.JSONName + "_unit" }
name={ field.JSONName + "_unit" }
style="max-width: 120px;"
if field.Required {
required
}
>
<option
value="minutes"
if components.GetInt32DisplayUnit(getTaskConfigInt32Field(config, field.JSONName)) == "minutes" {
selected
}
>
Minutes
</option>
<option
value="hours"
if components.GetInt32DisplayUnit(getTaskConfigInt32Field(config, field.JSONName)) == "hours" {
selected
}
>
Hours
</option>
<option
value="days"
if components.GetInt32DisplayUnit(getTaskConfigInt32Field(config, field.JSONName)) == "days" {
selected
}
>
Days
</option>
</select>
</div>
if field.Description != "" {
<div class="form-text text-muted">{ field.Description }</div>
}
</div>
} else if field.InputType == "checkbox" {
<!-- Checkbox field -->
<div class="mb-3">
<div class="form-check form-switch">
<input
class="form-check-input"
type="checkbox"
id={ field.JSONName }
name={ field.JSONName }
value="on"
if getTaskConfigBoolField(config, field.JSONName) {
checked
}
/>
<label class="form-check-label" for={ field.JSONName }>
<strong>{ field.DisplayName }</strong>
</label>
</div>
if field.Description != "" {
<div class="form-text text-muted">{ field.Description }</div>
}
</div>
} else if field.InputType == "text" {
<!-- Text field -->
<div class="mb-3">
<label for={ field.JSONName } class="form-label">
{ field.DisplayName }
if field.Required {
<span class="text-danger">*</span>
}
</label>
<input
type="text"
class="form-control"
id={ field.JSONName }
name={ field.JSONName }
value={ getTaskConfigStringField(config, field.JSONName) }
placeholder={ field.Placeholder }
if field.Required {
required
}
/>
if field.Description != "" {
<div class="form-text text-muted">{ field.Description }</div>
}
</div>
} else {
<!-- Number field -->
<div class="mb-3">
<label for={ field.JSONName } class="form-label">
{ field.DisplayName }
if field.Required {
<span class="text-danger">*</span>
}
</label>
<input
type="number"
class="form-control"
id={ field.JSONName }
name={ field.JSONName }
value={ fmt.Sprintf("%.6g", getTaskConfigFloatField(config, field.JSONName)) }
placeholder={ field.Placeholder }
if field.MinValue != nil {
min={ fmt.Sprintf("%v", field.MinValue) }
}
if field.MaxValue != nil {
max={ fmt.Sprintf("%v", field.MaxValue) }
}
step={ getTaskNumberStep(field) }
if field.Required {
required
}
/>
if field.Description != "" {
<div class="form-text text-muted">{ field.Description }</div>
}
</div>
}
}
// Typed field getters for task configs - avoiding interface{} where possible
func getTaskConfigBoolField(config interface{}, fieldName string) bool {
switch fieldName {
case "enabled":
// Use reflection only for the common 'enabled' field in BaseConfig
if value := getTaskFieldValue(config, fieldName); value != nil {
if boolVal, ok := value.(bool); ok {
return boolVal
}
}
return false
default:
// For other boolean fields, use reflection
if value := getTaskFieldValue(config, fieldName); value != nil {
if boolVal, ok := value.(bool); ok {
return boolVal
}
}
return false
}
}
func getTaskConfigInt32Field(config interface{}, fieldName string) int32 {
switch fieldName {
case "scan_interval_seconds", "max_concurrent":
// Common fields that should be int/int32
if value := getTaskFieldValue(config, fieldName); value != nil {
switch v := value.(type) {
case int32:
return v
case int:
return int32(v)
case int64:
return int32(v)
}
}
return 0
default:
// For other int fields, use reflection
if value := getTaskFieldValue(config, fieldName); value != nil {
switch v := value.(type) {
case int32:
return v
case int:
return int32(v)
case int64:
return int32(v)
case float64:
return int32(v)
}
}
return 0
}
}
func getTaskConfigFloatField(config interface{}, fieldName string) float64 {
if value := getTaskFieldValue(config, fieldName); value != nil {
switch v := value.(type) {
case float64:
return v
case float32:
return float64(v)
case int:
return float64(v)
case int32:
return float64(v)
case int64:
return float64(v)
}
}
return 0.0
}
func getTaskConfigStringField(config interface{}, fieldName string) string {
if value := getTaskFieldValue(config, fieldName); value != nil {
if strVal, ok := value.(string); ok {
return strVal
}
// Convert numbers to strings for form display
switch v := value.(type) {
case int:
return fmt.Sprintf("%d", v)
case int32:
return fmt.Sprintf("%d", v)
case int64:
return fmt.Sprintf("%d", v)
case float64:
return fmt.Sprintf("%.6g", v)
case float32:
return fmt.Sprintf("%.6g", v)
}
}
return ""
}
func getTaskNumberStep(field *config.Field) string {
if field.Type == config.FieldTypeFloat {
return "0.01"
}
return "1"
}
func getTaskFieldValue(config interface{}, fieldName string) interface{} {
if config == nil {
return nil
}
// Use reflection to get the field value from the config struct
configValue := reflect.ValueOf(config)
if configValue.Kind() == reflect.Ptr {
configValue = configValue.Elem()
}
if configValue.Kind() != reflect.Struct {
return nil
}
configType := configValue.Type()
for i := 0; i < configValue.NumField(); i++ {
field := configValue.Field(i)
fieldType := configType.Field(i)
// Handle embedded structs recursively (before JSON tag check)
if field.Kind() == reflect.Struct && fieldType.Anonymous {
if value := getTaskFieldValue(field.Interface(), fieldName); value != nil {
return value
}
continue
}
// Get JSON tag name
jsonTag := fieldType.Tag.Get("json")
if jsonTag == "" {
continue
}
// Remove options like ",omitempty"
if commaIdx := strings.Index(jsonTag, ","); commaIdx > 0 {
jsonTag = jsonTag[:commaIdx]
}
// Check if this is the field we're looking for
if jsonTag == fieldName {
return field.Interface()
}
}
return nil
}