gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add policy engine (#6970)

Browse Source
This commit is contained in:
Chris Lu
2025-07-13 16:21:36 -07:00
committed by GitHub
parent 1549ee2e15
commit 7cb1ca1308
33 changed files with 5565 additions and 195 deletions
Download Patch File Download Diff File Expand all files Collapse all files

279
weed/s3api/policy_engine/README_POLICY_ENGINE.md Normal file
View File

@@ -0,0 +1,279 @@
# SeaweedFS Policy Evaluation Engine
This document describes the comprehensive policy evaluation engine that has been added to SeaweedFS, providing AWS S3-compatible policy support while maintaining full backward compatibility with existing `identities.json` configuration.
## Overview
The policy engine provides:
- **Full AWS S3 policy compatibility** - JSON policies with conditions, wildcards, and complex logic
- **Backward compatibility** - Existing `identities.json` continues to work unchanged
- **Bucket policies** - Per-bucket access control policies
- **IAM policies** - User and group-level policies
- **Condition evaluation** - IP restrictions, time-based access, SSL-only, etc.
- **AWS-compliant evaluation order** - Explicit Deny > Explicit Allow > Default Deny
## Architecture
### Files Created
1. **`policy_engine/types.go`** - Core policy data structures and validation
2. **`policy_engine/conditions.go`** - Condition evaluators (StringEquals, IpAddress, etc.)
3. **`policy_engine/engine.go`** - Main policy evaluation engine
4. **`policy_engine/integration.go`** - Integration with existing IAM system
5. **`policy_engine/engine_test.go`** - Comprehensive tests
6. **`policy_engine/examples.go`** - Usage examples and documentation (excluded from builds)
7. **`policy_engine/wildcard_matcher.go`** - Optimized wildcard pattern matching
8. **`policy_engine/wildcard_matcher_test.go`** - Wildcard matching tests
### Key Components
```
PolicyEngine
├── Bucket Policies (per-bucket JSON policies)
├── User Policies (converted from identities.json + new IAM policies)
├── Condition Evaluators (IP, time, string, numeric, etc.)
└── Evaluation Logic (AWS-compliant precedence)
```
## Backward Compatibility
### Existing identities.json (No Changes Required)
Your existing configuration continues to work exactly as before:
```json
{
"identities": [
{
"name": "readonly_user",
"credentials": [{"accessKey": "key123", "secretKey": "secret123"}],
"actions": ["Read:public-bucket/*", "List:public-bucket"]
}
]
}
```
Legacy actions are automatically converted to AWS-style policies:
- `Read:bucket/*` → `s3:GetObject` on `arn:aws:s3:::bucket/*`
- `Write:bucket` → `s3:PutObject`, `s3:DeleteObject` on `arn:aws:s3:::bucket/*`
- `Admin` → `s3:*` on `arn:aws:s3:::*`
## New Capabilities
### 1. Bucket Policies
Set bucket-level policies using standard S3 API:
```bash
# Set bucket policy
curl -X PUT "http://localhost:8333/bucket?policy" \
-H "Authorization: AWS access_key:signature" \
-d '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucket/*"
}
]
}'
# Get bucket policy
curl "http://localhost:8333/bucket?policy"
# Delete bucket policy
curl -X DELETE "http://localhost:8333/bucket?policy"
```
### 2. Advanced Conditions
Support for all AWS condition operators:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::secure-bucket/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": ["192.168.1.0/24", "10.0.0.0/8"]
},
"Bool": {
"aws:SecureTransport": "true"
},
"DateGreaterThan": {
"aws:CurrentTime": "2023-01-01T00:00:00Z"
}
}
}
]
}
```
### 3. Supported Condition Operators
- **String**: `StringEquals`, `StringNotEquals`, `StringLike`, `StringNotLike`
- **Numeric**: `NumericEquals`, `NumericLessThan`, `NumericGreaterThan`, etc.
- **Date**: `DateEquals`, `DateLessThan`, `DateGreaterThan`, etc.
- **IP**: `IpAddress`, `NotIpAddress` (supports CIDR notation)
- **Boolean**: `Bool`
- **ARN**: `ArnEquals`, `ArnLike`
- **Null**: `Null`
### 4. Condition Keys
Standard AWS condition keys are supported:
- `aws:CurrentTime` - Current request time
- `aws:SourceIp` - Client IP address
- `aws:SecureTransport` - Whether HTTPS is used
- `aws:UserAgent` - Client user agent
- `s3:x-amz-acl` - Requested ACL
- `s3:VersionId` - Object version ID
- And many more...
## Policy Evaluation
### Evaluation Order (AWS-Compatible)
1. **Explicit Deny** - If any policy explicitly denies access → **DENY**
2. **Explicit Allow** - If any policy explicitly allows access → **ALLOW**
3. **Default Deny** - If no policy matches → **DENY**
### Policy Sources (Evaluated Together)
1. **Bucket Policies** - Stored per-bucket, highest priority
2. **User Policies** - Converted from `identities.json` + new IAM policies
3. **Legacy IAM** - For backward compatibility (lowest priority)
## Examples
### Public Read Bucket
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicRead",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::public-bucket/*"
}
]
}
```
### IP-Restricted Bucket
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": "arn:aws:s3:::secure-bucket/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": "192.168.1.0/24"
}
}
}
]
}
```
### SSL-Only Access
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": ["arn:aws:s3:::ssl-bucket/*", "arn:aws:s3:::ssl-bucket"],
"Condition": {
"Bool": {
"aws:SecureTransport": "false"
}
}
}
]
}
```
## Integration
### For Existing SeaweedFS Users
1. **No changes required** - Your existing setup continues to work
2. **Optional enhancement** - Add bucket policies for fine-grained control
3. **Gradual migration** - Move to full AWS policies over time
### For New Users
1. Start with either `identities.json` or AWS-style policies
2. Use bucket policies for complex access patterns
3. Full feature parity with AWS S3 policies
## Testing
Run the policy engine tests:
```bash
# Core policy tests
go test -v -run TestPolicyEngine
# Condition evaluator tests
go test -v -run TestConditionEvaluators
# Legacy compatibility tests
go test -v -run TestConvertIdentityToPolicy
# Validation tests
go test -v -run TestPolicyValidation
```
## Performance
- **Compiled patterns** - Regex patterns are pre-compiled for fast matching
- **Cached policies** - Policies are cached in memory with TTL
- **Early termination** - Evaluation stops on first explicit deny
- **Minimal overhead** - Backward compatibility with minimal performance impact
## Migration Path
### Phase 1: Backward Compatible (Current)
- Keep existing `identities.json` unchanged
- Add bucket policies as needed
- Legacy actions automatically converted to AWS policies
### Phase 2: Enhanced (Optional)
- Add advanced conditions to policies
- Use full AWS S3 policy features
- Maintain backward compatibility
### Phase 3: Full Migration (Future)
- Migrate to pure IAM policies
- Use AWS CLI/SDK for policy management
- Complete AWS S3 feature parity
## Compatibility
- ✅ **Full backward compatibility** with existing `identities.json`
- ✅ **AWS S3 API compatibility** for bucket policies
- ✅ **Standard condition operators** and keys
- ✅ **Proper evaluation precedence** (Deny > Allow > Default Deny)
- ✅ **Performance optimized** with caching and compiled patterns
The policy engine provides a seamless upgrade path from SeaweedFS's existing simple IAM system to full AWS S3-compatible policies, giving you the best of both worlds: simplicity for basic use cases and power for complex enterprise scenarios.