Commit Graph

1251 Commits

Author SHA1 Message Date
Chris Lu
69553e5ba6 convert error fromating to %w everywhere (#6995) 2025-07-16 23:39:27 -07:00
chrislu
64c5dde2f3 support multiple masters
fix https://github.com/seaweedfs/seaweedfs/issues/6988
2025-07-15 10:51:07 -07:00
Chris Lu
51543bbb87 Admin UI: Add message queue to admin UI (#6958)
* add a menu item "Message Queue"

* add a menu item "Message Queue"
  * move the "brokers" link under it.
  * add "topics", "subscribers". Add pages for them.

* refactor

* show topic details

* admin display publisher and subscriber info

* remove publisher and subscribers from the topic row pull down

* collecting more stats from publishers and subscribers

* fix layout

* fix publisher name

* add local listeners for mq broker and agent

* render consumer group offsets

* remove subscribers from left menu

* topic with retention

* support editing topic retention

* show retention when listing topics

* create bucket

* Update s3_buckets_templ.go

* embed the static assets into the binary

fix https://github.com/seaweedfs/seaweedfs/issues/6964
2025-07-11 10:19:27 -07:00
chalet
804979d68b [Enhancement] support fix for remote files with command fix (#6961) 2025-07-10 06:13:16 -07:00
chrislu
14859f0e8c add mq agent options to server.go 2025-07-09 09:02:26 -07:00
zuzuviewer
8fa1a69f8c * Fix undefined http serve behaiver (#6943) 2025-07-07 22:48:12 -07:00
chrislu
8493871406 remove sqlite due to GOOS support for freebsd and openbsd 2025-07-06 21:50:08 -07:00
Chris Lu
aa66852304 Admin UI add maintenance menu (#6944)
* add ui for maintenance

* valid config loading. fix workers page.

* refactor

* grpc between admin and workers

* add a long-running bidirectional grpc call between admin and worker
* use the grpc call to heartbeat
* use the grpc call to communicate
* worker can remove the http client
* admin uses http port + 10000 as its default grpc port

* one task one package

* handles connection failures gracefully with exponential backoff

* grpc with insecure tls

* grpc with optional tls

* fix detecting tls

* change time config from nano seconds to seconds

* add tasks with 3 interfaces

* compiles reducing hard coded

* remove a couple of tasks

* remove hard coded references

* reduce hard coded values

* remove hard coded values

* remove hard coded from templ

* refactor maintenance package

* fix import cycle

* simplify

* simplify

* auto register

* auto register factory

* auto register task types

* self register types

* refactor

* simplify

* remove one task

* register ui

* lazy init executor factories

* use registered task types

* DefaultWorkerConfig remove hard coded task types

* remove more hard coded

* implement get maintenance task

* dynamic task configuration

* "System Settings" should only have system level settings

* adjust menu for tasks

* ensure menu not collapsed

* render job configuration well

* use templ for ui of task configuration

* fix ordering

* fix bugs

* saving duration in seconds

* use value and unit for duration

* Delete WORKER_REFACTORING_PLAN.md

* Delete maintenance.json

* Delete custom_worker_example.go

* remove address from workers

* remove old code from ec task

* remove creating collection button

* reconnect with exponential backoff

* worker use security.toml

* start admin server with tls info from security.toml

* fix "weed admin" cli description
2025-07-06 13:57:02 -07:00
Konstantin Lebedev
93007c1842 [volume] refactor and add metrics for flight upload and download data limit condition (#6920)
* refactor concurrentDownloadLimit

* fix loop

* fix cmdServer

* fix: resolve conversation pr 6920

* Changes logging function (#6919)

* updated logging methods for stores

* updated logging methods for stores

* updated logging methods for filer

* updated logging methods for uploader and http_util

* updated logging methods for weed server

---------

Co-authored-by: akosov <a.kosov@kryptonite.ru>

* Improve lock ring (#6921)

* fix flaky lock ring test

* add more tests

* fix: build

* fix: rm import util/version

* fix: serverOptions

* refactoring

---------

Co-authored-by: Aleksey Kosov <rusyak777@list.ru>
Co-authored-by: akosov <a.kosov@kryptonite.ru>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2025-07-02 18:03:49 -07:00
Chris Lu
1db7c2b8aa Add credential storage (#6938)
* add credential store interface

* load credential.toml

* lint

* create credentialManager with explicit store type

* add type name

* InitializeCredentialManager

* remove unused functions

* fix missing import

* fix import

* fix nil configuration
2025-07-02 18:03:17 -07:00
Chris Lu
1defee3d68 Add admin component (#6928)
* init version

* relocate

* add s3 bucket link

* refactor handlers into weed/admin folder

* fix login logout

* adding favicon

* remove fall back to http get topology

* grpc dial option, disk total capacity

* show filer count

* fix each volume disk usage

* add filers to dashboard

* adding hosts, volumes, collections

* refactor code and menu

* remove "refresh" button

* fix data for collections

* rename cluster hosts into volume servers

* add masters, filers

* reorder

* adding file browser

* create folder and upload files

* add filer version, created at time

* remove mock data

* remove fields

* fix submenu item highlighting

* fix bucket creation

* purge files

* delete multiple

* fix bucket creation

* remove region from buckets

* add object store with buckets and users

* rendering permission

* refactor

* get bucket objects and size

* link to file browser

* add file size and count for collections page

* paginate the volumes

* fix possible SSRF

https://github.com/seaweedfs/seaweedfs/pull/6928/checks?check_run_id=45108469801

* Update weed/command/admin.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/command/admin.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix build

* import

* remove filer CLI option

* remove filer option

* remove CLI options

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-01 01:28:09 -07:00
chrislu
166e36bcd3 use telemetry.seaweedfs.com 2025-06-28 19:48:03 -07:00
Chris Lu
a1aab8a083 add telemetry (#6926)
* add telemetry

* fix go mod

* add default telemetry server url

* Update README.md

* replace with broker count instead of s3 count

* Update telemetry.pb.go

* github action to deploy
2025-06-28 14:11:55 -07:00
chrislu
5e79436498 reference seaweedfs.com 2025-06-26 11:09:17 -07:00
chrislu
ab49540d2b use master.toml value if not empty
fix https://github.com/seaweedfs/seaweedfs/issues/6922
2025-06-25 17:54:56 -07:00
chrislu
2f1b3d68d7 pass volume version when creating a volume 2025-06-19 01:15:25 -07:00
Erik Jacobson
77397be070 allow distro mount command to use disableXAttr (#6872) 2025-06-16 08:45:05 -07:00
chrislu
c79e73aa2a mount: complete fix for freebsd
fix https://github.com/seaweedfs/seaweedfs/issues/6645
2025-06-12 08:19:47 -07:00
chrislu
2f3de5e199 fix build 2025-06-03 22:50:45 -07:00
chrislu
bd4891a117 change version directory 2025-06-03 22:46:10 -07:00
Luna Yao
5e354a18a3 Update filer upsert sqlstring for postgresql (#6835) 2025-06-02 10:46:50 -07:00
Aleksey Kosov
283d9e0079 Add context with request (#6824) 2025-05-28 11:34:02 -07:00
Weihao Jiang
ea70d17c5f Make fuse command linux/MacOS only (#6811) 2025-05-23 08:21:46 -07:00
bwlfhu
f1181f1121 Fix mysql tls enable (#6807) 2025-05-22 21:55:21 -07:00
chrislu
f3d44b1eb6 Update fuse.go
Changed the signal from SIGUSR1 to SIGTERM. This should fix the compilation error since SIGTERM is available on all Unix-like systems including macOS. The functionality remains similar - it will still allow the master process to wait for a signal before exiting, just using a different signal type.
2025-05-22 11:32:08 -07:00
chrislu
25d6310307 fix error format 2025-05-22 09:56:31 -07:00
Weihao Jiang
874b4a5535 Ensure weed fuse master process exits after mounted (#6809)
* Ensure fuse master process wait for mounted

* Validate parent PID input in fuse command
2025-05-22 09:50:07 -07:00
Aleksey Kosov
165af32d6b added context to filer_client method calls (#6808)
Co-authored-by: akosov <a.kosov@kryptonite.ru>
2025-05-22 09:46:49 -07:00
Mohamed Sekour
93aed187e9 Add SFTP Server Support (#6753)
* Add SFTP Server Support

Signed-off-by: Mohamed Sekour <mohamed.sekour@exfo.com>

* fix s3 tests and helm lint

Signed-off-by: Mohamed Sekour <mohamed.sekour@exfo.com>

* increase  helm chart version

* adjust version

---------

Signed-off-by: Mohamed Sekour <mohamed.sekour@exfo.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2025-05-05 11:43:49 -07:00
ARibster
a2c5510ae1 Addendum to issue #6733 (#6756)
add s3.ip.bind command line parameter to filer
2025-05-05 10:30:54 -07:00
chrislu
ed15357c7a adds tls examples
related to https://github.com/seaweedfs/seaweedfs/pull/6738
2025-05-04 05:46:33 -07:00
ARibster
d424cbbcd5 fix: S3 Feature: please add s3.idleTimeout command line parameter #6746 (#6747)
ildeTimeout command line parameter for s3
2025-05-01 13:09:47 -07:00
ARibster
9f4be83e88 fix #6733 add s3.ip.bind command line parameter (#6741) 2025-04-28 04:49:01 -07:00
Konstantin Lebedev
aa9121aa9d Revert "[filer] void panic if s3opt.tlsCACertificate is nill" (#6694)
Revert "[filer] void panic if s3opt.tlsCACertificate is nill (#6685)"

This reverts commit 4dca59ed27.
2025-04-07 11:09:40 -07:00
Konstantin Lebedev
4dca59ed27 [filer] void panic if s3opt.tlsCACertificate is nill (#6685)
* fix issues/6684

* add s3.cacert.file option for filer
2025-04-07 09:45:49 -07:00
MaratKarimov
ba3afd1803 Tarantool filer store (#6669)
Co-authored-by: Marat Karimov <m.karimov@digitalms.ru>
2025-03-29 21:12:06 -07:00
Chris Lu
669a3917af add lua redis store to ensure atomicity (#6671) 2025-03-28 11:27:39 -07:00
chrislu
138b66231a fix help message 2025-03-25 09:30:56 -07:00
Karol Będkowski
95746f48e6 feat/fix: add s3.cacert.file/tlsVerifyClientCert to filer command (#6659)
This prevent crash filler with nil pointer dereference as s3 expect this
parameters.

New two parameters are added to filer command - copy of s3 parameters:
- s3.cacert.file - path to the TLS CA certificate file
- s3.tlsVerifyClientCert - whether to verify the client's certificate
2025-03-24 10:14:31 -07:00
Weihao Jiang
75ef324533 Allow standalone webdav server to bind specific address (#6403)
* Allow standalone webdav to bind specific address

* Rename bindIp to ipBind

* Bind embedded webdav IP to its parent
2025-03-17 23:01:26 -07:00
Chris Lu
02773a6107 Accumulated changes for message queue (#6600)
* rename

* set agent address

* refactor

* add agent sub

* pub messages

* grpc new client

* can publish records via agent

* send init message with session id

* fmt

* check cancelled request while waiting

* use sessionId

* handle possible nil stream

* subscriber process messages

* separate debug port

* use atomic int64

* less logs

* minor

* skip io.EOF

* rename

* remove unused

* use saved offsets

* do not reuse session, since always session id is new after restart

remove last active ts from SessionEntry

* simplify printing

* purge unused

* just proxy the subscription, skipping the session step

* adjust offset types

* subscribe offset type and possible value

* start after the known tsns

* avoid wrongly set startPosition

* move

* remove

* refactor

* typo

* fix

* fix changed path
2025-03-09 23:49:42 -07:00
lizhengui007
61249d8dde clear cache directory when mount exits (#6605)
Signed-off-by: lizhengui <lizhengui@virtaitech.com>
Co-authored-by: lizhengui <lizhengui@virtaitech.com>
2025-03-06 08:14:17 -08:00
chrislu
314f211260 Update filer.toml 2025-02-26 17:52:45 -08:00
chrislu
9bd4c47299 update redis support status 2025-02-26 17:48:44 -08:00
chrislu
4613186e8a not planning to redis3 2025-02-26 17:35:31 -08:00
Chris Lu
df436d342b add cassandra2 (#6582) 2025-02-26 17:29:22 -08:00
Karol Będkowski
441614b386 fix: s3 command ignore -tlsVerifyClientCert and -cacert.file arguments (#6547)
s3 command ignore tlsVerifyClientCert and cacert.file arguments from
command line. On startS3Server instead of use real values (in s3opt),
default values (from s3Options, always empty) are checked.

Now on right values are checked and if user provide this arguments
RequireAndVerifyClientCert is set and/or ca certificate is loaded.
2025-02-13 13:27:38 -08:00
Lisandro Pin
e8d8bfcccc Nit: remove missing newlines on weed shell commands output. (#6524)
Nit: remove missing newlines on `weed` commands output.
2025-02-07 10:27:04 -08:00
Ethan Mosbaugh
9ebc132ffc fix: bucket-hook fails with gnu wget (#6521) 2025-02-06 23:11:17 -08:00
Chris Lu
cc05874d06 Add message queue agent (#6463)
* scaffold message queue agent

* adjust proto, add mq_agent

* add agent client implementation

* remove unused function

* agent publish server implementation

* adding agent
2025-01-20 22:19:27 -08:00