gsh-digital-services/seaweedFS
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
11,501 Commits 1 Branch 0 Tags
bed0a64693cdfd9bbcd11ad6f813fbd35af08587
Commit Graph

913 Commits

Author SHA1 Message Date
Lisandro Pin
bed0a64693 New needle_map.CompactMap() implementation for reduced memory usage (#6842) 
* Rework `needle_map.CompactMap()` to maximize memory efficiency.

* Use a memory-efficient structure for `CompactMap` needle value entries.

This slightly complicates the code, but makes a **massive** difference
in memory efficiency - preliminary results show a ~30% reduction in
heap usage, with no measurable performance impact otherwise.

* Clean up type for `CompactMap` chunk IDs.

* Add a small comment description for `CompactMap()`.

* Add the old version of `CompactMap()` for comparison purposes.
 2025-06-05 14:03:29 -07:00
chrislu
35f0daa198 the isFsync parameter is essentially IsAsyncWrite and it needs to be turned off if s.isStopping 
d8c574a5ef (r159132764)
 2025-06-05 00:19:10 -07:00
chrislu
bd4891a117 change version directory 2025-06-03 22:46:10 -07:00
Chris Lu
7151a54b28 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2025-06-02 23:57:54 -07:00
Chris Lu
b25561d0d7 3.89 2025-06-02 23:56:58 -07:00
Chris Lu
d40746f34e fix insert beyond look back window (#6838) 2025-06-02 23:43:01 -07:00
Lisandro Pin
7204731749 Minor fix for the CompactMap() performance test. (#6836) 
Per-entry memory usage is based on `TotalAllocs`, which is incorrect - that
value is a cummulative of heap usage, which doesn't decrease when objects
are freeed.

`Allocs` is instead an accurate represeentation of actual memory usage
at the time metrics are reported.
 2025-06-02 17:09:01 -07:00
Chris Lu
90802cb201 revert part of d8c574a5ef (#6829) 2025-06-01 12:27:49 -07:00
Lisandro Pin
9ffc8bcb54 Further improve memory usage of needle_map.CompactMap(). (#6825) 2025-05-28 11:42:00 -07:00
Lisandro Pin
2e1506c31e Rewrite needle_map.CompactMap() for more efficient memory usage (#6813) 2025-05-23 07:05:08 -07:00
dongxufeng
ff878a542d correctly report volume with input/output error to master (#6790) 
* correctly capture io error and report to master

* code fix

* check io error by error.Is

---------

Co-authored-by: dongxu_feng <dongxu_feng@intsig.net>
 2025-05-15 00:56:43 -07:00
chrislu
d8c574a5ef fix fsync logic 2025-05-14 01:33:36 -07:00
Lisandro Pin
dddb0f0ae5 Fix update of SeaweedFS_volumeServer_volumes gauge metrics when EC shards are unmounted (#6776) 2025-05-09 10:15:34 -07:00
Quentin D.
2ae5b480a6 Use the correct constant when computing the offset in SearchNeedleFromSortedIndex (#6771) 
NeedleHeaderSize happen to have the same size as NeedleMapEntrySize, except when running the 5 bytes offset variant of Seaweedfs, because it does not contain OffsetSize. This causes ECX corruption on deletes, due to the drifting offset computation (offset is always computed on a basis of 16 bytes per record instead of 17 bytes)

Signed-off-by: Quentin Devos <4972091+Okhoshi@users.noreply.github.com>
 2025-05-09 08:47:53 -07:00
bwlfhu
0e08b83521 Fix volume ttl (#6683) 2025-04-02 07:59:21 -07:00
dependabot[bot]
216c52e377 chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679) 
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0

Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: gocloud.dev
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix error

* fix printing errors

* Update go.mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2025-03-31 21:42:54 -07:00
NinaLua
271505717e chore: make function comment match function name (#6607) 2025-03-07 00:45:47 -08:00
Aleksey Kosov
ef4eda0761 added re-generating and writing the Volume UUID if it is empty (#6568) 2025-02-24 07:58:43 -08:00
Lisandro Pin
7913681297 ec.encode: Display a warning on EC balancing if no replica placement settings are found. (#6487) 2025-01-29 08:50:19 -08:00
chrislu
be15fee8e7 show error if backend is mis-configured 
related to https://github.com/seaweedfs/seaweedfs/discussions/6472
 2025-01-23 09:18:54 -08:00
ludwigxia
5452405a81 cloud tier: support for Alibaba Cloud OSS (#6466) 2025-01-22 00:48:55 -08:00
chrislu
de09a82a32 go fmt 2024-12-19 19:25:27 -08:00
chrislu
bfd59dd579 adjust import 2024-12-19 19:25:06 -08:00
chrislu
ec155022e7 "golang.org/x/exp/slices" => "slices" and go fmt 2024-12-19 19:25:06 -08:00
coffeecloudgit
c531e2eb0e Update disk_location_ec.go (#6359) 
Add lock for l.ecVolumes
 2024-12-15 22:05:47 -08:00
coffeecloudgit
7eb8257126 Update store.go (#6358) 
Add lock for location.ecVolumes And single delete
 2024-12-15 22:05:22 -08:00
Konstantin Lebedev
ff1392f7f4 [shell] use constant for hdd of type (#6337) 
use constant for hdd of type
 2024-12-10 08:43:59 -08:00
chrislu
f3dde99796 adjust error message 2024-12-05 09:33:50 -08:00
Trim21
fd6c787c02 fix EcVolumes sorting in volume UI (#6275) 
* Update store_ec.go

* fix: EcVolume sorting not work

* use stdlib

* revert
 2024-11-23 22:43:08 -08:00
chrislu
75f5afa571 fix compilation 2024-11-21 00:41:22 -08:00
chrislu
1c143cfcea display actual offset value 2024-11-21 00:38:03 -08:00
chrislu
c9f3448692 ReadAt may return io.EOF t end of file 
related to https://github.com/seaweedfs/seaweedfs/issues/6219
 2024-11-21 00:37:38 -08:00
chrislu
be09d08eb6 adjust error message 2024-11-20 23:49:01 -08:00
chrislu
96f1a2e01a adjust error message 2024-11-19 14:16:06 -08:00
Lisandro Pin
f2db746690 Introduce logic to resolve volume replica placement within EC rebalancing. (#6254) 
* Rename `command_ec_encode_test.go` to `command_ec_common_test.go`.

All tests defined in this file are now for `command_ec_common.go`.

* Minor code cleanups.

- Fix broken `ec.balance` test.
- Rework integer ceiling division to not use floats, which can introduce precision errors.

* Introduce logic to resolve volume replica placement within EC rebalancing.

This will be used to make rebalancing logic topology-aware.

* Give shell.EcNode.dc a dedicated DataCenterId type.
 2024-11-18 18:05:06 -08:00
Kevin Bulteel
4a0dc883cf Fix/parse upload filename (#6241) 
* fix: parse filename in PUT + refactor

* fix: master iu public url with http

* fix: better parsing and handle disposition header

* fix: take mime type from file extension if not set
 2024-11-15 11:25:18 -08:00
Eugeniy E. Mikhailov
3003c9e17e added backward compatible CRC check and extra explanatory comments (#6233) 2024-11-12 13:56:14 -08:00
chrislu
9873b033d1 backward compatible vif loading 2024-10-28 19:44:30 -07:00
chrislu
ae5bd0667a rename proto field from DestroyTime to expire_at_sec 
For TTL volume converted into EC volume, this change may leave the volumes staying.
 2024-10-24 21:35:11 -07:00
chrislu
d38456dc06 do not delete collection metrics when unmount a volume 
fix https://github.com/seaweedfs/seaweedfs/issues/6139
 2024-10-22 00:44:52 -07:00
dsd
1e13b6879c fix(volume): to avoid duplicate write a same needle (#6138) 
fix WriteNeedleBlob to avoid duplicate write a same needle

Co-authored-by: 邓书东 <shudong_deng@hhnb2024010108.intsig.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2024-10-18 01:20:50 -07:00
chrislu
f11567816e Revert "fix: volume data integrity check (#6118)" 
This reverts commit e12b9ab746.
 2024-10-17 18:52:30 -07:00
wusong
e12b9ab746 fix: volume data integrity check (#6118) 
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2024-10-14 12:17:41 -07:00
chrislu
7dd480acad handle missing leading zero padding in replica 2024-10-01 10:16:21 -07:00
LHHDZ
4dc33cc143 fix unclaimed spaces calculation when volumePreallocate is enabled (#6063) 
the calculation of `unclaimedSpaces` only needs to subtract `unusedSpace` when `preallocate` is not enabled.

Signed-off-by: LHHDZ <shichanglin5@qq.com>
 2024-09-24 23:04:18 -07:00
Max Denushev
d056c0ddf2 fix(volume): don't persist RO state in specific cases (#6058) 
* fix(volume): don't persist RO state in specific cases

* fix(volume): writable always persist
 2024-09-24 16:15:54 -07:00
chrislu
4ec6eab004 fix max volume count auto setting 
related to https://github.com/seaweedfs/seaweedfs/issues/6041#issuecomment-2360448182
 2024-09-19 09:13:08 -07:00
Konstantin Lebedev
6d4f5c94ac [volume] remove truncate idx file if size not healthy (#6043) 2024-09-19 01:04:08 -07:00
Eugeniy E. Mikhailov
bc01f09e37 Do CRC check if the buffer contains the full needle data before it is sent (#5980) 2024-09-05 17:14:55 -07:00
Bruce
f9e141a412 persist readonly state to volume info (#5977) 2024-09-05 07:58:24 -07:00