Commit Graph

157 Commits

Author SHA1 Message Date
Chris Lu
3e5d34dd67 skip md5 validation if Content-MD5 is not provided 2026-01-18 15:04:56 -08:00
Chris Lu
a473278bfa Fix: Fail fast on unsupported volume versions (#8047)
* Fix: Fail fast when initializing volume with Version 0

* Fix: Fail fast when loading unsupported volume version (e.g. 0 or 4)

* Refactor: Use IsSupportedVersion helper function for version validation
2026-01-16 19:19:18 -08:00
promalert
9012069bd7 chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code

Signed-off-by: promalert <promalert@outlook.com>

* goimports -w .

---------

Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-07 13:06:08 -08:00
Chris Lu
ec1c27a4b3 storage/needle: add bounds check for WriteNeedleBlob buffer (#7973)
* storage/needle: add bounds check for WriteNeedleBlob buffer

* storage/needle: use int offsets when checking/writing Version3 timestamp

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-01-05 19:21:57 -08:00
Chris Lu
0ede657a34 fix: return error on size mismatch in ReadNeedleMeta for consistency (#7687)
* fix: return error on size mismatch in ReadNeedleMeta for consistency

When ReadNeedleMeta encounters a size mismatch at offset >= MaxPossibleVolumeSize,
it previously just continued without returning an error, potentially using wrong data.

This fix makes ReadNeedleMeta consistent with ReadBytes (needle_read.go), which
properly returns an error in both cases:
- ErrorSizeMismatch when offset < MaxPossibleVolumeSize (to trigger retry at offset+32GB)
- A descriptive error when offset >= MaxPossibleVolumeSize (after retry failed)

Fixes #7673

* refactor: use more accurate error message for size mismatch
2025-12-09 13:17:18 -08:00
Chris Lu
cea12ba3c4 fix: prevent makeslice panic in ReadNeedleMeta with corrupted needle (#7675)
* fix: prevent makeslice panic in ReadNeedleMeta with corrupted needle

When a needle's DataSize in the .dat file is corrupted to a very large
value, the calculation of metaSize can become negative, causing a panic
with 'makeslice: len out of range' when creating the metadata slice.

This fix adds validation to check if metaSize is negative before
creating the slice, returning a descriptive error instead of panicking.

Fixes #7475

* Update weed/storage/needle/needle_read_page.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-08 22:26:03 -08:00
Chris Lu
28ac536280 fix: normalize Windows backslash paths in weed admin file uploads (#7636)
fix: normalize Windows backslash paths in file uploads

When uploading files from a Windows client to a Linux server,
file paths containing backslashes were not being properly interpreted as
directory separators. This caused files intended for subdirectories to be
created in the root directory with backslashes in their filenames.

Changes:
- Add util.CleanWindowsPath and util.CleanWindowsPathBase helper functions
  in weed/util/fullpath.go for reusable path normalization
- Use path.Join/path.Clean/path.Base instead of filepath equivalents
  for URL path semantics (filepath is OS-specific)
- Apply normalization in weed admin handlers and filer upload parsing

Fixes #7628
2025-12-05 17:40:32 -08:00
Chris Lu
69553e5ba6 convert error fromating to %w everywhere (#6995) 2025-07-16 23:39:27 -07:00
chrislu
d2be5822a1 refactoring 2025-06-16 22:25:22 -07:00
chrislu
e71d681fee refactor 2025-06-11 20:46:13 -07:00
chrislu
7c4d98446b refactor 2025-06-11 20:13:06 -07:00
chrislu
f27e195354 refactoring 2025-06-11 20:13:06 -07:00
chrislu
33ecc8442e refactor 2025-06-08 22:11:09 -07:00
chrislu
60f11f6510 add a readme file for volume needle data layout 2025-06-07 15:52:51 -07:00
chrislu
61c4f01e05 refactor 2025-06-06 08:55:32 -07:00
chrislu
a489d99333 fix tests 2025-06-06 08:25:04 -07:00
chrislu
7439af0eca refactoring 2025-06-06 01:35:48 -07:00
chrislu
cc135c63f7 a bit refactoring 2025-06-06 01:26:54 -07:00
chrislu
c4695fc3b3 refactor needle write for different versions 2025-06-06 00:35:13 -07:00
dongxufeng
ff878a542d correctly report volume with input/output error to master (#6790)
* correctly capture io error and report to master

* code fix

* check io error by error.Is

---------

Co-authored-by: dongxu_feng <dongxu_feng@intsig.net>
2025-05-15 00:56:43 -07:00
bwlfhu
0e08b83521 Fix volume ttl (#6683) 2025-04-02 07:59:21 -07:00
chrislu
c9f3448692 ReadAt may return io.EOF t end of file
related to https://github.com/seaweedfs/seaweedfs/issues/6219
2024-11-21 00:37:38 -08:00
chrislu
be09d08eb6 adjust error message 2024-11-20 23:49:01 -08:00
Kevin Bulteel
4a0dc883cf Fix/parse upload filename (#6241)
* fix: parse filename in PUT + refactor

* fix: master iu public url with http

* fix: better parsing and handle disposition header

* fix: take mime type from file extension if not set
2024-11-15 11:25:18 -08:00
Eugeniy E. Mikhailov
3003c9e17e added backward compatible CRC check and extra explanatory comments (#6233) 2024-11-12 13:56:14 -08:00
augustazz
0b00706454 EC volume supports expiration and displays expiration message when executing volume.list (#5895)
* ec volume expire

* volume.list show DestroyTime

* comments

* code optimization

---------

Co-authored-by: xuwenfeng <xuwenfeng1@zto.com>
2024-08-16 00:20:00 -07:00
Konstantin Lebedev
5ffacbb6ea refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
Kevin Bulteel
a3dab1fcd0 feat: upload files directly without multipart/form-data (#5539) 2024-04-26 16:03:40 -07:00
Konstantin Lebedev
5189a09de0 [volume] Reduce the number of buffers for uploading one chunk (#5458) 2024-04-11 04:47:21 -07:00
skycope
316ae45795 can vacuum volume when size mismatch (#5200)
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-01-15 16:16:46 -08:00
skycope
0e8a54f6f6 fix write volume over size MaxPossibleVolumeSize (#5190)
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-01-11 20:23:46 -08:00
Konstantin Lebedev
a7fc723ae0 chore: add status code for request_total metrics (#5188) 2024-01-10 10:05:27 -08:00
zemul
bd6f2716d3 fix_read_tomb_needle_meta (#4263)
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-03-02 08:26:55 -08:00
Eric Yang
51d462f204 ADHOC: volume fsck using append at ns (#3906)
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
Konstantin Lebedev
6253058d9d ensure monotonic n.AppendAtNs in each place (#3880)
https://github.com/seaweedfs/seaweedfs/issues/3852

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-23 18:24:52 -07:00
Konstantin Lebedev
452202f35b ReadNeedleMeta recovery panic occurred (#3883)
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:28 -07:00
chrislu
e3ea544d40 skip reading needle if size is invalid
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:15 -07:00
Ryan Russell
277976bd76 refactor(storage): readability improvements (#3703)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 02:43:17 -07:00
chrislu
2bfc8970d2 refactor: move ReadNeedleDataInto into volume_read.go 2022-09-08 18:54:02 -07:00
Eric Yang
b324a6536c ADHOC: add read needle meta grpc (#3581)
* ADHOC: add read needle meta grpc

* add test

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-06 23:51:27 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
36d2756978 volume: avoid writing too much data for range requests in large files
related to https://github.com/chrislusf/seaweedfs/issues/3178
2022-06-14 11:16:47 -07:00
chrislu
11f1e7996b fix test 2022-06-05 18:50:41 -07:00
chrislu
056c480eb0 volume: checksum remove the hashing step in Value() 2022-06-05 15:24:02 -07:00
chrislu
ecef844dfc stream read large files 2022-06-05 11:54:04 -07:00
chrislu
685643d506 needle read into a writer 2022-06-04 19:10:52 -07:00
chrislu
663bc5dc23 add needle reading in chunks 2022-06-04 18:15:39 -07:00
chrislu
18b29f709c refactoring, split file 2022-06-03 23:31:14 -07:00
chrislu
433fde4b18 move error to a separate file
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Konstantin Lebedev
3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00