Rust volume server implementation with CI (#8539)

* Match Go gRPC client transport defaults

* Honor Go HTTP idle timeout

* Honor maintenanceMBps during volume copy

* Honor images.fix.orientation on uploads

* Honor cpuprofile when pprof is disabled

* Match Go memory status payloads

* Propagate request IDs across gRPC calls

* Format pending Rust source updates

* Match Go stats endpoint payloads

* Serve Go volume server UI assets

* Enforce Go HTTP whitelist guards

* Align Rust metrics admin-port test with Go behavior

* Format pending Rust server updates

* Honor access.ui without per-request JWT checks

* Honor keepLocalDatFile in tier upload shortcut

* Honor Go remote volume write mode

* Load tier backends from master config

* Check master config before loading volumes

* Remove vif files on volume destroy

* Delete remote tier data on volume destroy

* Honor vif version defaults and overrides

* Reject mismatched vif bytes offsets

* Load remote-only tiered volumes

* Report Go tail offsets in sync status

* Stream remote dat in incremental copy

* Honor collection vif for EC shard config

* Persist EC expireAtSec in vif metadata

* Stream remote volume reads through HTTP

* Serve HTTP ranges from backend source

* Match Go ReadAllNeedles scan order

* Match Go CopyFile zero-stop metadata

* Delete EC volumes with collection cleanup

* Drop deleted collection metrics

* Match Go tombstone ReadNeedleMeta

* Match Go TTL parsing: all-digit default to minutes, two-pass fit algorithm

* Match Go needle ID/cookie formatting and name size computation

* Match Go image ext checks: webp resize only, no crop; empty healthz body

* Match Go Prometheus metric names and add missing handler counter constants

* Match Go ReplicaPlacement short string parsing with zero-padding

* Add missing EC constants MAX_SHARD_COUNT and MIN_TOTAL_DISKS

* Add walk_ecx_stats for accurate EC volume file counts and size

* Match Go VolumeStatus dat file size, EC shard stats, and disk pct precision

* Match Go needle map: unconditional delete counter, fix redb idx walk offset

* Add CompactMapSegment overflow panic guard matching Go

* Match Go volume: vif creation, version from superblock, TTL expiry, dedup data_size, garbage_level fallback

* Match Go 304 Not Modified: return bare status with no headers

* Match Go JWT error message: use "wrong jwt" instead of detailed error

* Match Go read handler bare 400, delete error prefix, download throttle timeout

* Match Go pretty JSON 1-space indent and "Deletion Failed:" error prefix

* Match Go heartbeat: keep is_heartbeating on error, add EC shard identification

* Match Go needle ReadBytes V2: tolerate EOF on truncated body

* Match Go volume: cookie check on any existing needle, return DataSize, 128KB meta guard

* Match Go DeleteCollection: propagate destroy errors

* Match Go gRPC: BatchDelete no flag, IncrementalCopy error, FetchAndWrite concurrent, VolumeUnmount/DeleteCollection errors, tail draining, query error code

* Match Go Content-Disposition RFC 6266 formatting with RFC 2231 encoding

* Match Go Guard isWriteActive: combine whitelist and signing key check

* Match Go DeleteCollectionMetrics: use partial label matching

* Match Go heartbeat: send state-only delta on volume state changes

* Match Go ReadNeedleMeta paged I/O: read header+tail only, skip data; add EIO tracking

* Match Go ScrubVolume INDEX mode dispatch; add VolumeCopy preallocation and EC NeedleStatus TODOs

* Add read_ec_shard_needle for full needle reconstruction from local EC shards

* Make heartbeat master config helpers pub for VolumeCopy preallocation

* Match Go gRPC: VolumeCopy preallocation, EC NeedleStatus full read, error message wording

* Match Go HTTP responses: omitempty fields, 2-space JSON indent, JWT JSON error, delete pretty/JSONP, 304 Last-Modified, raw write error

* Match Go WriteNeedleBlob V3 timestamp patching, fix makeup_diff double padding, count==0 read handling

* Add rebuild_ecx_file for EC index reconstruction from data shards

* Match Go gRPC: tail header first-chunk-only, EC cleanup on failure, copy append mode, ecx rebuild, compact cancellation

* Add EC volume read and delete support in HTTP handlers

* Add per-shard EC mount/unmount, location predicate search, idx directory for EC

* Add CheckVolumeDataIntegrity on volume load matching Go

* Match Go gRPC: EC multi-disk placement, per-shard mount/unmount, no auto-mount on reconstruct, streaming ReadAll/EcShardRead, ReceiveFile cleanup, version check, proxy streaming, redirect Content-Type

* Match Go heartbeat metric accounting

* Match Go duplicate UUID heartbeat retries

* Delete expired EC volumes during heartbeat

* Match Go volume heartbeat pruning

* Honor master preallocate in volume max

* Report remote storage info in heartbeats

* Emit EC heartbeat deltas on shard changes

* Match Go throttle boundary: use <= instead of <, fix pretty JSON to 1-space

* Match Go write_needle_blob monotonic appendAtNs via get_append_at_ns

* Match Go VolumeUnmount: idempotent success when volume not found

* Match Go TTL Display: return empty string when unit is Empty

Go checks `t.Unit == Empty` separately and returns "" for TTLs
with nonzero count but Empty unit. Rust only checked is_empty()
(count==0 && unit==0), so count>0 with unit=0 would format as
"5 " instead of "".

* Match Go error behavior for truncated needle data in read_body_v2

Go's readNeedleDataVersion2 returns "index out of range %d" errors
(indices 1-7) when needle body or metadata fields are truncated.
Rust was silently tolerating truncation and returning Ok. Now returns
NeedleError::IndexOutOfRange with the matching index for each field.

* Match Go download throttle: return JSON error instead of plain text

* Match Go crop params: default x1/y1 to 0 when not provided

* Match Go ScrubEcVolume: accumulate total_files from EC shards

* Match Go ScrubVolume: count total_files even on scrub error

* Match Go VolumeEcShardsCopy: set ignore_source_file_not_found for .vif

* Match Go VolumeTailSender: send needle_header on every chunk

* Match Go read_super_block: apply replication override from .vif

* Match Go check_volume_data_integrity: verify all 10 entries, detect trailing corruption

* Match Go WriteNeedleBlob: dedup check before writing during replication

* handlers: use meta-only reads for HEAD

* handlers: align range parsing and responses with Go

* handlers: align upload parsing with Go

* deps: enable webp support

* Make 5bytes the default feature for idx entry compatibility

* Match Go TTL: preserve original unit when count fits in byte

* Fix EC locate_needle: use get_actual_size for full needle size

* Fix raw body POST: only parse multipart when Content-Type contains form-data

* Match Go ReceiveFile: return protocol errors in response body, not gRPC status

* add docs

* Match Go VolumeEcShardsCopy: append to .ecj file instead of truncating

* Match Go ParsePath: support _delta suffix on file IDs for sub-file addressing

* Match Go chunk manifest: add Accept-Ranges, Content-Disposition, filename fallback, MIME detection

* Match Go privateStoreHandler: use proper JSON error for unsupported methods

* Match Go Destroy: add only_empty parameter to reject non-empty volume deletion

* Fix compilation: set_read_only_persist and set_writable return ()

These methods fire-and-forget save_vif internally, so gRPC callers
should not try to chain .map_err() on the unit return type.

* Match Go SaveVolumeInfo: check writability and propagate errors in save_vif

* Match Go VolumeDelete: propagate only_empty to delete_volume for defense in depth

The gRPC VolumeDelete handler had a pre-check for only_empty but then
passed false to store.delete_volume(), bypassing the store-level check.
Go passes req.OnlyEmpty directly to DeleteVolume. Now Rust does the same
for defense in depth against TOCTOU races (though the store write lock
makes this unlikely).

* Match Go ProcessRangeRequest: return full content for empty/oversized ranges

Go returns nil from ProcessRangeRequest when ranges are empty or total
range size exceeds content length, causing the caller to serve the full
content as a normal 200 response. Rust was returning an empty 200 body.

* Match Go Query: quote JSON keys in output records

Go's ToJson produces valid JSON with quoted keys like {"name":"Alice"}.
Rust was producing invalid JSON with unquoted keys like {name:"Alice"}.

* Match Go VolumeCopy: reject when no suitable disk location exists

Go returns ErrVolumeNoSpaceLeft when no location matches the disk type
and has sufficient space. Rust had an unsafe fallback that silently
picked the first location regardless of type or available space.

* Match Go DeleteVolumeNeedle: check noWriteOrDelete before allowing delete

Go checks v.noWriteOrDelete before proceeding with needle deletion,
returning "volume is read only" if true. Rust was skipping this check.

* Match Go ReceiveFile: prefer HardDrive location for EC and use response-level write errors

Two fixes: (1) Go prefers HardDriveType disk location for EC volumes,
falling back to first location. Returns "no storage location available"
when no locations exist. (2) Write failures are now response-level
errors (in response body) instead of gRPC status errors, matching Go.

* Match Go CopyFile: sync EC volume journal to disk before copying

Go calls ecVolume.Sync() before copying EC volume files to ensure the
.ecj journal is flushed to disk. Added sync_to_disk() to EcVolume and
call it in the CopyFile EC branch.

* Match Go readSuperBlock: propagate replication parse errors

Go returns an error when parsing the replication string from the .vif
file fails. Rust was silently ignoring the parse failure and using the
super block's replication as-is.

* Match Go TTL expiry: remove append_at_ns > 0 guard

Go computes TTL expiry from AppendAtNs without guarding against zero.
When append_at_ns is 0, the expiry is epoch + TTL which is in the past,
correctly returning NotFound. Rust's extra guard skipped the check,
incorrectly returning success for such needles.

* Match Go delete_collection: skip volumes with compaction in progress

Go checks !v.isCompactionInProgress.Load() before destroying a volume
during collection deletion, skipping compacting volumes. Also changed
destroy errors to log instead of aborting the entire collection delete.

* Match Go MarkReadonly/MarkWritable: always notify master even on local error

Go always notifies the master regardless of whether the local
set_read_only_persist or set_writable step fails. The Rust code was
using `?` which short-circuited on error, skipping the final master
notification. Save the result and defer the `?` until after the
notify call.

* Match Go PostHandler: return 500 for all write errors

Go returns 500 (InternalServerError) for all write failures. Rust was
returning 404 for volume-not-found and 403 for read-only volumes.

* Match Go makeupDiff: validate .cpd compaction revision is old + 1

Go reads the new .cpd file's super block and verifies the compaction
revision is exactly old + 1. Rust only validated the old revision.

* Match Go VolumeStatus: check data backend before returning status

Go checks v.DataBackend != nil before building the status response,
returning an error if missing. Rust was silently returning size 0.

* Match Go PostHandler: always include mime field in upload response JSON

Go always serializes the mime field even when empty ("mime":""). Rust was
omitting it when empty due to Option<String> with skip_serializing_if.

* Match Go FindFreeLocation: account for EC shards in free slot calculation

Go subtracts EC shard equivalents when computing available volume slots.
Rust was only comparing volume count, potentially over-counting free
slots on locations with many EC shards.

* Match Go privateStoreHandler: use INVALID as metrics label for unsupported methods

Go records the method as INVALID in metrics for unsupported HTTP methods.
Rust was using the actual method name.

* Match Go volume: add commit_compact guard and scrub data size validation

Two fixes: (1) commit_compact now checks/sets is_compacting flag to
prevent concurrent commits, matching Go's CompareAndSwap guard.
(2) scrub now validates total needle sizes against .dat file size.

* Match Go gRPC: fix TailSender error propagation, EcShardsInfo all slots, EcShardRead .ecx check

Three fixes: (1) VolumeTailSender now propagates binary search errors
instead of silently falling back to start. (2) VolumeEcShardsInfo
returns entries for all shard slots including unmounted. (3)
VolumeEcShardRead checks .ecx index for deletions instead of .ecj.

* Match Go metrics: add BuildInfo gauge and connection tracking functions

Go exposes a BuildInfo Prometheus metric with version labels, and tracks
open connections via stats.ConnectionOpen/Close. Added both to Rust.

* Match Go NeedleMap.Delete: use !is_deleted() instead of is_valid()

Go's CompactMap.Delete checks !IsDeleted() not IsValid(), so needles
with size==0 (live but anomalous) can still be deleted. The Rust code
was using is_valid() which returns false for size==0, preventing
deletion of such needles.

* Match Go fitTtlCount: always normalize TTL to coarsest unit

Go's fitTtlCount always converts to seconds first, then finds the
coarsest unit that fits in one byte (e.g., 120m → 2h). Rust had an
early return for count<=255 that skipped normalization, producing
different binary encodings for the same duration.

* Match Go BuildInfo metric: correct name and add missing labels

Go uses SeaweedFS_build_info (Namespace=SeaweedFS, Subsystem=build,
Name=info) with labels [version, commit, sizelimit, goos, goarch].
Rust had SeaweedFS_volumeServer_buildInfo with only [version].

* Match Go HTTP handlers: fix UploadResult fields, DiskStatus JSON, chunk manifest ETag

- UploadResult.mime: add skip_serializing_if to omit empty MIME (Go uses omitempty)
- UploadResult.contentMd5: only include when request provided Content-MD5 header
- Content-MD5 response header: only set when request provided it
- DiskStatuses: use camelCase field names (percentFree, percentUsed, diskType)
  to match Go's protobuf JSON marshaling
- Chunk manifest: preserve needle ETag in expanded response headers

* Match Go volume: fix version(), integrity check, scrub, and commit_compact

- version(): use self.version() instead of self.super_block.version in
  read_all_needles, check_volume_data_integrity, scan_raw_needles_from
  to respect volumeInfo.version override
- check_volume_data_integrity: initialize healthy_index_size to idx_size
  (matching Go) and continue on EOF instead of returning error
- scrub(): count deleted needles in total_read since they still occupy
  space in the .dat file (matches Go's totalRead += actualSize for deleted)
- commit_compact: clean up .cpd/.cpx files on makeup_diff failure
  (matches Go's error path cleanup)

* Match Go write queue: add 4MB batch byte limit

Go's startWorker breaks the batch at either 128 requests or 4MB of
accumulated write data. Rust only had the 128-request limit, allowing
large writes to accumulate unbounded latency.

* Add TTL normalization tests for Go parity verification

Test that fit_ttl_count normalizes 120m→2h, 24h→1d, 7d→1w even
when count fits in a byte, matching Go's fitTtlCount behavior.

* Match Go FindFreeLocation: account for EC shards in free slot calculation

Go's free volume count subtracts both regular volumes and EC volumes
from max_volume_count. Rust was only counting regular volumes, which
could over-report available slots when EC shards are mounted.

* Match Go EC volume: mark deletions in .ecx and replay .ecj at startup

Go's DeleteNeedleFromEcx marks needles as deleted in the .ecx index
in-place (writing TOMBSTONE_FILE_SIZE at the size field) in addition
to appending to the .ecj journal. Go's RebuildEcxFile replays .ecj
entries into .ecx on startup, then removes the .ecj file.

Rust was only appending to .ecj without marking .ecx, which meant
deleted EC needles remained readable via .ecx binary search. This
fix:
- Opens .ecx in read/write mode (was read-only)
- Adds mark_needle_deleted_in_ecx: binary search + in-place write
- Calls it from journal_delete before appending to .ecj
- Adds rebuild_ecx_from_journal: replays .ecj into .ecx on startup

* Match Go check_all_ec_shards_deleted: use MAX_SHARD_COUNT instead of hardcoded 14

Go's TotalShardsCount is DataShardsCount + ParityShardsCount = 14 by
default, but custom EC configs via .vif can have more shards (up to
MaxShardCount = 32). Using MAX_SHARD_COUNT ensures all shard files
are checked regardless of EC configuration.

* Match Go EC locate: subtract 1 from shard size and use datFileSize override

Go's LocateEcShardNeedleInterval passes shard.ecdFileSize-1 to
LocateData (shards are padded, -1 avoids overcounting large block
rows). When datFileSize is known, Go uses datFileSize/DataShards
instead. Rust was passing the raw shard file size without adjustment.

* Fix TTL parsing and DiskStatus field names to match Go exactly

TTL::read: Go's ReadTTL preserves the original unit (7d stays 7d,
not 1w) and errors on count > 255. The previous normalization change
was incorrect — Go only normalizes internally via fitTtlCount, not
during string parsing.

DiskStatus: Go uses encoding/json on protobuf structs, which reads
the json struct tags (snake_case: percent_free, percent_used,
disk_type), not the protobuf JSON names (camelCase). Revert to
snake_case to match Go's actual output.

* Fix heartbeat: check leader != current master before redirect, process duplicated UUIDs first

Match Go's volume_grpc_client_to_master.go behavior:
1. Only trigger leader redirect when the leader address differs from the
   current master (prevents unnecessary reconnect loops when master confirms
   its own address).
2. Process duplicated_uuids before leader redirect check, matching Go's
   ordering where duplicate UUID detection takes priority.

* Remove SetState version check to match Go behavior

Go's SetState unconditionally applies the state without any version
mismatch check. The Rust version had an extra optimistic concurrency
check that would reject valid requests from Go clients that don't
track versions.

* Fix TTL::read() to normalize via fit_ttl_count matching Go's ReadTTL

Go's ReadTTL calls fitTtlCount which converts to seconds and normalizes
to the coarsest unit that fits in a byte count (e.g. 120m->2h, 7d->1w,
24h->1d). The Rust version was preserving the original unit, producing
different binary encodings on disk and in heartbeat messages.

* Always return Content-MD5 header and JSON field on successful writes

Go always sets Content-MD5 in the response regardless of whether the
request included it. The Rust version was conditionally including it
only when the request provided Content-MD5.

* Include name and size in UploadResult JSON even when empty/zero

Go's encoding/json always includes empty strings and zero values in
the upload response. The Rust version was using skip_serializing_if
to omit them, causing JSON structure differences.

* Include deleted needles in scan_raw_needles_from to match Go

Go's ScanVolumeFileFrom visits ALL needles including deleted ones.
Skipping deleted entries during incremental copy would cause tombstones
to not be propagated, making deleted files reappear on the receiving side.

* Match Go NeedleMap.Delete: always write tombstone to idx file

Go's NeedleMap.Delete unconditionally writes a tombstone entry to the
idx file and updates metrics, even if the needle doesn't exist or is
already deleted. This is important for replication where every delete
operation must produce an idx write. The Rust version was skipping the
tombstone write for non-existent or already-deleted needles.

* Limit MIME type to 255 bytes matching Go's CreateNeedleFromRequest

* Title-case Seaweed-* pair keys to match Go HTTP header canonicalization

* Unify DiskType::Hdd into HardDrive to match Go's single HardDriveType

* Skip tombstone entries in walk_ecx_stats total_size matching Go's Raw()

* Return EMPTY TTL when computed seconds is zero matching Go's fitTtlCount

* Include disk-space-low in Volume.is_read_only() matching Go

* Log error on CIDR parse failure in whitelist matching Go's glog.Errorf

* Log cookie mismatch in gRPC Query matching Go's V(0).Infof

* Fix is_expired volume_size comparison to use < matching Go

Go checks `volumeSize < super_block.SuperBlockSize` (strict less-than),
but Rust used `<=`. This meant Rust would fail to expire a volume that
is exactly SUPER_BLOCK_SIZE bytes.

* Apply Go's JWT expiry defaults: 10s write, 60s read

Go calls v.SetDefault("jwt.signing.expires_after_seconds", 10) and
v.SetDefault("jwt.signing.read.expires_after_seconds", 60). Rust
defaulted to 0 for both, which meant tokens would never expire when
security.toml has a signing key but omits expires_after_seconds.

* Stop [grpc.volume].ca from overriding [grpc].ca matching Go

Go reads the gRPC CA file only from config.GetString("grpc.ca"), i.e.
the [grpc] section. The [grpc.volume] section only provides cert and
key. Rust was also reading ca from [grpc.volume] which would silently
override the [grpc].ca value when both were present.

* Fix free_volume_count to use EC shard count matching Go

Was counting EC volumes instead of EC shards, which underestimates EC
space usage. One EC volume with 14 shards uses ~1.4 volume slots, not 1.
Now uses Go's formula: ((max - volumes) * DataShardsCount - ecShardCount) / DataShardsCount.

* Include preallocate in compaction space check matching Go

Go uses max(preallocate, estimatedCompactSize) for the free space check.
Rust was only using the estimated volume size, which could start a
compaction that fails mid-way if preallocate exceeds the volume size.

* Check gzip magic bytes before setting Content-Encoding matching Go

Go checks both Accept-Encoding contains "gzip" AND IsGzippedContent
(data starts with 0x1f 0x8b) before setting Content-Encoding: gzip.
Rust only checked Accept-Encoding, which could incorrectly declare
gzip encoding for non-gzip compressed data.

* Only set upload response name when needle HasName matching Go

Go checks reqNeedle.HasName() before setting ret.Name. Rust always set
the name from the filename variable, which could return the fid portion
of the path as the name for raw PUT requests without a filename.

* Treat MaxVolumeCount==0 as unlimited matching Go's hasFreeDiskLocation

Go's hasFreeDiskLocation returns true immediately when MaxVolumeCount
is 0, treating it as unlimited. Rust was computing effective_free as
<= 0 for max==0, rejecting the location. This could fail volume
creation during early startup before the first heartbeat adjusts max.

* Read lastAppendAtNs from deleted V3 entries in integrity check

Go's doCheckAndFixVolumeData reads AppendAtNs from both live entries
(verifyNeedleIntegrity) and deleted tombstones (verifyDeletedNeedleIntegrity).
Rust was skipping deleted entries, which could result in a stale
last_append_at_ns if the last index entry is a deletion.

* Return empty body for empty/oversized range requests matching Go

Go's ProcessRangeRequest returns nil (empty body, 200 OK) when
parsed ranges are empty or combined range size exceeds total content
size. The Rust buffered path incorrectly returned the full file data
for both cases. The streaming path already handled this correctly.

* Dispatch ScrubEcVolume by mode matching Go's INDEX/LOCAL/FULL

Go's ScrubEcVolume switches on mode: INDEX calls v.ScrubIndex()
(ecx integrity only), LOCAL calls v.ScrubLocal(), FULL calls
vs.store.ScrubEcVolume(). Rust was ignoring the mode and always
running verify_ec_shards. Now INDEX mode checks ecx index integrity
(sorted overlap detection + file size validation) without shard I/O,
while LOCAL/FULL modes run the existing shard verification.

* Fix TTL test expectation: 7d normalizes to 1w matching Go's fitTtlCount

Go's ReadTTL calls fitTtlCount which normalizes to the coarsest unit
that fits: 7 days = 1 week, so "7d" becomes {Count:1, Unit:Week}
which displays as "1w". Both Go and Rust normalize identically.

* Add version mismatch check to SetState matching Go's State.Update

Go's State.Update compares the incoming version with the stored
version and returns "version mismatch" error if they differ. This
provides optimistic concurrency control. The Rust implementation
was accepting any version unconditionally.

* Use unquoted keys in Query JSON output matching Go's json.ToJson

Go's json.ToJson produces records with unquoted keys like
{score:12} not {"score":12}. This is a custom format used
internally by SeaweedFS for query results.

* Fix TTL test expectation in VolumeNeedleStatus: 7d normalizes to 1w

Same normalization as the HTTP test: Go's ReadTTL calls fitTtlCount
which converts 7 days to 1 week.

* Include ETag header in 304 Not Modified responses matching Go behavior

Go sets ETag on the response writer (via SetEtag) before the
If-Modified-Since and If-None-Match conditional checks, so both
304 response paths include the ETag header. The Rust implementation
was only adding ETag to 200 responses.

* Remove needle-name fallback in chunk manifest filename resolution

Go's tryHandleChunkedFile only falls back from URL filename to
manifest name. Rust had an extra fallback to needle.name that
Go does not perform, which could produce different
Content-Disposition filenames for chunk manifests.

* Validate JWT nbf (Not Before) claim matching Go's jwt-go/v5

Go's jwt.ParseWithClaims validates the nbf claim when present,
rejecting tokens whose nbf is in the future. The Rust jsonwebtoken
crate defaults validate_nbf to false, so tokens with future nbf
were incorrectly accepted.

* Set isHeartbeating to true at startup matching Go's VolumeServer init

Go unconditionally sets isHeartbeating: true in the VolumeServer
struct literal. Rust was starting with false when masters are
configured, causing /healthz to return 503 until the first
heartbeat succeeds.

* Call store.close() on shutdown matching Go's Shutdown()

Go's Shutdown() calls vs.store.Close() which closes all volumes
and flushes file handles. The Rust server was relying on process
exit for cleanup, which could leave data unflushed.

* Include server ID in maintenance mode error matching Go's format

Go returns "volume server %s is in maintenance mode" with the
store ID. Rust was returning a generic "maintenance mode" message.

* Fix DiskType test: use HardDrive variant matching Go's HddType=""

Go maps both "" and "hdd" to HardDriveType (empty string). The
Rust enum variant is HardDrive, not Hdd. The test referenced a
nonexistent Hdd variant causing compilation failure.

* Do not include ETag in 304 responses matching Go's GetOrHeadHandler

Go sets ETag at L235 AFTER the If-Modified-Since and If-None-Match
304 return paths, so Go's 304 responses do not include the ETag header.
The Rust code was incorrectly including ETag in both 304 response paths.

* Return 400 on malformed query strings in PostHandler matching Go's ParseForm

Go's r.ParseForm() returns HTTP 400 with "form parse error: ..." when
the query string is malformed. Rust was silently falling back to empty
query params via unwrap_or_default().

* Load EC volume version from .vif matching Go's NewEcVolume

Go sets ev.Version = needle.Version(volumeInfo.Version) from the .vif
file. Rust was always using Version::current() (V3), which would produce
wrong needle actual size calculations for volumes created with V1 or V2.

* Sync .ecx file before close matching Go's EcVolume.Close

Go calls ev.ecxFile.Sync() before closing to ensure in-place deletion
marks are flushed to disk. Without this, deletion marks written via
MarkNeedleDeleted could be lost on crash.

* Validate SuperBlock extra data size matching Go's Bytes() guard

Go checks extraSize > 256*256-2 and calls glog.Fatalf to prevent
corrupt super block headers. Rust was silently truncating via u16 cast,
which would write an incorrect extra_size field.

* Update quinn-proto 0.11.13 -> 0.11.14 to fix GHSA-6xvm-j4wr-6v98

Fixes Dependency Review CI failure: quinn-proto < 0.11.14 is vulnerable
to unauthenticated remote DoS via panic in QUIC transport parameter
parsing.

* Skip TestMultipartUploadUsesFormFieldsForTimestampAndTTL for Go server

Go's r.FormValue() cannot read multipart text fields after
r.MultipartReader() consumes the body, so ts/ttl sent as multipart
form fields only work with the Rust volume server. Skip this test
when VOLUME_SERVER_IMPL != "rust" to fix CI failure.

* Flush .ecx in EC volume sync_to_disk matching Go's Sync()

Go's EcVolume.Sync() flushes both the .ecj journal and the .ecx index
to disk. The Rust version only flushed .ecj, leaving in-place deletion
marks in .ecx unpersisted until close(). This could cause data
inconsistency if the server crashes after marking a needle deleted in
.ecx but before close().

* Remove .vif file in EC volume destroy matching Go's Destroy()

Go's EcVolume.Destroy() removes .ecx, .ecj, and .vif files. The Rust
version only removed .ecx and .ecj, leaving orphaned .vif files on
disk after EC volume destruction (e.g., after TTL expiry).

* Fix is_expired to use <= for SuperBlockSize check matching Go

Go checks contentSize <= SuperBlockSize to detect empty volumes (no
needles). Rust used < which would incorrectly allow a volume with
exactly SuperBlockSize bytes (header only, no data) to proceed to
the TTL expiry check and potentially be marked as expired.

* Fix read_append_at_ns to read timestamps from tombstone entries

Go reads the full needle body for all entries including tombstones
(deleted needles with size=0) to extract the actual AppendAtNs
timestamp. The Rust version returned 0 early for size <= 0 entries,
which would cause the binary search in incremental copy to produce
incorrect results for positions containing deleted needles.

Now uses get_actual_size to compute the on-disk size (which handles
tombstones correctly) and only returns 0 when the actual size is 0.

* Add X-Request-Id response header matching Go's requestIDMiddleware

Go sets both X-Request-Id and x-amz-request-id response headers.
The Rust server only set x-amz-request-id, missing X-Request-Id.

* Add skip_serializing_if for UploadResult name and size fields

Go's UploadResult uses json:"name,omitempty" and json:"size,omitempty",
omitting these fields from JSON when they are zero values (empty
string / 0). The Rust struct always serialized them, producing
"name":"" and "size":0 where Go would omit them.

* Support JSONP/pretty-print for write success responses

Go's writeJsonQuiet checks for callback (JSONP) and pretty query
parameters on all JSON responses including write success. The Rust
write success path used axum::Json directly, bypassing JSONP and
pretty-print support. Now uses json_result_with_query to match Go.

* Include actual limit in file size limit error message

Go returns "file over the limited %d bytes" with the actual limit
value included. Rust returned a generic "file size limit exceeded"
without the limit value, making it harder to debug.

* Extract extension from 2-segment URL paths for image operations

Go's parseURLPath extracts the file extension from all URL formats
including 2-segment paths like /vid,fid.jpg. The Rust version only
handled 3-segment paths (/vid/fid/filename.ext), so extensions in
2-segment paths were lost. This caused image resize/crop operations
requested via query params to be silently skipped for those paths.

* Add size_hint to TrackedBody so throttled downloads get Content-Length

TrackedBody (used for download throttling) did not implement
size_hint(), causing HTTP/1.1 to fall back to chunked transfer
encoding instead of setting Content-Length. Go always sets
Content-Length explicitly for non-range responses.

* Add Last-Modified, pairs, and S3 headers to chunk manifest responses

Go sets Last-Modified, needle pairs, and S3 pass-through headers on
the response writer BEFORE calling tryHandleChunkedFile. Since the
Rust chunk manifest handler created fresh response headers and
returned early, these headers were missing from chunk manifest
responses. Now passes last_modified_str into the chunk manifest
handler and applies pairs and S3 pass-through query params
(response-cache-control, response-content-encoding, etc.) to the
chunk manifest response headers.

* Fix multipart fallback to use first part data when no filename

Go reads the first part's data unconditionally, then looks for a
part with a filename. If none found, Go uses the first part's data
(with empty filename). Rust only captured parts with filenames, so
when no part had a filename it fell back to the raw multipart body
bytes (including boundary delimiters), producing corrupt needle data.

* Set HasName and HasMime flags for empty values matching Go

Go's CreateNeedleFromRequest sets HasName and HasMime flags even when
the filename or MIME type is empty (len < 256 is true for len 0).
Rust skipped empty values, causing the on-disk needle format to
differ: Go-written needles include extra bytes for the empty name/mime
size fields, changing the serialized needle size in the idx entry.
This ensures binary format compatibility between Go and Rust servers.

* Add is_stopping guard to vacuum_volume_commit matching Go

Go's CommitCompactVolume (store_vacuum.go L53-54) checks
s.isStopping before committing compaction to prevent file
swaps during shutdown. The Rust handler was missing this
check, which could allow compaction commits while the
server is stopping.

* Remove disk_type from required status fields since Go omits it

Go's default DiskType is "" (HardDriveType), and protobuf's omitempty
tag causes empty strings to be dropped from JSON output.

* test: honor rust env in dual volume harness

* grpc: notify master after volume lifecycle changes

* http: proxy to replicas before download-limit timeout

* test: pass readMode to rust volume harnesses

* fix store free-location predicate selection

* fix volume copy disk placement and heartbeat notification

* fix chunk manifest delete replication

* fix write replication to survive client disconnects

* fix download limit proxy and wait flow

* fix crop gating for streamed reads

* fix upload limit wait counter behavior

* fix chunk manifest image transforms

* fix has_resize_ops to check width/height > 0 instead of is_some()

Go's shouldResizeImages condition is `width > 0 || height > 0`, so
`?width=0` correctly evaluates to false. Rust was using `is_some()`
which made `?width=0` evaluate to true, unnecessarily disabling
streaming reads for those requests.

* fix Content-MD5 to only compute and return when provided by client

Go only computes the MD5 of uncompressed data when a Content-MD5
header or multipart field is provided. Rust was always computing and
returning it. Also fix the mismatch error message to include size,
matching Go's format.

* fix save_vif to compute ExpireAtSec from TTL

Go's SaveVolumeInfo always computes ExpireAtSec = now + ttlSeconds
when the volume has a TTL. The save_vif path (used by set_read_only
and set_writable) was missing this computation, causing .vif files
to be written without the correct expiration timestamp for TTL volumes.

* fix set_writable to not modify no_write_can_delete

Go's MarkVolumeWritable only sets noWriteOrDelete=false and persists.
Rust was additionally setting no_write_can_delete=has_remote_file,
which could incorrectly change the write mode for remote-file volumes
when the master explicitly asks to make the volume writable.

* fix write_needle_blob_and_index to error on too-small V3 blob

Go returns an error when the needle blob is too small for timestamp
patching. Rust was silently skipping the patch and writing the blob
with a stale/zero timestamp, which could cause data integrity issues
during incremental replication that relies on AppendAtNs ordering.

* fix VolumeEcShardsToVolume to validate dataShards range

Go validates that dataShards is > 0 and <= MaxShardCount before
proceeding with EC-to-volume reconstruction. Without this check,
a zero or excessively large data_shards value could cause confusing
downstream failures.

* fix destroy to use VolumeError::NotEmpty instead of generic Io error

The dedicated NotEmpty variant exists in the enum but was not being
used. This makes error matching consistent with Go's ErrVolumeNotEmpty.

* fix SetState to persist state to disk with rollback on failure

Go's State.Update saves VolumeServerState to a state.pb file after
each SetState call, and rolls back the in-memory state if persistence
fails. Rust was only updating in-memory atomics, so maintenance mode
would be lost on server restart. Now saves protobuf-encoded state.pb
and loads it on startup.

* fix VolumeTierMoveDatToRemote to close local dat backend after upload

Go calls v.LoadRemoteFile() after saving volume info, which closes
the local DataBackend before transitioning to remote storage. Without
this, the volume holds a stale file handle to the deleted local .dat
file, causing reads to fail until server restart.

* fix VolumeTierMoveDatFromRemote to close remote dat backend after download

Go calls v.DataBackend.Close() and sets DataBackend=nil after removing
the remote file reference. Without this, the stale remote backend
state lingers and reads may not discover the newly downloaded local
.dat file until server restart.

* fix redirect to use internal url instead of public_url

Go's proxyReqToTargetServer builds the redirect Location header from
loc.Url (the internal URL), not publicUrl. Using public_url could
cause redirect failures when internal and external URLs differ.

* fix redirect test and add state_file_path to integration test

Update redirect unit test to expect internal url (matching the
previous fix). Add missing state_file_path field to the integration
test VolumeServerState constructor.

* fix FetchAndWriteNeedle to await all writes before checking errors

Go uses a WaitGroup to await all writes (local + replicas) before
checking errors. Rust was short-circuiting on local write failure,
which could leave replica writes in-flight without waiting for
completion.

* fix shutdown to send deregister heartbeat before pre_stop delay

Go's StopHeartbeat() closes stopChan immediately on interrupt, causing
the heartbeat goroutine to send the deregister heartbeat right away,
before the preStopSeconds delay. Rust was only setting is_stopping=true
without waking the heartbeat loop, so the deregister was delayed until
after the pre_stop sleep. Now we call volume_state_notify.notify_one()
to wake the heartbeat immediately.

* fix heartbeat response ordering to check duplicate UUIDs first

Go processes heartbeat responses in this order: DuplicatedUuids first,
then volume options (prealloc/size limit), then leader redirect. Rust
was applying volume options before checking for duplicate UUIDs, which
meant volume option changes would take effect even when the response
contained a duplicate UUID error that should cause an immediate return.

* the test thread was blocked

* fix(deps): update aws-lc-sys 0.38.0 → 0.39.0 to resolve security advisories

Bumps aws-lc-rs 1.16.1 → 1.16.2, pulling in aws-lc-sys 0.39.0 which
fixes GHSA-394x-vwmw-crm3 (X.509 Name Constraints wildcard/unicode
bypass) and GHSA-9f94-5g5w-gf6r (CRL Distribution Point scope check
logic error).

* fix: match Go Content-MD5 mismatch error message format

Go uses "Content-MD5 did not match md5 of file data expected [X]
received [Y] size Z" while Rust had a shorter format. Match the
exact Go error string so clients see identical messages.

* fix: match Go Bearer token length check (> 7, not >= 7)

Go requires len(bearer) > 7 ensuring at least one char after
"Bearer ". Rust used >= 7 which would accept an empty token.

* fix(deps): drop legacy rustls 0.21 to resolve rustls-webpki GHSA-pwjx-qhcg-rvj4

aws-sdk-s3's default "rustls" feature enables tls-rustls in
aws-smithy-runtime, which pulls in legacy-rustls-ring (rustls 0.21
→ rustls-webpki 0.101.7, moderate CRL advisory). Replace with
explicit default-https-client which uses only rustls 0.23 /
rustls-webpki 0.103.9.

* fix: use uploaded filename for auto-compression extension detection

Go extracts the file extension from pu.FileName (the uploaded
filename) for auto-compression decisions. Rust was using the URL
path, which typically has no extension for SeaweedFS file IDs.

* fix: add CRC legacy Value() backward-compat check on needle read

Go double-checks CRC: n.Checksum != crc && uint32(n.Checksum) !=
crc.Value(). The Value() path is a deprecated transform for compat
with seaweed versions prior to commit 056c480eb. Rust had the
legacy_value() method but wasn't using it in validation.

* fix: remove /stats/* endpoints to match Go (commented out since L130)

Go's volume_server.go has the /stats/counter, /stats/memory, and
/stats/disk endpoints commented out (lines 130-134). Remove them
from the Rust router along with the now-unused whitelist_guard
middleware.

* fix: filter application/octet-stream MIME for chunk manifests

Go's tryHandleChunkedFile (L334) filters out application/octet-stream
from chunk manifest MIME types, falling back to extension-based
detection. Rust was returning the stored MIME as-is for manifests.

* fix: VolumeMarkWritable returns error before notifying master

Go returns early at L200 if MarkVolumeWritable fails, before
reaching the master notification at L206. Rust was notifying master
even on failure, creating inconsistent state where master thinks
the volume is writable but local marking failed.

* fix: check volume existence before maintenance in MarkReadonly/Writable

Go's VolumeMarkReadonly (L239-241) and VolumeMarkWritable (L253-255)
look up the volume first, then call makeVolumeReadonly/Writable which
checks maintenance. Rust was checking maintenance first, returning
"maintenance mode" instead of "not found" for missing volumes.

* feat: implement ScrubVolume mark_broken_volumes_readonly (PR #8360)

Add the mark_broken_volumes_readonly flag from PR #8360:
- Sync proto field (tag 3) to local volume_server.proto
- After scrubbing, if flag is set, call makeVolumeReadonly on each
  broken volume (notify master, mark local readonly, notify again)
- Collect errors via joined error semantics matching Go's errors.Join
- Factor out make_volume_readonly helper reused by both
  VolumeMarkReadonly and ScrubVolume

Also refactors VolumeMarkReadonly to use the shared helper.

* fix(deps): update rustls-webpki 0.103.9 → 0.103.10 (GHSA-pwjx-qhcg-rvj4)

CRL Distribution Point matching logic fix for moderate severity
advisory about CRLs not considered authoritative.

* test: update integration tests for removed /stats/* endpoints

Replace tests that expected /stats/* routes to return 200/401 with
tests confirming they now fall through to the store handler (400),
matching Go's commented-out stats endpoints.

* docs: fix misleading comment about default offset feature

The comment said "4-byte offsets unless explicitly built with 5-byte
support" but the default feature enables 5bytes. This is intentional
for production parity with Go -tags 5BytesOffset builds. Fix the
comment to match reality.

seaweed-volume/vendor/reed-solomon-erasure/.cargo-ok

@@ -0,0 +1 @@
+{"v":1}

seaweed-volume/vendor/reed-solomon-erasure/.cargo_vcs_info.json

@@ -0,0 +1,6 @@
+{
+  "git": {
+    "sha1": "a1ca49de5384445b68ade7d72f31f0379c199943"
+  },
+  "path_in_vcs": ""
+}

seaweed-volume/vendor/reed-solomon-erasure/.gitattributes

@@ -0,0 +1,3 @@
+BackBlaze_JavaReedSolomon/* linguist-vendored
+KlausPost_reedsolomon/* linguist-vendored
+NicolasT_reedsolomon/* linguist-vendored

seaweed-volume/vendor/reed-solomon-erasure/.gitignore

@@ -0,0 +1,2 @@
+/target/
+/Cargo.lock

seaweed-volume/vendor/reed-solomon-erasure/CHANGELOG.md

@@ -0,0 +1,181 @@
+## 6.0.0
+- Use LruCache instead of InversionTree for caching data decode matrices
+  - See [PR #104](https://github.com/rust-rse/reed-solomon-erasure/pull/104)
+- Minor code duplication
+  - See [PR #102](https://github.com/rust-rse/reed-solomon-erasure/pull/102)
+- Dependencies update
+  - Updated `smallvec` from `0.6.1` to `1.8.0`
+
+## 5.0.3
+- Fixed cross build bug for aarch64 with simd-accel
+  - See [PR #100](https://github.com/rust-rse/reed-solomon-erasure/pull/100)
+
+## 5.0.2
+* Add support for `RUST_REED_SOLOMON_ERASURE_ARCH` environment variable and stop using `native` architecture for SIMD code
+  - See [PR #98](https://github.com/rust-rse/reed-solomon-erasure/pull/98)
+
+## 5.0.1
+- The `simd-accel` feature now builds on M1 Macs
+  - See [PR #92](https://github.com/rust-rse/reed-solomon-erasure/pull/92)
+- Minor code cleanup
+
+## 5.0.0
+- Merged several PRs
+- Not fully reviewed as I am no longer maintaining this crate
+
+## 4.0.2
+- Updated build.rs to respect RUSTFLAGS's target-cpu if available
+  - See [PR #75](https://github.com/darrenldl/reed-solomon-erasure/pull/75)
+- Added AVX512 support
+  - See [PR #69](https://github.com/darrenldl/reed-solomon-erasure/pull/69)
+- Disabled SIMD acceleration when MSVC is being used to build the library
+  - See [PR #67](https://github.com/darrenldl/reed-solomon-erasure/pull/67)
+- Dependencies update
+  - Updated `smallvec` from `0.6` to `1.2`
+
+## 4.0.1
+- Updated SIMD C code for Windows compatibility
+  - Removed include of `unistd.h` in `simd_c/reedsolomon.c`
+  - Removed GCC `nonnull` attribute in `simd_c/reedsolomon.h`
+  - See PR [#63](https://github.com/darrenldl/reed-solomon-erasure/pull/63) [#64](https://github.com/darrenldl/reed-solomon-erasure/pull/64) for details
+- Replaced use of `libc::uint8_t` in `src/galois_8.rs` with `u8`
+
+## 4.0.0
+- Major API restructure: removed `Shard` type in favor of generic functions
+- The logic of this crate is now generic over choice of finite field
+- The SIMD acceleration feature for GF(2^8) is now activated with the `simd-accel` Cargo feature. Pure-rust behavior is default.
+- Ran rustfmt
+- Adds a GF(2^16) implementation
+
+## 3.1.2 (not published)
+- Doc fix
+  - Added space before parantheses in code comments and documentation
+- Disabled SIMD C code for Android and iOS targets entirely
+
+## 3.1.1
+- Fixed `Matrix::augment`
+  - The error checking code was incorrect
+  - Since this method is used in internal code only, and the only use case is a correct use case, the error did not lead to any bugs
+- Fixed benchmark data
+  - Previously used MB=10^6 bytes while I should have used MB=2^20 bytes
+  - Table in README has been updated accordingly
+    - The `>= 2.1.0` data is obtained by measuring again with the corrected `rse-benchmark` code
+    - The `2.0.X` and `1.X.X` data are simply adjusted by mutiplying `10^6` then dividing by `2^20`
+- Dependencies update
+  - Updated `rand` from `0.4` to `0.5.4`
+- Added special handling in `build.rs` for CC options on Android and iOS
+  - `-march=native` is not available for GCC on Android, see issue #23
+
+## 3.1.0
+- Impl'd `std::error::Error` for `reed_solomon_erasure::Error` and `reed_solomon_erasure::SBSError`
+  - See issue [#17](https://github.com/darrenldl/reed-solomon-erasure/issues/17), suggested by [DrPeterVanNostrand](https://github.com/DrPeterVanNostrand)
+- Added fuzzing suite
+  - No code changes due to this as no bugs were found
+- Upgraded InversionTree QuickCheck test
+  - No code changes due to this as no bugs were found
+- Upgraded test suite for main codec methods (e.g. encode, reconstruct)
+  - A lot of heavy QuickCheck tests were added
+  - No code changes due to this as no bugs were found
+- Upgraded test suite for ShardByShard methods
+  - A lot of heavy QuickCheck tests were added
+  - No code changes due to this as no bugs were found
+- Minor code refactoring in `reconstruct_internal` method
+  - This means `reconstruct` and related methods are slightly more optimized
+
+## 3.0.3
+- Added QuickCheck tests to the test suite
+  - InversionTree is heavily tested now
+- No code changes as no bugs were found
+- Deps update
+  - Updated rayon from 0.9 to 1.0
+
+## 3.0.2
+- Same as 3.0.1, but 3.0.1 had unapplied changes
+
+## 3.0.1 (yanked)
+- Updated doc for `with_buffer` variants of verifying methods
+  - Stated explicitly that the buffer contains the correct parity shards after a successful call
+- Added tests for the above statement
+
+## 3.0.0
+- Added `with_buffer` variants for verifying methods
+  - This gives user the option of reducing heap allocation(s)
+- Core code clean up, improvements, and review, added more AUDIT comments
+- Improved shard utils
+- Added code to remove leftover parity shards in `reconstruct_data_shards`
+  - This means one fewer gotcha of using the methods
+- `ShardByShard` code review and overhaul
+- `InversionTree` code review and improvements
+
+## 2.4.0
+- Added more flexibility for `convert_2D_slices` macro
+  - Now accepts expressions rather than just identifiers
+  - The change requires change of syntax
+
+## 2.3.3
+- Replaced all slice splitting functions in `misc_utils` with std lib ones or rayon ones
+  - This means there are fewer heap allocations in general
+
+## 2.3.2
+- Made `==`(`eq`) for `ReedSolomon` more reasonable
+  - Previously `==` would compare
+    - data shard count
+    - parity shard count
+    - total shard count
+    - internal encoding matrix
+    - internal `ParallelParam`
+  - Now it only compares
+    - data shard count
+    - parity shard count
+
+## 2.3.1
+- Added info on encoding behaviour to doc
+
+## 2.3.0
+- Made Reed-Solomon codec creation methods return error instead of panic when shard numbers are not correct
+
+## 2.2.0
+- Fixed SBS error checking code
+- Documentation fixes and polishing
+- Renamed `Error::InvalidShardsIndicator` to `Error::InvalidShardFlags`
+- Added more details to documentation on error handling
+- Error handling code overhaul and checks for all method variants
+- Dead commented out code cleanup and indent fix
+
+## 2.1.0
+- Added Nicolas's SIMD C code files, gaining major speedup on supported CPUs
+- Added support for "shard by shard" encoding, allowing easier streamed encoding
+- Added functions for shard by shard encoding
+
+## 2.0.0
+- Complete rewrite of most code following Klaus Post's design
+- Added optimsations (parallelism, loop unrolling)
+- 4-5x faster than `1.X.X`
+
+## 1.1.1
+- Documentation polish
+- Added documentation badge to README
+- Optimised internal matrix related operations
+  - This largely means `decode_missing` is faster
+
+## 1.1.0
+- Added more helper functions
+- Added more tests
+
+## 1.0.1
+- Added more tests
+- Fixed decode_missing
+  - Previously may reconstruct the missing shards with incorrect length
+
+## 1.0.0
+- Added more tests
+- Added integration with Codecov (via kcov)
+- Code refactoring
+- Added integration with Coveralls (via kcov)
+
+## 0.9.1
+- Code restructuring
+- Added documentation
+
+## 0.9.0
+- Base version

seaweed-volume/vendor/reed-solomon-erasure/Cargo.toml

@@ -0,0 +1,87 @@
+# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
+#
+# When uploading crates to the registry Cargo will automatically
+# "normalize" Cargo.toml files for maximal compatibility
+# with all versions of Cargo and also rewrite `path` dependencies
+# to registry (e.g., crates.io) dependencies.
+#
+# If you are reading this file be aware that the original Cargo.toml
+# will likely look very different (and much more reasonable).
+# See Cargo.toml.orig for the original contents.
+
+[package]
+edition = "2018"
+name = "reed-solomon-erasure"
+version = "6.0.0"
+authors = ["Darren Ldl <darrenldldev@gmail.com>"]
+build = "build.rs"
+exclude = [
+    "appveyor.yml",
+    ".travis.yml",
+]
+description = "Rust implementation of Reed-Solomon erasure coding"
+homepage = "https://github.com/darrenldl/reed-solomon-erasure"
+documentation = "https://docs.rs/reed-solomon-erasure"
+readme = "README.md"
+keywords = [
+    "reed-solomon",
+    "erasure",
+]
+categories = ["encoding"]
+license = "MIT"
+repository = "https://github.com/darrenldl/reed-solomon-erasure"
+
+[[bench]]
+name = "reconstruct"
+
+[dependencies.libc]
+version = "0.2"
+optional = true
+
+[dependencies.libm]
+version = "0.2.1"
+
+[dependencies.lru]
+version = "0.16.3"
+
+[dependencies.parking_lot]
+version = "0.11.2"
+optional = true
+
+[dependencies.smallvec]
+version = "1.2"
+
+[dependencies.spin]
+version = "0.9.2"
+features = ["spin_mutex"]
+default-features = false
+
+[dev-dependencies.quickcheck]
+version = "0.9"
+
+[dev-dependencies.rand]
+version = "0.7.2"
+
+[build-dependencies.cc]
+version = "1.0"
+optional = true
+
+[features]
+default = ["std"]
+simd-accel = [
+    "cc",
+    "libc",
+]
+std = ["parking_lot"]
+
+[badges.appveyor]
+repository = "darrenldl/reed-solomon-erasure"
+
+[badges.codecov]
+repository = "darrenldl/reed-solomon-erasure"
+
+[badges.coveralls]
+repository = "darrenldl/reed-solomon-erasure"
+
+[badges.travis-ci]
+repository = "darrenldl/reed-solomon-erasure"

seaweed-volume/vendor/reed-solomon-erasure/Cargo.toml.orig

@@ -0,0 +1,56 @@
+[package]
+name= "reed-solomon-erasure"
+version = "6.0.0"
+authors = ["Darren Ldl <darrenldldev@gmail.com>"]
+edition = "2018"
+build = "build.rs"
+exclude = [
+    "appveyor.yml",
+    ".travis.yml"
+]
+
+description = "Rust implementation of Reed-Solomon erasure coding"
+
+documentation = "https://docs.rs/reed-solomon-erasure"
+homepage= "https://github.com/darrenldl/reed-solomon-erasure"
+repository= "https://github.com/darrenldl/reed-solomon-erasure"
+
+readme= "README.md"
+
+keywords= ["reed-solomon", "erasure"]
+
+categories= ["encoding"]
+
+license = "MIT"
+
+[features]
+default = ["std"] # simd off by default
+std = ["parking_lot"]
+simd-accel = ["cc", "libc"]
+
+[badges]
+travis-ci = { repository = "darrenldl/reed-solomon-erasure" }
+appveyor= { repository = "darrenldl/reed-solomon-erasure" }
+codecov = { repository = "darrenldl/reed-solomon-erasure" }
+coveralls = { repository = "darrenldl/reed-solomon-erasure" }
+
+[dependencies]
+libc = { version = "0.2", optional = true }
+# `log2()` impl for `no_std`
+libm = "0.2.1"
+lru = "0.16.3"
+# Efficient `Mutex` implementation for `std` environment
+parking_lot = { version = "0.11.2", optional = true }
+smallvec = "1.2"
+# `Mutex` implementation for `no_std` environment with the same high-level API as `parking_lot`
+spin = { version = "0.9.2", default-features = false, features = ["spin_mutex"] }
+
+[dev-dependencies]
+rand = "0.7.2"
+quickcheck = "0.9"
+
+[build-dependencies]
+cc = { version = "1.0", optional = true }
+
+[[bench]]
+name = "reconstruct"

seaweed-volume/vendor/reed-solomon-erasure/LICENSE

@@ -0,0 +1,24 @@
+MIT License
+
+Copyright (c) 2017 Darren Ldl
+Copyright (c) 2015, 2016 Nicolas Trangez
+Copyright (c) 2015 Klaus Post
+Copyright (c) 2015 Backblaze
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

seaweed-volume/vendor/reed-solomon-erasure/README.md

@@ -0,0 +1,166 @@
+# reed-solomon-erasure
+[![Build Status](https://travis-ci.org/darrenldl/reed-solomon-erasure.svg?branch=master)](https://travis-ci.org/darrenldl/reed-solomon-erasure)
+[![Build status](https://ci.appveyor.com/api/projects/status/47c0emjoa9bhpjlb/branch/master?svg=true)](https://ci.appveyor.com/project/darrenldl/reed-solomon-erasure/branch/master)
+[![codecov](https://codecov.io/gh/darrenldl/reed-solomon-erasure/branch/master/graph/badge.svg)](https://codecov.io/gh/darrenldl/reed-solomon-erasure)
+[![Coverage Status](https://coveralls.io/repos/github/darrenldl/reed-solomon-erasure/badge.svg?branch=master)](https://coveralls.io/github/darrenldl/reed-solomon-erasure?branch=master)
+[![Crates](https://img.shields.io/crates/v/reed-solomon-erasure.svg)](https://crates.io/crates/reed-solomon-erasure)
+[![Documentation](https://docs.rs/reed-solomon-erasure/badge.svg)](https://docs.rs/reed-solomon-erasure)
+[![dependency status](https://deps.rs/repo/github/darrenldl/reed-solomon-erasure/status.svg)](https://deps.rs/repo/github/darrenldl/reed-solomon-erasure)
+
+Rust implementation of Reed-Solomon erasure coding
+
+WASM builds are also available, see section **WASM usage** below for details
+
+This is a port of [BackBlaze's Java implementation](https://github.com/Backblaze/JavaReedSolomon), [Klaus Post's Go implementation](https://github.com/klauspost/reedsolomon), and [Nicolas Trangez's Haskell implementation](https://github.com/NicolasT/reedsolomon).
+
+Version `1.X.X` copies BackBlaze's implementation, and is less performant as there were fewer places where parallelism could be added.
+
+Version `>= 2.0.0` copies Klaus Post's implementation. The SIMD C code is copied from Nicolas Trangez's implementation with minor modifications.
+
+See [Notes](#notes) and [License](#license) section for details.
+
+## WASM usage
+
+See [here](wasm/README.md) for details
+
+## Rust usage
+Add the following to your `Cargo.toml` for the normal version (pure Rust version)
+```toml
+[dependencies]
+reed-solomon-erasure = "4.0"
+```
+or the following for the version which tries to utilise SIMD
+```toml
+[dependencies]
+reed-solomon-erasure = { version = "4.0", features = [ "simd-accel" ] }
+```
+and the following to your crate root
+```rust
+extern crate reed_solomon_erasure;
+```
+
+NOTE: `simd-accel` is tuned for Haswell+ processors on x86-64 and not in any way for other architectures, set
+environment variable `RUST_REED_SOLOMON_ERASURE_ARCH` during build to force compilation of C code for specific architecture (`-march` flag in
+GCC/Clang). Even on x86-64 you can achieve better performance by setting it to `native`, but it will stop running on
+older CPUs, YMMV.
+
+## Example
+```rust
+#[macro_use(shards)]
+extern crate reed_solomon_erasure;
+
+use reed_solomon_erasure::galois_8::ReedSolomon;
+// or use the following for Galois 2^16 backend
+// use reed_solomon_erasure::galois_16::ReedSolomon;
+
+fn main () {
+    let r = ReedSolomon::new(3, 2).unwrap(); // 3 data shards, 2 parity shards
+
+    let mut master_copy = shards!(
+        [0, 1,  2,  3],
+        [4, 5,  6,  7],
+        [8, 9, 10, 11],
+        [0, 0,  0,  0], // last 2 rows are parity shards
+        [0, 0,  0,  0]
+    );
+
+    // Construct the parity shards
+    r.encode(&mut master_copy).unwrap();
+
+    // Make a copy and transform it into option shards arrangement
+    // for feeding into reconstruct_shards
+    let mut shards: Vec<_> = master_copy.iter().cloned().map(Some).collect();
+
+    // We can remove up to 2 shards, which may be data or parity shards
+    shards[0] = None;
+    shards[4] = None;
+
+    // Try to reconstruct missing shards
+    r.reconstruct(&mut shards).unwrap();
+
+    // Convert back to normal shard arrangement
+    let result: Vec<_> = shards.into_iter().filter_map(|x| x).collect();
+
+    assert!(r.verify(&result).unwrap());
+    assert_eq!(master_copy, result);
+}
+```
+
+## Benchmark it yourself
+You can test performance under different configurations quickly (e.g. data parity shards ratio, parallel parameters)
+by cloning this repo: https://github.com/darrenldl/rse-benchmark
+
+`rse-benchmark` contains a copy of this library (usually a fully functional dev version), so you only need to adjust `main.rs`
+then do `cargo run --release` to start the benchmark.
+
+## Performance
+Version `1.X.X`, `2.0.0` do not utilise SIMD.
+
+Version `2.1.0` onward uses Nicolas's C files for SIMD operations.
+
+Machine: laptop with `Intel(R) Core(TM) i5-3337U CPU @ 1.80GHz (max 2.70GHz) 2 Cores 4 Threads`
+
+Below shows the result of one of the test configurations, other configurations show similar results in terms of ratio.
+
+|Configuration| Klaus Post's | >= 2.1.0 && < 4.0.0 | 2.0.X | 1.X.X |
+|---|---|---|---|---|
+| 10x2x1M | ~7800MB/s |~4500MB/s | ~1000MB/s | ~240MB/s |
+
+Versions `>= 4.0.0` have not been benchmarked thoroughly yet
+
+## Changelog
+[Changelog](CHANGELOG.md)
+
+## Contributions
+Contributions are welcome. Note that by submitting contributions, you agree to license your work under the same license used by this project as stated in the LICENSE file.
+
+## Credits
+#### Library overhaul and Galois 2^16 backend
+Many thanks to the following people for overhaul of the library and introduction of Galois 2^16 backend
+
+  - [@drskalman](https://github.com/drskalman)
+
+  - Jeff Burdges [@burdges](https://github.com/burdges)
+
+  - Robert Habermeier [@rphmeier](https://github.com/rphmeier)
+
+#### WASM builds
+Many thanks to Nazar Mokrynskyi [@nazar-pc](https://github.com/nazar-pc) for submitting his package for WASM builds
+
+He is the original author of the files stored in `wasm` folder. The files may have been modified by me later.
+
+#### AVX512 support
+Many thanks to [@sakridge](https://github.com/sakridge) for adding support for AVX512 (see [PR #69](https://github.com/darrenldl/reed-solomon-erasure/pull/69))
+
+#### build.rs improvements
+Many thanks to [@ryoqun](https://github.com/ryoqun) for improving the usability of the library in the context of cross-compilation (see [PR #75](https://github.com/darrenldl/reed-solomon-erasure/pull/75))
+
+#### no_std support
+Many thanks to Nazar Mokrynskyi [@nazar-pc](https://github.com/nazar-pc) for adding `no_std` support (see [PR #90](https://github.com/darrenldl/reed-solomon-erasure/pull/90))
+
+#### Testers
+Many thanks to the following people for testing and benchmarking on various platforms
+
+  - Laurențiu Nicola [@lnicola](https://github.com/lnicola/) (platforms: Linux, Intel)
+
+  - Roger Andersen [@hexjelly](https://github.com/hexjelly) (platforms: Windows, AMD)
+
+## Notes
+#### Code quality review
+If you'd like to evaluate the quality of this library, you may find audit comments helpful.
+
+Simply search for "AUDIT" to see the dev notes that are aimed at facilitating code reviews.
+
+#### Implementation notes
+The `1.X.X` implementation mostly copies [BackBlaze's Java implementation](https://github.com/Backblaze/JavaReedSolomon).
+
+`2.0.0` onward mostly copies [Klaus Post's Go implementation](https://github.com/klauspost/reedsolomon), and copies C files from [Nicolas Trangez's Haskell implementation](https://github.com/NicolasT/reedsolomon).
+
+The test suite for all versions copies [Klaus Post's Go implementation](https://github.com/klauspost/reedsolomon) as basis.
+
+## License
+#### Nicolas Trangez's Haskell Reed-Solomon implementation
+The C files for SIMD operations are copied (with no/minor modifications) from [Nicolas Trangez's Haskell implementation](https://github.com/NicolasT/reedsolomon), and are under the same MIT License as used by NicolasT's project
+
+#### TL;DR
+All files are released under the MIT License

seaweed-volume/vendor/reed-solomon-erasure/benches/reconstruct.rs

@@ -0,0 +1,108 @@
+#![feature(test)]
+
+extern crate test;
+
+use {
+    rand::{prelude::*, Rng},
+    reed_solomon_erasure::galois_8::Field,
+    test::Bencher,
+};
+
+type ReedSolomon = reed_solomon_erasure::ReedSolomon<Field>;
+
+const SHARD_SIZE: usize = 1024;
+
+fn run_reconstruct_bench(bencher: &mut Bencher, num_data_shards: usize, num_parity_shards: usize) {
+    let mut rng = rand::thread_rng();
+    let mut shards = vec![vec![0u8; SHARD_SIZE]; num_data_shards + num_parity_shards];
+    for shard in &mut shards[..num_data_shards] {
+        rng.fill(&mut shard[..]);
+    }
+    let reed_solomon = ReedSolomon::new(num_data_shards, num_parity_shards).unwrap();
+    reed_solomon.encode(&mut shards[..]).unwrap();
+    let shards: Vec<_> = shards.into_iter().map(Some).collect();
+
+    bencher.iter(|| {
+        let mut shards = shards.clone();
+        for _ in 0..num_parity_shards {
+            *shards.choose_mut(&mut rng).unwrap() = None;
+        }
+        reed_solomon.reconstruct(&mut shards[..]).unwrap();
+        assert!(shards.iter().all(Option::is_some));
+    });
+}
+
+#[bench]
+fn bench_reconstruct_2_2(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 2, 2)
+}
+
+#[bench]
+fn bench_reconstruct_4_2(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 4, 2)
+}
+
+#[bench]
+fn bench_reconstruct_4_4(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 4, 4)
+}
+
+#[bench]
+fn bench_reconstruct_8_2(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 8, 2)
+}
+
+#[bench]
+fn bench_reconstruct_8_4(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 8, 4)
+}
+
+#[bench]
+fn bench_reconstruct_8_8(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 8, 8)
+}
+
+#[bench]
+fn bench_reconstruct_16_2(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 16, 2)
+}
+
+#[bench]
+fn bench_reconstruct_16_4(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 16, 4)
+}
+
+#[bench]
+fn bench_reconstruct_16_8(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 16, 8)
+}
+
+#[bench]
+fn bench_reconstruct_16_16(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 16, 16)
+}
+
+#[bench]
+fn bench_reconstruct_32_2(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 32, 2)
+}
+
+#[bench]
+fn bench_reconstruct_32_4(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 32, 4)
+}
+
+#[bench]
+fn bench_reconstruct_32_8(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 32, 8)
+}
+
+#[bench]
+fn bench_reconstruct_32_16(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 32, 16)
+}
+
+#[bench]
+fn bench_reconstruct_32_32(bencher: &mut Bencher) {
+    run_reconstruct_bench(bencher, 32, 32)
+}

seaweed-volume/vendor/reed-solomon-erasure/build.rs

@@ -0,0 +1,196 @@
+use std::env;
+use std::fs::File;
+use std::io::Write;
+use std::path::Path;
+
+#[cfg(feature = "simd-accel")]
+extern crate cc;
+
+const FIELD_SIZE: usize = 256;
+
+const GENERATING_POLYNOMIAL: usize = 29;
+
+fn gen_log_table(polynomial: usize) -> [u8; FIELD_SIZE] {
+    let mut result: [u8; FIELD_SIZE] = [0; FIELD_SIZE];
+    let mut b: usize = 1;
+
+    for log in 0..FIELD_SIZE - 1 {
+        result[b] = log as u8;
+
+        b = b << 1;
+
+        if FIELD_SIZE <= b {
+            b = (b - FIELD_SIZE) ^ polynomial;
+        }
+    }
+
+    result
+}
+
+const EXP_TABLE_SIZE: usize = FIELD_SIZE * 2 - 2;
+
+fn gen_exp_table(log_table: &[u8; FIELD_SIZE]) -> [u8; EXP_TABLE_SIZE] {
+    let mut result: [u8; EXP_TABLE_SIZE] = [0; EXP_TABLE_SIZE];
+
+    for i in 1..FIELD_SIZE {
+        let log = log_table[i] as usize;
+        result[log] = i as u8;
+        result[log + FIELD_SIZE - 1] = i as u8;
+    }
+
+    result
+}
+
+fn multiply(log_table: &[u8; FIELD_SIZE], exp_table: &[u8; EXP_TABLE_SIZE], a: u8, b: u8) -> u8 {
+    if a == 0 || b == 0 {
+        0
+    } else {
+        let log_a = log_table[a as usize];
+        let log_b = log_table[b as usize];
+        let log_result = log_a as usize + log_b as usize;
+        exp_table[log_result]
+    }
+}
+
+fn gen_mul_table(
+    log_table: &[u8; FIELD_SIZE],
+    exp_table: &[u8; EXP_TABLE_SIZE],
+) -> [[u8; FIELD_SIZE]; FIELD_SIZE] {
+    let mut result: [[u8; FIELD_SIZE]; FIELD_SIZE] = [[0; 256]; 256];
+
+    for a in 0..FIELD_SIZE {
+        for b in 0..FIELD_SIZE {
+            result[a][b] = multiply(log_table, exp_table, a as u8, b as u8);
+        }
+    }
+
+    result
+}
+
+fn gen_mul_table_half(
+    log_table: &[u8; FIELD_SIZE],
+    exp_table: &[u8; EXP_TABLE_SIZE],
+) -> ([[u8; 16]; FIELD_SIZE], [[u8; 16]; FIELD_SIZE]) {
+    let mut low: [[u8; 16]; FIELD_SIZE] = [[0; 16]; FIELD_SIZE];
+    let mut high: [[u8; 16]; FIELD_SIZE] = [[0; 16]; FIELD_SIZE];
+
+    for a in 0..low.len() {
+        for b in 0..low.len() {
+            let mut result = 0;
+            if !(a == 0 || b == 0) {
+                let log_a = log_table[a];
+                let log_b = log_table[b];
+                result = exp_table[log_a as usize + log_b as usize];
+            }
+            if (b & 0x0F) == b {
+                low[a][b] = result;
+            }
+            if (b & 0xF0) == b {
+                high[a][b >> 4] = result;
+            }
+        }
+    }
+    (low, high)
+}
+
+macro_rules! write_table {
+    (1D => $file:ident, $table:ident, $name:expr, $type:expr) => {{
+        let len = $table.len();
+        let mut table_str = String::from(format!("pub static {}: [{}; {}] = [", $name, $type, len));
+
+        for v in $table.iter() {
+            let str = format!("{}, ", v);
+            table_str.push_str(&str);
+        }
+
+        table_str.push_str("];\n");
+
+        $file.write_all(table_str.as_bytes()).unwrap();
+    }};
+    (2D => $file:ident, $table:ident, $name:expr, $type:expr) => {{
+        let rows = $table.len();
+        let cols = $table[0].len();
+        let mut table_str = String::from(format!(
+            "pub static {}: [[{}; {}]; {}] = [",
+            $name, $type, cols, rows
+        ));
+
+        for a in $table.iter() {
+            table_str.push_str("[");
+            for b in a.iter() {
+                let str = format!("{}, ", b);
+                table_str.push_str(&str);
+            }
+            table_str.push_str("],\n");
+        }
+
+        table_str.push_str("];\n");
+
+        $file.write_all(table_str.as_bytes()).unwrap();
+    }};
+}
+
+fn write_tables() {
+    let log_table = gen_log_table(GENERATING_POLYNOMIAL);
+    let exp_table = gen_exp_table(&log_table);
+    let mul_table = gen_mul_table(&log_table, &exp_table);
+
+    let out_dir = env::var("OUT_DIR").unwrap();
+    let dest_path = Path::new(&out_dir).join("table.rs");
+    let mut f = File::create(&dest_path).unwrap();
+
+    write_table!(1D => f, log_table,      "LOG_TABLE",      "u8");
+    write_table!(1D => f, exp_table,      "EXP_TABLE",      "u8");
+    write_table!(2D => f, mul_table,      "MUL_TABLE",      "u8");
+
+    if cfg!(feature = "simd-accel") {
+        let (mul_table_low, mul_table_high) = gen_mul_table_half(&log_table, &exp_table);
+
+        write_table!(2D => f, mul_table_low,  "MUL_TABLE_LOW",  "u8");
+        write_table!(2D => f, mul_table_high, "MUL_TABLE_HIGH", "u8");
+    }
+}
+
+#[cfg(all(
+    feature = "simd-accel",
+    any(target_arch = "x86_64", target_arch = "aarch64"),
+    not(target_env = "msvc"),
+    not(any(target_os = "android", target_os = "ios"))
+))]
+fn compile_simd_c() {
+    let mut build = cc::Build::new();
+    build.opt_level(3);
+
+    match env::var("RUST_REED_SOLOMON_ERASURE_ARCH") {
+        Ok(arch) => {
+            // Use explicitly specified environment variable as architecture.
+            build.flag(&format!("-march={}", arch));
+        }
+        Err(_error) => {
+            // On x86-64 enabling Haswell architecture unlocks useful instructions and improves performance
+            // dramatically while allowing it to run ony modern CPU.
+            match env::var("CARGO_CFG_TARGET_ARCH").unwrap().as_str(){
+                "x86_64"  => { build.flag(&"-march=haswell"); },
+                _         => ()
+            }
+        }
+    }
+
+    build
+        .flag("-std=c11")
+        .file("simd_c/reedsolomon.c")
+        .compile("reedsolomon");
+}
+
+#[cfg(not(all(
+    feature = "simd-accel",
+    any(target_arch = "x86_64", target_arch = "aarch64"),
+    not(target_env = "msvc"),
+    not(any(target_os = "android", target_os = "ios"))
+)))]
+fn compile_simd_c() {}
+
+fn main() {
+    compile_simd_c();
+    write_tables();
+}

seaweed-volume/vendor/reed-solomon-erasure/sage/galois_ext_test.sage

@@ -0,0 +1,26 @@
+GF256.<a> = FiniteField(256)
+R.<x> = GF256[x]
+ext_poly = R.irreducible_element(2,algorithm="first_lexicographic" )
+ExtField.<b> = GF256.extension(ext_poly)
+print ExtField
+print len(ExtField)
+
+x^2 + a*x + a^7
+
+e1 = (a^7 + a^6 + a^4 + a)*b + a^3 + a^2 + a + 1
+e2 = (a^7 + a^5 + a^2)*b + a^7 + a^4 + a^3 + a
+
+print "e1: ", e1
+print "e2: ", e2
+
+print "e1 + e2: ", e1 + e2
+#(a^6 + a^5 + a^4 + a^2 + a)*b + a^7 + a^4 + a^2 + 1
+
+print "e1 * e2: ", e1 * e2
+#(a^4 + a^2 + a + 1)*b + a^7 + a^5 + a^3 + a
+
+print "e1 / e2: ", e1 / e2
+#(a^7 + a^6 + a^5 + a^4 + a^3 + a^2 + 1)*b + a^6 + a^3 + a
+
+print "1/b: ", 1/b
+#(a^4 + a^3 + a + 1)*b + a^5 + a^4 + a^2 + a

seaweed-volume/vendor/reed-solomon-erasure/simd_c/reedsolomon.c

@@ -0,0 +1,574 @@
+/* reedsolomon.c - SIMD-optimized Galois-field multiplication routines
+ *
+ * Copyright (c) 2015, 2016 Nicolas Trangez
+ * Copyright (c) 2015 Klaus Post
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE
+ */
+
+#if HAVE_CONFIG_H
+# include "config.h"
+#endif
+
+#include <stdint.h>
+#include <string.h>
+
+//#if defined(__SSE2__) && __SSE2__ && defined(HAVE_EMMINTRIN_H) && HAVE_EMMINTRIN_H
+//#ifdef __SSE2__
+#if defined(__SSE2__) && __SSE2__
+# define USE_SSE2 1
+# undef VECTOR_SIZE
+# define VECTOR_SIZE 16
+# include <emmintrin.h>
+#else
+# define USE_SSE2 0
+#endif
+
+//#if defined(__SSSE3__) && __SSSE3__ && defined(HAVE_TMMINTRIN_H) && HAVE_TMMINTRIN_H
+//#ifdef __SSSE3__
+#if defined(__SSSE3__) && __SSSE3__
+# define USE_SSSE3 1
+# undef VECTOR_SIZE
+# define VECTOR_SIZE 16
+# include <tmmintrin.h>
+#else
+# define USE_SSSE3 0
+#endif
+
+//#if defined(__AVX2__) && __AVX2__ && defined(HAVE_IMMINTRIN_H) && HAVE_IMMINTRIN_H
+//#ifdef __AVX2__
+#if defined(__AVX2__) && __AVX2__
+# define USE_AVX2 1
+# undef VECTOR_SIZE
+# define VECTOR_SIZE 32
+# include <immintrin.h>
+#else
+# define USE_AVX2 0
+#endif
+
+
+#if defined(__AVX512F__) && __AVX512F__
+# define USE_AVX512 1
+# undef VECTOR_SIZE
+# define VECTOR_SIZE 64
+# include <immintrin.h>
+#else
+# define USE_AVX512 0
+#endif
+
+
+/*#if ((defined(__ARM_NEON__) && __ARM_NEON__) \
+        || (defined(__ARM_NEON) && __ARM_NEON) \
+        || (defined(__aarch64__) && __aarch64__)) \
+        && defined(HAVE_ARM_NEON_H) && HAVE_ARM_NEON_H*/
+#if ((defined(__ARM_NEON__) && __ARM_NEON__)      \
+     || (defined(__ARM_NEON) && __ARM_NEON)       \
+     || (defined(__aarch64__) && __aarch64__))
+# define USE_ARM_NEON 1
+#undef VECTOR_SIZE
+# define VECTOR_SIZE 16
+# include <arm_neon.h>
+#else
+# define USE_ARM_NEON 0
+#endif
+
+//#if defined(__ALTIVEC__) && __ALTIVEC__ && defined(HAVE_ALTIVEC_H) && HAVE_ALTIVEC_H
+#if defined(__ALTIVEC__) && __ALTIVEC__
+# define USE_ALTIVEC 1
+# undef VECTOR_SIZE
+# define VECTOR_SIZE 16
+# include <altivec.h>
+#else
+# define USE_ALTIVEC 0
+#endif
+
+#ifndef VECTOR_SIZE
+/* 'Generic' code */
+# define VECTOR_SIZE 16
+#endif
+
+# define USE_ALIGNED_ACCESS 0
+# define ALIGNED_ACCESS __attribute__((unused))
+# define UNALIGNED_ACCESS
+
+#include "reedsolomon.h"
+
+#if defined(HAVE_FUNC_ATTRIBUTE_HOT) && HAVE_FUNC_ATTRIBUTE_HOT
+# define HOT_FUNCTION   __attribute__((hot))
+#else
+# define HOT_FUNCTION
+#endif
+
+#if defined(HAVE_FUNC_ATTRIBUTE_CONST) && HAVE_FUNC_ATTRIBUTE_CONST
+# define CONST_FUNCTION __attribute__((const))
+#else
+# define CONST_FUNCTION
+#endif
+
+#if defined(HAVE_FUNC_ATTRIBUTE_ALWAYS_INLINE) && HAVE_FUNC_ATTRIBUTE_ALWAYS_INLINE
+# define ALWAYS_INLINE  inline __attribute__((always_inline))
+#else
+# define ALWAYS_INLINE  inline
+#endif
+
+#if defined(HAVE_FUNC_ATTRIBUTE_FORCE_ALIGN_ARG_POINTER) && HAVE_FUNC_ATTRIBUTE_FORCE_ALIGN_ARG_POINTER
+# define FORCE_ALIGN_ARG_POINTER __attribute__((force_align_arg_pointer))
+#else
+# define FORCE_ALIGN_ARG_POINTER
+#endif
+
+#define CONCAT_HELPER(a, b)     a ## b
+#define CONCAT(a, b)            CONCAT_HELPER(a, b)
+
+typedef uint8_t v16u8v __attribute__((vector_size(16), aligned(1)));
+typedef uint64_t v2u64v __attribute__((vector_size(16), aligned(1)));
+
+#define T(t, n) t n[VSIZE / 8 / sizeof(t)]
+#define T1(t, n) t n
+
+#define VSIZE 128
+typedef union {
+        T(uint8_t, u8);
+        T(uint64_t, u64);
+#if USE_SSE2
+        T1(__m128i, m128i);
+#endif
+#if USE_ARM_NEON
+        T1(uint8x16_t, uint8x16);
+        T1(uint8x8x2_t, uint8x8x2);
+#endif
+#if USE_ALTIVEC
+        T1(__vector uint8_t, uint8x16);
+        T1(__vector uint64_t, uint64x2);
+#endif
+        T1(v16u8v, v16u8);
+        T1(v2u64v, v2u64);
+} v128 __attribute__((aligned(1)));
+#undef VSIZE
+
+#define VSIZE 256
+typedef union {
+        T(uint8_t, u8);
+#if USE_AVX2
+        __m256i m256i;
+#endif
+} v256 __attribute__((aligned(1)));
+#undef VSIZE
+
+#define VSIZE 512
+typedef union {
+        T(uint8_t, u8);
+#if USE_AVX512
+        __m512i m512i;
+#endif
+} v512 __attribute__((aligned(1)));
+
+#undef T
+#undef T1
+
+#if VECTOR_SIZE == 16
+typedef v128 v;
+#elif VECTOR_SIZE == 32
+typedef v256 v;
+#elif VECTOR_SIZE == 64
+typedef v512 v;
+#else
+# error Unsupported VECTOR_SIZE
+#endif
+
+static ALWAYS_INLINE UNALIGNED_ACCESS v128 loadu_v128(const uint8_t *in) {
+#if USE_SSE2
+        const v128 result = { .m128i = _mm_loadu_si128((const __m128i *)in) };
+#else
+        v128 result;
+        memcpy(&result.u64, in, sizeof(result.u64));
+#endif
+
+        return result;
+}
+
+static ALWAYS_INLINE UNALIGNED_ACCESS v loadu_v(const uint8_t *in) {
+#if USE_AVX512
+        const v512 result = { .m512i = _mm512_loadu_si512((const __m512i *)in) };
+#elif USE_AVX2
+        const v256 result = { .m256i = _mm256_loadu_si256((const __m256i *)in) };
+#else
+        const v128 result = loadu_v128(in);
+#endif
+
+        return result;
+}
+
+static ALWAYS_INLINE ALIGNED_ACCESS v load_v(const uint8_t *in) {
+#if USE_AVX512
+        const v512 result = { .m512i = _mm512_load_si512((const __m512i *)in) };
+#elif USE_AVX2
+        const v256 result = { .m256i = _mm256_load_si256((const __m256i *)in) };
+#elif USE_SSE2
+        const v128 result = { .m128i = _mm_load_si128((const __m128i *)in) };
+#elif USE_ARM_NEON
+        const v128 result = { .uint8x16 = vld1q_u8(in) };
+#elif USE_ALTIVEC
+        const v128 result = { .uint8x16 = vec_ld(0, in) };
+#else
+        const v128 result = loadu_v128(in);
+#endif
+
+        return result;
+}
+
+static ALWAYS_INLINE CONST_FUNCTION v set1_epi8_v(const uint8_t c) {
+#if USE_AVX512
+        const v512 result = { .m512i = _mm512_set1_epi8(c) };
+#elif USE_AVX2
+        const v256 result = { .m256i = _mm256_set1_epi8(c) };
+#elif USE_SSE2
+        const v128 result = { .m128i = _mm_set1_epi8(c) };
+#elif USE_ARM_NEON
+        const v128 result = { .uint8x16 = vdupq_n_u8(c) };
+#elif USE_ALTIVEC
+        const v128 result = { .uint8x16 = { c, c, c, c, c, c, c, c,
+                                            c, c, c, c, c, c, c, c } };
+#else
+        uint64_t c2 = c,
+                 tmp = (c2 << (7 * 8)) |
+                       (c2 << (6 * 8)) |
+                       (c2 << (5 * 8)) |
+                       (c2 << (4 * 8)) |
+                       (c2 << (3 * 8)) |
+                       (c2 << (2 * 8)) |
+                       (c2 << (1 * 8)) |
+                       (c2 << (0 * 8));
+        const v128 result = { .u64 = { tmp, tmp } };
+#endif
+
+        return result;
+}
+
+static ALWAYS_INLINE CONST_FUNCTION v srli_epi64_v(const v in /*, const unsigned int n*/) {
+        // TODO: Hard code n to 4 to avoid build issues on M1 Macs (the
+        //       `USE_ARM_NEON` path below) where apple clang is failing to
+        //       recognize the constant `n`.
+        //
+        //       See https://github.com/rust-rse/reed-solomon-erasure/pull/92
+        //
+        #define n 4
+#if USE_AVX512
+        const v512 result = { .m512i = _mm512_srli_epi64(in.m512i, n) };
+#elif USE_AVX2
+        const v256 result = { .m256i = _mm256_srli_epi64(in.m256i, n) };
+#elif USE_SSE2
+        const v128 result = { .m128i = _mm_srli_epi64(in.m128i, n) };
+#elif USE_ARM_NEON
+        const v128 result = { .uint8x16 = vshrq_n_u8(in.uint8x16, n) };
+#elif USE_ALTIVEC
+# if RS_HAVE_VEC_VSRD
+        const v128 shift = { .v2u64 = { n, n } },
+                   result = { .uint64x2 = vec_vsrd(in.v2u64, shift.v2u64) };
+# else
+        const v128 result = { .v2u64 = in.v2u64 >> n };
+# endif
+#else
+        const v128 result = { .u64 = { in.u64[0] >> n,
+                                       in.u64[1] >> n } };
+#endif
+        #undef n
+        return result;
+}
+
+static ALWAYS_INLINE CONST_FUNCTION v and_v(const v a, const v b) {
+#if USE_AVX512
+        const v512 result = { .m512i = _mm512_and_si512(a.m512i, b.m512i) };
+#elif USE_AVX2
+        const v256 result = { .m256i = _mm256_and_si256(a.m256i, b.m256i) };
+#elif USE_SSE2
+        const v128 result = { .m128i = _mm_and_si128(a.m128i, b.m128i) };
+#elif USE_ARM_NEON
+        const v128 result = { .uint8x16 = vandq_u8(a.uint8x16, b.uint8x16) };
+#elif USE_ALTIVEC
+        const v128 result = { .uint8x16 = vec_and(a.uint8x16, b.uint8x16) };
+#else
+        const v128 result = { .v2u64 = a.v2u64 & b.v2u64 };
+#endif
+
+        return result;
+}
+
+static ALWAYS_INLINE CONST_FUNCTION v xor_v(const v a, const v b) {
+#if USE_AVX512
+        const v512 result = { .m512i = _mm512_xor_si512(a.m512i, b.m512i) };
+#elif USE_AVX2
+        const v256 result = { .m256i = _mm256_xor_si256(a.m256i, b.m256i) };
+#elif USE_SSE2
+        const v128 result = { .m128i = _mm_xor_si128(a.m128i, b.m128i) };
+#elif USE_ARM_NEON
+        const v128 result = { .uint8x16 = veorq_u8(a.uint8x16, b.uint8x16) };
+#elif USE_ALTIVEC
+        const v128 result = { .uint8x16 = vec_xor(a.uint8x16, b.uint8x16) };
+#else
+        const v128 result = { .v2u64 = a.v2u64 ^ b.v2u64 };
+#endif
+
+        return result;
+}
+
+static ALWAYS_INLINE CONST_FUNCTION v shuffle_epi8_v(const v vec, const v mask) {
+#if USE_AVX512
+        const v512 result = { .m512i = _mm512_shuffle_epi8(vec.m512i, mask.m512i) };
+#elif USE_AVX2
+        const v256 result = { .m256i = _mm256_shuffle_epi8(vec.m256i, mask.m256i) };
+#elif USE_SSSE3
+        const v128 result = { .m128i = _mm_shuffle_epi8(vec.m128i, mask.m128i) };
+#elif USE_ARM_NEON
+# if defined(RS_HAVE_VQTBL1Q_U8) && RS_HAVE_VQTBL1Q_U8
+        const v128 result = { .uint8x16 = vqtbl1q_u8(vec.uint8x16, mask.uint8x16) };
+# else
+        /* There's no NEON instruction mapping 1-to-1 to _mm_shuffle_epi8, but
+         * this should have the same result...
+         */
+        const v128 result = { .uint8x16 = vcombine_u8(vtbl2_u8(vec.uint8x8x2,
+                                                               vget_low_u8(mask.uint8x16)),
+                                                      vtbl2_u8(vec.uint8x8x2,
+                                                               vget_high_u8(mask.uint8x16))) };
+
+# endif
+#elif USE_ALTIVEC
+        const v128 zeros = set1_epi8_v(0),
+                   result = { .uint8x16 = vec_perm(vec.uint8x16, zeros.uint8x16, mask.uint8x16) };
+#elif defined(RS_HAVE_BUILTIN_SHUFFLE) && RS_HAVE_BUILTIN_SHUFFLE
+        const v16u8v zeros = { 0, 0, 0, 0, 0, 0, 0, 0
+                             , 0, 0, 0, 0, 0, 0, 0, 0 };
+        const v128 result = { .v16u8 = __builtin_shuffle(vec.v16u8, zeros, mask.v16u8) };
+#else
+        v128 result = { .u64 = { 0, 0 } };
+
+# define DO_BYTE(i) \
+        result.u8[i] = mask.u8[i] & 0x80 ? 0 : vec.u8[mask.u8[i] & 0x0F];
+
+        DO_BYTE( 0); DO_BYTE( 1); DO_BYTE( 2); DO_BYTE( 3);
+        DO_BYTE( 4); DO_BYTE( 5); DO_BYTE( 6); DO_BYTE( 7);
+        DO_BYTE( 8); DO_BYTE( 9); DO_BYTE(10); DO_BYTE(11);
+        DO_BYTE(12); DO_BYTE(13); DO_BYTE(14); DO_BYTE(15);
+#endif
+
+        return result;
+}
+
+static ALWAYS_INLINE UNALIGNED_ACCESS void storeu_v(uint8_t *out, const v vec) {
+#if USE_AVX512
+        _mm512_storeu_si512((__m512i *)out, vec.m512i);
+#elif USE_AVX2
+        _mm256_storeu_si256((__m256i *)out, vec.m256i);
+#elif USE_SSE2
+        _mm_storeu_si128((__m128i *)out, vec.m128i);
+#else
+        memcpy(out, &vec.u64, sizeof(vec.u64));
+#endif
+}
+
+static ALWAYS_INLINE ALIGNED_ACCESS void store_v(uint8_t *out, const v vec) {
+#if USE_AVX512
+        _mm512_store_si512((__m512i *)out, vec.m512i);
+#elif USE_AVX2
+        _mm256_store_si256((__m256i *)out, vec.m256i);
+#elif USE_SSE2
+        _mm_store_si128((__m128i *)out, vec.m128i);
+#elif USE_ARM_NEON
+        vst1q_u8(out, vec.uint8x16);
+#elif USE_ALTIVEC
+        vec_st(vec.uint8x16, 0, out);
+#else
+        storeu_v(out, vec);
+#endif
+}
+
+static ALWAYS_INLINE CONST_FUNCTION v replicate_v128_v(const v128 vec) {
+#if USE_AVX512
+        const v512 result = { .m512i = _mm512_broadcast_i32x4(vec.m128i) };
+#elif USE_AVX2
+        const v256 result = { .m256i = _mm256_broadcastsi128_si256(vec.m128i) };
+#else
+        const v128 result = vec;
+#endif
+
+        return result;
+}
+
+
+//+build !noasm !appengine
+
+// Copyright 2015, Klaus Post, see LICENSE for details.
+
+// Based on http://www.snia.org/sites/default/files2/SDC2013/presentations/NewThinking/EthanMiller_Screaming_Fast_Galois_Field%20Arithmetic_SIMD%20Instructions.pdf
+// and http://jerasure.org/jerasure/gf-complete/tree/master
+
+/*
+// func galMulSSSE3Xor(low, high, in, out []byte)
+TEXT ·galMulSSSE3Xor(SB), 7, $0
+    MOVQ    low+0(FP),SI        // SI: &low
+    MOVQ    high+24(FP),DX      // DX: &high
+    MOVOU  (SI), X6             // X6 low
+    MOVOU  (DX), X7             // X7: high
+    MOVQ    $15, BX             // BX: low mask
+    MOVQ    BX, X8
+    PXOR    X5, X5
+    MOVQ    in+48(FP),SI        // R11: &in
+    MOVQ    in_len+56(FP),R9    // R9: len(in)
+    MOVQ    out+72(FP), DX      // DX: &out
+    PSHUFB  X5, X8              // X8: lomask (unpacked)
+    SHRQ    $4, R9              // len(in) / 16
+    CMPQ    R9 ,$0
+    JEQ     done_xor
+loopback_xor:
+    MOVOU  (SI),X0   // in[x]
+    MOVOU  (DX),X4   // out[x]
+    MOVOU   X0, X1   // in[x]
+    MOVOU   X6, X2   // low copy
+    MOVOU   X7, X3   // high copy
+    PSRLQ   $4, X1   // X1: high input
+    PAND    X8, X0   // X0: low input
+    PAND    X8, X1   // X0: high input
+    PSHUFB  X0, X2   // X2: mul low part
+    PSHUFB  X1, X3   // X3: mul high part
+    PXOR    X2, X3   // X3: Result
+    PXOR    X4, X3   // X3: Result xor existing out
+    MOVOU   X3, (DX) // Store
+    ADDQ    $16, SI  // in+=16
+    ADDQ    $16, DX  // out+=16
+    SUBQ    $1, R9
+    JNZ     loopback_xor
+done_xor:
+    RET
+
+// func galMulSSSE3(low, high, in, out []byte)
+TEXT ·galMulSSSE3(SB), 7, $0
+    MOVQ    low+0(FP),SI        // SI: &low
+    MOVQ    high+24(FP),DX      // DX: &high
+    MOVOU   (SI), X6            // X6 low
+    MOVOU   (DX), X7            // X7: high
+    MOVQ    $15, BX             // BX: low mask
+    MOVQ    BX, X8
+    PXOR    X5, X5
+    MOVQ    in+48(FP),SI        // R11: &in
+    MOVQ    in_len+56(FP),R9    // R9: len(in)
+    MOVQ    out+72(FP), DX      // DX: &out
+    PSHUFB  X5, X8              // X8: lomask (unpacked)
+    SHRQ    $4, R9              // len(in) / 16
+    CMPQ    R9 ,$0
+    JEQ     done
+loopback:
+    MOVOU  (SI),X0   // in[x]
+    MOVOU   X0, X1   // in[x]
+    MOVOU   X6, X2   // low copy
+    MOVOU   X7, X3   // high copy
+    PSRLQ   $4, X1   // X1: high input
+    PAND    X8, X0   // X0: low input
+    PAND    X8, X1   // X0: high input
+    PSHUFB  X0, X2   // X2: mul low part
+    PSHUFB  X1, X3   // X3: mul high part
+    PXOR    X2, X3   // X3: Result
+    MOVOU   X3, (DX) // Store
+    ADDQ    $16, SI  // in+=16
+    ADDQ    $16, DX  // out+=16
+    SUBQ    $1, R9
+    JNZ     loopback
+done:
+    RET
+*/
+
+static ALWAYS_INLINE v reedsolomon_gal_mul_v(
+        const v low_mask_unpacked,
+        const v low_vector,
+        const v high_vector,
+
+        v (*modifier)(const v new, const v old),
+
+        const v in_x,
+        const v old) {
+        const v low_input = and_v(in_x, low_mask_unpacked),
+                in_x_shifted = srli_epi64_v(in_x /*, 4*/),
+                high_input = and_v(in_x_shifted, low_mask_unpacked),
+
+                mul_low_part = shuffle_epi8_v(low_vector, low_input),
+                mul_high_part = shuffle_epi8_v(high_vector, high_input),
+
+                new = xor_v(mul_low_part, mul_high_part),
+                result = modifier(new, old);
+
+        return result;
+}
+
+static ALWAYS_INLINE PROTO_RETURN reedsolomon_gal_mul_impl(
+        PROTO_ARGS,
+        v (*modifier)(const v new, const v old)) {
+        const v low_mask_unpacked = set1_epi8_v(0x0f);
+
+        const v128 low_vector128 = loadu_v128(low),
+                   high_vector128 = loadu_v128(high);
+        const v low_vector = replicate_v128_v(low_vector128),
+                high_vector = replicate_v128_v(high_vector128);
+
+        size_t done = 0;
+
+#if USE_ALIGNED_ACCESS
+# define LOAD(addr) load_v(addr)
+# define STORE(addr, vec) store_v(addr, vec)
+#else
+# define LOAD(addr) loadu_v(addr)
+# define STORE(addr, vec) storeu_v(addr, vec)
+#endif
+
+#if RS_HAVE_CLANG_LOOP_UNROLL
+# pragma clang loop unroll(enable)
+#endif
+        for(size_t x = 0; x < len / sizeof(v); x++) {
+                const v in_x = LOAD(&in[done]),
+                        old = LOAD(&out[done]),
+                        result = reedsolomon_gal_mul_v(
+                                        low_mask_unpacked,
+                                        low_vector, high_vector,
+                                        modifier,
+                                        in_x,
+                                        old);
+
+                STORE(&out[done], result);
+
+                done += sizeof(v);
+        }
+
+        return done;
+}
+
+static ALWAYS_INLINE CONST_FUNCTION v noop(const v new, const v old __attribute__((__unused__))) {
+        return new;
+}
+
+#ifdef HOT
+HOT_FUNCTION
+#endif
+FORCE_ALIGN_ARG_POINTER PROTO(reedsolomon_gal_mul) {
+        return reedsolomon_gal_mul_impl(low, high, in, out, len, noop);
+}
+
+#ifdef HOT
+HOT_FUNCTION
+#endif
+FORCE_ALIGN_ARG_POINTER PROTO(reedsolomon_gal_mul_xor) {
+        return reedsolomon_gal_mul_impl(low, high, in, out, len, xor_v);
+}

seaweed-volume/vendor/reed-solomon-erasure/simd_c/reedsolomon.h

@@ -0,0 +1,54 @@
+/* reedsolomon.h - SIMD-optimized Galois-field multiplication routines
+ *
+ * Copyright (c) 2015, 2016 Nicolas Trangez
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE
+ */
+
+#include <stdint.h>
+
+#if HAVE_CONFIG_H
+# include "config.h"
+#endif
+
+#define PROTO_RETURN size_t
+#define PROTO_ARGS                              \
+        const uint8_t low[16],                  \
+        const uint8_t high[16],                 \
+        const uint8_t *restrict const in,       \
+        uint8_t *restrict const out,            \
+        const size_t len
+#define PROTO(name)                     \
+        PROTO_RETURN                    \
+        name (PROTO_ARGS)
+
+PROTO(reedsolomon_gal_mul);
+PROTO(reedsolomon_gal_mul_xor);
+
+typedef enum {
+        REEDSOLOMON_CPU_GENERIC = 0,
+        REEDSOLOMON_CPU_SSE2 = 1,
+        REEDSOLOMON_CPU_SSSE3 = 2,
+        REEDSOLOMON_CPU_AVX = 3,
+        REEDSOLOMON_CPU_AVX2 = 4,
+        REEDSOLOMON_CPU_NEON = 5,
+        REEDSOLOMON_CPU_ALTIVEC = 6,
+} reedsolomon_cpu_support;
+
+reedsolomon_cpu_support reedsolomon_determine_cpu_support(void);

seaweed-volume/vendor/reed-solomon-erasure/src/core.rs

@@ -0,0 +1,927 @@
+extern crate alloc;
+
+use alloc::sync::Arc;
+use alloc::vec;
+use alloc::vec::Vec;
+use core::num::NonZeroUsize;
+
+use smallvec::SmallVec;
+
+use crate::errors::Error;
+use crate::errors::SBSError;
+
+use crate::matrix::Matrix;
+
+use lru::LruCache;
+
+#[cfg(feature = "std")]
+use parking_lot::Mutex;
+#[cfg(not(feature = "std"))]
+use spin::Mutex;
+
+use super::Field;
+use super::ReconstructShard;
+
+const DATA_DECODE_MATRIX_CACHE_CAPACITY: usize = 254;
+
+// /// Parameters for parallelism.
+// #[derive(PartialEq, Debug, Clone, Copy)]
+// pub struct ParallelParam {
+//     /// Number of bytes to split the slices into for computations
+//     /// which can be done in parallel.
+//     ///
+//     /// Default is 32768.
+//     pub bytes_per_encode: usize,
+// }
+
+// impl ParallelParam {
+//     /// Create a new `ParallelParam` with the given split arity.
+//     pub fn new(bytes_per_encode: usize) -> ParallelParam {
+//         ParallelParam { bytes_per_encode }
+//     }
+// }
+
+// impl Default for ParallelParam {
+//     fn default() -> Self {
+//         ParallelParam::new(32768)
+//     }
+// }
+
+/// Bookkeeper for shard by shard encoding.
+///
+/// This is useful for avoiding incorrect use of
+/// `encode_single` and `encode_single_sep`
+///
+/// # Use cases
+///
+/// Shard by shard encoding is useful for streamed data encoding
+/// where you do not have all the needed data shards immediately,
+/// but you want to spread out the encoding workload rather than
+/// doing the encoding after everything is ready.
+///
+/// A concrete example would be network packets encoding,
+/// where encoding packet by packet as you receive them may be more efficient
+/// than waiting for N packets then encode them all at once.
+///
+/// # Example
+///
+/// ```
+/// # #[macro_use] extern crate reed_solomon_erasure;
+/// # use reed_solomon_erasure::*;
+/// # fn main () {
+/// use reed_solomon_erasure::galois_8::Field;
+/// let r: ReedSolomon<Field> = ReedSolomon::new(3, 2).unwrap();
+///
+/// let mut sbs = ShardByShard::new(&r);
+///
+/// let mut shards = shards!([0u8,  1,  2,  3,  4],
+///                          [5,  6,  7,  8,  9],
+///                          // say we don't have the 3rd data shard yet
+///                          // and we want to fill it in later
+///                          [0,  0,  0,  0,  0],
+///                          [0,  0,  0,  0,  0],
+///                          [0,  0,  0,  0,  0]);
+///
+/// // encode 1st and 2nd data shard
+/// sbs.encode(&mut shards).unwrap();
+/// sbs.encode(&mut shards).unwrap();
+///
+/// // fill in 3rd data shard
+/// shards[2][0] = 10.into();
+/// shards[2][1] = 11.into();
+/// shards[2][2] = 12.into();
+/// shards[2][3] = 13.into();
+/// shards[2][4] = 14.into();
+///
+/// // now do the encoding
+/// sbs.encode(&mut shards).unwrap();
+///
+/// assert!(r.verify(&shards).unwrap());
+/// # }
+/// ```
+#[derive(PartialEq, Debug)]
+pub struct ShardByShard<'a, F: 'a + Field> {
+    codec: &'a ReedSolomon<F>,
+    cur_input: usize,
+}
+
+impl<'a, F: 'a + Field> ShardByShard<'a, F> {
+    /// Creates a new instance of the bookkeeping struct.
+    pub fn new(codec: &'a ReedSolomon<F>) -> ShardByShard<'a, F> {
+        ShardByShard {
+            codec,
+            cur_input: 0,
+        }
+    }
+
+    /// Checks if the parity shards are ready to use.
+    pub fn parity_ready(&self) -> bool {
+        self.cur_input == self.codec.data_shard_count
+    }
+
+    /// Resets the bookkeeping data.
+    ///
+    /// You should call this when you have added and encoded
+    /// all data shards, and have finished using the parity shards.
+    ///
+    /// Returns `SBSError::LeftoverShards` when there are shards encoded
+    /// but parity shards are not ready to use.
+    pub fn reset(&mut self) -> Result<(), SBSError> {
+        if self.cur_input > 0 && !self.parity_ready() {
+            return Err(SBSError::LeftoverShards);
+        }
+
+        self.cur_input = 0;
+
+        Ok(())
+    }
+
+    /// Resets the bookkeeping data without checking.
+    pub fn reset_force(&mut self) {
+        self.cur_input = 0;
+    }
+
+    /// Returns the current input shard index.
+    pub fn cur_input_index(&self) -> usize {
+        self.cur_input
+    }
+
+    fn return_ok_and_incre_cur_input(&mut self) -> Result<(), SBSError> {
+        self.cur_input += 1;
+        Ok(())
+    }
+
+    fn sbs_encode_checks<U: AsRef<[F::Elem]> + AsMut<[F::Elem]>>(
+        &mut self,
+        slices: &mut [U],
+    ) -> Result<(), SBSError> {
+        let internal_checks = |codec: &ReedSolomon<F>, data: &mut [U]| {
+            check_piece_count!(all => codec, data);
+            check_slices!(multi => data);
+
+            Ok(())
+        };
+
+        if self.parity_ready() {
+            return Err(SBSError::TooManyCalls);
+        }
+
+        match internal_checks(self.codec, slices) {
+            Ok(()) => Ok(()),
+            Err(e) => Err(SBSError::RSError(e)),
+        }
+    }
+
+    fn sbs_encode_sep_checks<T: AsRef<[F::Elem]>, U: AsRef<[F::Elem]> + AsMut<[F::Elem]>>(
+        &mut self,
+        data: &[T],
+        parity: &mut [U],
+    ) -> Result<(), SBSError> {
+        let internal_checks = |codec: &ReedSolomon<F>, data: &[T], parity: &mut [U]| {
+            check_piece_count!(data => codec, data);
+            check_piece_count!(parity => codec, parity);
+            check_slices!(multi => data, multi => parity);
+
+            Ok(())
+        };
+
+        if self.parity_ready() {
+            return Err(SBSError::TooManyCalls);
+        }
+
+        match internal_checks(self.codec, data, parity) {
+            Ok(()) => Ok(()),
+            Err(e) => Err(SBSError::RSError(e)),
+        }
+    }
+
+    /// Constructs the parity shards partially using the current input data shard.
+    ///
+    /// Returns `SBSError::TooManyCalls` when all input data shards
+    /// have already been filled in via `encode`
+    pub fn encode<T, U>(&mut self, mut shards: T) -> Result<(), SBSError>
+    where
+        T: AsRef<[U]> + AsMut<[U]>,
+        U: AsRef<[F::Elem]> + AsMut<[F::Elem]>,
+    {
+        let shards = shards.as_mut();
+        self.sbs_encode_checks(shards)?;
+
+        self.codec.encode_single(self.cur_input, shards).unwrap();
+
+        self.return_ok_and_incre_cur_input()
+    }
+
+    /// Constructs the parity shards partially using the current input data shard.
+    ///
+    /// Returns `SBSError::TooManyCalls` when all input data shards
+    /// have already been filled in via `encode`
+    pub fn encode_sep<T: AsRef<[F::Elem]>, U: AsRef<[F::Elem]> + AsMut<[F::Elem]>>(
+        &mut self,
+        data: &[T],
+        parity: &mut [U],
+    ) -> Result<(), SBSError> {
+        self.sbs_encode_sep_checks(data, parity)?;
+
+        self.codec
+            .encode_single_sep(self.cur_input, data[self.cur_input].as_ref(), parity)
+            .unwrap();
+
+        self.return_ok_and_incre_cur_input()
+    }
+}
+
+/// Reed-Solomon erasure code encoder/decoder.
+///
+/// # Common error handling
+///
+/// ## For `encode`, `encode_shards`, `verify`, `verify_shards`, `reconstruct`, `reconstruct_data`, `reconstruct_shards`, `reconstruct_data_shards`
+///
+/// Return `Error::TooFewShards` or `Error::TooManyShards`
+/// when the number of provided shards
+/// does not match the codec's one.
+///
+/// Return `Error::EmptyShard` when the first shard provided is
+/// of zero length.
+///
+/// Return `Error::IncorrectShardSize` when the provided shards
+/// are of different lengths.
+///
+/// ## For `reconstruct`, `reconstruct_data`, `reconstruct_shards`, `reconstruct_data_shards`
+///
+/// Return `Error::TooFewShardsPresent` when there are not
+/// enough shards for reconstruction.
+///
+/// Return `Error::InvalidShardFlags` when the number of flags does not match
+/// the total number of shards.
+///
+/// # Variants of encoding methods
+///
+/// ## `sep`
+///
+/// Methods ending in `_sep` takes an immutable reference to data shards,
+/// and a mutable reference to parity shards.
+///
+/// They are useful as they do not need to borrow the data shards mutably,
+/// and other work that only needs read-only access to data shards can be done
+/// in parallel/concurrently during the encoding.
+///
+/// Following is a table of all the `sep` variants
+///
+/// | not `sep` | `sep` |
+/// | --- | --- |
+/// | `encode_single` | `encode_single_sep` |
+/// | `encode`        | `encode_sep` |
+///
+/// The `sep` variants do similar checks on the provided data shards and
+/// parity shards.
+///
+/// Return `Error::TooFewDataShards`, `Error::TooManyDataShards`,
+/// `Error::TooFewParityShards`, or `Error::TooManyParityShards` when applicable.
+///
+/// ## `single`
+///
+/// Methods containing `single` facilitate shard by shard encoding, where
+/// the parity shards are partially constructed using one data shard at a time.
+/// See `ShardByShard` struct for more details on how shard by shard encoding
+/// can be useful.
+///
+/// They are prone to **misuse**, and it is recommended to use the `ShardByShard`
+/// bookkeeping struct instead for shard by shard encoding.
+///
+/// The ones that are also `sep` are **ESPECIALLY** prone to **misuse**.
+/// Only use them when you actually need the flexibility.
+///
+/// Following is a table of all the shard by shard variants
+///
+/// | all shards at once | shard by shard |
+/// | --- | --- |
+/// | `encode` | `encode_single` |
+/// | `encode_sep` | `encode_single_sep` |
+///
+/// The `single` variants do similar checks on the provided data shards and parity shards,
+/// and also do index check on `i_data`.
+///
+/// Return `Error::InvalidIndex` if `i_data >= data_shard_count`.
+///
+/// # Encoding behaviour
+/// ## For `encode`
+///
+/// You do not need to clear the parity shards beforehand, as the methods
+/// will overwrite them completely.
+///
+/// ## For `encode_single`, `encode_single_sep`
+///
+/// Calling them with `i_data` being `0` will overwrite the parity shards
+/// completely. If you are using the methods correctly, then you do not need
+/// to clear the parity shards beforehand.
+///
+/// # Variants of verifying methods
+///
+/// `verify` allocate sa buffer on the heap of the same size
+/// as the parity shards, and encode the input once using the buffer to store
+/// the computed parity shards, then check if the provided parity shards
+/// match the computed ones.
+///
+/// `verify_with_buffer`, allows you to provide
+/// the buffer to avoid making heap allocation(s) for the buffer in every call.
+///
+/// The `with_buffer` variants also guarantee that the buffer contains the correct
+/// parity shards if the result is `Ok(_)` (i.e. it does not matter whether the
+/// verification passed or not, as long as the result is not an error, the buffer
+/// will contain the correct parity shards after the call).
+///
+/// Following is a table of all the `with_buffer` variants
+///
+/// | not `with_buffer` | `with_buffer` |
+/// | --- | --- |
+/// | `verify` | `verify_with_buffer` |
+///
+/// The `with_buffer` variants also check the dimensions of the buffer and return
+/// `Error::TooFewBufferShards`, `Error::TooManyBufferShards`, `Error::EmptyShard`,
+/// or `Error::IncorrectShardSize` when applicable.
+///
+#[derive(Debug)]
+pub struct ReedSolomon<F: Field> {
+    data_shard_count: usize,
+    parity_shard_count: usize,
+    total_shard_count: usize,
+    matrix: Matrix<F>,
+    data_decode_matrix_cache: Mutex<LruCache<Vec<usize>, Arc<Matrix<F>>>>,
+}
+
+impl<F: Field> Clone for ReedSolomon<F> {
+    fn clone(&self) -> ReedSolomon<F> {
+        ReedSolomon::new(self.data_shard_count, self.parity_shard_count)
+            .expect("basic checks already passed as precondition of existence of self")
+    }
+}
+
+impl<F: Field> PartialEq for ReedSolomon<F> {
+    fn eq(&self, rhs: &ReedSolomon<F>) -> bool {
+        self.data_shard_count == rhs.data_shard_count
+            && self.parity_shard_count == rhs.parity_shard_count
+    }
+}
+
+impl<F: Field> ReedSolomon<F> {
+    // AUDIT
+    //
+    // Error detection responsibilities
+    //
+    // Terminologies and symbols:
+    //   X =A, B, C=> Y: X delegates error checking responsibilities A, B, C to Y
+    //   X:= A, B, C: X needs to handle responsibilities A, B, C
+    //
+    // Encode methods
+    //
+    // `encode_single`:=
+    //   - check index `i_data` within range [0, data shard count)
+    //   - check length of `slices` matches total shard count exactly
+    //   - check consistency of length of individual slices
+    // `encode_single_sep`:=
+    //   - check index `i_data` within range [0, data shard count)
+    //   - check length of `parity` matches parity shard count exactly
+    //   - check consistency of length of individual parity slices
+    //   - check length of `single_data` matches length of first parity slice
+    // `encode`:=
+    //   - check length of `slices` matches total shard count exactly
+    //   - check consistency of length of individual slices
+    // `encode_sep`:=
+    //   - check length of `data` matches data shard count exactly
+    //   - check length of `parity` matches parity shard count exactly
+    //   - check consistency of length of individual data slices
+    //   - check consistency of length of individual parity slices
+    //   - check length of first parity slice matches length of first data slice
+    //
+    // Verify methods
+    //
+    // `verify`:=
+    //   - check length of `slices` matches total shard count exactly
+    //   - check consistency of length of individual slices
+    //
+    //   Generates buffer then passes control to verify_with_buffer
+    //
+    // `verify_with_buffer`:=
+    //   - check length of `slices` matches total shard count exactly
+    //   - check length of `buffer` matches parity shard count exactly
+    //   - check consistency of length of individual slices
+    //   - check consistency of length of individual slices in buffer
+    //   - check length of first slice in buffer matches length of first slice
+    //
+    // Reconstruct methods
+    //
+    // `reconstruct` =ALL=> `reconstruct_internal`
+    // `reconstruct_data`=ALL=> `reconstruct_internal`
+    // `reconstruct_internal`:=
+    //   - check length of `slices` matches total shard count exactly
+    //   - check consistency of length of individual slices
+    //   - check length of `slice_present` matches length of `slices`
+
+    fn get_parity_rows(&self) -> SmallVec<[&[F::Elem]; 32]> {
+        let mut parity_rows = SmallVec::with_capacity(self.parity_shard_count);
+        let matrix = &self.matrix;
+        for i in self.data_shard_count..self.total_shard_count {
+            parity_rows.push(matrix.get_row(i));
+        }
+
+        parity_rows
+    }
+
+    fn build_matrix(data_shards: usize, total_shards: usize) -> Matrix<F> {
+        let vandermonde = Matrix::vandermonde(total_shards, data_shards);
+
+        let top = vandermonde.sub_matrix(0, 0, data_shards, data_shards);
+
+        vandermonde.multiply(&top.invert().unwrap())
+    }
+
+    /// Creates a new instance of Reed-Solomon erasure code encoder/decoder.
+    ///
+    /// Returns `Error::TooFewDataShards` if `data_shards == 0`.
+    ///
+    /// Returns `Error::TooFewParityShards` if `parity_shards == 0`.
+    ///
+    /// Returns `Error::TooManyShards` if `data_shards + parity_shards > F::ORDER`.
+    pub fn new(data_shards: usize, parity_shards: usize) -> Result<ReedSolomon<F>, Error> {
+        if data_shards == 0 {
+            return Err(Error::TooFewDataShards);
+        }
+        if parity_shards == 0 {
+            return Err(Error::TooFewParityShards);
+        }
+        if data_shards + parity_shards > F::ORDER {
+            return Err(Error::TooManyShards);
+        }
+
+        let total_shards = data_shards + parity_shards;
+
+        let matrix = Self::build_matrix(data_shards, total_shards);
+
+        Ok(ReedSolomon {
+            data_shard_count: data_shards,
+            parity_shard_count: parity_shards,
+            total_shard_count: total_shards,
+            matrix,
+            data_decode_matrix_cache: Mutex::new(LruCache::new(
+                NonZeroUsize::new(DATA_DECODE_MATRIX_CACHE_CAPACITY).unwrap(),
+            )),
+        })
+    }
+
+    pub fn data_shard_count(&self) -> usize {
+        self.data_shard_count
+    }
+
+    pub fn parity_shard_count(&self) -> usize {
+        self.parity_shard_count
+    }
+
+    pub fn total_shard_count(&self) -> usize {
+        self.total_shard_count
+    }
+
+    fn code_some_slices<T: AsRef<[F::Elem]>, U: AsMut<[F::Elem]>>(
+        &self,
+        matrix_rows: &[&[F::Elem]],
+        inputs: &[T],
+        outputs: &mut [U],
+    ) {
+        for i_input in 0..self.data_shard_count {
+            self.code_single_slice(matrix_rows, i_input, inputs[i_input].as_ref(), outputs);
+        }
+    }
+
+    fn code_single_slice<U: AsMut<[F::Elem]>>(
+        &self,
+        matrix_rows: &[&[F::Elem]],
+        i_input: usize,
+        input: &[F::Elem],
+        outputs: &mut [U],
+    ) {
+        outputs.iter_mut().enumerate().for_each(|(i_row, output)| {
+            let matrix_row_to_use = matrix_rows[i_row][i_input];
+            let output = output.as_mut();
+
+            if i_input == 0 {
+                F::mul_slice(matrix_row_to_use, input, output);
+            } else {
+                F::mul_slice_add(matrix_row_to_use, input, output);
+            }
+        })
+    }
+
+    fn check_some_slices_with_buffer<T, U>(
+        &self,
+        matrix_rows: &[&[F::Elem]],
+        inputs: &[T],
+        to_check: &[T],
+        buffer: &mut [U],
+    ) -> bool
+    where
+        T: AsRef<[F::Elem]>,
+        U: AsRef<[F::Elem]> + AsMut<[F::Elem]>,
+    {
+        self.code_some_slices(matrix_rows, inputs, buffer);
+
+        let at_least_one_mismatch_present = buffer
+            .iter_mut()
+            .enumerate()
+            .map(|(i, expected_parity_shard)| {
+                expected_parity_shard.as_ref() == to_check[i].as_ref()
+            })
+            .any(|x| !x); // find the first false (some slice is different from the expected one)
+        !at_least_one_mismatch_present
+    }
+
+    /// Constructs the parity shards partially using only the data shard
+    /// indexed by `i_data`.
+    ///
+    /// The slots where the parity shards sit at will be overwritten.
+    ///
+    /// # Warning
+    ///
+    /// You must apply this method on the data shards in strict sequential order (0..data shard count),
+    /// otherwise the parity shards will be incorrect.
+    ///
+    /// It is recommended to use the `ShardByShard` bookkeeping struct instead of this method directly.
+    pub fn encode_single<T, U>(&self, i_data: usize, mut shards: T) -> Result<(), Error>
+    where
+        T: AsRef<[U]> + AsMut<[U]>,
+        U: AsRef<[F::Elem]> + AsMut<[F::Elem]>,
+    {
+        let slices = shards.as_mut();
+
+        check_slice_index!(data => self, i_data);
+        check_piece_count!(all=> self, slices);
+        check_slices!(multi => slices);
+
+        // Get the slice of output buffers.
+        let (mut_input, output) = slices.split_at_mut(self.data_shard_count);
+
+        let input = mut_input[i_data].as_ref();
+
+        self.encode_single_sep(i_data, input, output)
+    }
+
+    /// Constructs the parity shards partially using only the data shard provided.
+    ///
+    /// The data shard must match the index `i_data`.
+    ///
+    /// The slots where the parity shards sit at will be overwritten.
+    ///
+    /// # Warning
+    ///
+    /// You must apply this method on the data shards in strict sequential order (0..data shard count),
+    /// otherwise the parity shards will be incorrect.
+    ///
+    /// It is recommended to use the `ShardByShard` bookkeeping struct instead of this method directly.
+    pub fn encode_single_sep<U: AsRef<[F::Elem]> + AsMut<[F::Elem]>>(
+        &self,
+        i_data: usize,
+        single_data: &[F::Elem],
+        parity: &mut [U],
+    ) -> Result<(), Error> {
+        check_slice_index!(data => self, i_data);
+        check_piece_count!(parity => self, parity);
+        check_slices!(multi => parity, single => single_data);
+
+        let parity_rows = self.get_parity_rows();
+
+        // Do the coding.
+        self.code_single_slice(&parity_rows, i_data, single_data, parity);
+
+        Ok(())
+    }
+
+    /// Constructs the parity shards.
+    ///
+    /// The slots where the parity shards sit at will be overwritten.
+    pub fn encode<T, U>(&self, mut shards: T) -> Result<(), Error>
+    where
+        T: AsRef<[U]> + AsMut<[U]>,
+        U: AsRef<[F::Elem]> + AsMut<[F::Elem]>,
+    {
+        let slices: &mut [U] = shards.as_mut();
+
+        check_piece_count!(all => self, slices);
+        check_slices!(multi => slices);
+
+        // Get the slice of output buffers.
+        let (input, output) = slices.split_at_mut(self.data_shard_count);
+
+        self.encode_sep(&*input, output)
+    }
+
+    /// Constructs the parity shards using a read-only view into the
+    /// data shards.
+    ///
+    /// The slots where the parity shards sit at will be overwritten.
+    pub fn encode_sep<T: AsRef<[F::Elem]>, U: AsRef<[F::Elem]> + AsMut<[F::Elem]>>(
+        &self,
+        data: &[T],
+        parity: &mut [U],
+    ) -> Result<(), Error> {
+        check_piece_count!(data => self, data);
+        check_piece_count!(parity => self, parity);
+        check_slices!(multi => data, multi => parity);
+
+        let parity_rows = self.get_parity_rows();
+
+        // Do the coding.
+        self.code_some_slices(&parity_rows, data, parity);
+
+        Ok(())
+    }
+
+    /// Checks if the parity shards are correct.
+    ///
+    /// This is a wrapper of `verify_with_buffer`.
+    pub fn verify<T: AsRef<[F::Elem]>>(&self, slices: &[T]) -> Result<bool, Error> {
+        check_piece_count!(all => self, slices);
+        check_slices!(multi => slices);
+
+        let slice_len = slices[0].as_ref().len();
+
+        let mut buffer: SmallVec<[Vec<F::Elem>; 32]> =
+            SmallVec::with_capacity(self.parity_shard_count);
+
+        for _ in 0..self.parity_shard_count {
+            buffer.push(vec![F::zero(); slice_len]);
+        }
+
+        self.verify_with_buffer(slices, &mut buffer)
+    }
+
+    /// Checks if the parity shards are correct.
+    pub fn verify_with_buffer<T, U>(&self, slices: &[T], buffer: &mut [U]) -> Result<bool, Error>
+    where
+        T: AsRef<[F::Elem]>,
+        U: AsRef<[F::Elem]> + AsMut<[F::Elem]>,
+    {
+        check_piece_count!(all => self, slices);
+        check_piece_count!(parity_buf => self, buffer);
+        check_slices!(multi => slices, multi => buffer);
+
+        let data = &slices[0..self.data_shard_count];
+        let to_check = &slices[self.data_shard_count..];
+
+        let parity_rows = self.get_parity_rows();
+
+        Ok(self.check_some_slices_with_buffer(&parity_rows, data, to_check, buffer))
+    }
+
+    /// Reconstructs all shards.
+    ///
+    /// The shards marked not present are only overwritten when no error
+    /// is detected. All provided shards must have the same length.
+    ///
+    /// This means if the method returns an `Error`, then nothing is touched.
+    ///
+    /// `reconstruct`, `reconstruct_data`, `reconstruct_shards`,
+    /// `reconstruct_data_shards` share the same core code base.
+    pub fn reconstruct<T: ReconstructShard<F>>(&self, slices: &mut [T]) -> Result<(), Error> {
+        self.reconstruct_internal(slices, false)
+    }
+
+    /// Reconstructs only the data shards.
+    ///
+    /// The shards marked not present are only overwritten when no error
+    /// is detected. All provided shards must have the same length.
+    ///
+    /// This means if the method returns an `Error`, then nothing is touched.
+    ///
+    /// `reconstruct`, `reconstruct_data`, `reconstruct_shards`,
+    /// `reconstruct_data_shards` share the same core code base.
+    pub fn reconstruct_data<T: ReconstructShard<F>>(&self, slices: &mut [T]) -> Result<(), Error> {
+        self.reconstruct_internal(slices, true)
+    }
+
+    fn get_data_decode_matrix(
+        &self,
+        valid_indices: &[usize],
+        invalid_indices: &[usize],
+    ) -> Arc<Matrix<F>> {
+        {
+            let mut cache = self.data_decode_matrix_cache.lock();
+            if let Some(entry) = cache.get(invalid_indices) {
+                return entry.clone();
+            }
+        }
+        // Pull out the rows of the matrix that correspond to the shards that
+        // we have and build a square matrix. This matrix could be used to
+        // generate the shards that we have from the original data.
+        let mut sub_matrix = Matrix::new(self.data_shard_count, self.data_shard_count);
+        for (sub_matrix_row, &valid_index) in valid_indices.iter().enumerate() {
+            for c in 0..self.data_shard_count {
+                sub_matrix.set(sub_matrix_row, c, self.matrix.get(valid_index, c));
+            }
+        }
+        // Invert the matrix, so we can go from the encoded shards back to the
+        // original data. Then pull out the row that generates the shard that
+        // we want to decode. Note that since this matrix maps back to the
+        // original data, it can be used to create a data shard, but not a
+        // parity shard.
+        let data_decode_matrix = Arc::new(sub_matrix.invert().unwrap());
+        // Cache the inverted matrix for future use keyed on the indices of the
+        // invalid rows.
+        {
+            let data_decode_matrix = data_decode_matrix.clone();
+            let mut cache = self.data_decode_matrix_cache.lock();
+            cache.put(Vec::from(invalid_indices), data_decode_matrix);
+        }
+        data_decode_matrix
+    }
+
+    fn reconstruct_internal<T: ReconstructShard<F>>(
+        &self,
+        shards: &mut [T],
+        data_only: bool,
+    ) -> Result<(), Error> {
+        check_piece_count!(all => self, shards);
+
+        let data_shard_count = self.data_shard_count;
+
+        // Quick check: are all of the shards present?  If so, there's
+        // nothing to do.
+        let mut number_present = 0;
+        let mut shard_len = None;
+
+        for shard in shards.iter_mut() {
+            if let Some(len) = shard.len() {
+                if len == 0 {
+                    return Err(Error::EmptyShard);
+                }
+                number_present += 1;
+                if let Some(old_len) = shard_len {
+                    if len != old_len {
+                        // mismatch between shards.
+                        return Err(Error::IncorrectShardSize);
+                    }
+                }
+                shard_len = Some(len);
+            }
+        }
+
+        if number_present == self.total_shard_count {
+            // Cool.  All of the shards are there.  We don't
+            // need to do anything.
+            return Ok(());
+        }
+
+        // More complete sanity check
+        if number_present < data_shard_count {
+            return Err(Error::TooFewShardsPresent);
+        }
+
+        let shard_len = shard_len.expect("at least one shard present; qed");
+
+        // Pull out an array holding just the shards that
+        // correspond to the rows of the submatrix.  These shards
+        // will be the input to the decoding process that re-creates
+        // the missing data shards.
+        //
+        // Also, create an array of indices of the valid rows we do have
+        // and the invalid rows we don't have.
+        //
+        // The valid indices are used to construct the data decode matrix,
+        // the invalid indices are used to key the data decode matrix
+        // in the data decode matrix cache.
+        //
+        // We only need exactly N valid indices, where N = `data_shard_count`,
+        // as the data decode matrix is a N x N matrix, thus only needs
+        // N valid indices for determining the N rows to pick from
+        // `self.matrix`.
+        let mut sub_shards: SmallVec<[&[F::Elem]; 32]> = SmallVec::with_capacity(data_shard_count);
+        let mut missing_data_slices: SmallVec<[&mut [F::Elem]; 32]> =
+            SmallVec::with_capacity(self.parity_shard_count);
+        let mut missing_parity_slices: SmallVec<[&mut [F::Elem]; 32]> =
+            SmallVec::with_capacity(self.parity_shard_count);
+        let mut valid_indices: SmallVec<[usize; 32]> = SmallVec::with_capacity(data_shard_count);
+        let mut invalid_indices: SmallVec<[usize; 32]> = SmallVec::with_capacity(data_shard_count);
+
+        // Separate the shards into groups
+        for (matrix_row, shard) in shards.iter_mut().enumerate() {
+            // get or initialize the shard so we can reconstruct in-place,
+            // but if we are only reconstructing data shard,
+            // do not initialize if the shard is not a data shard
+            let shard_data = if matrix_row >= data_shard_count && data_only {
+                shard.get().ok_or(None)
+            } else {
+                shard.get_or_initialize(shard_len).map_err(Some)
+            };
+
+            match shard_data {
+                Ok(shard) => {
+                    if sub_shards.len() < data_shard_count {
+                        sub_shards.push(shard);
+                        valid_indices.push(matrix_row);
+                    } else {
+                        // Already have enough shards in `sub_shards`
+                        // as we only need N shards, where N = `data_shard_count`,
+                        // for the data decode matrix
+                        //
+                        // So nothing to do here
+                    }
+                }
+                Err(None) => {
+                    // the shard data is not meant to be initialized here,
+                    // but we should still note it missing.
+                    invalid_indices.push(matrix_row);
+                }
+                Err(Some(x)) => {
+                    // initialized missing shard data.
+                    let shard = x?;
+                    if matrix_row < data_shard_count {
+                        missing_data_slices.push(shard);
+                    } else {
+                        missing_parity_slices.push(shard);
+                    }
+
+                    invalid_indices.push(matrix_row);
+                }
+            }
+        }
+
+        let data_decode_matrix = self.get_data_decode_matrix(&valid_indices, &invalid_indices);
+
+        // Re-create any data shards that were missing.
+        //
+        // The input to the coding is all of the shards we actually
+        // have, and the output is the missing data shards. The computation
+        // is done using the special decode matrix we just built.
+        let mut matrix_rows: SmallVec<[&[F::Elem]; 32]> =
+            SmallVec::with_capacity(self.parity_shard_count);
+
+        for i_slice in invalid_indices
+            .iter()
+            .cloned()
+            .take_while(|i| i < &data_shard_count)
+        {
+            matrix_rows.push(data_decode_matrix.get_row(i_slice));
+        }
+
+        self.code_some_slices(&matrix_rows, &sub_shards, &mut missing_data_slices);
+
+        if data_only {
+            Ok(())
+        } else {
+            // Now that we have all of the data shards intact, we can
+            // compute any of the parity that is missing.
+            //
+            // The input to the coding is ALL of the data shards, including
+            // any that we just calculated.  The output is whichever of the
+            // parity shards were missing.
+            let mut matrix_rows: SmallVec<[&[F::Elem]; 32]> =
+                SmallVec::with_capacity(self.parity_shard_count);
+            let parity_rows = self.get_parity_rows();
+
+            for i_slice in invalid_indices
+                .iter()
+                .cloned()
+                .skip_while(|i| i < &data_shard_count)
+            {
+                matrix_rows.push(parity_rows[i_slice - data_shard_count]);
+            }
+            {
+                // Gather up all the data shards.
+                // old data shards are in `sub_shards`,
+                // new ones are in `missing_data_slices`.
+                let mut i_old_data_slice = 0;
+                let mut i_new_data_slice = 0;
+
+                let mut all_data_slices: SmallVec<[&[F::Elem]; 32]> =
+                    SmallVec::with_capacity(data_shard_count);
+
+                let mut next_maybe_good = 0;
+                let mut push_good_up_to = move |data_slices: &mut SmallVec<_>, up_to| {
+                    // if next_maybe_good == up_to, this loop is a no-op.
+                    for _ in next_maybe_good..up_to {
+                        // push all good indices we just skipped.
+                        data_slices.push(sub_shards[i_old_data_slice]);
+                        i_old_data_slice += 1;
+                    }
+
+                    next_maybe_good = up_to + 1;
+                };
+
+                for i_slice in invalid_indices
+                    .iter()
+                    .cloned()
+                    .take_while(|i| i < &data_shard_count)
+                {
+                    push_good_up_to(&mut all_data_slices, i_slice);
+                    all_data_slices.push(missing_data_slices[i_new_data_slice]);
+                    i_new_data_slice += 1;
+                }
+                push_good_up_to(&mut all_data_slices, data_shard_count);
+
+                // Now do the actual computation for the missing
+                // parity shards
+                self.code_some_slices(&matrix_rows, &all_data_slices, &mut missing_parity_slices);
+            }
+
+            Ok(())
+        }
+    }
+}

seaweed-volume/vendor/reed-solomon-erasure/src/errors.rs

@@ -0,0 +1,158 @@
+use core::fmt::Formatter;
+
+#[derive(PartialEq, Debug, Clone, Copy)]
+pub enum Error {
+    TooFewShards,
+    TooManyShards,
+    TooFewDataShards,
+    TooManyDataShards,
+    TooFewParityShards,
+    TooManyParityShards,
+    TooFewBufferShards,
+    TooManyBufferShards,
+    IncorrectShardSize,
+    TooFewShardsPresent,
+    EmptyShard,
+    InvalidShardFlags,
+    InvalidIndex,
+}
+
+impl Error {
+    fn to_string(&self) -> &str {
+        match *self {
+            Error::TooFewShards=> "The number of provided shards is smaller than the one in codec",
+            Error::TooManyShards => "The number of provided shards is greater than the one in codec",
+            Error::TooFewDataShards => "The number of provided data shards is smaller than the one in codec",
+            Error::TooManyDataShards => "The number of provided data shards is greater than the one in codec",
+            Error::TooFewParityShards => "The number of provided parity shards is smaller than the one in codec",
+            Error::TooManyParityShards => "The number of provided parity shards is greater than the one in codec",
+            Error::TooFewBufferShards => "The number of provided buffer shards is smaller than the number of parity shards in codec",
+            Error::TooManyBufferShards => "The number of provided buffer shards is greater than the number of parity shards in codec",
+            Error::IncorrectShardSize => "At least one of the provided shards is not of the correct size",
+            Error::TooFewShardsPresent => "The number of shards present is smaller than number of parity shards, cannot reconstruct missing shards",
+            Error::EmptyShard => "The first shard provided is of zero length",
+            Error::InvalidShardFlags => "The number of flags does not match the total number of shards",
+            Error::InvalidIndex => "The data shard index provided is greater or equal to the number of data shards in codec",
+        }
+    }
+}
+
+impl core::fmt::Display for Error {
+    fn fmt(&self, f: &mut Formatter) -> Result<(), core::fmt::Error> {
+        write!(f, "{}", self.to_string())
+    }
+}
+
+#[cfg(feature = "std")]
+impl std::error::Error for Error {
+    fn description(&self) -> &str {
+        self.to_string()
+    }
+}
+
+#[derive(PartialEq, Debug, Clone, Copy)]
+pub enum SBSError {
+    TooManyCalls,
+    LeftoverShards,
+    RSError(Error),
+}
+
+impl SBSError {
+    fn to_string(&self) -> &str {
+        match *self {
+            SBSError::TooManyCalls => "Too many calls",
+            SBSError::LeftoverShards => "Leftover shards",
+            SBSError::RSError(ref e) => e.to_string(),
+        }
+    }
+}
+
+impl core::fmt::Display for SBSError {
+    fn fmt(&self, f: &mut Formatter) -> Result<(), core::fmt::Error> {
+        write!(f, "{}", self.to_string())
+    }
+}
+
+#[cfg(feature = "std")]
+impl std::error::Error for SBSError {
+    fn description(&self) -> &str {
+        self.to_string()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use crate::errors::Error;
+    use crate::errors::SBSError;
+
+    #[test]
+    fn test_error_to_string_is_okay() {
+        assert_eq!(
+            Error::TooFewShards.to_string(),
+            "The number of provided shards is smaller than the one in codec"
+        );
+        assert_eq!(
+            Error::TooManyShards.to_string(),
+            "The number of provided shards is greater than the one in codec"
+        );
+        assert_eq!(
+            Error::TooFewDataShards.to_string(),
+            "The number of provided data shards is smaller than the one in codec"
+        );
+        assert_eq!(
+            Error::TooManyDataShards.to_string(),
+            "The number of provided data shards is greater than the one in codec"
+        );
+        assert_eq!(
+            Error::TooFewParityShards.to_string(),
+            "The number of provided parity shards is smaller than the one in codec"
+        );
+        assert_eq!(
+            Error::TooManyParityShards.to_string(),
+            "The number of provided parity shards is greater than the one in codec"
+        );
+        assert_eq!(
+            Error::TooFewBufferShards.to_string(),
+            "The number of provided buffer shards is smaller than the number of parity shards in codec"
+        );
+        assert_eq!(
+            Error::TooManyBufferShards.to_string(),
+            "The number of provided buffer shards is greater than the number of parity shards in codec"
+        );
+        assert_eq!(
+            Error::IncorrectShardSize.to_string(),
+            "At least one of the provided shards is not of the correct size"
+        );
+        assert_eq!(Error::TooFewShardsPresent.to_string(), "The number of shards present is smaller than number of parity shards, cannot reconstruct missing shards");
+        assert_eq!(
+            Error::EmptyShard.to_string(),
+            "The first shard provided is of zero length"
+        );
+        assert_eq!(
+            Error::InvalidShardFlags.to_string(),
+            "The number of flags does not match the total number of shards"
+        );
+        assert_eq!(
+            Error::InvalidIndex.to_string(),
+            "The data shard index provided is greater or equal to the number of data shards in codec"
+        );
+    }
+
+    #[test]
+    fn test_sbserror_to_string_is_okay() {
+        assert_eq!(SBSError::TooManyCalls.to_string(), "Too many calls");
+        assert_eq!(SBSError::LeftoverShards.to_string(), "Leftover shards");
+    }
+
+    #[cfg(feature = "std")]
+    #[test]
+    fn test_error_display_does_not_panic() {
+        println!("{}", Error::TooFewShards);
+    }
+
+    #[cfg(feature = "std")]
+    #[test]
+    fn test_sbserror_display_does_not_panic() {
+        println!("{}", SBSError::TooManyCalls);
+    }
+}

seaweed-volume/vendor/reed-solomon-erasure/src/galois_16.rs

@@ -0,0 +1,412 @@
+//! GF(2^16) implementation.
+//!
+//! More accurately, this is a `GF((2^8)^2)` implementation which builds an extension
+//! field of `GF(2^8)`, as defined in the `galois_8` module.
+
+use crate::galois_8;
+use core::ops::{Add, Div, Mul, Sub};
+
+// the irreducible polynomial used as a modulus for the field.
+// print R.irreducible_element(2,algorithm="first_lexicographic" )
+// x^2 + a*x + a^7
+//
+// hopefully it is a fast polynomial
+const EXT_POLY: [u8; 3] = [1, 2, 128];
+
+/// The field GF(2^16).
+#[derive(Debug, Default, Copy, Clone, PartialEq, Eq)]
+pub struct Field;
+
+impl crate::Field for Field {
+    const ORDER: usize = 65536;
+
+    type Elem = [u8; 2];
+
+    fn add(a: [u8; 2], b: [u8; 2]) -> [u8; 2] {
+        (Element(a) + Element(b)).0
+    }
+
+    fn mul(a: [u8; 2], b: [u8; 2]) -> [u8; 2] {
+        (Element(a) * Element(b)).0
+    }
+
+    fn div(a: [u8; 2], b: [u8; 2]) -> [u8; 2] {
+        (Element(a) / Element(b)).0
+    }
+
+    fn exp(elem: [u8; 2], n: usize) -> [u8; 2] {
+        Element(elem).exp(n).0
+    }
+
+    fn zero() -> [u8; 2] {
+        [0; 2]
+    }
+
+    fn one() -> [u8; 2] {
+        [0, 1]
+    }
+
+    fn nth_internal(n: usize) -> [u8; 2] {
+        [(n >> 8) as u8, n as u8]
+    }
+}
+
+/// Type alias of ReedSolomon over GF(2^8).
+pub type ReedSolomon = crate::ReedSolomon<Field>;
+
+/// Type alias of ShardByShard over GF(2^8).
+pub type ShardByShard<'a> = crate::ShardByShard<'a, Field>;
+
+/// An element of `GF(2^16)`.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+struct Element(pub [u8; 2]);
+
+impl Element {
+    // Create the zero element.
+    fn zero() -> Self {
+        Element([0, 0])
+    }
+
+    // A constant element evaluating to `n`.
+    fn constant(n: u8) -> Element {
+        Element([0, n])
+    }
+
+    // Whether this is the zero element.
+    fn is_zero(&self) -> bool {
+        self.0 == [0; 2]
+    }
+
+    fn exp(mut self, n: usize) -> Element {
+        if n == 0 {
+            Element::constant(1)
+        } else if self == Element::zero() {
+            Element::zero()
+        } else {
+            let x = self;
+            for _ in 1..n {
+                self = self * x;
+            }
+
+            self
+        }
+    }
+
+    // reduces from some polynomial with degree <= 2.
+    #[inline]
+    fn reduce_from(mut x: [u8; 3]) -> Self {
+        if x[0] != 0 {
+            // divide x by EXT_POLY and use remainder.
+            // i = 0 here.
+            // c*x^(i+j)  = a*x^i*b*x^j
+            x[1] ^= galois_8::mul(EXT_POLY[1], x[0]);
+            x[2] ^= galois_8::mul(EXT_POLY[2], x[0]);
+        }
+
+        Element([x[1], x[2]])
+    }
+
+    fn degree(&self) -> usize {
+        if self.0[0] != 0 {
+            1
+        } else {
+            0
+        }
+    }
+}
+
+impl From<[u8; 2]> for Element {
+    fn from(c: [u8; 2]) -> Self {
+        Element(c)
+    }
+}
+
+impl Default for Element {
+    fn default() -> Self {
+        Element::zero()
+    }
+}
+
+impl Add for Element {
+    type Output = Element;
+
+    fn add(self, other: Self) -> Element {
+        Element([self.0[0] ^ other.0[0], self.0[1] ^ other.0[1]])
+    }
+}
+
+impl Sub for Element {
+    type Output = Element;
+
+    fn sub(self, other: Self) -> Element {
+        self.add(other)
+    }
+}
+
+impl Mul for Element {
+    type Output = Element;
+
+    fn mul(self, rhs: Self) -> Element {
+        // FOIL; our elements are linear at most, with two coefficients
+        let out: [u8; 3] = [
+            galois_8::mul(self.0[0], rhs.0[0]),
+            galois_8::add(
+                galois_8::mul(self.0[1], rhs.0[0]),
+                galois_8::mul(self.0[0], rhs.0[1]),
+            ),
+            galois_8::mul(self.0[1], rhs.0[1]),
+        ];
+
+        Element::reduce_from(out)
+    }
+}
+
+impl Mul<u8> for Element {
+    type Output = Element;
+
+    fn mul(self, rhs: u8) -> Element {
+        Element([galois_8::mul(rhs, self.0[0]), galois_8::mul(rhs, self.0[1])])
+    }
+}
+
+impl Div for Element {
+    type Output = Element;
+
+    fn div(self, rhs: Self) -> Element {
+        self * rhs.inverse()
+    }
+}
+
+// helpers for division.
+
+#[derive(Debug)]
+enum EgcdRhs {
+    Element(Element),
+    ExtPoly,
+}
+
+impl Element {
+    // compute extended euclidean algorithm against an element of self,
+    // where the GCD is known to be constant.
+    fn const_egcd(self, rhs: EgcdRhs) -> (u8, Element, Element) {
+        if self.is_zero() {
+            let rhs = match rhs {
+                EgcdRhs::Element(elem) => elem,
+                EgcdRhs::ExtPoly => panic!("const_egcd invoked with divisible"),
+            };
+            (rhs.0[1], Element::constant(0), Element::constant(1))
+        } else {
+            let (cur_quotient, cur_remainder) = match rhs {
+                EgcdRhs::Element(rhs) => rhs.polynom_div(self),
+                EgcdRhs::ExtPoly => Element::div_ext_by(self),
+            };
+
+            // GCD is constant because EXT_POLY is irreducible
+            let (g, x, y) = cur_remainder.const_egcd(EgcdRhs::Element(self));
+            (g, y + (cur_quotient * x), x)
+        }
+    }
+
+    // divide EXT_POLY by self.
+    fn div_ext_by(rhs: Self) -> (Element, Element) {
+        if rhs.degree() == 0 {
+            // dividing by constant is the same as multiplying by another constant.
+            // and all constant multiples of EXT_POLY are in the equivalence class
+            // of 0.
+            return (Element::zero(), Element::zero());
+        }
+
+        // divisor is ensured linear here.
+        // now ensure divisor is monic.
+        let leading_mul_inv = galois_8::div(1, rhs.0[0]);
+
+        let monictized = rhs * leading_mul_inv;
+        let mut poly = EXT_POLY;
+
+        for i in 0..2 {
+            let coef = poly[i];
+            for j in 1..2 {
+                if rhs.0[j] != 0 {
+                    poly[i + j] ^= galois_8::mul(monictized.0[j], coef);
+                }
+            }
+        }
+
+        let remainder = Element::constant(poly[2]);
+        let quotient = Element([poly[0], poly[1]]) * leading_mul_inv;
+
+        (quotient, remainder)
+    }
+
+    fn polynom_div(self, rhs: Self) -> (Element, Element) {
+        let divisor_degree = rhs.degree();
+        if rhs.is_zero() {
+            panic!("divide by 0");
+        } else if self.degree() < divisor_degree {
+            // If divisor's degree (len-1) is bigger, all dividend is a remainder
+            (Element::zero(), self)
+        } else if divisor_degree == 0 {
+            // divide by constant.
+            let invert = galois_8::div(1, rhs.0[1]);
+            let quotient = Element([
+                galois_8::mul(invert, self.0[0]),
+                galois_8::mul(invert, self.0[1]),
+            ]);
+
+            (quotient, Element::zero())
+        } else {
+            // self degree is at least divisor degree, divisor degree not 0.
+            // therefore both are 1.
+            debug_assert_eq!(self.degree(), divisor_degree);
+            debug_assert_eq!(self.degree(), 1);
+
+            // ensure rhs is constant.
+            let leading_mul_inv = galois_8::div(1, rhs.0[0]);
+            let monic = Element([
+                galois_8::mul(leading_mul_inv, rhs.0[0]),
+                galois_8::mul(leading_mul_inv, rhs.0[1]),
+            ]);
+
+            let leading_coeff = self.0[0];
+            let mut remainder = self.0[1];
+
+            if monic.0[1] != 0 {
+                remainder ^= galois_8::mul(monic.0[1], self.0[0]);
+            }
+
+            (
+                Element::constant(galois_8::mul(leading_mul_inv, leading_coeff)),
+                Element::constant(remainder),
+            )
+        }
+    }
+
+    /// Convert the inverse of this field element. Panics if zero.
+    fn inverse(self) -> Element {
+        if self.is_zero() {
+            panic!("Cannot invert 0");
+        }
+
+        // first step of extended euclidean algorithm.
+        // done here because EXT_POLY is outside the scope of `Element`.
+        let (gcd, y) = {
+            // self / EXT_POLY = (0, self)
+            let remainder = self;
+
+            // GCD is constant because EXT_POLY is irreducible
+            let (g, x, _) = remainder.const_egcd(EgcdRhs::ExtPoly);
+
+            (g, x)
+        };
+
+        // we still need to normalize it by dividing by the gcd
+        if gcd != 0 {
+            // EXT_POLY is irreducible so the GCD will always be constant.
+            // EXT_POLY*x + self*y = gcd
+            // self*y = gcd - EXT_POLY*x
+            //
+            // EXT_POLY*x is representative of the equivalence class of 0.
+            let normalizer = galois_8::div(1, gcd);
+            y * normalizer
+        } else {
+            // self is equivalent to zero.
+            panic!("Cannot invert 0");
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use quickcheck::Arbitrary;
+
+    impl Arbitrary for Element {
+        fn arbitrary<G: quickcheck::Gen>(gen: &mut G) -> Self {
+            let a = u8::arbitrary(gen);
+            let b = u8::arbitrary(gen);
+
+            Element([a, b])
+        }
+    }
+
+    quickcheck! {
+        fn qc_add_associativity(a: Element, b: Element, c: Element) -> bool {
+            a + (b + c) == (a + b) + c
+        }
+
+        fn qc_mul_associativity(a: Element, b: Element, c: Element) -> bool {
+            a * (b * c) == (a * b) * c
+        }
+
+        fn qc_additive_identity(a: Element) -> bool {
+            let zero = Element::zero();
+            a - (zero - a) == zero
+        }
+
+        fn qc_multiplicative_identity(a: Element) -> bool {
+            a.is_zero() || {
+                let one = Element([0, 1]);
+                (one / a) * a == one
+            }
+        }
+
+        fn qc_add_commutativity(a: Element, b: Element) -> bool {
+            a + b == b + a
+        }
+
+        fn qc_mul_commutativity(a: Element, b: Element) -> bool {
+            a * b == b * a
+        }
+
+        fn qc_add_distributivity(a: Element, b: Element, c: Element) -> bool {
+            a * (b + c) == (a * b) + (a * c)
+        }
+
+        fn qc_inverse(a: Element) -> bool {
+            a.is_zero() || {
+                let inv = a.inverse();
+                a * inv == Element::constant(1)
+            }
+        }
+
+        fn qc_exponent_1(a: Element, n: u8) -> bool {
+            a.is_zero() || n == 0 || {
+                let mut b = a.exp(n as usize);
+                for _ in 1..n {
+                    b = b / a;
+                }
+
+                a == b
+            }
+        }
+
+        fn qc_exponent_2(a: Element, n: u8) -> bool {
+            a.is_zero() || {
+                let mut res = true;
+                let mut b = Element::constant(1);
+
+                for i in 0..n {
+                    res = res && b == a.exp(i as usize);
+                    b = b * a;
+                }
+
+                res
+            }
+        }
+
+        fn qc_exp_zero_is_one(a: Element) -> bool {
+            a.exp(0) == Element::constant(1)
+        }
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_div_b_is_0() {
+        let _ = Element([1, 0]) / Element::zero();
+    }
+
+    #[test]
+    fn zero_to_zero_is_one() {
+        assert_eq!(Element::zero().exp(0), Element::constant(1))
+    }
+}

seaweed-volume/vendor/reed-solomon-erasure/src/galois_8.rs

@@ -0,0 +1,621 @@
+//! Implementation of GF(2^8): the finite field with 2^8 elements.
+
+include!(concat!(env!("OUT_DIR"), "/table.rs"));
+
+/// The field GF(2^8).
+#[derive(Debug, Default, Copy, Clone, PartialEq, Eq)]
+pub struct Field;
+
+impl crate::Field for Field {
+    const ORDER: usize = 256;
+    type Elem = u8;
+
+    fn add(a: u8, b: u8) -> u8 {
+        add(a, b)
+    }
+
+    fn mul(a: u8, b: u8) -> u8 {
+        mul(a, b)
+    }
+
+    fn div(a: u8, b: u8) -> u8 {
+        div(a, b)
+    }
+
+    fn exp(elem: u8, n: usize) -> u8 {
+        exp(elem, n)
+    }
+
+    fn zero() -> u8 {
+        0
+    }
+
+    fn one() -> u8 {
+        1
+    }
+
+    fn nth_internal(n: usize) -> u8 {
+        n as u8
+    }
+
+    fn mul_slice(c: u8, input: &[u8], out: &mut [u8]) {
+        mul_slice(c, input, out)
+    }
+
+    fn mul_slice_add(c: u8, input: &[u8], out: &mut [u8]) {
+        mul_slice_xor(c, input, out)
+    }
+}
+
+/// Type alias of ReedSolomon over GF(2^8).
+pub type ReedSolomon = crate::ReedSolomon<Field>;
+
+/// Type alias of ShardByShard over GF(2^8).
+pub type ShardByShard<'a> = crate::ShardByShard<'a, Field>;
+
+/// Add two elements.
+pub fn add(a: u8, b: u8) -> u8 {
+    a ^ b
+}
+
+/// Subtract `b` from `a`.
+#[cfg(test)]
+pub fn sub(a: u8, b: u8) -> u8 {
+    a ^ b
+}
+
+/// Multiply two elements.
+pub fn mul(a: u8, b: u8) -> u8 {
+    MUL_TABLE[a as usize][b as usize]
+}
+
+/// Divide one element by another. `b`, the divisor, may not be 0.
+pub fn div(a: u8, b: u8) -> u8 {
+    if a == 0 {
+        0
+    } else if b == 0 {
+        panic!("Divisor is 0")
+    } else {
+        let log_a = LOG_TABLE[a as usize];
+        let log_b = LOG_TABLE[b as usize];
+        let mut log_result = log_a as isize - log_b as isize;
+        if log_result < 0 {
+            log_result += 255;
+        }
+        EXP_TABLE[log_result as usize]
+    }
+}
+
+/// Compute a^n.
+pub fn exp(a: u8, n: usize) -> u8 {
+    if n == 0 {
+        1
+    } else if a == 0 {
+        0
+    } else {
+        let log_a = LOG_TABLE[a as usize];
+        let mut log_result = log_a as usize * n;
+        while 255 <= log_result {
+            log_result -= 255;
+        }
+        EXP_TABLE[log_result]
+    }
+}
+
+const PURE_RUST_UNROLL: isize = 4;
+
+macro_rules! return_if_empty {
+    (
+        $len:expr
+    ) => {
+        if $len == 0 {
+            return;
+        }
+    };
+}
+
+#[cfg(not(all(
+    feature = "simd-accel",
+    any(target_arch = "x86_64", target_arch = "aarch64"),
+    not(target_env = "msvc"),
+    not(any(target_os = "android", target_os = "ios"))
+)))]
+pub fn mul_slice(c: u8, input: &[u8], out: &mut [u8]) {
+    mul_slice_pure_rust(c, input, out);
+}
+
+#[cfg(not(all(
+    feature = "simd-accel",
+    any(target_arch = "x86_64", target_arch = "aarch64"),
+    not(target_env = "msvc"),
+    not(any(target_os = "android", target_os = "ios"))
+)))]
+pub fn mul_slice_xor(c: u8, input: &[u8], out: &mut [u8]) {
+    mul_slice_xor_pure_rust(c, input, out);
+}
+
+fn mul_slice_pure_rust(c: u8, input: &[u8], out: &mut [u8]) {
+    let mt = &MUL_TABLE[c as usize];
+    let mt_ptr: *const u8 = &mt[0];
+
+    assert_eq!(input.len(), out.len());
+
+    let len: isize = input.len() as isize;
+    return_if_empty!(len);
+
+    let mut input_ptr: *const u8 = &input[0];
+    let mut out_ptr: *mut u8 = &mut out[0];
+
+    let mut n: isize = 0;
+    unsafe {
+        assert_eq!(4, PURE_RUST_UNROLL);
+        if len > PURE_RUST_UNROLL {
+            let len_minus_unroll = len - PURE_RUST_UNROLL;
+            while n < len_minus_unroll {
+                *out_ptr = *mt_ptr.offset(*input_ptr as isize);
+                *out_ptr.offset(1) = *mt_ptr.offset(*input_ptr.offset(1) as isize);
+                *out_ptr.offset(2) = *mt_ptr.offset(*input_ptr.offset(2) as isize);
+                *out_ptr.offset(3) = *mt_ptr.offset(*input_ptr.offset(3) as isize);
+
+                input_ptr = input_ptr.offset(PURE_RUST_UNROLL);
+                out_ptr = out_ptr.offset(PURE_RUST_UNROLL);
+                n += PURE_RUST_UNROLL;
+            }
+        }
+        while n < len {
+            *out_ptr = *mt_ptr.offset(*input_ptr as isize);
+
+            input_ptr = input_ptr.offset(1);
+            out_ptr = out_ptr.offset(1);
+            n += 1;
+        }
+    }
+    /* for n in 0..input.len() {
+     *   out[n] = mt[input[n] as usize]
+     * }
+     */
+}
+
+fn mul_slice_xor_pure_rust(c: u8, input: &[u8], out: &mut [u8]) {
+    let mt = &MUL_TABLE[c as usize];
+    let mt_ptr: *const u8 = &mt[0];
+
+    assert_eq!(input.len(), out.len());
+
+    let len: isize = input.len() as isize;
+    return_if_empty!(len);
+
+    let mut input_ptr: *const u8 = &input[0];
+    let mut out_ptr: *mut u8 = &mut out[0];
+
+    let mut n: isize = 0;
+    unsafe {
+        assert_eq!(4, PURE_RUST_UNROLL);
+        if len > PURE_RUST_UNROLL {
+            let len_minus_unroll = len - PURE_RUST_UNROLL;
+            while n < len_minus_unroll {
+                *out_ptr ^= *mt_ptr.offset(*input_ptr as isize);
+                *out_ptr.offset(1) ^= *mt_ptr.offset(*input_ptr.offset(1) as isize);
+                *out_ptr.offset(2) ^= *mt_ptr.offset(*input_ptr.offset(2) as isize);
+                *out_ptr.offset(3) ^= *mt_ptr.offset(*input_ptr.offset(3) as isize);
+
+                input_ptr = input_ptr.offset(PURE_RUST_UNROLL);
+                out_ptr = out_ptr.offset(PURE_RUST_UNROLL);
+                n += PURE_RUST_UNROLL;
+            }
+        }
+        while n < len {
+            *out_ptr ^= *mt_ptr.offset(*input_ptr as isize);
+
+            input_ptr = input_ptr.offset(1);
+            out_ptr = out_ptr.offset(1);
+            n += 1;
+        }
+    }
+    /* for n in 0..input.len() {
+     *   out[n] ^= mt[input[n] as usize];
+     * }
+     */
+}
+
+#[cfg(test)]
+fn slice_xor(input: &[u8], out: &mut [u8]) {
+    assert_eq!(input.len(), out.len());
+
+    let len: isize = input.len() as isize;
+    return_if_empty!(len);
+
+    let mut input_ptr: *const u8 = &input[0];
+    let mut out_ptr: *mut u8 = &mut out[0];
+
+    let mut n: isize = 0;
+    unsafe {
+        assert_eq!(4, PURE_RUST_UNROLL);
+        if len > PURE_RUST_UNROLL {
+            let len_minus_unroll = len - PURE_RUST_UNROLL;
+            while n < len_minus_unroll {
+                *out_ptr ^= *input_ptr;
+                *out_ptr.offset(1) ^= *input_ptr.offset(1);
+                *out_ptr.offset(2) ^= *input_ptr.offset(2);
+                *out_ptr.offset(3) ^= *input_ptr.offset(3);
+
+                input_ptr = input_ptr.offset(PURE_RUST_UNROLL);
+                out_ptr = out_ptr.offset(PURE_RUST_UNROLL);
+                n += PURE_RUST_UNROLL;
+            }
+        }
+        while n < len {
+            *out_ptr ^= *input_ptr;
+
+            input_ptr = input_ptr.offset(1);
+            out_ptr = out_ptr.offset(1);
+            n += 1;
+        }
+    }
+    /* for n in 0..input.len() {
+     *   out[n] ^= input[n]
+     * }
+     */
+}
+
+#[cfg(all(
+    feature = "simd-accel",
+    any(target_arch = "x86_64", target_arch = "aarch64"),
+    not(target_env = "msvc"),
+    not(any(target_os = "android", target_os = "ios"))
+))]
+extern "C" {
+    fn reedsolomon_gal_mul(
+        low: *const u8,
+        high: *const u8,
+        input: *const u8,
+        out: *mut u8,
+        len: libc::size_t,
+    ) -> libc::size_t;
+
+    fn reedsolomon_gal_mul_xor(
+        low: *const u8,
+        high: *const u8,
+        input: *const u8,
+        out: *mut u8,
+        len: libc::size_t,
+    ) -> libc::size_t;
+}
+
+#[cfg(all(
+    feature = "simd-accel",
+    any(target_arch = "x86_64", target_arch = "aarch64"),
+    not(target_env = "msvc"),
+    not(any(target_os = "android", target_os = "ios"))
+))]
+pub fn mul_slice(c: u8, input: &[u8], out: &mut [u8]) {
+    let low: *const u8 = &MUL_TABLE_LOW[c as usize][0];
+    let high: *const u8 = &MUL_TABLE_HIGH[c as usize][0];
+
+    assert_eq!(input.len(), out.len());
+
+    let input_ptr: *const u8 = &input[0];
+    let out_ptr: *mut u8 = &mut out[0];
+    let size: libc::size_t = input.len();
+
+    let bytes_done: usize =
+        unsafe { reedsolomon_gal_mul(low, high, input_ptr, out_ptr, size) as usize };
+
+    mul_slice_pure_rust(c, &input[bytes_done..], &mut out[bytes_done..]);
+}
+
+#[cfg(all(
+    feature = "simd-accel",
+    any(target_arch = "x86_64", target_arch = "aarch64"),
+    not(target_env = "msvc"),
+    not(any(target_os = "android", target_os = "ios"))
+))]
+pub fn mul_slice_xor(c: u8, input: &[u8], out: &mut [u8]) {
+    let low: *const u8 = &MUL_TABLE_LOW[c as usize][0];
+    let high: *const u8 = &MUL_TABLE_HIGH[c as usize][0];
+
+    assert_eq!(input.len(), out.len());
+
+    let input_ptr: *const u8 = &input[0];
+    let out_ptr: *mut u8 = &mut out[0];
+    let size: libc::size_t = input.len();
+
+    let bytes_done: usize =
+        unsafe { reedsolomon_gal_mul_xor(low, high, input_ptr, out_ptr, size) as usize };
+
+    mul_slice_xor_pure_rust(c, &input[bytes_done..], &mut out[bytes_done..]);
+}
+
+#[cfg(test)]
+mod tests {
+    extern crate alloc;
+
+    use alloc::vec;
+
+    use super::*;
+    use crate::tests::fill_random;
+    use rand;
+
+    static BACKBLAZE_LOG_TABLE: [u8; 256] = [
+        //-1,    0,    1,   25,    2,   50,   26,  198,
+        // first value is changed from -1 to 0
+        0, 0, 1, 25, 2, 50, 26, 198, 3, 223, 51, 238, 27, 104, 199, 75, 4, 100, 224, 14, 52, 141,
+        239, 129, 28, 193, 105, 248, 200, 8, 76, 113, 5, 138, 101, 47, 225, 36, 15, 33, 53, 147,
+        142, 218, 240, 18, 130, 69, 29, 181, 194, 125, 106, 39, 249, 185, 201, 154, 9, 120, 77,
+        228, 114, 166, 6, 191, 139, 98, 102, 221, 48, 253, 226, 152, 37, 179, 16, 145, 34, 136, 54,
+        208, 148, 206, 143, 150, 219, 189, 241, 210, 19, 92, 131, 56, 70, 64, 30, 66, 182, 163,
+        195, 72, 126, 110, 107, 58, 40, 84, 250, 133, 186, 61, 202, 94, 155, 159, 10, 21, 121, 43,
+        78, 212, 229, 172, 115, 243, 167, 87, 7, 112, 192, 247, 140, 128, 99, 13, 103, 74, 222,
+        237, 49, 197, 254, 24, 227, 165, 153, 119, 38, 184, 180, 124, 17, 68, 146, 217, 35, 32,
+        137, 46, 55, 63, 209, 91, 149, 188, 207, 205, 144, 135, 151, 178, 220, 252, 190, 97, 242,
+        86, 211, 171, 20, 42, 93, 158, 132, 60, 57, 83, 71, 109, 65, 162, 31, 45, 67, 216, 183,
+        123, 164, 118, 196, 23, 73, 236, 127, 12, 111, 246, 108, 161, 59, 82, 41, 157, 85, 170,
+        251, 96, 134, 177, 187, 204, 62, 90, 203, 89, 95, 176, 156, 169, 160, 81, 11, 245, 22, 235,
+        122, 117, 44, 215, 79, 174, 213, 233, 230, 231, 173, 232, 116, 214, 244, 234, 168, 80, 88,
+        175,
+    ];
+
+    #[test]
+    fn log_table_same_as_backblaze() {
+        for i in 0..256 {
+            assert_eq!(LOG_TABLE[i], BACKBLAZE_LOG_TABLE[i]);
+        }
+    }
+
+    #[test]
+    fn test_associativity() {
+        for a in 0..256 {
+            let a = a as u8;
+            for b in 0..256 {
+                let b = b as u8;
+                for c in 0..256 {
+                    let c = c as u8;
+                    let x = add(a, add(b, c));
+                    let y = add(add(a, b), c);
+                    assert_eq!(x, y);
+                    let x = mul(a, mul(b, c));
+                    let y = mul(mul(a, b), c);
+                    assert_eq!(x, y);
+                }
+            }
+        }
+    }
+
+    quickcheck! {
+        fn qc_add_associativity(a: u8, b: u8, c: u8) -> bool {
+            add(a, add(b, c)) == add(add(a, b), c)
+        }
+
+        fn qc_mul_associativity(a: u8, b: u8, c: u8) -> bool {
+            mul(a, mul(b, c)) == mul(mul(a, b), c)
+        }
+    }
+
+    #[test]
+    fn test_identity() {
+        for a in 0..256 {
+            let a = a as u8;
+            let b = sub(0, a);
+            let c = sub(a, b);
+            assert_eq!(c, 0);
+            if a != 0 {
+                let b = div(1, a);
+                let c = mul(a, b);
+                assert_eq!(c, 1);
+            }
+        }
+    }
+
+    quickcheck! {
+        fn qc_additive_identity(a: u8) -> bool {
+            sub(a, sub(0, a)) == 0
+        }
+
+        fn qc_multiplicative_identity(a: u8) -> bool {
+            if a == 0 { true }
+            else      { mul(a, div(1, a)) == 1 }
+        }
+    }
+
+    #[test]
+    fn test_commutativity() {
+        for a in 0..256 {
+            let a = a as u8;
+            for b in 0..256 {
+                let b = b as u8;
+                let x = add(a, b);
+                let y = add(b, a);
+                assert_eq!(x, y);
+                let x = mul(a, b);
+                let y = mul(b, a);
+                assert_eq!(x, y);
+            }
+        }
+    }
+
+    quickcheck! {
+        fn qc_add_commutativity(a: u8, b: u8) -> bool {
+            add(a, b) == add(b, a)
+        }
+
+        fn qc_mul_commutativity(a: u8, b: u8) -> bool {
+            mul(a, b) == mul(b, a)
+        }
+    }
+
+    #[test]
+    fn test_distributivity() {
+        for a in 0..256 {
+            let a = a as u8;
+            for b in 0..256 {
+                let b = b as u8;
+                for c in 0..256 {
+                    let c = c as u8;
+                    let x = mul(a, add(b, c));
+                    let y = add(mul(a, b), mul(a, c));
+                    assert_eq!(x, y);
+                }
+            }
+        }
+    }
+
+    quickcheck! {
+        fn qc_add_distributivity(a: u8, b: u8, c: u8) -> bool {
+            mul(a, add(b, c)) == add(mul(a, b), mul(a, c))
+        }
+    }
+
+    #[test]
+    fn test_exp() {
+        for a in 0..256 {
+            let a = a as u8;
+            let mut power = 1u8;
+            for j in 0..256 {
+                let x = exp(a, j);
+                assert_eq!(x, power);
+                power = mul(power, a);
+            }
+        }
+    }
+
+    #[test]
+    fn test_galois() {
+        assert_eq!(mul(3, 4), 12);
+        assert_eq!(mul(7, 7), 21);
+        assert_eq!(mul(23, 45), 41);
+
+        let input = [
+            0, 1, 2, 3, 4, 5, 6, 10, 50, 100, 150, 174, 201, 255, 99, 32, 67, 85, 200, 199, 198,
+            197, 196, 195, 194, 193, 192, 191, 190, 189, 188, 187, 186, 185,
+        ];
+        let mut output1 = vec![0; input.len()];
+        let mut output2 = vec![0; input.len()];
+        mul_slice(25, &input, &mut output1);
+        let expect = [
+            0x0, 0x19, 0x32, 0x2b, 0x64, 0x7d, 0x56, 0xfa, 0xb8, 0x6d, 0xc7, 0x85, 0xc3, 0x1f,
+            0x22, 0x7, 0x25, 0xfe, 0xda, 0x5d, 0x44, 0x6f, 0x76, 0x39, 0x20, 0xb, 0x12, 0x11, 0x8,
+            0x23, 0x3a, 0x75, 0x6c, 0x47,
+        ];
+        for i in 0..input.len() {
+            assert_eq!(expect[i], output1[i]);
+        }
+        mul_slice(25, &input, &mut output2);
+        for i in 0..input.len() {
+            assert_eq!(expect[i], output2[i]);
+        }
+
+        let expect_xor = [
+            0x0, 0x2d, 0x5a, 0x77, 0xb4, 0x99, 0xee, 0x2f, 0x79, 0xf2, 0x7, 0x51, 0xd4, 0x19, 0x31,
+            0xc9, 0xf8, 0xfc, 0xf9, 0x4f, 0x62, 0x15, 0x38, 0xfb, 0xd6, 0xa1, 0x8c, 0x96, 0xbb,
+            0xcc, 0xe1, 0x22, 0xf, 0x78,
+        ];
+        mul_slice_xor(52, &input, &mut output1);
+        for i in 0..input.len() {
+            assert_eq!(expect_xor[i], output1[i]);
+        }
+        mul_slice_xor(52, &input, &mut output2);
+        for i in 0..input.len() {
+            assert_eq!(expect_xor[i], output2[i]);
+        }
+
+        let expect = [
+            0x0, 0xb1, 0x7f, 0xce, 0xfe, 0x4f, 0x81, 0x9e, 0x3, 0x6, 0xe8, 0x75, 0xbd, 0x40, 0x36,
+            0xa3, 0x95, 0xcb, 0xc, 0xdd, 0x6c, 0xa2, 0x13, 0x23, 0x92, 0x5c, 0xed, 0x1b, 0xaa,
+            0x64, 0xd5, 0xe5, 0x54, 0x9a,
+        ];
+        mul_slice(177, &input, &mut output1);
+        for i in 0..input.len() {
+            assert_eq!(expect[i], output1[i]);
+        }
+        mul_slice(177, &input, &mut output2);
+        for i in 0..input.len() {
+            assert_eq!(expect[i], output2[i]);
+        }
+
+        let expect_xor = [
+            0x0, 0xc4, 0x95, 0x51, 0x37, 0xf3, 0xa2, 0xfb, 0xec, 0xc5, 0xd0, 0xc7, 0x53, 0x88,
+            0xa3, 0xa5, 0x6, 0x78, 0x97, 0x9f, 0x5b, 0xa, 0xce, 0xa8, 0x6c, 0x3d, 0xf9, 0xdf, 0x1b,
+            0x4a, 0x8e, 0xe8, 0x2c, 0x7d,
+        ];
+        mul_slice_xor(117, &input, &mut output1);
+        for i in 0..input.len() {
+            assert_eq!(expect_xor[i], output1[i]);
+        }
+        mul_slice_xor(117, &input, &mut output2);
+        for i in 0..input.len() {
+            assert_eq!(expect_xor[i], output2[i]);
+        }
+
+        assert_eq!(exp(2, 2), 4);
+        assert_eq!(exp(5, 20), 235);
+        assert_eq!(exp(13, 7), 43);
+    }
+
+    #[test]
+    fn test_slice_add() {
+        let length_list = [16, 32, 34];
+        for len in length_list.iter() {
+            let mut input = vec![0; *len];
+            fill_random(&mut input);
+            let mut output = vec![0; *len];
+            fill_random(&mut output);
+            let mut expect = vec![0; *len];
+            for i in 0..expect.len() {
+                expect[i] = input[i] ^ output[i];
+            }
+            slice_xor(&input, &mut output);
+            for i in 0..expect.len() {
+                assert_eq!(expect[i], output[i]);
+            }
+            fill_random(&mut output);
+            for i in 0..expect.len() {
+                expect[i] = input[i] ^ output[i];
+            }
+            slice_xor(&input, &mut output);
+            for i in 0..expect.len() {
+                assert_eq!(expect[i], output[i]);
+            }
+        }
+    }
+
+    #[test]
+    fn test_div_a_is_0() {
+        assert_eq!(0, div(0, 100));
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_div_b_is_0() {
+        div(1, 0);
+    }
+
+    #[test]
+    fn test_same_as_maybe_ffi() {
+        let len = 10_003;
+        for _ in 0..100 {
+            let c = rand::random::<u8>();
+            let mut input = vec![0; len];
+            fill_random(&mut input);
+            {
+                let mut output = vec![0; len];
+                fill_random(&mut output);
+                let mut output_copy = output.clone();
+
+                mul_slice(c, &input, &mut output);
+                mul_slice(c, &input, &mut output_copy);
+
+                assert_eq!(output, output_copy);
+            }
+            {
+                let mut output = vec![0; len];
+                fill_random(&mut output);
+                let mut output_copy = output.clone();
+
+                mul_slice_xor(c, &input, &mut output);
+                mul_slice_xor(c, &input, &mut output_copy);
+
+                assert_eq!(output, output_copy);
+            }
+        }
+    }
+}

seaweed-volume/vendor/reed-solomon-erasure/src/lib.rs

@@ -0,0 +1,200 @@
+//! This crate provides an encoder/decoder for Reed-Solomon erasure code.
+//!
+//! Please note that erasure coding means errors are not directly detected or corrected,
+//! but missing data pieces (shards) can be reconstructed given that
+//! the configuration provides high enough redundancy.
+//!
+//! You will have to implement error detection separately (e.g. via checksums)
+//! and simply leave out the corrupted shards when attempting to reconstruct
+//! the missing data.
+#![allow(dead_code)]
+#![cfg_attr(not(feature = "std"), no_std)]
+
+#[cfg(test)]
+#[macro_use]
+extern crate quickcheck;
+
+#[cfg(test)]
+extern crate rand;
+
+extern crate smallvec;
+
+#[cfg(feature = "simd-accel")]
+extern crate libc;
+
+use ::core::iter;
+use ::core::iter::FromIterator;
+
+#[macro_use]
+mod macros;
+
+mod core;
+mod errors;
+mod matrix;
+
+#[cfg(test)]
+mod tests;
+
+pub mod galois_16;
+pub mod galois_8;
+
+pub use crate::errors::Error;
+pub use crate::errors::SBSError;
+
+pub use crate::core::ReedSolomon;
+pub use crate::core::ShardByShard;
+
+// TODO: Can be simplified once https://github.com/rust-lang/rfcs/issues/2505 is resolved
+#[cfg(not(feature = "std"))]
+use libm::log2f as log2;
+#[cfg(feature = "std")]
+fn log2(n: f32) -> f32 {
+    n.log2()
+}
+
+/// A finite field to perform encoding over.
+pub trait Field: Sized {
+    /// The order of the field. This is a limit on the number of shards
+    /// in an encoding.
+    const ORDER: usize;
+
+    /// The representational type of the field.
+    type Elem: Default + Clone + Copy + PartialEq + ::core::fmt::Debug;
+
+    /// Add two elements together.
+    fn add(a: Self::Elem, b: Self::Elem) -> Self::Elem;
+
+    /// Multiply two elements together.
+    fn mul(a: Self::Elem, b: Self::Elem) -> Self::Elem;
+
+    /// Divide a by b. Panics is b is zero.
+    fn div(a: Self::Elem, b: Self::Elem) -> Self::Elem;
+
+    /// Raise `a` to the n'th power.
+    fn exp(a: Self::Elem, n: usize) -> Self::Elem;
+
+    /// The "zero" element or additive identity.
+    fn zero() -> Self::Elem;
+
+    /// The "one" element or multiplicative identity.
+    fn one() -> Self::Elem;
+
+    fn nth_internal(n: usize) -> Self::Elem;
+
+    /// Yield the nth element of the field. Panics if n >= ORDER.
+    /// Assignment is arbitrary but must be unique to `n`.
+    fn nth(n: usize) -> Self::Elem {
+        if n >= Self::ORDER {
+            let pow = log2(Self::ORDER as f32) as usize;
+            panic!("{} out of bounds for GF(2^{}) member", n, pow)
+        }
+
+        Self::nth_internal(n)
+    }
+
+    /// Multiply a slice of elements by another. Writes into the output slice.
+    ///
+    /// # Panics
+    /// Panics if the output slice does not have equal length to the input.
+    fn mul_slice(elem: Self::Elem, input: &[Self::Elem], out: &mut [Self::Elem]) {
+        assert_eq!(input.len(), out.len());
+
+        for (i, o) in input.iter().zip(out) {
+            *o = Self::mul(elem.clone(), i.clone())
+        }
+    }
+
+    /// Multiply a slice of elements by another, adding each result to the corresponding value in
+    /// `out`.
+    ///
+    /// # Panics
+    /// Panics if the output slice does not have equal length to the input.
+    fn mul_slice_add(elem: Self::Elem, input: &[Self::Elem], out: &mut [Self::Elem]) {
+        assert_eq!(input.len(), out.len());
+
+        for (i, o) in input.iter().zip(out) {
+            *o = Self::add(o.clone(), Self::mul(elem.clone(), i.clone()))
+        }
+    }
+}
+
+/// Something which might hold a shard.
+///
+/// This trait is used in reconstruction, where some of the shards
+/// may be unknown.
+pub trait ReconstructShard<F: Field> {
+    /// The size of the shard data; `None` if empty.
+    fn len(&self) -> Option<usize>;
+
+    /// Get a mutable reference to the shard data, returning `None` if uninitialized.
+    fn get(&mut self) -> Option<&mut [F::Elem]>;
+
+    /// Get a mutable reference to the shard data, initializing it to the
+    /// given length if it was `None`. Returns an error if initialization fails.
+    fn get_or_initialize(
+        &mut self,
+        len: usize,
+    ) -> Result<&mut [F::Elem], Result<&mut [F::Elem], Error>>;
+}
+
+impl<F: Field, T: AsRef<[F::Elem]> + AsMut<[F::Elem]> + FromIterator<F::Elem>> ReconstructShard<F>
+    for Option<T>
+{
+    fn len(&self) -> Option<usize> {
+        self.as_ref().map(|x| x.as_ref().len())
+    }
+
+    fn get(&mut self) -> Option<&mut [F::Elem]> {
+        self.as_mut().map(|x| x.as_mut())
+    }
+
+    fn get_or_initialize(
+        &mut self,
+        len: usize,
+    ) -> Result<&mut [F::Elem], Result<&mut [F::Elem], Error>> {
+        let is_some = self.is_some();
+        let x = self
+            .get_or_insert_with(|| iter::repeat(F::zero()).take(len).collect())
+            .as_mut();
+
+        if is_some {
+            Ok(x)
+        } else {
+            Err(Ok(x))
+        }
+    }
+}
+
+impl<F: Field, T: AsRef<[F::Elem]> + AsMut<[F::Elem]>> ReconstructShard<F> for (T, bool) {
+    fn len(&self) -> Option<usize> {
+        if !self.1 {
+            None
+        } else {
+            Some(self.0.as_ref().len())
+        }
+    }
+
+    fn get(&mut self) -> Option<&mut [F::Elem]> {
+        if !self.1 {
+            None
+        } else {
+            Some(self.0.as_mut())
+        }
+    }
+
+    fn get_or_initialize(
+        &mut self,
+        len: usize,
+    ) -> Result<&mut [F::Elem], Result<&mut [F::Elem], Error>> {
+        let x = self.0.as_mut();
+        if x.len() == len {
+            if self.1 {
+                Ok(x)
+            } else {
+                Err(Ok(x))
+            }
+        } else {
+            Err(Err(Error::IncorrectShardSize))
+        }
+    }
+}

seaweed-volume/vendor/reed-solomon-erasure/src/macros.rs

@@ -0,0 +1,245 @@
+/// Constructs vector of shards.
+///
+/// # Example
+/// ```rust
+/// # #[macro_use] extern crate reed_solomon_erasure;
+/// # use reed_solomon_erasure::*;
+/// # fn main () {
+/// let shards: Vec<Vec<u8>> = shards!([1, 2, 3],
+///                                    [4, 5, 6]);
+/// # }
+/// ```
+#[macro_export]
+macro_rules! shards {
+    (
+        $( [ $( $x:expr ),* ] ),*
+    ) => {{
+        vec![ $( vec![ $( $x ),* ] ),* ]
+    }}
+}
+
+/// Makes it easier to work with 2D slices, arrays, etc.
+///
+/// # Examples
+/// ## Byte arrays on stack to `Vec<&[u8]>`
+/// ```rust
+/// # #[macro_use] extern crate reed_solomon_erasure;
+/// # fn main () {
+/// let array: [[u8; 3]; 2] = [[1, 2, 3],
+///                             [4, 5, 6]];
+///
+/// let refs: Vec<&[u8]> =
+///     convert_2D_slices!(array =>to_vec &[u8]);
+/// # }
+/// ```
+/// ## Byte arrays on stack to `Vec<&mut [u8]>` (borrow mutably)
+/// ```rust
+/// # #[macro_use] extern crate reed_solomon_erasure;
+/// # fn main () {
+/// let mut array: [[u8; 3]; 2] = [[1, 2, 3],
+///                                [4, 5, 6]];
+///
+/// let refs: Vec<&mut [u8]> =
+///     convert_2D_slices!(array =>to_mut_vec &mut [u8]);
+/// # }
+/// ```
+/// ## Byte arrays on stack to `SmallVec<[&mut [u8]; 32]>` (borrow mutably)
+/// ```rust
+/// # #[macro_use] extern crate reed_solomon_erasure;
+/// # extern crate smallvec;
+/// # use smallvec::SmallVec;
+/// # fn main () {
+/// let mut array: [[u8; 3]; 2] = [[1, 2, 3],
+///                                [4, 5, 6]];
+///
+/// let refs: SmallVec<[&mut [u8]; 32]> =
+///     convert_2D_slices!(array =>to_mut SmallVec<[&mut [u8]; 32]>,
+///                        SmallVec::with_capacity);
+/// # }
+/// ```
+/// ## Shard array to `SmallVec<[&mut [u8]; 32]>` (borrow mutably)
+/// ```rust
+/// # #[macro_use] extern crate reed_solomon_erasure;
+/// # extern crate smallvec;
+/// # use smallvec::SmallVec;
+/// # fn main () {
+/// let mut shards = shards!([1, 2, 3],
+///                          [4, 5, 6]);
+///
+/// let refs: SmallVec<[&mut [u8]; 32]> =
+///     convert_2D_slices!(shards =>to_mut SmallVec<[&mut [u8]; 32]>,
+///                        SmallVec::with_capacity);
+/// # }
+/// ```
+/// ## Shard array to `Vec<&mut [u8]>` (borrow mutably) into `SmallVec<[&mut [u8]; 32]>` (move)
+/// ```rust
+/// # #[macro_use] extern crate reed_solomon_erasure;
+/// # extern crate smallvec;
+/// # use smallvec::SmallVec;
+/// # fn main () {
+/// let mut shards = shards!([1, 2, 3],
+///                          [4, 5, 6]);
+///
+/// let refs1 = convert_2D_slices!(shards =>to_mut_vec &mut [u8]);
+///
+/// let refs2: SmallVec<[&mut [u8]; 32]> =
+///     convert_2D_slices!(refs1 =>into SmallVec<[&mut [u8]; 32]>,
+///                        SmallVec::with_capacity);
+/// # }
+/// ```
+#[macro_export]
+macro_rules! convert_2D_slices {
+    (
+        $slice:expr =>into_vec $dst_type:ty
+    ) => {
+        convert_2D_slices!($slice =>into Vec<$dst_type>,
+                           Vec::with_capacity)
+    };
+    (
+        $slice:expr =>to_vec $dst_type:ty
+    ) => {
+        convert_2D_slices!($slice =>to Vec<$dst_type>,
+                           Vec::with_capacity)
+    };
+    (
+        $slice:expr =>to_mut_vec $dst_type:ty
+    ) => {
+        convert_2D_slices!($slice =>to_mut Vec<$dst_type>,
+                           Vec::with_capacity)
+    };
+    (
+        $slice:expr =>into $dst_type:ty, $with_capacity:path
+    ) => {{
+        let mut result: $dst_type =
+            $with_capacity($slice.len());
+        for i in $slice.into_iter() {
+            result.push(i);
+        }
+        result
+    }};
+    (
+        $slice:expr =>to $dst_type:ty, $with_capacity:path
+    ) => {{
+        let mut result: $dst_type =
+            $with_capacity($slice.len());
+        for i in $slice.iter() {
+            result.push(i);
+        }
+        result
+    }};
+    (
+        $slice:expr =>to_mut $dst_type:ty, $with_capacity:path
+    ) => {{
+        let mut result: $dst_type =
+            $with_capacity($slice.len());
+        for i in $slice.iter_mut() {
+            result.push(i);
+        }
+        result
+    }}
+}
+
+macro_rules! check_slices {
+    (
+        multi => $slices:expr
+    ) => {{
+        let size = $slices[0].as_ref().len();
+        if size == 0 {
+            return Err(Error::EmptyShard);
+        }
+        for slice in $slices.iter() {
+            if slice.as_ref().len() != size {
+                return Err(Error::IncorrectShardSize);
+            }
+        }
+    }};
+    (
+        single => $slice_left:expr, single => $slice_right:expr
+    ) => {{
+        if $slice_left.as_ref().len() != $slice_right.as_ref().len() {
+            return Err(Error::IncorrectShardSize);
+        }
+    }};
+    (
+        multi => $slices:expr, single => $single:expr
+    ) => {{
+        check_slices!(multi => $slices);
+
+        check_slices!(single => $slices[0], single => $single);
+    }};
+    (
+        multi => $slices_left:expr, multi => $slices_right:expr
+    ) => {{
+        check_slices!(multi => $slices_left);
+        check_slices!(multi => $slices_right);
+
+        check_slices!(single => $slices_left[0], single => $slices_right[0]);
+    }}
+}
+
+macro_rules! check_slice_index {
+    (
+        all => $codec:expr, $index:expr
+    ) => {{
+        if $index >= $codec.total_shard_count {
+            return Err(Error::InvalidIndex);
+        }
+    }};
+    (
+        data => $codec:expr, $index:expr
+    ) => {{
+        if $index >= $codec.data_shard_count {
+            return Err(Error::InvalidIndex);
+        }
+    }};
+    (
+        parity => $codec:expr, $index:expr
+    ) => {{
+        if $index >= $codec.parity_shard_count {
+            return Err(Error::InvalidIndex);
+        }
+    }};
+}
+
+macro_rules! check_piece_count {
+    (
+        all => $codec:expr, $pieces:expr
+    ) => {{
+        if $pieces.as_ref().len() < $codec.total_shard_count {
+            return Err(Error::TooFewShards);
+        }
+        if $pieces.as_ref().len() > $codec.total_shard_count {
+            return Err(Error::TooManyShards);
+        }
+    }};
+    (
+        data => $codec:expr, $pieces:expr
+    ) => {{
+        if $pieces.as_ref().len() < $codec.data_shard_count {
+            return Err(Error::TooFewDataShards);
+        }
+        if $pieces.as_ref().len() > $codec.data_shard_count {
+            return Err(Error::TooManyDataShards);
+        }
+    }};
+    (
+        parity => $codec:expr, $pieces:expr
+    ) => {{
+        if $pieces.as_ref().len() < $codec.parity_shard_count {
+            return Err(Error::TooFewParityShards);
+        }
+        if $pieces.as_ref().len() > $codec.parity_shard_count {
+            return Err(Error::TooManyParityShards);
+        }
+    }};
+    (
+        parity_buf => $codec:expr, $pieces:expr
+    ) => {{
+        if $pieces.as_ref().len() < $codec.parity_shard_count {
+            return Err(Error::TooFewBufferShards);
+        }
+        if $pieces.as_ref().len() > $codec.parity_shard_count {
+            return Err(Error::TooManyBufferShards);
+        }
+    }};
+}

seaweed-volume/vendor/reed-solomon-erasure/src/matrix.rs

@@ -0,0 +1,425 @@
+#![allow(dead_code)]
+extern crate alloc;
+
+use alloc::vec;
+use alloc::vec::Vec;
+
+use crate::Field;
+use smallvec::SmallVec;
+
+#[derive(Debug)]
+pub enum Error {
+    SingularMatrix,
+}
+
+macro_rules! acc {
+    (
+        $m:ident, $r:expr, $c:expr
+    ) => {
+        $m.data[$r * $m.col_count + $c]
+    };
+}
+
+pub fn flatten<T>(m: Vec<Vec<T>>) -> Vec<T> {
+    let mut result: Vec<T> = Vec::with_capacity(m.len() * m[0].len());
+    for row in m {
+        for v in row {
+            result.push(v);
+        }
+    }
+    result
+}
+
+#[derive(PartialEq, Debug, Clone)]
+pub struct Matrix<F: Field> {
+    row_count: usize,
+    col_count: usize,
+    data: SmallVec<[F::Elem; 1024]>, // store in flattened structure
+                                     // the smallvec can hold a matrix of size up to 32x32 in stack
+}
+
+fn calc_matrix_row_start_end(col_count: usize, row: usize) -> (usize, usize) {
+    let start = row * col_count;
+    let end = start + col_count;
+
+    (start, end)
+}
+
+impl<F: Field> Matrix<F> {
+    fn calc_row_start_end(&self, row: usize) -> (usize, usize) {
+        calc_matrix_row_start_end(self.col_count, row)
+    }
+
+    pub fn new(rows: usize, cols: usize) -> Matrix<F> {
+        let data = SmallVec::from_vec(vec![F::zero(); rows * cols]);
+
+        Matrix {
+            row_count: rows,
+            col_count: cols,
+            data,
+        }
+    }
+
+    pub fn new_with_data(init_data: Vec<Vec<F::Elem>>) -> Matrix<F> {
+        let rows = init_data.len();
+        let cols = init_data[0].len();
+
+        for r in init_data.iter() {
+            if r.len() != cols {
+                panic!("Inconsistent row sizes")
+            }
+        }
+
+        let data = SmallVec::from_vec(flatten(init_data));
+
+        Matrix {
+            row_count: rows,
+            col_count: cols,
+            data,
+        }
+    }
+
+    #[cfg(test)]
+    pub fn make_random(size: usize) -> Matrix<F>
+    where
+        rand::distributions::Standard: rand::distributions::Distribution<F::Elem>,
+    {
+        let mut vec: Vec<Vec<F::Elem>> = vec![vec![Default::default(); size]; size];
+        for v in vec.iter_mut() {
+            crate::tests::fill_random(v);
+        }
+
+        Matrix::new_with_data(vec)
+    }
+
+    pub fn identity(size: usize) -> Matrix<F> {
+        let mut result = Self::new(size, size);
+        for i in 0..size {
+            acc!(result, i, i) = F::one();
+        }
+        result
+    }
+
+    pub fn col_count(&self) -> usize {
+        self.col_count
+    }
+
+    pub fn row_count(&self) -> usize {
+        self.row_count
+    }
+
+    pub fn get(&self, r: usize, c: usize) -> F::Elem {
+        acc!(self, r, c).clone()
+    }
+
+    pub fn set(&mut self, r: usize, c: usize, val: F::Elem) {
+        acc!(self, r, c) = val;
+    }
+
+    pub fn multiply(&self, rhs: &Matrix<F>) -> Matrix<F> {
+        if self.col_count != rhs.row_count {
+            panic!(
+                "Colomn count on left is different from row count on right, lhs: {}, rhs: {}",
+                self.col_count, rhs.row_count
+            )
+        }
+        let mut result = Self::new(self.row_count, rhs.col_count);
+        for r in 0..self.row_count {
+            for c in 0..rhs.col_count {
+                let mut val = F::zero();
+                for i in 0..self.col_count {
+                    let mul = F::mul(acc!(self, r, i).clone(), acc!(rhs, i, c).clone());
+
+                    val = F::add(val, mul);
+                }
+                acc!(result, r, c) = val;
+            }
+        }
+        result
+    }
+
+    pub fn augment(&self, rhs: &Matrix<F>) -> Matrix<F> {
+        if self.row_count != rhs.row_count {
+            panic!(
+                "Matrices do not have the same row count, lhs: {}, rhs: {}",
+                self.row_count, rhs.row_count
+            )
+        }
+        let mut result = Self::new(self.row_count, self.col_count + rhs.col_count);
+        for r in 0..self.row_count {
+            for c in 0..self.col_count {
+                acc!(result, r, c) = acc!(self, r, c).clone();
+            }
+            let self_column_count = self.col_count;
+            for c in 0..rhs.col_count {
+                acc!(result, r, self_column_count + c) = acc!(rhs, r, c).clone();
+            }
+        }
+
+        result
+    }
+
+    pub fn sub_matrix(&self, rmin: usize, cmin: usize, rmax: usize, cmax: usize) -> Matrix<F> {
+        let mut result = Self::new(rmax - rmin, cmax - cmin);
+        for r in rmin..rmax {
+            for c in cmin..cmax {
+                acc!(result, r - rmin, c - cmin) = acc!(self, r, c).clone();
+            }
+        }
+        result
+    }
+
+    pub fn get_row(&self, row: usize) -> &[F::Elem] {
+        let (start, end) = self.calc_row_start_end(row);
+
+        &self.data[start..end]
+    }
+
+    pub fn swap_rows(&mut self, r1: usize, r2: usize) {
+        let (r1_s, _) = self.calc_row_start_end(r1);
+        let (r2_s, _) = self.calc_row_start_end(r2);
+
+        if r1 == r2 {
+            return;
+        } else {
+            for i in 0..self.col_count {
+                self.data.swap(r1_s + i, r2_s + i);
+            }
+        }
+    }
+
+    pub fn is_square(&self) -> bool {
+        self.row_count == self.col_count
+    }
+
+    pub fn gaussian_elim(&mut self) -> Result<(), Error> {
+        for r in 0..self.row_count {
+            if acc!(self, r, r) == F::zero() {
+                for r_below in r + 1..self.row_count {
+                    if acc!(self, r_below, r) != F::zero() {
+                        self.swap_rows(r, r_below);
+                        break;
+                    }
+                }
+            }
+            // If we couldn't find one, the matrix is singular.
+            if acc!(self, r, r) == F::zero() {
+                return Err(Error::SingularMatrix);
+            }
+            // Scale to 1.
+            if acc!(self, r, r) != F::one() {
+                let scale = F::div(F::one(), acc!(self, r, r).clone());
+                for c in 0..self.col_count {
+                    acc!(self, r, c) = F::mul(scale, acc!(self, r, c).clone());
+                }
+            }
+            // Make everything below the 1 be a 0 by subtracting
+            // a multiple of it.  (Subtraction and addition are
+            // both exclusive or in the Galois field.)
+            for r_below in r + 1..self.row_count {
+                if acc!(self, r_below, r) != F::zero() {
+                    let scale = acc!(self, r_below, r).clone();
+                    for c in 0..self.col_count {
+                        acc!(self, r_below, c) = F::add(
+                            acc!(self, r_below, c).clone(),
+                            F::mul(scale, acc!(self, r, c).clone()),
+                        );
+                    }
+                }
+            }
+        }
+
+        // Now clear the part above the main diagonal.
+        for d in 0..self.row_count {
+            for r_above in 0..d {
+                if acc!(self, r_above, d) != F::zero() {
+                    let scale = acc!(self, r_above, d).clone();
+                    for c in 0..self.col_count {
+                        acc!(self, r_above, c) = F::add(
+                            acc!(self, r_above, c).clone(),
+                            F::mul(scale, acc!(self, d, c).clone()),
+                        );
+                    }
+                }
+            }
+        }
+        Ok(())
+    }
+
+    pub fn invert(&self) -> Result<Matrix<F>, Error> {
+        if !self.is_square() {
+            panic!("Trying to invert a non-square matrix")
+        }
+
+        let row_count = self.row_count;
+        let col_count = self.col_count;
+
+        let mut work = self.augment(&Self::identity(row_count));
+        work.gaussian_elim()?;
+
+        Ok(work.sub_matrix(0, row_count, col_count, col_count * 2))
+    }
+
+    pub fn vandermonde(rows: usize, cols: usize) -> Matrix<F> {
+        let mut result = Self::new(rows, cols);
+
+        for r in 0..rows {
+            // doesn't matter what `r_a` is as long as it's unique.
+            // then the vandermonde matrix is invertible.
+            let r_a = F::nth(r);
+            for c in 0..cols {
+                acc!(result, r, c) = F::exp(r_a, c);
+            }
+        }
+
+        result
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    extern crate alloc;
+
+    use alloc::vec;
+
+    use super::Matrix;
+    use crate::galois_8;
+
+    macro_rules! matrix {
+        (
+            $(
+                [ $( $x:expr ),+ ]
+            ),*
+        ) => (
+            Matrix::<galois_8::Field>::new_with_data(vec![ $( vec![$( $x ),*] ),* ])
+        );
+        ($rows:expr, $cols:expr) => (Matrix::new($rows, $cols));
+    }
+
+    #[test]
+    fn test_matrix_col_count() {
+        let m1 = matrix!([1, 0, 0]);
+        let m2 = matrix!([0, 0, 0], [0, 0, 0]);
+        let m3: Matrix<galois_8::Field> = Matrix::new(1, 4);
+
+        assert_eq!(3, m1.col_count());
+        assert_eq!(3, m2.col_count());
+        assert_eq!(4, m3.col_count());
+    }
+
+    #[test]
+    fn test_matrix_row_count() {
+        let m1 = matrix!([1, 0, 0]);
+        let m2 = matrix!([0, 0, 0], [0, 0, 0]);
+        let m3: Matrix<galois_8::Field> = Matrix::new(1, 4);
+
+        assert_eq!(1, m1.row_count());
+        assert_eq!(2, m2.row_count());
+        assert_eq!(1, m3.row_count());
+    }
+
+    #[test]
+    fn test_matrix_swap_rows() {
+        {
+            let mut m1 = matrix!([1, 2, 3], [4, 5, 6], [7, 8, 9]);
+            let expect = matrix!([7, 8, 9], [4, 5, 6], [1, 2, 3]);
+            m1.swap_rows(0, 2);
+            assert_eq!(expect, m1);
+        }
+        {
+            let mut m1 = matrix!([1, 2, 3], [4, 5, 6], [7, 8, 9]);
+            let expect = m1.clone();
+            m1.swap_rows(0, 0);
+            assert_eq!(expect, m1);
+            m1.swap_rows(1, 1);
+            assert_eq!(expect, m1);
+            m1.swap_rows(2, 2);
+            assert_eq!(expect, m1);
+        }
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_inconsistent_row_sizes() {
+        matrix!([1, 0, 0], [0, 1], [0, 0, 1]);
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_incompatible_multiply() {
+        let m1 = matrix!([0, 1], [0, 1], [0, 1]);
+        let m2 = matrix!([0, 1, 2]);
+
+        m1.multiply(&m2);
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_incompatible_augment() {
+        let m1 = matrix!([0, 1]);
+        let m2 = matrix!([0, 1], [2, 3]);
+
+        m1.augment(&m2);
+    }
+
+    #[test]
+    fn test_matrix_identity() {
+        let m1 = Matrix::identity(3);
+        let m2 = matrix!([1, 0, 0], [0, 1, 0], [0, 0, 1]);
+        assert_eq!(m1, m2);
+    }
+
+    #[test]
+    fn test_matrix_multiply() {
+        let m1 = matrix!([1, 2], [3, 4]);
+        let m2 = matrix!([5, 6], [7, 8]);
+        let actual = m1.multiply(&m2);
+        let expect = matrix!([11, 22], [19, 42]);
+        assert_eq!(actual, expect);
+    }
+
+    #[test]
+    fn test_matrix_inverse_pass_cases() {
+        {
+            // Test case validating inverse of the input Matrix.
+            let m = matrix!([56, 23, 98], [3, 100, 200], [45, 201, 123])
+                .invert()
+                .unwrap();
+            let expect = matrix!([175, 133, 33], [130, 13, 245], [112, 35, 126]);
+            assert_eq!(m, expect);
+        }
+        {
+            // Test case validating inverse of the input Matrix.
+            let m = matrix!(
+                [1, 0, 0, 0, 0],
+                [0, 1, 0, 0, 0],
+                [0, 0, 0, 1, 0],
+                [0, 0, 0, 0, 1],
+                [7, 7, 6, 6, 1]
+            )
+            .invert()
+            .unwrap();
+            let expect = matrix!(
+                [1, 0, 0, 0, 0],
+                [0, 1, 0, 0, 0],
+                [123, 123, 1, 122, 122],
+                [0, 0, 1, 0, 0],
+                [0, 0, 0, 1, 0]
+            );
+            assert_eq!(m, expect);
+        }
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_matrix_inverse_non_square() {
+        // Test case with a non-square matrix.
+        matrix!([56, 23], [3, 100], [45, 201]).invert().unwrap();
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_matrix_inverse_singular() {
+        matrix!([4, 2], [12, 6]).invert().unwrap();
+    }
+}

seaweed-volume/vendor/reed-solomon-erasure/src/tests/galois_16.rs

@@ -0,0 +1,489 @@
+extern crate alloc;
+
+use alloc::vec;
+use alloc::vec::Vec;
+
+use super::{fill_random, option_shards_into_shards, shards_into_option_shards};
+use crate::galois_16::ReedSolomon;
+
+macro_rules! make_random_shards {
+    ($per_shard:expr, $size:expr) => {{
+        let mut shards = Vec::with_capacity(20);
+        for _ in 0..$size {
+            shards.push(vec![[0; 2]; $per_shard]);
+        }
+
+        for s in shards.iter_mut() {
+            fill_random(s);
+        }
+
+        shards
+    }};
+}
+
+#[test]
+fn correct_field_order_restriction() {
+    const ORDER: usize = 1 << 16;
+
+    assert!(ReedSolomon::new(ORDER, 1).is_err());
+    assert!(ReedSolomon::new(1, ORDER).is_err());
+
+    // way too slow, because it needs to build a 65536*65536 vandermonde matrix
+    // assert!(ReedSolomon::new(ORDER - 1, 1).is_ok());
+    assert!(ReedSolomon::new(1, ORDER - 1).is_ok());
+}
+
+quickcheck! {
+    fn qc_encode_verify_reconstruct_verify(data: usize,
+                                           parity: usize,
+                                           corrupt: usize,
+                                           size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let corrupt = corrupt % (parity + 1);
+
+        let mut corrupt_pos_s = Vec::with_capacity(corrupt);
+        for _ in 0..corrupt {
+            let mut pos = rand::random::<usize>() % (data + parity);
+
+            while let Some(_) = corrupt_pos_s.iter().find(|&&x| x == pos) {
+                pos = rand::random::<usize>() % (data + parity);
+            }
+
+            corrupt_pos_s.push(pos);
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        {
+            let mut refs =
+                convert_2D_slices!(expect =>to_mut_vec &mut [[u8; 2]]);
+
+            r.encode(&mut refs).unwrap();
+        }
+
+        let expect = expect;
+
+        let mut shards = expect.clone();
+
+        // corrupt shards
+        for &p in corrupt_pos_s.iter() {
+            fill_random(&mut shards[p]);
+        }
+        let mut slice_present = vec![true; data + parity];
+        for &p in corrupt_pos_s.iter() {
+            slice_present[p] = false;
+        }
+
+        // reconstruct
+        {
+            let mut refs: Vec<_> = shards.iter_mut()
+                .map(|i| &mut i[..])
+                .zip(slice_present.iter().cloned())
+                .collect();
+
+            r.reconstruct(&mut refs[..]).unwrap();
+        }
+
+        ({
+            let refs =
+                convert_2D_slices!(expect =>to_vec &[[u8; 2]]);
+
+            r.verify(&refs).unwrap()
+        })
+            &&
+            expect == shards
+            &&
+            ({
+                let refs =
+                    convert_2D_slices!(shards =>to_vec &[[u8; 2]]);
+
+                r.verify(&refs).unwrap()
+            })
+    }
+
+    fn qc_encode_verify_reconstruct_verify_shards(data: usize,
+                                                  parity: usize,
+                                                  corrupt: usize,
+                                                  size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let corrupt = corrupt % (parity + 1);
+
+        let mut corrupt_pos_s = Vec::with_capacity(corrupt);
+        for _ in 0..corrupt {
+            let mut pos = rand::random::<usize>() % (data + parity);
+
+            while let Some(_) = corrupt_pos_s.iter().find(|&&x| x == pos) {
+                pos = rand::random::<usize>() % (data + parity);
+            }
+
+            corrupt_pos_s.push(pos);
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        r.encode(&mut expect).unwrap();
+
+        let expect = expect;
+
+        let mut shards = shards_into_option_shards(expect.clone());
+
+        // corrupt shards
+        for &p in corrupt_pos_s.iter() {
+            shards[p] = None;
+        }
+
+        // reconstruct
+        r.reconstruct(&mut shards).unwrap();
+
+        let shards = option_shards_into_shards(shards);
+
+        r.verify(&expect).unwrap()
+            && expect == shards
+            && r.verify(&shards).unwrap()
+    }
+
+    fn qc_verify(data: usize,
+                 parity: usize,
+                 corrupt: usize,
+                 size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let corrupt = corrupt % (parity + 1);
+
+        let mut corrupt_pos_s = Vec::with_capacity(corrupt);
+        for _ in 0..corrupt {
+            let mut pos = rand::random::<usize>() % (data + parity);
+
+            while let Some(_) = corrupt_pos_s.iter().find(|&&x| x == pos) {
+                pos = rand::random::<usize>() % (data + parity);
+            }
+
+            corrupt_pos_s.push(pos);
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        {
+            let mut refs =
+                convert_2D_slices!(expect =>to_mut_vec &mut [[u8; 2]]);
+
+            r.encode(&mut refs).unwrap();
+        }
+
+        let expect = expect;
+
+        let mut shards = expect.clone();
+
+        // corrupt shards
+        for &p in corrupt_pos_s.iter() {
+            fill_random(&mut shards[p]);
+        }
+
+        ({
+            let refs =
+                convert_2D_slices!(expect =>to_vec &[[u8; 2]]);
+
+            r.verify(&refs).unwrap()
+        })
+            &&
+            ((corrupt > 0 && expect != shards)
+             || (corrupt == 0 && expect == shards))
+            &&
+            ({
+                let refs =
+                    convert_2D_slices!(shards =>to_vec &[[u8; 2]]);
+
+                (corrupt > 0 && !r.verify(&refs).unwrap())
+                    || (corrupt == 0 && r.verify(&refs).unwrap())
+            })
+    }
+
+    fn qc_verify_shards(data: usize,
+                        parity: usize,
+                        corrupt: usize,
+                        size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let corrupt = corrupt % (parity + 1);
+
+        let mut corrupt_pos_s = Vec::with_capacity(corrupt);
+        for _ in 0..corrupt {
+            let mut pos = rand::random::<usize>() % (data + parity);
+
+            while let Some(_) = corrupt_pos_s.iter().find(|&&x| x == pos) {
+                pos = rand::random::<usize>() % (data + parity);
+            }
+
+            corrupt_pos_s.push(pos);
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        r.encode(&mut expect).unwrap();
+
+        let expect = expect;
+
+        let mut shards = expect.clone();
+
+        // corrupt shards
+        for &p in corrupt_pos_s.iter() {
+            fill_random(&mut shards[p]);
+        }
+
+        r.verify(&expect).unwrap()
+            &&
+            ((corrupt > 0 && expect != shards)
+             || (corrupt == 0 && expect == shards))
+            &&
+            ((corrupt > 0 && !r.verify(&shards).unwrap())
+             || (corrupt == 0 && r.verify(&shards).unwrap()))
+    }
+
+    fn qc_encode_sep_same_as_encode(data: usize,
+                                    parity: usize,
+                                    size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        let mut shards = expect.clone();
+
+        {
+            let mut refs =
+                convert_2D_slices!(expect =>to_mut_vec &mut [[u8; 2]]);
+
+            r.encode(&mut refs).unwrap();
+        }
+
+        let expect = expect;
+
+        {
+            let (data, parity) = shards.split_at_mut(data);
+
+            let data_refs =
+                convert_2D_slices!(data =>to_mut_vec &[[u8; 2]]);
+
+            let mut parity_refs =
+                convert_2D_slices!(parity =>to_mut_vec &mut [[u8; 2]]);
+
+            r.encode_sep(&data_refs, &mut parity_refs).unwrap();
+        }
+
+        let shards = shards;
+
+        expect == shards
+    }
+
+    fn qc_encode_sep_same_as_encode_shards(data: usize,
+                                           parity: usize,
+                                           size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        let mut shards = expect.clone();
+
+        r.encode(&mut expect).unwrap();
+
+        let expect = expect;
+
+        {
+            let (data, parity) = shards.split_at_mut(data);
+
+            r.encode_sep(data, parity).unwrap();
+        }
+
+        let shards = shards;
+
+        expect == shards
+    }
+
+    fn qc_encode_single_same_as_encode(data: usize,
+                                       parity: usize,
+                                       size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        let mut shards = expect.clone();
+
+        {
+            let mut refs =
+                convert_2D_slices!(expect =>to_mut_vec &mut [[u8; 2]]);
+
+            r.encode(&mut refs).unwrap();
+        }
+
+        let expect = expect;
+
+        {
+            let mut refs =
+                convert_2D_slices!(shards =>to_mut_vec &mut [[u8; 2]]);
+
+            for i in 0..data {
+                r.encode_single(i, &mut refs).unwrap();
+            }
+        }
+
+        let shards = shards;
+
+        expect == shards
+    }
+
+    fn qc_encode_single_same_as_encode_shards(data: usize,
+                                              parity: usize,
+                                              size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        let mut shards = expect.clone();
+
+        r.encode(&mut expect).unwrap();
+
+        let expect = expect;
+
+        for i in 0..data {
+            r.encode_single(i, &mut shards).unwrap();
+        }
+
+        let shards = shards;
+
+        expect == shards
+    }
+
+    fn qc_encode_single_sep_same_as_encode(data: usize,
+                                           parity: usize,
+                                           size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        let mut shards = expect.clone();
+
+        {
+            let mut refs =
+                convert_2D_slices!(expect =>to_mut_vec &mut [[u8; 2]]);
+
+            r.encode(&mut refs).unwrap();
+        }
+
+        let expect = expect;
+
+        {
+            let (data_shards, parity_shards) = shards.split_at_mut(data);
+
+            let data_refs =
+                convert_2D_slices!(data_shards =>to_mut_vec &[[u8; 2]]);
+
+            let mut parity_refs =
+                convert_2D_slices!(parity_shards =>to_mut_vec &mut [[u8; 2]]);
+
+            for i in 0..data {
+                r.encode_single_sep(i, data_refs[i], &mut parity_refs).unwrap();
+            }
+        }
+
+        let shards = shards;
+
+        expect == shards
+    }
+
+    fn qc_encode_single_sep_same_as_encode_shards(data: usize,
+                                                  parity: usize,
+                                                  size: usize) -> bool {
+        let data = 1 + data % 255;
+        let mut parity = 1 + parity % 255;
+        if data + parity > 256 {
+            parity -= data + parity - 256;
+        }
+
+        let size = 1 + size % 1_000_000;
+
+        let r = ReedSolomon::new(data, parity).unwrap();
+
+        let mut expect = make_random_shards!(size, data + parity);
+        let mut shards = expect.clone();
+
+        r.encode(&mut expect).unwrap();
+
+        let expect = expect;
+
+        {
+            let (data_shards, parity_shards) = shards.split_at_mut(data);
+
+            for i in 0..data {
+                r.encode_single_sep(i, &data_shards[i], parity_shards).unwrap();
+            }
+        }
+
+        let shards = shards;
+
+        expect == shards
+    }
+}