New needle_map.CompactMap() implementation for reduced memory usage (#6842)

* Rework `needle_map.CompactMap()` to maximize memory efficiency.

* Use a memory-efficient structure for `CompactMap` needle value entries.

This slightly complicates the code, but makes a **massive** difference
in memory efficiency - preliminary results show a ~30% reduction in
heap usage, with no measurable performance impact otherwise.

* Clean up type for `CompactMap` chunk IDs.

* Add a small comment description for `CompactMap()`.

* Add the old version of `CompactMap()` for comparison purposes.
This commit is contained in:
Lisandro Pin
2025-06-05 23:03:29 +02:00
committed by GitHub
parent d8ddc22fc2
commit bed0a64693
7 changed files with 1344 additions and 485 deletions

View File

@@ -5,11 +5,12 @@ package needle_map
import (
"fmt"
"github.com/seaweedfs/seaweedfs/weed/storage/types"
"github.com/stretchr/testify/assert"
"log"
"os"
"testing"
"github.com/seaweedfs/seaweedfs/weed/storage/types"
"github.com/stretchr/testify/assert"
)
func Test5bytesIndexLoading(t *testing.T) {