New needle_map.CompactMap() implementation for reduced memory usage (#6842)

* Rework `needle_map.CompactMap()` to maximize memory efficiency.

* Use a memory-efficient structure for `CompactMap` needle value entries.

This slightly complicates the code, but makes a **massive** difference
in memory efficiency - preliminary results show a ~30% reduction in
heap usage, with no measurable performance impact otherwise.

* Clean up type for `CompactMap` chunk IDs.

* Add a small comment description for `CompactMap()`.

* Add the old version of `CompactMap()` for comparison purposes.
This commit is contained in:
Lisandro Pin
2025-06-05 23:03:29 +02:00
committed by GitHub
parent d8ddc22fc2
commit bed0a64693
7 changed files with 1344 additions and 485 deletions

View File

@@ -43,6 +43,7 @@ func TestMemoryUsage(t *testing.T) {
PrintMemUsage(totalRowCount)
now := time.Now()
fmt.Printf("\tCompactMap = %s", m.String())
fmt.Printf("\tTaken = %v\n", now.Sub(startTime))
startTime = now
}