Goal
Bring idle RAM of a long-lived pond mcp server toward the <500 MiB target. Peak stays comfortably under the 2 GiB ceiling.
Measured baseline (real 2.1M-message corpus, local FS)
cargo bench --bench serve_mem_bench after serving fts + vector + sql + get, model idle-unloaded:
- idle floor: 877 MiB phys_footprint (the macOS "Memory"/Jetsam metric)
vmmap anatomy of that floor:
| Component |
MiB |
Reclaimable? |
| Live Lance caches (index IVF/PQ + FTS postings) |
~200 |
no - needed for search |
MALLOC_LARGE (empty) - freed, retained by allocator |
~351 |
soft (kernel reclaims under pressure) |
MALLOC_SMALL (empty) - freed, retained by allocator |
~285 |
soft |
| base process / stacks / __DATA |
~40 |
no |
.rmm resident meta map (mmap) |
0 dirty |
file-backed, not in footprint |
Key finding: ~636 MiB of the floor is freed-but-retained transient memory, not live data. The system allocator's high-water mark is set by the largest transient peaks, and it holds those regions. Only ~200 MiB (the Lance caches) is genuinely live - already under target.
Dead end - do NOT retry: swapping the global allocator
Measured on the same corpus; all WORSE than the system allocator:
| Allocator |
idle floor PF |
| system libmalloc (current) |
877 |
| mimalloc (default) |
1525 |
mimalloc MIMALLOC_PURGE_DELAY=0 |
1042 |
jemalloc dirty_decay_ms:0,muzzy_decay_ms:0 |
1756 |
macOS phys_footprint is tightly coupled to the system libmalloc's MADV_FREE_REUSABLE path. Third-party allocators reserve/retain arenas that count against the footprint and return poorly on Darwin (jemalloc has long-standing poor macOS support). malloc_zone_pressure_relief also does not help (it MADV_FREEs lazily; pages stay in the footprint until kernel pressure). This avenue is closed - the fix must reduce our own transient peaks.
Proposed work (the real levers)
The retention tracks the high-water mark of two transient peaks:
1. Stream the rowmap build (biggest single lever)
Store::collect_row_metas materializes all 2.1M RowMetaEntry - each with an owned search_text String - into one Vec (~1 GB), then RowMetaMap::build grows a second giant blob while that Vec is still alive: 2145 MiB transient, leaving ~320 MiB of MALLOC_LARGE (empty) retention.
Restructure to stream: process the scan in BLOCK_ROWS (256) chunks - compress + write each block, drop its text - keeping only the small per-row headers + dictionaries in RAM. Target peak ~400 MiB.
Blocker to untangle: the dict indexes (session_index/project_index/etc) currently hold immutable borrows into entries, so progressive search_text free won't compile. Own the dict keys up front (a few thousand short strings) so entries can be consumed in a single pass.
2. Ship the E5 embedder as f16 safetensors
The model ships as f32 safetensors, mmap'd then cast to f16 on load - the full f32 file pages in alongside the f16 copy: ~470 MiB transient. Pre-convert to an f16 safetensors asset and load it directly.
3. (Optional, correct-sizing) metadata cache cap 128 -> 32 MiB
LOCAL_METADATA_CACHE_BYTES is a 128 MiB ceiling the working set never reaches (the corpus has tens of fragments per table; live metadata is a few MiB). Cutting it to 32 is safe correct-sizing but will NOT move the idle floor. Regression risk only if a future migration shrinks TARGET_FRAGMENT_BYTES (many small fragments inflate fragment metadata).
Note on reachability
No single fix reaches <500. Build streaming removes ~320 MiB; the f16 model removes more of the load-transient retention; they need to stack. The ~200 MiB live Lance cache floor (index, needed for search) is the practical lower bound for a server that has served a vector query.
Tooling already in place
benches/serve_mem_bench.rs reports idle floor + peak with per-phase PF/RSS attribution.Store::lance_cache_bytes() reports live Lance index+metadata cache occupancy.
Goal
Bring idle RAM of a long-lived
pond mcpserver toward the <500 MiB target. Peak stays comfortably under the 2 GiB ceiling.Measured baseline (real 2.1M-message corpus, local FS)
cargo bench --bench serve_mem_benchafter serving fts + vector + sql + get, model idle-unloaded:vmmapanatomy of that floor:MALLOC_LARGE (empty)- freed, retained by allocatorMALLOC_SMALL (empty)- freed, retained by allocator.rmmresident meta map (mmap)Key finding: ~636 MiB of the floor is freed-but-retained transient memory, not live data. The system allocator's high-water mark is set by the largest transient peaks, and it holds those regions. Only ~200 MiB (the Lance caches) is genuinely live - already under target.
Dead end - do NOT retry: swapping the global allocator
Measured on the same corpus; all WORSE than the system allocator:
MIMALLOC_PURGE_DELAY=0dirty_decay_ms:0,muzzy_decay_ms:0macOS
phys_footprintis tightly coupled to the system libmalloc'sMADV_FREE_REUSABLEpath. Third-party allocators reserve/retain arenas that count against the footprint and return poorly on Darwin (jemalloc has long-standing poor macOS support).malloc_zone_pressure_reliefalso does not help (it MADV_FREEs lazily; pages stay in the footprint until kernel pressure). This avenue is closed - the fix must reduce our own transient peaks.Proposed work (the real levers)
The retention tracks the high-water mark of two transient peaks:
1. Stream the rowmap build (biggest single lever)
Store::collect_row_metasmaterializes all 2.1MRowMetaEntry- each with an ownedsearch_textString - into oneVec(~1 GB), thenRowMetaMap::buildgrows a second giant blob while that Vec is still alive: 2145 MiB transient, leaving ~320 MiB ofMALLOC_LARGE (empty)retention.Restructure to stream: process the scan in
BLOCK_ROWS(256) chunks - compress + write each block, drop its text - keeping only the small per-row headers + dictionaries in RAM. Target peak ~400 MiB.Blocker to untangle: the dict indexes (
session_index/project_index/etc) currently hold immutable borrows intoentries, so progressivesearch_textfree won't compile. Own the dict keys up front (a few thousand short strings) soentriescan be consumed in a single pass.2. Ship the E5 embedder as f16 safetensors
The model ships as f32 safetensors, mmap'd then cast to f16 on load - the full f32 file pages in alongside the f16 copy: ~470 MiB transient. Pre-convert to an f16 safetensors asset and load it directly.
3. (Optional, correct-sizing) metadata cache cap 128 -> 32 MiB
LOCAL_METADATA_CACHE_BYTESis a 128 MiB ceiling the working set never reaches (the corpus has tens of fragments per table; live metadata is a few MiB). Cutting it to 32 is safe correct-sizing but will NOT move the idle floor. Regression risk only if a future migration shrinksTARGET_FRAGMENT_BYTES(many small fragments inflate fragment metadata).Note on reachability
No single fix reaches <500. Build streaming removes ~320 MiB; the f16 model removes more of the load-transient retention; they need to stack. The ~200 MiB live Lance cache floor (index, needed for search) is the practical lower bound for a server that has served a vector query.
Tooling already in place
benches/serve_mem_bench.rsreports idle floor + peak with per-phase PF/RSS attribution.Store::lance_cache_bytes()reports live Lance index+metadata cache occupancy.