Skip to content

Commit 164ca2c

Browse files
ZocoLiniclaude
andcommitted
perf(dash-spv): size storage segments per item type
Every segment cache held 50 000 items, whatever an item costs. Near the tip a block segment holds tens of MB of decoded blocks, while a header segment spanning the same heights is 5.6 MB and a filter header one 1.6 MB. `Persistable::ITEMS_PER_SEGMENT` lets each type choose: headers 10 000 (~1.1 MB per segment), filter headers 50 000 (~1.6 MB), filters 2 000 (~2 MB near the tip) and blocks 1 000. This changes the on-disk layout of the header, filter and block segments. Storage written with 50 000-item segments is misread by this layout and has to be deleted until a migration or a versioned folder lands. Mainnet restore, mainnet.100mbi.100ms, #1015/#1016/#1014 applied, two resident segments, jemalloc heap profiling, wallet identical in every run (14114383 sat, 7112 records, 13389 addresses): segment items time peak RSS segment caches 50 000 for every type 7.0 min 818 MiB 332 MiB 5 000 for every type 7.3-8.3 min 554-687 MiB 23-106 MiB 1 000 for every type 9.2-12.2 min 453-577 MiB 2-66 MiB per type (this commit) 7.5 min 538 MiB 38 MiB With 1 000 items everywhere, the header and filter header phases paid an fsync per evicted segment (105-141 s and 254-332 s instead of ~60 s and ~150 s). Blocks at 500 items saved ~18 MiB of cache but reloaded 50 % more block segments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DruChNTWXwJoWPartZwCf
1 parent 5374756 commit 164ca2c

1 file changed

Lines changed: 10 additions & 1 deletion

File tree

‎dash-spv/src/storage/segments.rs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ use crate::{
2727
pub trait Persistable: Sized + Encodable + Decodable + PartialEq + Clone {
2828
const SEGMENT_PREFIX: &'static str = "segment";
2929
const DATA_FILE_EXTENSION: &'static str = "dat";
30+
const ITEMS_PER_SEGMENT: u32;
3031

3132
fn segment_file_name(segment_id: u32) -> String {
3233
format!("{}_{:04}.{}", Self::SEGMENT_PREFIX, segment_id, Self::DATA_FILE_EXTENSION)
@@ -36,12 +37,16 @@ pub trait Persistable: Sized + Encodable + Decodable + PartialEq + Clone {
3637
}
3738

3839
impl Persistable for Vec<u8> {
40+
const ITEMS_PER_SEGMENT: u32 = 2_000;
41+
3942
fn sentinel() -> Self {
4043
vec![]
4144
}
4245
}
4346

4447
impl Persistable for HashedBlockHeader {
48+
const ITEMS_PER_SEGMENT: u32 = 10_000;
49+
4550
fn sentinel() -> Self {
4651
let header = BlockHeader {
4752
version: Version::from_consensus(i32::MAX), // Invalid version
@@ -57,12 +62,16 @@ impl Persistable for HashedBlockHeader {
5762
}
5863

5964
impl Persistable for FilterHeader {
65+
const ITEMS_PER_SEGMENT: u32 = 50_000;
66+
6067
fn sentinel() -> Self {
6168
FilterHeader::from_byte_array([0u8; 32])
6269
}
6370
}
6471

6572
impl Persistable for HashedBlock {
73+
const ITEMS_PER_SEGMENT: u32 = 1_000;
74+
6675
fn sentinel() -> Self {
6776
let block = Block {
6877
header: *HashedBlockHeader::sentinel().header(),
@@ -562,7 +571,7 @@ pub struct Segment<I: Persistable> {
562571
}
563572

564573
impl<I: Persistable> Segment<I> {
565-
const ITEMS_PER_SEGMENT: u32 = 50_000;
574+
const ITEMS_PER_SEGMENT: u32 = I::ITEMS_PER_SEGMENT;
566575

567576
fn new(segment_id: u32, mut items: Vec<I>, state: SegmentState) -> Self {
568577
debug_assert!(items.len() <= Self::ITEMS_PER_SEGMENT as usize);

0 commit comments

Comments
 (0)