MEID-001 (Ledger & Replication)

[bmuddha]

Ledger & Replication — design overview

Two components and how they work together:

• magicblock-ledger — the engine's durable record of what executed:
  transactions, block boundaries, and execution metadata, organized so old
  history is cheap to drop.
• magicblock-replicator — copies a leader's durable ledger state to a
  follower, either as a live stream of new blockstore bytes or, when the
  follower has fallen too far behind, as an accountsdb snapshot. Still WIP
  — see Current status.

The interesting part is the seam between them: replication doesn't reach into
ledger internals, it rides on two things the ledger already publishes — a
durable byte cursor per superblock, and a broadcast that fires whenever that
cursor advances. Most of this document is about that seam. The ledger's own
internals are summarized only as far as you need them to follow the transfer.

---

1. Ledger essentials

The ledger is an append-only log split into superblocks. Each superblock is
a self-contained directory, so retention is just "delete the oldest directory"
rather than a compaction over one shared store. Superblocks are numbered
sequentially with contiguous, non-overlapping slot ranges; the newest is the
head, where all appends land.

<ledger root>/
├── ledger.meta                 # mmap header: head id, retained count, slot range, totals
├── superblock-000000001/       # oldest retained (purged first under retention)
│   ├── superblock.meta         # mmap header: durable cursors, slot range, sealed flag
│   ├── blockstore.db           # ordered stream of txns + block markers + seal  ← replicated
│   ├── executions.db           # per-txn execution header + compressed details
│   ├── accountsdb.tar.zst      # accounts snapshot (only after this SB is sealed)  ← replicated
│   └── index/                  # LMDB: transactions / slots / accounts
├── superblock-000000002/
└── superblock-000000003/       # active head

What matters for replication:

• blockstore.db is the authoritative ordered log — a flat wincode stream
  of typed entries. Transaction entries are raw tx bytes; a Block entry
  (slot, hash, time) delimits the transactions before it; a Superblock
  seal (id, checksum) terminates the segment and records the accountsdb
  checksum at seal time. This byte stream is what a follower streams and
  replays.

  blockstore.db  (append-only, grows left → right)
  ┌────┬────┬───────┬────┬────┬───────┬────┬───────┬──────────────┐
  │ Tx │ Tx │ Block │ Tx │ Tx │ Block │ Tx │ Block │  Superblock  │
  │    │    │ slot N│    │    │ slot+1│    │ slot+2│    (seal)    │
  └────┴────┴───────┴────┴────┴───────┴────┴───────┴──────────────┘
   \___ block N ___/ \__ block N+1 __/ \_ blk+2 _/        │
                                                          └─ closes the segment,
                                                             carries the checksum
  

• Durable cursors live in superblock.meta (cursors.blockstore,
  cursors.executions) — the exact number of bytes of each data file that are
  fsync'd and safe to read. Nothing, including a replication peer, is ever
  served past a published cursor.

• accountsdb.tar.zst appears in a superblock directory only after that
  superblock is sealed. It's the accounts snapshot the keeper archives on
  finalize_superblock, and it's what the snapshot path ships.

The rest — executions.db layout (a cheap wincode header plus zstd-compressed
details), the LMDB index/ (signature / slot / account lookups pointing back
into the data files by packed offset+size spans), and the reader worker pool
that serves point queries — is internal to the ledger and doesn't participate in
the transfer. It's covered in the crate README.

The write ordering that replication depends on

A single writer thread owns the append path, and the order it commits things in
is the whole reason a published cursor can be trusted:

 per block boundary:
   1. append the Block marker to blockstore.db
   2. fsync both data files              → bytes are durable
   3. commit + flush the LMDB index
   4. publish durable cursors into superblock.meta
   5. update ledger.meta
   6. broadcast BlockstorePosition {superblock, offset}   ← replication wakes here

Data is durable (step 2) before the cursor is published (step 4) and before
the broadcast (step 6). So any BlockstorePosition a follower ever sees points
only at bytes that are fully written and on disk — there's no window to read a
half-written entry. A superblock seal runs the same sync-then-publish sequence,
then trims file slack and rotates to the next directory.

---

2. How replication rides on the ledger

BlockstorePosition { superblock, offset } is the currency of the whole
protocol. It names a byte in the blockstore stream, and its ordering is
lexicographic over (superblock, offset) — which is exactly on-disk append
order, including across rotations. That single fact makes "everything after the
follower's position" unambiguous even when it spans multiple superblock files.

The leader exposes two ledger facilities to the transfer worker:

• the position broadcast (LedgerHandle.position) — the same channel the
  appender publishes to at step 6 above. A replication server subscribes to it
  and wakes on every committed block.
• per-superblock cursors and files — to open a superblock's blockstore.db
  and know how far it's valid, and to locate a sealed accountsdb.tar.zst.

        LEADER                                          FOLLOWER
  ┌────────────────────────┐                      ┌────────────────────┐
  │ ReplicationDispatcher  │   accept (async)     │ ReplicationClient  │
  │  (async accept loop)   │◀──────── TCP ───────▶│                    │
  └──────────┬─────────────┘                      └────────────────────┘
             │ one per connection
             ▼ blocking OS thread + current-thread runtime
  ┌────────────────────────┐   subscribes to ledger position broadcast,
  │ ReplicationServer      │   reads blockstore.db bytes, serves accountsdb.tar.zst
  └────────────────────────┘

Wire framing

Control messages (handshake and the leader's decision) are wincode behind a
little-endian u32 length prefix, capped at 64 KiB before allocation. Bulk
payloads — the snapshot archive, or a run of blockstore bytes — follow the
selected response with no framing and no size cap.

Handshake: stream or snapshot?

The follower opens by reporting its last durable cursor. The leader compares it
against its own head cursor and picks a mode. The two "snapshot" branches are
exactly where the ledger's retention meets replication: if the follower's
starting superblock has already been purged, or is more than two behind, there
are no bytes left to stream and only a snapshot can catch it up.

  follower ──▶ HandshakeRequest { version, position }

  leader:
     ├─ version mismatch ...........................▶ Err
     ├─ requested > leader's head cursor ...........▶ Err  (asks for data we don't have)
     ├─ requested SB is >2 behind head .............▶ SNAPSHOT
     ├─ requested SB no longer retained ............▶ SNAPSHOT   (retention already dropped it)
     ├─ requested offset > that file's length ......▶ Err
     └─ otherwise ..................................▶ STREAM from requested position

Stream path

The leader replies Stream(position), opens that superblock's blockstore.db,
and then just tails it — every time the position broadcast fires, it forwards
the new byte range. Crossing a superblock boundary is handled by the
BlockstorePosition ordering: send the sealed tail of the old file, then
reopen the next superblock at offset 0 and continue.

  leader ──▶ Stream(position);  open blockstore.db at offset
  loop on position broadcast:
     same superblock     → send bytes [current.offset .. new.offset)
     superblock rolled   → send sealed tail, reopen next blockstore.db at 0, continue

Because the broadcast only carries positions the appender has already published,
the leader never races ahead of durable data — the write ordering in §1 is what
makes this safe without any extra coordination.

Snapshot path

When the follower is too far behind, byte catch-up is pointless, so the leader
ships the whole accounts state as of its current superblock:

  leader ──▶ Snapshot { len, superblock }
  leader ──▶ raw accountsdb.tar.zst bytes   (exactly len, unframed)

  follower:
     write len bytes into superblock-<id>/accountsdb.tar.zst
     record a local SuperblockSeal { id, checksum }

That archive isn't produced by the replicator — it's the file the keeper already
wrote into the superblock directory when it sealed that superblock
(finalize_superblock exports accountsdb, a background archiver tars and
zstd-compresses it). The replicator just serves the existing file. The seal
entry sitting at the end of the corresponding blockstore.db carries the
accountsdb checksum for that same superblock, which is what lets the follower
verify the snapshot later.

Rejoining after a snapshot

The archive is a complete accountsdb image — both the persisted store (the
mutable, ER-owned accounts) and the volatile store serialized as volatile.db
(the in-memory accounts: sysvars and other chain/service-owned state). The
snapshot captures state as of one superblock boundary, and the live tip is ahead of it.

To close that gap the follower replays the blockstore.db entries recorded
after the snapshot's boundary, forward through a sequencer, up to the tip.