`quereus-plugin-optimystic` implements neither `beginSchemaBatch` nor `endSchemaBatch`: a cold `apply schema` makes ~190 raw-storage calls per object, ~90% of them redundant reads

[risavian]

Affected versions: @optimystic/quereus-plugin-optimystic@0.17.0 (latest published at time of
writing), against @quereus/quereus 4.4.1 and 4.5.0.

Summary

Quereus lets a virtual-table module opt into folding an entire APPLY SCHEMA migration into a
single substrate commit, via the optional beginSchemaBatch / endSchemaBatch module hooks
(packages/quereus/src/vtab/module.ts:473). Quereus's own @quereus/isolation implements them and
documents the consequence of not doing so, in isolation-module.ts:

A batching-capable underlying module folds the whole APPLY SCHEMA into a single substrate commit
by opening a batch here that its subsequent create/destroy/alter callbacks (which IsolationModule
forwards to the underlying) join. Without this forward the underlying is never reached and
silently falls back to per-DDL commits.

@optimystic/quereus-plugin-optimystic implements neither hook. Confirmed both by grep (0 hits
across the published package) and at runtime off db.schemaManager.allModules():

module          beginSchemaBatch   endSchemaBatch
memory          false              false
optimystic      false              false

So an APPLY SCHEMA against the optimystic vtab is executed as a series of independent DDL
statements with no batch context — runBatchedMigrationLoop awaits each
db._execWithinTransaction(ddl) in turn, and the module is never told they belong together.

The consequence turned out to be read amplification rather than commit count (measurements
below): without a batch-scoped context, each table's initialisation re-reads revision, block and
metadata state that the same APPLY SCHEMA already read moments earlier. Applying a real
schema of 54 tables + 13 indexes issues 13,025 raw-storage calls — 194 per created object —
of which 89.3% are provably redundant. This is the dominant cost of first-time schema creation,
and it is reproducible with the published packages and a synthetic schema (see below).

Measured impact

All figures below marked [repro] come from the script in the next section, which uses only
published packages and a synthetic schema — you can run it directly. Figures marked
[app schema] come from a real 92 KB / 54-table / 14-view application schema that is not
public; they are included only as corroboration and are not needed to see the problem.

Read amplification

[repro] One cold apply schema creating 54 tables + 13 indexes issues 13,025 IRawStorage
calls — 194.4 per created object — accounting for 76% of the apply:

op	calls	total
listRevisions	1,676	83.5 ms
getMaterializedBlock	1,676	69.4 ms
getMetadata	5,974	12.1 ms
saveMaterializedBlock	303	7.1 ms
listPendingTransactions	1,720	2.7 ms
getPendingTransaction	920	2.7 ms
savePendingTransaction	178	2.1 ms
saveMetadata	222	0.7 ms
saveRevision	178	0.4 ms
promotePendingTransaction	178	0.4 ms
TOTAL	13,025	181.1 ms

Note only 178 saveRevision / promotePendingTransaction — roughly 178 transactions for 67
objects. The dominant cost is repeated READS, not commit count. Each object's initialisation
re-reads revision, block and metadata state that the same APPLY SCHEMA already read.

calls / object is ~constant as the schema scales (194.4 at 54 tables, 181.8 at 10), so total cost
is ~linear in object count.

Confirmed win from implementing the hooks

[repro] Prototyping a batch-scoped read context — memoising those reads for the duration of one
APPLY SCHEMA, invalidating on write:

	baseline	with batch read context
apply wall time	~186–213 ms	~110–125 ms
speedup (median of 5)	—	1.63x (range 1.55–1.91x)
raw-storage time	181.1 ms	47.6 ms
read cache hits / misses	—	10,677 / 1,286

89.3% of the reads a cold apply performs are redundant. The elided work is provably
unnecessary rather than merely cacheable: both runs produce an identical catalog (all 54 tables,
name + column-count fingerprint) and an identical successful write/read round-trip
(write+read OK (Amount=7)).

[app schema] The same experiment on the real 92 KB schema: 14,447 storage calls (~198/object,
68% of the apply), 11,913 / 1,400 cache hits/misses (89.5% redundant), 1.83x median speedup.

Cost relative to other substrates

[app schema] Identical schema and identical 88 generated statements, local transactor with
MemoryRawStorage (no libp2p, no disk):

substrate	total apply	per statement
quereus built-in in-memory vtab	24 ms	0.3 ms
real LevelDB via @quereus/plugin-leveldb	~60 ms	0.7 ms
optimystic vtab	~300–590 ms	~4–7 ms

Why each read is expensive

[app schema] CPU profile of the apply window:

	self time	share
structuredClone (V8 builtin)	100.6 ms	27.7%
decodeJson (@optimystic/db-p2p)	87.0 ms	23.9%
rangeRevisions (db-p2p)	12.1 ms	3.3%
encodeJson (db-p2p)	7.1 ms	2.0%

By package: @optimystic/db-p2p 37.1%, node internals 38.7%, @quereus/quereus 16.4%,
@optimystic/db-core 5.6%. Every redundant read pays a JSON decode plus a structuredClone.

This is much worse on React Native/Hermes. structuredClone is not native there — it is
polyfilled in JS (e.g. @ungap/structured-clone) — and Hermes has no JIT. A path hit ~1,700+
times per create, paying a JS deep-clone plus a JS JSON decode each time, is the most plausible
explanation for a multi-second schema creation on device. Not measured on device — stated as a
hypothesis, not a result.

Reproduction

Self-contained — published packages, synthetic schema, no external files. Applies to main with
tables + indexes only, deliberately, so this stays independent of quereus's separate
non-main-schema view/assertion defect.

npm init -y && npm pkg set type=module
npm i @quereus/quereus@4.5.0 @optimystic/quereus-plugin-optimystic@0.17.0 @optimystic/db-p2p@0.17.0
node repro.mjs               # 54 tables + 13 indexes
TABLES=10 INDEXES=3 node repro.mjs

The script instruments IRawStorage, runs the apply once as-is, then runs it again with a
batch-scoped read context prototyped at the storage seam, and checks the two runs produce the
same catalog and a working write/read round-trip.

repro.mjs

// Self-contained reproduction for Optimystic issue #8.
//
//   `quereus-plugin-optimystic` implements neither `beginSchemaBatch` nor `endSchemaBatch`.
//   A cold `apply schema` makes ~200 IRawStorage calls per created object, ~90% of which are
//   redundant reads within that same apply.
//
// Published packages only, synthetic schema, no external files:
//
//   npm init -y && npm pkg set type=module
//   npm i @quereus/quereus@4.5.0 @optimystic/quereus-plugin-optimystic@0.17.0 @optimystic/db-p2p@0.17.0
//   node repro.mjs                # default 54 tables / 13 indexes
//   TABLES=10 node repro.mjs      # scale it — cost is ~linear in object count
//
// Applies to `main` with tables+indexes only, deliberately: that keeps this focused on the
// optimystic module and avoids quereus's separate non-main-schema view/assertion defect.

import { Database, registerPlugin } from '@quereus/quereus';
import optimysticPlugin from '@optimystic/quereus-plugin-optimystic/plugin';
import { MemoryRawStorage } from '@optimystic/db-p2p';

const TABLES = Number(process.env.TABLES ?? 54);
const INDEXES = Number(process.env.INDEXES ?? 13);

// ---------------------------------------------------------------------------
// synthetic schema of roughly the shape a real app schema has
// ---------------------------------------------------------------------------
function buildSchema(tables, indexes) {
	const parts = [];
	for (let i = 0; i < tables; i++) {
		parts.push(`
  table T${i} (
    Id text,
    Name text,
    Amount integer,
    CreatedAt datetime,
    primary key (Id),
    constraint T${i}NameNotEmpty check (Name is null or Name <> '')
  );`);
	}
	for (let i = 0; i < indexes; i++) {
		parts.push(`\n  index T${i}ByName on T${i} (Name);`);
	}
	return parts.join('');
}

const DDL = buildSchema(TABLES, INDEXES);
const WRAPPED = `declare schema main {${DDL}\n}\napply schema main;`;

// ---------------------------------------------------------------------------
// IRawStorage instrumentation
//
// NOTE: walk the WHOLE prototype chain. MemoryRawStorage's own prototype carries only
// `constructor`; the storage methods live one level further up. A single
// Object.getPrototypeOf() instruments nothing and reports a misleading "0 calls".
// Also preserve async-generator-ness — several methods are AsyncGeneratorFunctions and
// callers do `yield* storage.listRevisions(...)`.
// ---------------------------------------------------------------------------
function methodNames(obj) {
	const names = new Set();
	for (let p = Object.getPrototypeOf(obj); p && p !== Object.prototype; p = Object.getPrototypeOf(p)) {
		for (const n of Object.getOwnPropertyNames(p)) if (n !== 'constructor') names.add(n);
	}
	for (const n of Object.keys(obj)) names.add(n);
	return [...names].filter((n) => typeof obj[n] === 'function');
}

function instrument(storage) {
	const stats = new Map();
	const bump = (name, t) => {
		const s = stats.get(name) ?? { calls: 0, totalMs: 0 };
		s.calls++;
		s.totalMs += performance.now() - t;
		stats.set(name, s);
	};
	for (const name of methodNames(storage)) {
		const real = storage[name];
		const bound = real.bind(storage);
		const kind = real.constructor?.name;
		if (kind === 'AsyncGeneratorFunction') {
			storage[name] = async function* (...a) {
				const t = performance.now();
				try { yield* bound(...a); } finally { bump(name, t); }
			};
		} else {
			storage[name] = (...a) => {
				const t = performance.now();
				const out = bound(...a);
				if (out && typeof out.then === 'function') return out.finally(() => bump(name, t));
				bump(name, t);
				return out;
			};
		}
	}
	return stats;
}

// ---------------------------------------------------------------------------
// The proposed mechanism, prototyped at the IRawStorage seam: while a schema batch is
// open, memoise reads and invalidate them on the corresponding write. Outside a batch it
// is a pass-through, mirroring the beginSchemaBatch contract.
//
// This is a DEMONSTRATION of the available win, not a proposed patch — a real fix belongs
// inside the module (one transaction + one read context for the whole APPLY SCHEMA).
// ---------------------------------------------------------------------------
const READS = {
	getMetadata: (a) => `md:${a[0]}`,
	listRevisions: (a) => `rev:${a[0]}:${JSON.stringify(a.slice(1))}`,
	getMaterializedBlock: (a) => `mb:${a[0]}:${JSON.stringify(a.slice(1))}`,
	getPendingTransaction: (a) => `pt:${a[0]}:${JSON.stringify(a.slice(1))}`,
	listPendingTransactions: (a) => `lpt:${a[0]}:${JSON.stringify(a.slice(1))}`,
};
const WRITES = {
	saveMetadata: ['md', 'rev', 'mb'],
	saveRevision: ['rev', 'mb'],
	saveMaterializedBlock: ['mb'],
	savePendingTransaction: ['pt', 'lpt'],
	deletePendingTransaction: ['pt', 'lpt'],
	promotePendingTransaction: ['md', 'rev', 'mb', 'pt', 'lpt'],
};

function withBatchReadContext(storage) {
	let open = false;
	const cache = new Map();
	const counters = { hits: 0, misses: 0 };

	const invalidate = (blockId, namespaces) => {
		for (const key of [...cache.keys()]) {
			const [ns, id] = key.split(':');
			if (namespaces.includes(ns) && (id === String(blockId) || blockId === undefined)) cache.delete(key);
		}
	};

	for (const [name, keyOf] of Object.entries(READS)) {
		const real = storage[name];
		if (typeof real !== 'function') continue;
		const bound = real.bind(storage);
		if (real.constructor?.name === 'AsyncGeneratorFunction') {
			storage[name] = async function* (...a) {
				if (!open) { yield* bound(...a); return; }
				const k = keyOf(a);
				let arr = cache.get(k);
				if (arr === undefined) {
					counters.misses++;
					arr = [];
					for await (const x of bound(...a)) arr.push(x);
					cache.set(k, arr);
				} else counters.hits++;
				yield* arr;
			};
		} else {
			storage[name] = async (...a) => {
				if (!open) return bound(...a);
				const k = keyOf(a);
				if (cache.has(k)) { counters.hits++; return cache.get(k); }
				counters.misses++;
				const v = await bound(...a);
				cache.set(k, v);
				return v;
			};
		}
	}
	for (const [name, dirties] of Object.entries(WRITES)) {
		const real = storage[name];
		if (typeof real !== 'function') continue;
		const bound = real.bind(storage);
		storage[name] = async (...a) => {
			const r = await bound(...a);
			if (open) invalidate(a[0], dirties);
			return r;
		};
	}
	return {
		counters,
		begin() { open = true; cache.clear(); },
		end() { open = false; cache.clear(); },
	};
}

// ---------------------------------------------------------------------------
async function run({ batched }) {
	const storage = new MemoryRawStorage();
	const gate = withBatchReadContext(storage);
	const stats = instrument(storage);

	const db = new Database();
	const networkName = `repro-${batched ? 'batched' : 'baseline'}`;
	const result = optimysticPlugin(db, {
		default_transactor: 'local',
		default_key_network: 'libp2p',
		default_network_name: networkName,
		enable_cache: true,
		rawStorageFactory: () => storage,
	});
	for (const v of result.vtables ?? []) db.registerModule(v.name, v.module, v.auxData);
	for (const f of result.functions ?? []) db.registerFunction(f.schema);
	for (const c of result.collations ?? []) db.registerCollation(c.name, c.func, c.normalizer);

	db.setDefaultVtabName('optimystic');
	db.setDefaultVtabArgs({ networkName, transactor: 'local', keyNetwork: 'libp2p' });
	await result.hydrate(db);

	if (batched) gate.begin();
	const t0 = performance.now();
	await db.exec(WRAPPED);
	const applyMs = performance.now() - t0;
	if (batched) gate.end();

	// correctness: identical catalog + a working write/read round-trip
	const schema = db.schemaManager.getSchema('main');
	const names = [...(schema?.tables?.keys?.() ?? [])].filter((n) => /^t\d+$/i.test(n)).sort();
	const fingerprint = names.map((n) => `${n}:${schema.tables.get(n)?.columns?.length ?? '?'}`).join(',');
	let roundTrip;
	try {
		await db.exec(`insert into T0 (Id, Name, Amount, CreatedAt) values ('a', 'alice', 7, '2026-01-01T00:00:00')`);
		const row = await db.get(`select Amount from T0 where Id = 'a'`);
		roundTrip = `write+read OK (Amount=${row?.Amount})`;
	} catch (e) {
		roundTrip = 'FAILED: ' + String(e?.message ?? e).split('\n')[0].slice(0, 70);
	}

	try { await db.close(); } catch { /* ignore */ }
	return { applyMs, stats, counters: gate.counters, tableCount: names.length, fingerprint, roundTrip };
}

const fmt = (x) => `${x.toFixed(1)}ms`;

console.log('=== Optimystic #8 repro: quereus 4.5.0 + quereus-plugin-optimystic 0.17.0 ===');
console.log(`schema: ${TABLES} tables + ${INDEXES} indexes, transactor=local, MemoryRawStorage, no libp2p`);
console.log(`node ${process.version} ${process.platform}/${process.arch}\n`);

// 1. the gap itself
{
	const db = new Database();
	const r = optimysticPlugin(db, {
		default_transactor: 'local', default_key_network: 'libp2p',
		default_network_name: 'hookcheck', enable_cache: true,
		rawStorageFactory: () => new MemoryRawStorage(),
	});
	for (const v of r.vtables ?? []) db.registerModule(v.name, v.module, v.auxData);
	const mod = (r.vtables ?? []).find((v) => v.name === 'optimystic')?.module;
	console.log('hook presence on the optimystic module:');
	console.log(`  beginSchemaBatch : ${typeof mod?.beginSchemaBatch}`);
	console.log(`  endSchemaBatch   : ${typeof mod?.endSchemaBatch}`);
	try { await db.close(); } catch { /* ignore */ }
}

const base = await run({ batched: false });

console.log('\n--- BASELINE raw-storage traffic for one cold apply ---');
const rows = [...base.stats.entries()].sort((a, b) => b[1].totalMs - a[1].totalMs).filter(([, s]) => s.calls);
const totalCalls = rows.reduce((a, [, s]) => a + s.calls, 0);
const totalMs = rows.reduce((a, [, s]) => a + s.totalMs, 0);
console.log(`${'op'.padEnd(28)}${'calls'.padStart(8)}${'total'.padStart(11)}`);
for (const [n, s] of rows) console.log(`${n.padEnd(28)}${String(s.calls).padStart(8)}${fmt(s.totalMs).padStart(11)}`);
console.log(`${'TOTAL'.padEnd(28)}${String(totalCalls).padStart(8)}${fmt(totalMs).padStart(11)}`);
console.log(`\napply wall time        : ${fmt(base.applyMs)}`);
console.log(`raw storage share      : ${((totalMs / base.applyMs) * 100).toFixed(1)}%`);
console.log(`objects created        : ${TABLES + INDEXES}`);
console.log(`storage calls / object : ${(totalCalls / (TABLES + INDEXES)).toFixed(1)}`);

const proto = await run({ batched: true });
const protoCalls = [...proto.stats.values()].reduce((a, s) => a + s.calls, 0);
const protoMs = [...proto.stats.values()].reduce((a, s) => a + s.totalMs, 0);
const hits = proto.counters.hits, misses = proto.counters.misses;

console.log('\n--- WITH a batch-scoped read context ---');
console.log(`apply wall time        : ${fmt(proto.applyMs)}`);
console.log(`raw storage time       : ${fmt(protoMs)}  (baseline ${fmt(totalMs)})`);
console.log(`read cache hits/misses : ${hits} / ${misses}  => ${((hits / (hits + misses)) * 100).toFixed(1)}% of reads redundant`);

console.log('\n=== result ===');
console.log(`  speedup        : ${(base.applyMs / proto.applyMs).toFixed(2)}x  (${fmt(base.applyMs)} -> ${fmt(proto.applyMs)})`);
console.log(`  tables created : ${base.tableCount} vs ${proto.tableCount}`);
console.log(`  catalog match  : ${base.fingerprint === proto.fingerprint ? 'IDENTICAL' : 'MISMATCH'}`);
console.log(`  round-trip     : "${base.roundTrip}" vs "${proto.roundTrip}" ${base.roundTrip === proto.roundTrip ? '(identical)' : '(MISMATCH)'}`);

Output (@quereus/quereus@4.5.0 + @optimystic/quereus-plugin-optimystic@0.17.0, node 24, linux/arm64):

=== Optimystic #8 repro: quereus 4.5.0 + quereus-plugin-optimystic 0.17.0 ===
schema: 54 tables + 13 indexes, transactor=local, MemoryRawStorage, no libp2p
node v24.15.0 linux/arm64

hook presence on the optimystic module:
  beginSchemaBatch : undefined
  endSchemaBatch   : undefined

--- BASELINE raw-storage traffic for one cold apply ---
op                             calls      total
listRevisions                   1692     74.7ms
getMaterializedBlock            1692     60.5ms
getMetadata                     6043     11.4ms
saveMaterializedBlock            308      6.0ms
getPendingTransaction            930      2.5ms
listPendingTransactions         1739      2.4ms
savePendingTransaction           183      2.1ms
saveMetadata                     230      0.5ms
promotePendingTransaction        183      0.4ms
saveRevision                     183      0.3ms
TOTAL                          13183    161.0ms

apply wall time        : 232.8ms
raw storage share      : 69.2%
objects created        : 67
storage calls / object : 196.8

--- WITH a batch-scoped read context ---
apply wall time        : 127.1ms
raw storage time       : 54.1ms  (baseline 161.0ms)
read cache hits/misses : 10677 / 1286  => 89.3% of reads redundant

=== result ===
  speedup        : 1.83x  (232.8ms -> 127.1ms)
  tables created : 54 vs 54
  catalog match  : IDENTICAL
  round-trip     : "write+read OK (Amount=7)" vs "write+read OK (Amount=7)" (identical)

Speedup over 5 consecutive runs: 1.69 / 1.55 / 1.57 / 1.63 / 1.91 -> median 1.63x.

calls / object stays roughly constant as the schema scales (194.4 at 54 tables, 181.8 at
10 tables, with 87.8% of reads redundant at the smaller size), so the cost is ~linear in object
count — consistent with per-object read amplification rather than a fixed startup cost.

Root cause

@optimystic/quereus-plugin-optimystic@0.17.0's virtual-table module object defines no
beginSchemaBatch / endSchemaBatch. Quereus's beginSchemaBatchAll skips modules without the
hook (packages/quereus/src/runtime/emit/schema-declarative.ts), so the module never learns that a
batch of DDL is in flight — it has no scope in which to reuse a transaction or cache a read.

The plugin's own source already documents the adjacent case, at src/optimystic-module.ts:193:

every cold-start connect() after hydrate() re-writes a byte-identical schema and re-reads it
back — one transaction per table+index, which dominates post-hydrate cold-start time

That short-circuit only fires when a schema is already persisted, so it helps a warm reopen but
misses every object on a cold create — which is exactly the case measured here.

Note this also affects @quereus/store's StoreModule, which likewise does not implement the
hooks — so createIsolatedStoreModule's IsolationModule (which does implement them, as a pure
delegate to underlying) forwards into a dead end. That half is a quereus-side gap; this issue is
about the optimystic module.

Suggested fix

Implement the two hooks on the optimystic vtab module, and use them to establish a
batch-scoped read context (not merely a shared write transaction — the measurements above
show reads are the cost):

async beginSchemaBatch(db, schemaName) {
  // open ONE transaction for the whole APPLY SCHEMA, and a per-batch read context that
  // memoises revision / materialized-block / metadata lookups
},
async endSchemaBatch(db, schemaName, error) {
  // commit once on success; discard on failure; drop the read context either way
}

Contract points worth honouring, per quereus's docs on the hooks:

• endSchemaBatch fires exactly once per successful beginSchemaBatch, on both the success
  and failure paths (error is set on failure).
• A module that owns no tables in schemaName should treat both as no-ops.
• Subsequent create/destroy/alter callbacks during the loop join the open batch.

Two follow-ups worth considering independently of the hooks, since batching only hides them:

1. ~198 raw-storage calls per created object looks high on its own merits. A single
   CREATE TABLE re-reading revision and block state ~25 times suggests missing memoisation
   inside table initialisation regardless of any batch.
2. structuredClone on every decode is a heavy default for a storage layer, and is
   pathological on runtimes where it is polyfilled (React Native/Hermes). If the clone is
   defensive copying, consider making it opt-in or using a cheaper structural copy for
   internally-owned values.

Severity

High for first-time schema creation on the optimystic substrate, and disproportionately so on
React Native/Hermes where structuredClone is polyfilled in JS. It does not affect steady-state
reads/writes, and it does not affect warm reopen: composeStrand's
hydrate-before-apply step means a re-attach diffs against a hydrated catalog, emits zero DDL, and
costs ~30 ms instead of ~430 ms (a 92.6% saving, measured). So the cost lands entirely on the
first apply per store — which is exactly the user-visible "create" action.

Unmeasured

Per-commit latency on a real distributed/networked transactor, on real LevelDB raw storage, and
under Hermes were not measured (no Android device available on the test host). The commit count
and the ~10× in-memory ratio are measured facts; the absolute on-device figure is not.

Side observation (separate issue material)

On the @quereus/plugin-leveldb path, closing the database logs
[StoreModule] Failed to persist catalog DDL after schema change: Database failed to open. It
reproduces in an uninstrumented control, so it is pre-existing and not caused by the harness. That
belongs to quereus/@quereus/store, not Optimystic, but it means catalog DDL persistence is
failing silently on that path.

[kjeib]

On-device (React Native / rn-leveldb) A/B: this amplification is a first-run blocker for a real schema

The issue lists the on-device figure as Unmeasured ("no Android device available on the test
host"). Here it is, from the Sereus Health app — plus a clean before/after that shows the read
amplification is not a micro-optimization on RN but a hard blocker once a schema has more than a
handful of objects.

Setup (identical across both arms)

• Same Android x86_64 emulator, same app, same 22-object schema (9 tables + 13 indexes).
• Release (Hermes) APKs, built the same way; storage is rn-leveldb via
  @optimystic/db-p2p-storage-rn (LevelDBRawStorage). Fresh install each run (empty store → cold
  apply schema). Node is solo (empty bootstrap, no listen addrs).
• Timed CadreNode.addStrand({ founder: true }) (which runs the cold apply schema) from its start
  log to the "strand ready" log.

Result

Stack	apply schema path	Cold-apply time
cadre-core 0.10 / optimystic 0.22 (StrandConfig.mode: 'bootstrap')	bootstrap-mode local transactor	~2.9 s ✅
cadre-core 0.12 / optimystic 0.27 (mode removed)	coordinated commit (cohort-of-1)	did not complete in 47 min ❌

Same schema, same device. The 0.12 arm was reproduced across debug, release, and a websockets-only
transport config (ruling out the relay/webRTC transports as a factor) — all >10 min, none completed.
optimystic:db-p2p:* shows a steady stream of solo commits (cohortSize: 1) at ~3–4/sec, ~30% CPU
with the rest spent waiting on native rn-leveldb calls — i.e. exactly this issue's ~194
raw-storage-calls-per-object read amplification, except each call is now a native-bridge + LevelDB
round-trip. For 22 objects that is enough to make founding effectively non-terminating.

Why the two arms differ so much

The amplification described in this issue was present but masked in the 0.10 stack: cadre-core's
now-removed StrandConfig.mode: 'bootstrap' routed schema apply through a local transactor that
did not incur the coordinated per-DDL read amplification. cadre-core 0.12 removed mode ("solo/local
commit is now automatic at the storage layer"), so cold apply schema now flows through the coordinated
commit path — and this issue's amplification is what lands on it. So on RN there are effectively two
compounding problems:

1. This issue (optimystic): cold apply schema does ~194 raw-storage calls/object with no
   beginSchemaBatch/endSchemaBatch; on RN each call is a native round-trip, so the cost is
   multi-minute and scales with object count.
2. cadre-core 0.12 regression (may deserve its own tracking issue): removing the bootstrap-mode
   local transactor exposed (1) to the founder/first-apply path with no fast-path replacement, turning
   a ~3 s operation into a non-terminating one for a non-trivial schema.

Fixing either would unblock the host app; fixing (1) — the beginSchemaBatch/endSchemaBatch
hooks this issue already proposes — is the more general fix and would also help the non-solo case.

Caveat

Figures are an x86_64 emulator (slow virtual disk is a worst case for this I/O-bound path); a real
arm64 device's flash storage would be faster. But the 0.10-vs-0.12 ratio (~3 s vs >47 min) is a
same-device comparison and is not an emulator artifact.

[n8allan]

Thanks to both of you — the original report and the on-device A/B are unusually good, and the
follow-up caught us out on a claim we had not yet made publicly. Here is where this stands, what
we could reproduce, and the one thing we cannot answer without a few more log lines from the device.

The amplification this issue was filed about is largely fixed — but not by the hooks

withReadCache landed between 0.17 and 0.27 and now fronts a node's raw storage
(libp2p-node-base.ts, resolveStorage). We rebuilt @risavian's exact workload at HEAD (0.27.0) —
solo libp2p node, KvRawStorage over an instrumented driver, counting below the cache:

schema	driver calls	per object	commits
54 tables + 13 indexes (this issue's)	1,575	23.5	80

Against the reported 13,025 / 194.4 per object — an 88% cut. The harness is measuring the same
thing you were: it produces 178 promote/putRevision, exactly the 178 you reported for that
schema.

beginSchemaBatch / endSchemaBatch are still unimplemented. Your diagnosis was right and your
proposed fix is still not built; the symptom was addressed on a different axis. We are not claiming
we did what you asked. The hooks remain the better shape for the commit-count and round-trip half
of the problem, which a read cache cannot touch.

We could not reproduce the non-termination

@kjeib — we built the coordinated (cohort-of-1) path your 0.12 arm takes and ran your 22-object
schema through it on Node:

arm	wall	commits
9t + 13i, local transactor	0.16 s	84
9t + 13i, network transactor, solo cohort	2.07 s	35
200t + 50i, network	31 s	300

Roughly linear, no stall.

One thing your report lets us pin down: ~3–4 solo commits/sec is the normal rate of this workload
on slow storage, not a loop signature. Injecting 5 ms per storage call — an emulator-plausible
figure for a native-bridge LevelDB round trip — reproduces your log rate exactly:

t=2.0s  commits=7  (+3.5/s)
t=4.0s  commits=15 (+4.0/s)
t=6.1s  commits=22 (+3.4/s)

...and finishes in 11.45 s, having issued 35 commits total. That is the crux: 22 objects budgets
~35 commits. A steady 3–4/sec sustained for 47 minutes is on the order of 9,000 commits — ~250x
more than the schema justifies. So the rate is explained and the volume is not.

We ruled out two candidate mechanisms:

• Timeout-induced retry storm — squeezing NetworkTransactor.timeoutMs to 50 ms produced an
  identical 35 commits.
• Read-cache thrash — the working set is 688 KB for 67 objects, well inside React Native's
  8 MB budget (platformDefaultBytes). Even forced down to a 256 KB budget, driver calls only go
  1,575 → 2,524.

What we need from the device

The decisive question is whether the same blocks are being committed over and over, or whether
cadre-core's founding path is doing far more work than the 22 declared objects. These are
different bugs and the fix differs. Five things, roughly in order of usefulness:

1. commit:solo-cohort block ids — the one that settles it.

DEBUG=optimystic:db-p2p:coordinator-repo:*

Every commit logs { blockId, cohortSize, soleIsSelf }. Please capture ~60 s of it from deep
into the run (say 10+ minutes in, not the first minute) and tell us: how many distinct blockId
values appear in that window, versus how many lines?

• Distinct ids, few repeats → founding is doing genuine work far exceeding 22 objects; the problem
  is workload volume and this issue's batching hooks are the right lever.
• A small set of ids cycling → a commit loop, which is a defect on our side and takes priority over
  everything else in this issue.

Also worth reporting: soleIsSelf. If it is false (or cohortSize: 0), routing is degraded
rather than genuinely solo, which is a different failure again.

2. Is it progressing at all? Cumulative count of distinct blockIds at, say, the 2-minute and
the 20-minute mark. Flat ⇒ stuck; climbing ⇒ slow but live.

3. Storage calls per object on device. Wrapping the driver you hand the node is enough:

const counts = {};
const counted = new Proxy(realDriver, {
  get(t, p, r) {
    const v = Reflect.get(t, p, r);
    if (typeof v !== 'function') return v;
    return (...a) => { counts[String(p)] = (counts[String(p)] ?? 0) + 1; return v.apply(t, a); };
  },
});
// ...pass `new KvRawStorage(counted)` as the node's `storage`, dump `counts` periodically

If you see ~23 calls/object, storage is behaving as it does here and the cost is elsewhere. If you
see ~194, the read cache is not attached on your build — which would be its own bug and would
explain a great deal.

4. Confirm the cache actually attached. After ~60 s:

import { defaultCachePool } from '@optimystic/db-p2p';
console.log(JSON.stringify(defaultCachePool().stats()));

An empty stores: [] means the node never wrapped your storage — check whether the storage you pass
already reports readCached, and whether the node is built through createLibp2pNode.

5. findCluster timing. DEBUG=optimystic:db-p2p:cluster-tx:* logs cluster-tx:cluster-members
once per commit. On Node this is already the single most expensive seam we measured: 178 calls,
2,455 ms, 13.8 ms/call — 49% of a 5-second apply, on a solo node with zero peers. Every commit
pays it through getClusterPeerIds. If wall-clock gaps between consecutive cluster-members lines
on device are in the hundreds of ms or seconds, that alone could account for the cliff. We are
opening a separate issue to make this cheap when the cohort is known to be just this node,
independent of how the above resolves.

Also

@kjeib's point (2) — that cadre-core 0.12 removed StrandConfig.mode: 'bootstrap' without a
fast-path replacement — is not ours to fix, but it is the proximate cause of the regression and
does deserve its own tracking issue on that side.

We are adding regression gates for the coordinated cold-apply path (driver calls/object,
commits/object, and a two-scale check that calls/object does not grow with schema size). Today only
the local transactor is guarded, which is precisely why the path that broke on your device drifted
unnoticed.

[kjeib]

Device diagnostics — and a correction: the non-termination is a control-network read-repair loop, not the schema amplification

@n8allan — thank you, this was exactly the right set of probes. I captured a clean, foreground-pinned
22-minute run on the x86_64 emulator (stock RN config: profile: 'transaction', solo control network,
empty bootstrap, LevelDBRawStorage; DEBUG=optimystic:db-p2p:coordinator-repo:*,cluster-tx:* plus a
30 s defaultCachePool().stats() log). It overturns my first comment's headline — the schema
amplification is not what hangs founding. Here is what the device shows, against your five asks:

Correction to my earlier comment: I attributed the non-termination to this issue's read
amplification. That was wrong. The amplification is real but it completes — the hang is a separate
control-network read-repair loop (below). My earlier "steady 3–4 commits/sec ⇒ ~9,000 commits" figure
came from contaminated runs where Android had backgrounded the app and paused/retried the JS
thread; disregard it. The clean numbers:

#4 — the read cache IS attached and working. defaultCachePool().stats() shows
stores:[{label:"control",…},{label:"<strandId>",…}], and hits climb continuously
14,081 (30 s) → 52,221 (22.5 min). So reads are served from cache; read amplification is not the
on-device cost. (This is also why manually wrapping our storage in CachedRawStorage changed nothing —
the node already wraps it.)

#1/#2 — commits are bounded and front-loaded, not looping. commit:solo-cohort over the whole
run: 120 total, 44 distinct blockIds, and 116 of them in the first 2 minutes, then ~flat
(2 commits per 8-min window after). Every commit is cohortSize: 1, soleIsSelf: true — genuinely solo.
That matches your ~35-commit budget once control-network blocks are included. So the commit path is
fine on device.

#5 — findCluster is not on the solo path. cluster-tx:cluster-members: 0 occurrences. The
solo path short-circuits via cluster-fetch:solo-self-skip (thousands of them). So the
getClusterPeerIds seam you flagged isn't our cost here.

The actual defect — a non-converging control read-repair loop. After the 2-minute commit burst,
addStrand({ founder: true }) never returns and the node spins for the entire remaining 20+ minutes
in:

cluster-tx:read-repair-triggered → cluster-fetch:solo-self-skip → cluster-tx:read-repair-noop   (repeat)

Whole-run totals: 3,880 read-repair-triggered, 3,879 read-repair-noop (every one a no-op),
across only 9 distinct blockIds (~430 repairs/block, ~29 ops/sec sustained — the same rate the cache
hits climb at). The 9 blocks are all control-network rows, not the strand schema:
default/Strand (~2,000×), default/CadrePeer (~1,450×), default/Revocation, default/OwnerKey,
their _uniq_*.stampid indexes, and a few hashed control blocks. No errors, no timeouts; soleIsSelf
stays true throughout.

Per your own rubric — "a small set of ids cycling → a commit loop, a defect on our side, top
priority" — this is that, on the read-repair path: a solo node repeatedly triggering read-repair on
its control tables, each finding nothing to repair, never settling, and (apparently) blocking founding
from completing.

What would help pin it down further

I stopped at coordinator-repo/cluster-tx per your ask. If you point me at the namespace(s) that
would show what drives the repeated control reads (control-cohort reconcile? a founder-side
membership/consistency wait? strand bring-up polling the control DB?), I'll re-run and capture — the
repro is 100% reliable here (~2 min to reach the loop, then indefinite). A driver-call Proxy (#3) is
also easy to add if the calls/object number is still useful, though #4 already shows the cache is
carrying the reads.

Related

Filed the cadre-core-side proximate cause (removal of StrandConfig.mode: 'bootstrap' with no
fast-path replacement) as gotchoices/sereus#8, per your suggestion — though this read-repair loop
looks like the more immediate on-device blocker and is squarely Optimystic-side.

[n8allan]

0.28.0 is out — with a measured warning that it probably does not fix the device case

@optimystic/quereus-plugin-optimystic@0.28.0 and @optimystic/db-p2p@0.28.0 are on npm.

What landed that touches this issue

Profiling the cold apply schema on a solo node found that ~49% of it was one syscall.
findCluster puts this node's own addresses into every cluster record it builds, and it got them
from libp2p.getMultiaddrs(), which re-derives announce addresses from os.networkInterfaces() — a
full network-interface sweep, measured at 3.19 ms of a 3.49 ms call. Every commit resolves a
cohort (CoordinatorRepo.commit → getClusterPeerIds → findCluster), so a cold apply was paying
one interface sweep per commit to recompute an identical answer.

Those addresses are now derived once and held until libp2p reports self:peer:update.

	0.27.0	0.28.0
findCluster, solo node	3.85 ms/call	0.010 ms/call
cold apply, 9 tables + 13 indexes	2.07 s	0.21 s
cold apply, 54 tables + 13 indexes	4.89 s	0.61 s

Why you should not expect that to move your number

@kjeib — before you spend a build on this: the cost I removed is very likely not a cost you were
paying. Those figures come from a node with a TCP listener. The expensive part is expanding a
wildcard listen address (0.0.0.0) across the machine's interfaces, and your arm — websockets-only,
empty bootstrap, no listen addrs — never does that. Measured directly, on the same machine:

node configuration	getMultiaddrs()
TCP listener (what I originally benchmarked)	4.44 ms/call
websockets-only, no listen addrs (your shape)	0.001 ms/call
os.networkInterfaces() alone, for reference	3.37 ms/call

With no listen addresses there is nothing to expand, getMultiaddrs() returns an empty list, and
the sweep never happens. So this is a real fix for service/reference peers that listen on TCP and
commit — and close to a no-op for an edge client shaped like yours. Please do re-test, since it
costs you only a version bump, but if the 47 minutes is unchanged that is the expected result, not a
failed fix.

I would rather tell you that up front than have you read a speedup table and conclude the bug was
addressed.

What is still NOT in 0.28.0

• beginSchemaBatch / endSchemaBatch are still unimplemented. @risavian's original ask stands.
• No fix for the non-termination, because we still cannot explain it. Everything in the previous
  comment holds: the coordinated cohort-of-1 path completes here in 0.21 s for your 22-object
  schema; retry storms and read-cache thrash are both ruled out by measurement; and 3–4 commits/sec
  is reproducibly the normal rate at ~5 ms per storage call rather than a loop signature. 22
  objects budgets ~35 commits. 47 minutes at that rate is ~9,000. The rate is explained, the
  volume is not, and that gap is the whole bug.

Also in this release, and relevant to you

The plugin now builds against Quereus 4.19 (you reported against 4.4.1 / 4.5.0). 4.19 retired
TableSchema.estimatedRows in favour of statistics, so a plugin built for the older engine will
not type-check against it. If you are pinned to an older Quereus, upgrade both together.

There are now regression gates on the coordinated cold-apply path — raw-storage calls per object,
commits per object, cohort lookups per commit, and a two-scale check that per-object cost does not
grow with schema size. Previously only the local transactor was guarded, which is precisely why
the path your app switched onto could drift without anyone noticing. That is the process half of
this issue, and it is the part I am most confident was worth doing.

Still the one thing that would settle it

The request from the previous comment is unchanged and is still the blocker:

DEBUG=optimystic:db-p2p:coordinator-repo:*

From a ~60 s window taken deep into the run (10+ minutes in, not the first minute): how many
distinct blockId values appear, versus how many lines?

• Few distinct ids, cycling → a commit loop on our side, and this becomes our top-priority defect.
• Many distinct ids → founding is doing far more work than the 22 declared objects, and the batching
  hooks in this issue are the right lever after all.

Also worth a glance while you are in there: if those logs carry repeated pending conflict: block … held by unresolved action(s) … rejections, say so — we have a separate tracked defect with that
signature. I do not think it is your cause (its mechanism needs a multi-member cohort, and
CoordinatorRepo short-circuits a cohort of 1 straight to local storage before that gate), but it
is cheap to rule out from logs you already have.

[kjeib]

The "settle it" data (deep-in window) + pending conflict ruled out

Thanks @n8allan — and appreciated the honesty on 0.28.0. Answering your two remaining probes from the
same clean 22-min 0.27 capture (I'll explain below why re-capturing on 0.28 wouldn't change this).

Deep-in coordinator-repo:* — distinct blockIds vs lines (60 s windows, 14–17 min in):

window	coordinator-repo lines	distinct blockIds	commits
14–15 min	441	5	0
15–16 min	535	8	2
16–17 min	441	5	0

That is your "few distinct ids, cycling → a commit loop on our side, top-priority defect" case —
except the cycling is on the read-repair path, not commits. Each ~440-line minute is
read-repair-triggered → cluster-fetch:solo-self-skip → read-repair-noop repeating over the same
handful of control-network blocks: default/Strand, default/CadrePeer, default/Revocation,
and two hashed control blocks. Whole run: 3,880 read-repair-triggered, 3,879 read-repair-noop
(every one a no-op), across 9 distinct blockIds; commits are 120 total and finish in the first ~2 min.
So the ~9,000-commit figure was never real (it was my contaminated early runs) — the volume is
read-repairs, not commits, and it never converges.

pending conflict / unresolved action: 0 occurrences in the whole capture — so that tracked
defect is ruled out here, as you suspected.

Why not re-capture on 0.28.0

Two reasons, and I'd rather be straight about them:

1. You've said 0.28 has no fix for the non-termination, and its actual change (the
   findCluster/getMultiaddrs interface-sweep) is one our shape (ws-only, no listen addrs) doesn't
   pay. Since the read-repair loop lives in a different path, the deep-in picture above would be
   identical on 0.28 — no new signal for this question.
2. We can't cleanly run 0.28 yet regardless: @serfab/cadre-core@0.12.0 pins @optimystic/* ^0.27
   and @quereus/quereus ^4.18, and there's no cadre-core release paired with optimystic 0.28 /
   quereus 4.19. Is a cadre-core release targeting the 0.28/4.19 line planned? That's what would
   let us adopt the fixes and do a real on-device re-test.

Happy to capture more

The repro is 100% reliable (~2 min to reach the loop, then indefinite). If you point me at the
namespace that shows what schedules the repeated control reads — whatever drives read-repair to
re-fire on default/Strand / CadrePeer / Revocation on a solo node — I'll capture it. That
scheduler is the thing to find.

[n8allan]

Found it. That capture was decisive — thank you.

Your read-repair observation was the whole answer, and it is ours. Tracked as a fix ticket now, with
your figures as the reference data.

The defect

CoordinatorRepo.fetchBlockFromCluster has three exits. Two of them call markBlocksSeen(blockId),
which stamps lastSeenCommitMs — and that stamp is the only thing that arms the read-repair
window. The third exit is the solo short-circuit, the one printing your cluster-fetch:solo-self-skip:

// Solo-cluster short-circuit: the only responsible peer is us. There is no
// remote to sync from, so skip the callback entirely.
this.log('cluster-fetch:solo-self-skip', { blockId });
return { absence: 'confirmed' };        // <-- returns WITHOUT arming the window

Skipping the fetch is right — there is no remote, and on a node with no listen addresses dialling
self can hang, which is exactly your configuration. What is wrong is that it returns without
recording that the check happened. So shouldReadRepair in lazy mode reads lastSeen == null,
returns true, and does so again on the next read, and the next:

read → window says stale → consult → solo skip → no-op → window still says stale.

Nothing in that cycle can make progress and nothing bounds it. That is your 3,880 triggered /
3,879 no-ops, and it is why the count never converges while commits finish in two minutes.

Reproduced here

Against CoordinatorRepo directly with a deterministic clock — nine reads, one second apart, so
reads 2–9 all fall inside the 10 s window and must not re-consult:

	reads	read-repair-triggered	read-repair-noop	solo-self-skip
current	9	9	9	9
with the solo exit arming the window	9	1	1	1

The 1:1 trigger/no-op ratio matches your 3,880/3,879, which is what convinces me the probe and your
device are the same defect rather than two things that look alike.

One methodological note, since it cost me a wrong answer first: my initial probe stepped the clock
60 s between reads, and showed 9 triggers both before and after the fix — because a lapsed window
is supposed to re-trigger. The bug is only visible on reads taken inside the window. If you ever
probe this yourself, that is the trap.

Why it took a device to find

It needs a cohort of exactly one that is self. Any multi-node deployment takes the consult path and
arms the window on the way out, so our whole test suite missed it. And your previous stack routed
schema creation through a local transactor that never entered the coordinated read path — so
cadre-core 0.12 did not introduce this, it exposed it. The defect is older than your report.

That also retires the last of my earlier speculation: it was never a commit loop, and the ~9,000
figure was never the thing to explain. You were right to correct it.

On your two questions

The scheduler you asked for is shouldReadRepair / lastSeenCommitMs in coordinator-repo.ts —
there is no separate timer. Read-repair is driven entirely by reads, which is why it looks like a
background loop: something in the founding path keeps reading those control blocks, and every read
re-arms nothing. You do not need to capture anything further for this; the ticket has what it needs.

A cadre-core release paired with optimystic 0.28 / quereus 4.19 is not my call — @n8allan will
have to answer that one. Flagging that it is the thing blocking you from adopting anything we ship,
including this fix once it lands.

Also worth your time

@optimystic/db-p2p now routes every service's logs through the package's own optimystic:db-p2p:*
tree — six services (including cluster and repo, both on the commit path) previously logged under
libp2p's db-p2p:* root and were invisible to the documented filter. That is #16, and it is the
reason an earlier reporter read a zero log count as proof a path never ran. Unreleased as of writing.
When you next capture, DEBUG=optimystic:db-p2p:* will actually show you everything.

No fix date promised yet — but the diagnosis is closed, and it is on the board rather than in a
comment thread.

[n8allan]

Root cause of the on-device hang is found and fixed on main. Your 60-second window table is what made it findable — thank you for taking that measurement, it was decisive.

Read the scope carefully, because this issue holds two separate things and only one of them is closed.

What your data said, and what it turned out to be

Your capture: 3,880 cluster-tx:read-repair-triggered against 3,879 cluster-tx:read-repair-noop, across 9 distinct blockIds, each interleaved with cluster-fetch:solo-self-skip, while commits had all finished inside the first two minutes. Every consult a no-op. Never converging.

That 1:1 ratio was the tell. fetchBlockFromCluster has three exits, and two of them call markBlocksSeen, which stamps the timestamp that is the only thing arming the lazy read-repair window:

• cluster-fetch:local-current — arms it.
• the normal end of a consult — arms it.
• the solo short-circuit — did not.

The solo short-circuit fires when the cohort is exactly one peer and that peer is you. Skipping the fetch is correct — there is no remote to sync from — but returning without stamping the block left lastSeenCommitMs unset, so shouldReadRepair read null and returned true on that read, and on every read after it, forever. Read → "stale" → consult → solo skip → no-op → still "stale". Nothing in the loop could make progress and nothing bounded it.

Reproduced against CoordinatorRepo with a deterministic clock, cohort ['self-peer'], nine reads one second apart (all inside the 10 s window, so reads 2–9 must not re-consult):

	reads	triggered	no-op
before	9	9	9
after	9	1	1

Same 1:1 ratio you measured on device, which is the evidence the probe and your report are the same defect.

Why it was invisible to us for so long: it needs a cohort of exactly one, which is self. Any multi-node deployment takes the consult path and arms the window on the way out. Your previous stack routed schema creation through a local transactor that never entered the coordinated read path — so upgrading exposed this, it did not introduce it. The defect is older than your report.

The fix arms the window on that exit. It ships with a spec that we verified fails without it — reverting the one line fails 4 tests. There is a deliberate asymmetry recorded at the site: the neighbouring peerIds.length === 0 exit stays unarmed, because an empty cohort is a routing failure that can recover at any moment, not a settled answer.

What is NOT fixed: this issue's title

beginSchemaBatch / endSchemaBatch are still unimplemented, and the ticket for them is still open. What has changed on the cost side, measured on Node:

	before	now
raw-storage driver calls per object (67-object schema)	~194	23.5
cold apply schema, 22 objects	2.07 s	0.21 s
cold apply schema, 67 objects	4.89 s	0.61 s

Most of that came from a separate defect — findCluster was re-deriving this node's own addresses on every commit through os.networkInterfaces(), a blocking syscall, once per commit. Caveat we measured after the fact and would rather you heard from us than discovered yourselves: that win is real for listening peers and close to nothing for a dial-only mobile client (4.44 ms/call with a TCP listener versus 0.001 ms/call for a websockets-only node with no listen addresses). It is not the reason your device was slow.

We also added five cost gates over the coordinated cold-apply path so this cannot regress unnoticed, since nothing guarded that path when it regressed on you.

Still true, and the reason this issue stays open: per-object ITransactor.get grows with schema size (40.9 / 55.0 / 80.6 at 22 / 67 / 250 objects) because each created object re-reads a growing catalog. The read cache absorbs it before it reaches storage, which is why the driver-call figure above is flat while this rises. Batch-scoped context is what would stop the re-read, and that is exactly what the schema-batch hooks are for.

Availability

None of this is in 0.28.0. It is on main, unreleased. When it ships, the thing to re-run is your own capture: on a solo node the read-repair-triggered count should drop to roughly one per block per 10 s window, and cluster-fetch:solo-self-skip should stop being a per-read event.

[n8allan]

Released. Everything described above is in 0.29.0, published today across all six packages (db-core, db-p2p, db-p2p-storage-*, quereus-plugin-optimystic, quereus-plugin-crypto). The previous comment said "none of this is in 0.28.0, it is on main, unreleased" — that is no longer the case.

The scope split in that comment is unchanged: the solo read-repair non-termination is fixed, this issue's title is not. beginSchemaBatch / endSchemaBatch are still unimplemented and the ticket for them is still open, so per-object ITransactor.get still grows with schema size for the reason given above.

What to re-run, unchanged from the last comment: your own capture. On a solo node, cluster-tx:read-repair-triggered should drop to roughly one per block per 10 s window, and cluster-fetch:solo-self-skip should stop being a per-read event. If it does not, that is a new finding rather than a residue of this one, and worth its own capture.

[kjeib]

Re-ran the Sereus Chat capture on 0.29.0. The read-repair fix is clearly working — and it
uncovers something underneath it that I think is a new finding, per your "if it does not, that is a
new finding rather than a residue" note.

The fix is engaged

Same app, same 4-table schema, fresh party, solo node, RN over rn-leveldb, debug on both times.

	0.28.0	0.29.0
owner genesis	9.5 s	797 ms
cluster-fetch:solo-self-skip	per read, unbounded	mostly one per block per window

Genesis alone is a 12× improvement, so the window is arming.

But the apply still does not converge — and one block is why

Strand apply began 08:50:09. At 09:10 — 20 minutes — it had not attached and the app was
still alive. 790 skips over 37 distinct blocks, so per-block repeat gaps are measurable:

	n	median gap
repeats at/over the window	296	14.32 s ✅
repeats inside the window	225	0.141 s ❌

183 of those 225 are under one second; the minimum is 8 ms. And they are not spread evenly —
one block dominates:

block	sub-window repeats / total repeats
default/Revocation	179 / 229
optimystic/schema	16 / 19
default/CadrePeer	8 / 60
default/Strand	4 / 70
default/CadrePeer/index/_uniq_7.stampid	4 / 5

default/Strand and default/CadrePeer behave — overwhelmingly at-window, which is the fix doing
its job. default/Revocation re-enters the armed exit in a sub-second loop and was still doing so
at 09:03:46 (17 ms apart), thirteen minutes in.

A contiguous slice, so the shape is visible — a full findCluster per iteration:

db-core:batch-coordinator  createBatches blockIds=1 batches=1 excluded=0   +407ms
db-p2p:storage-repo        get blockIds=1                                  +369ms
libp2p-key-network         findCluster:start  key=ZGVmYXVsdC9S
libp2p-key-network         findCluster:done   …
coordinator-repo           cluster-fetch:solo-self-skip { blockId: 'default/Revocation' }  +410ms
db-p2p:storage-repo        get blockIds=1                                    +7ms
db-core:network-transactor get:done blockIds=1 ms=25                         +25ms
db-core:network-transactor get blockIds=1                                     +0ms

So markBlocksSeen is being called at that exit — the other blocks prove it — yet for this one the
next read does not see the stamp. I cannot tell from outside whether the stamp is keyed more
narrowly than the re-check (block vs block+context), or whether something clears it for a block that
is repeatedly absent. default/Revocation is presumably empty on a fresh party, which may be the
distinguishing property: the blocks that behave are ones that have content.

To rule out your other candidate explicitly: this is the self-cohort exit, not the
peerIds.length === 0 one. Libp2pKeyPeerNetwork.findCluster always includes self, and every
sample shows the solo-self log line, which the empty-cohort exit does not emit.

Scope note

Understood that beginSchemaBatch/endSchemaBatch remain unimplemented and that per-object get
still grows with schema size — I am not reporting the cost here. At 4 tables, 20 minutes without
converging still reads as non-termination rather than cost, and the sub-window loop above is the
mechanism I can actually point at.

Happy to run a build with extra instrumentation around shouldReadRepair / markBlocksSeen if that
would settle it faster than reasoning from the outside — this is our only blocker, so we have every
reason to.

Environment

@optimystic/* 0.29.0 (all five, via yarn resolutions — @serfab/cadre-core@0.12.0 still
pins ^0.27.0, so they do not arrive otherwise), @serfab/cadre-core 0.12.0,
@quereus/quereus 4.19.0, p2p-fret 1.0.0-beta.4, libp2p 3.3.8. React Native (Hermes), Android
emulator API 37, LevelDBRawStorage over rn-leveldb. Solo, founder, no relay, no cohort peers.