v0.13.0

Tool analytics stop paying the JSON tax: the common query shapes now run on three narrow derived columns instead of the multi-GB variant_data blob, turning remote S3 tool GROUP BYs from hard >30s timeouts into ~9s answers (local: 1,693ms -> 48ms, ~35x). Existing stores upgrade themselves in place on first open - no re-ingest, no manual step - but once migrated they are unreadable by older pond binaries, so upgrade every machine that shares a store together.

🛠 Breaking Changes

• parts: [breaking] materialize tool columns + in-place schema migration; per-query SQL timeout (#100) (5490122)
  • What breaks: the parts table gains three derived nullable columns - tool_name, call_id, is_failure - plus a BTree scalar index on tool_name. pond <= 0.12.2 enforces strict schema equality, so a store first opened by 0.13.0 becomes unreadable by older binaries, and there is no downgrade path once migrated. variant_data stays the verbatim source of truth; the new columns are derived from it at write time.
  • What upgrading requires: nothing manual. The first open by a 0.13.0 binary migrates the store in place: a one-time backfill derives the three columns from stored variant_data (seconds on a local store; one add_columns commit on a remote/S3 store), announced by a single stderr notice. If multiple machines sync into one shared store, upgrade all of them before the first 0.13.0 open - any host still on <= 0.12.2 loses access the moment the store migrates.
  • Pre-0.13.0 .pond archives keep restoring unchanged: the columns are derived at the read boundary and the archive file is never modified.
  • Also in this change: a per-query SQL timeout - timeout_seconds on pond_sql_query, --timeout on pond sql (default 30s, clamp 1..600); the timeout error names the knob and steers toward the native columns.
  • Measured on the full real corpus (10,840 sessions / 1.81M messages / ~608K tool-call parts) over S3: tool GROUP BY timeout -> 8.5-9.7s, failure-rate self-join timeout -> 23-24s, indexed tool_name = 'Bash' point filter 4.7-8.0s. Ingest and read benchmarks unchanged within noise.

📚 Documentation

• update README for clarity and structure, add new sections on usage and maintenance (8298db5)
• launch fold, memory-tool FAQ, OG/meta, vocs 2.3.3, reference-page fixes (a3a3e65)
• readme: launch fold - hook quote, live search demo, real prompts (d53226f)

Full Changelog: v0.12.2...v0.13.0

v0.12.2

🎉 New Features

• deps: upgrade lance 7.0.0 -> 8.0.0 (56e968e)

Full Changelog: v0.12.1...v0.12.2

v0.12.1

Sync status now reports genuine work only, and forked subagent transcripts are no longer silently dropped. Verified end to end against the full real corpus (11k+ sessions / 1.8M messages) on both a local store and the S3 backend: ingestion is byte-identical to v0.12.0 except for the recovered data.

🎉 New Features

• status: pending preview for claude-desktop-app and opencode (fa41ddf) - these adapters now report an accurate per-session pending count instead of "pending unknown".

🐛 Bug Fixes

• sync: stop reporting provably-synced or empty sessions as pending (6a05cd1) - a source whose stored watermark already covers it (or that a bounded whole-source scan proves holds nothing ingestible) no longer counts as pending, so a clean store reports "up to date" instead of a permanent phantom floor (real corpus: claude-code 43 -> 0 false pending, codex-cli 4 -> 0). Skip signals derive only from stored data; anything the gate cannot cheaply classify still re-reads.
• claude-code: ingest forked subagent transcripts (06a2d27) - a /fork subagent transcript (Claude Code >= 2.1.117) opens with a fork-context-ref header row that carries no sessionId, which the adapter rejected as "line 1 missing sessionId" - silently dropping the entire forked conversation. The id is now taken from the first row that carries one (subagents derive it from the path regardless), recovering the full transcript with lossless native restore. Real corpus: 1 of 7,843 subagent transcripts affected, recovered as 16 messages, every other row byte-identical.

Full Changelog: v0.12.0...v0.12.1

v0.12.0

Onboarding and multi-machine sync: a first run no longer looks like it hangs, the scheduled sync no longer races a manual one, and pond status finally reports this host's own relationship to the store. Verified end to end on a fresh install (macOS and Linux) plus cold-context first-run testing.

🛠 Breaking Changes

• pond status --format json output shape changed - the adapters field is renamed to source_agents (count of distinct source agents in the store), and schedule changes from a string to an object {active, backend, every}. A new top-level pond_version field carries the producing binary's version so consumers can pin the format going forward. Scripts parsing status JSON must update these keys. (5e56aad)

🎉 New Features

• sync: per-host single-flight lock (a local flock in the state dir, never on the Lance store - cross-host writers stay pure OCC); a second sync waits and names the holder, --no-wait skips cleanly (exit 0) and is what the scheduled job passes so ticks never queue. Adds --dry-run (per-adapter freshness preview, writes nothing) and --format json (one summary document on stdout for every outcome, progress on stderr). Every long phase now has a live face - rowmap-build spinner, model-download stage line, per-adapter bar with a recent-rate ETA, inline-embed counter - plus a ~30s heartbeat off-TTY. pond status gains a local section (per-adapter sources + pending-sync counts, last sync outcome incl. a surfaced scheduled failure, next scheduled run) and --hosts fleet view; pond init runs the first sync in the foreground and registers the schedule only after it completes, so a fresh timer never races it. (26a7c7a)
• nix: canonical flake shipped in-repo; releases are the single binary host (a4f5e09)

🐛 Bug Fixes

• cli: first-run onboarding polish - the ~500 MB embedding-model download now announces itself before it starts (previously a silent multi-minute "hang"); pond status no longer fuses long adapter names with their counts and reads "semantic ready (brute-force; index builds at scale)" instead of the alarming "below activation threshold"; empty-store search points at pond init rather than blaming filters that were never set; no-adapters states name pond adapters discover; message deltas are labelled "searchable" so the searchable-vs-total gap stops reading as data loss. (43d53f5)
• cli: pond status --format json emits a JSON error document on the store-open failure path instead of empty stdout (matching sync --format json); vector search reads "N nearest messages" with a --mode fts caveat so a gibberish query no longer looks like confident relevance; and pond sql/pond search/pond get error text renders CLI verbs instead of the shared module's MCP tool/resource names. (9423535)

🚜 Refactor

• sync: address PR review findings - close the Ctrl-C window in init schedule registration, reject un-embeddable XDG_STATE_HOME paths, and DRY the shared status/heartbeat helpers (29a4b3a)

Full Changelog: v0.11.2...v0.12.0

v0.11.2

🐛 Bug Fixes

• index: consolidate FTS delta segments by rebuild, never merge (3a46cea)
• index: guard FTS folds against all-null tails; honest pending counts (a7a82f9)
• optimize: make --rebuild reachable when the fold is broken; document claude.ai import (7695f3f)

🚀 Performance

• sync: escalating peek window, parallel peek, skip no-op sessions merge (11d1037)

🧹 Chores

• repo: move gitleaks/release-plz configs + git hooks under .github/, moon-manage hook setup (2323744)

Full Changelog: v0.11.1...v0.11.2

v0.11.1

🚀 Performance

• sync: cut remote sync from ~80-520s to ~44s
• sync: eliminate compaction churn + batch scalar folds + live progress

🚜 Refactor

• bench: rename copy_bench -> write_bench, add write-path profiler

📚 Documentation

• add logo to README

Full Changelog: v0.11.0...v0.11.1

v0.11.0

The write path becomes append-only - incremental index folds, inline embed at ingest, and delta-only copy - and remote pond_search drops from ~8s to sub-second.

🛠 Breaking Changes

• Append-only write path. pond sync now embeds each message inline in its ingest commit, and both sync and copy fold the scalar indexes incrementally (optimize_indices(append)) instead of a full create_index(replace=true) rebuild - so the whole-source-column index rebuild that dominated each sync/copy tail (~520s on the real corpus) is gone. pond copy carries only absent-or-grown sessions and appends a grown session's delta rows through the shared ingest write path (append, not merge-insert), keeping remote copies bandwidth-bound rather than commit-latency-bound.

🚀 Performance

• search: warm remote pond_search drops from ~7.9s to sub-second (best 224ms) on the full S3 corpus (11,788 sessions / 2.14M messages). Two stages did work the query never needed. has_embeddings answered "does this store have embeddings?" with an IsNotNull(vector) scan of the entire vector column (6.8-11.7s per query); it now reads the manifest (index presence, ~0ms) and only falls back to a LIMIT 1 probe when no index exists. Per-hit part summaries fetched file blobs from S3 and scanned parts once per session sequentially; they now skip the blob (the label rides variant_data metadata) and run concurrently. The real retrieval - embed + IVF probe + hydrate - is ~0.1s.
• search (#75): from_date/to_date returned empty on remote stores because the messages_timestamp_zonemap mis-prunes the tz-aware timestamp column (ScalarValue::partial_cmp across the tz mismatch prunes every zone). The index is dropped and date bounds run as a refine over the candidate set. Stores that already built it: run pond optimize --drop-index messages_timestamp_zonemap once, or date filters stay empty there.

📚 Documentation

• add remote read-path cold-start plan and drop stale prewarm comment figures

🧹 Chores

• bench batch/commit sweeps, append-only write-path plan, AIMD hands-off rule
• enforce changelog header taxonomy (pre-commit + moon + CI); backfill 0.10.1/0.10.2

Full Changelog: v0.10.2...v0.11.0

v0.10.2

A follow-up to the 0.10.1 sync work: the embed stage stops scanning wide columns to find its backlog.

🚀 Performance

• embed: the per-sync backlog check no longer scans full columns - model-swap detection is a LIMIT 1 read and the backlog gate is a manifest-only count (idle embed-only 2.24s -> 0.67s).
• embed: the worker's pending scan filters the co-set, ~50x narrower embedding_model column instead of decoding the 1.2 GB Float16 vector column to locate unembedded rows (a whole-table vector decode -> 149 KB).

Full Changelog: v0.10.1...v0.10.2

v0.10.1

A sync performance and correctness release: incremental pond sync no longer re-reads the whole corpus on every run.

🚀 Performance

• Incremental pond sync is dramatically faster. Two compounding fixes to the freshness path:
  • Claude Code appends trailing metadata rows (last-prompt, permission-mode, bridge-session, ...) with no timestamp, so the watermark peek returned None and ~2,000 of ~9,800 sessions never fresh-skipped - re-decoding ~1.18M already-stored rows every sync. The peek now walks back to the last timestamped row. Measured on the real corpus: claude-code import 20.1s -> 1.76s, rows re-decoded 1.18M -> 10.5k, fresh-skips 7,863 -> 9,823.
  • The resident rowmap now delta-extends across embedding's fragment rewrites (keyed on the stable row ids already enabled) instead of rewriting a full ~283 MB base every sync.

🐛 Bug Fixes

• sync: rebuild the rowmap when the base version's manifest was reclaimed by the cleanup retention window, instead of silently re-reading every source on every sync forever.
• build: gate the RLIMIT_NOFILE bump to Unix so the Windows cross-build compiles.
• schedule: gate ScheduleEvery::secs/from_secs to Unix (dead-code on the Windows target).
• ci: point pnpm/action-setup at docs/site/package.json so the docs site deploys.

Full Changelog: v0.10.0...v0.10.1

v0.10.0

This release rebuilds the admin CLI, the retrieval model, and the sync/copy write path - and makes remote-S3 operation dramatically faster.

🛠 Breaking Changes

• cli: retire the overloaded "sources" - [sources.*]/pond sources -> [adapters.*]/pond adapters; config auto-migrates on pond init
• cli: pond storage use is switch-only (a pure pointer flip, copies nothing); data copy moves to the new top-level pond copy; bare pond storage removed (use pond status)
• cli: remove pond export / pond import - snapshot to a .pond archive or .jsonl stream and restore from one is now pond copy --to <file> / pond copy --from <file>
• cli: split maintenance out of sync - pond sync runs import+embed+index by default; the new pond optimize verb runs embed+index on demand, and sync --no-optimize defers to it
• cli: pond sync no longer discovers or auto-enables adapters - enabling is the explicit job of pond adapters/pond init, so a scheduled sync can never grow the adapter set
• cli: verb convergence + flag renames - --config->--config-file, init --schedule->--every, sync --source-dir->--path, sync stage update-indexes->index, --format pretty->text; --storage-path/--config-file are now root-global selectors
• cli: pond copy requires explicit endpoints, adds the @ (configured store) and local keywords; self-verifying with an id-set completeness check (exit 0 SYNCED / exit 6 missing rows)
• storage: first-class pond creds {add,list,delete} for URL-scoped credential sets; pond init captures remote creds inline (masked prompt, never argv)
• search: drop server-side hybrid fusion for single-arm retrieval - mode=vector (default) or mode=fts, plus --sort-by recency
• search: vector index IVF_PQ -> IVF_SQ (drop the refine pass); FTS moves from character-ngram to a word simple tokenizer with English stemming
• sync: durable idempotent-replay sync/copy with a cheap messages-based S3 oracle and sync --verify; resident per-session max_ts watermark replaces the version-resolution oracle
• tools: redesign pond_get/pond_sql_query and unify the transcript renderer
• copy: append fast-path + per-table write plan (absent rows append, grown rows merge)

🎉 New Features

• cli: pond skill prints the bundled agent-onboarding SKILL.md, in lockstep with the binary
• copy: incremental store-to-store copy with no temp staging; streams the source scan straight into the destination
• search: resident per-message meta cache (mmap'd, LSM version-delta refresh) shared across pond processes
• storage: self-verifying migrate and pond storage verify

🚀 Performance

Measured on the real ~2M-message S3 corpus (Hetzner nbg1); baseline = pre-optimization on this branch.

Sync & status:

• per-session staleness oracle: 79s warm / 133s cold -> ~1s / ~4s (messages-based key replaces the versions() per-manifest fetch storm)
• warm re-sync of the full corpus: ~928s -> ~25s (append fast-path + the new oracle)
• status -v: 130s -> ~14s; the stale-embedding count that runs in every default sync: 59.5s -> ~7s

Copy:

• append fast-path vs merge-insert for absent rows: 5.47x faster (13.8 min vs 75.7 min full-corpus; 62 vs 2,685 objects written)
• streaming store-to-store, no temp staging: 1.92x faster (24.1s -> 12.6s, local 500-session set); unchanged-source re-copy ~90 ms

Search:

• FTS arm latency 2043ms -> 2ms p50 (6667ms -> 125ms p95) and ~60 -> 3 object GETs/query via the resident row-key map; per-query S3 bytes -81% (6.0 -> 1.16 MB)
• FTS index 1.14 GB -> 41 MB (28x) and query RAM 2248 -> 379 MB (5.9x) after the word-tokenizer switch; English Success@3 31/111 -> 66/111 (2.1x)
• FTS cold query (full corpus) 76s -> 27s p50 (148.9s -> 48.5s p95); cold server prewarm 175-442s -> ~81s
• vector arm ~393 -> 0-1 object GETs/query after IVF_PQ -> IVF_SQ and dropping the refine pass
• bounded server RAM lowers sql cold 18s -> 5.9s
• resident row-key map: 281.7 MiB for 2.1M messages, built in ~3.8s, removing the per-query hydration scan (and a remote Lance decoder panic)

📚 Documentation

• spec: add the session-movement-complete completeness rule, the session-erasure exception, and micro-batch live-write
• migrate the docs site from mdBook to vocs (pond.locker); correct the search model to single-arm across spec/README/site; refresh SKILL.md for agent ergonomics

🐛 Bug Fixes

• cli: phantom embed backlog, progress-bar wrapping, and verify memory
• sync: restore the per-session staleness watermark from the row version
• build: gate the RLIMIT_NOFILE bump to Unix so the x86_64-pc-windows-gnu release binary builds

Full Changelog: v0.9.0...v0.10.0