Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,12 @@ The wizard prompts (`pond init`, source discovery, etc.) go through cliclack/dia

## Search scope is intentional - do not "fix" it

- `search_text` (the column the FTS index, embeddings, and `pond_search` all use, built in `sessions.rs::search_text`) carries ONLY user- and assistant-role conversational text (plus file metadata). Tool calls, tool results, reasoning, and system/tool-role messages are deliberately excluded. This is a deliberate signal-quality choice, NOT a gap or a bug: that excluded content almost never holds anything findable beyond what the conversational text already says, and indexing it would pollute both results and the caller's context. Never "improve" search by folding tool/reasoning/system content into `search_text`. Tool-body archaeology goes through `parts.variant_data` via `pond_sql_query`.
- Subagent sessions are excluded from default `pond_search` results for the same reason (low unique signal, high pollution); they remain reachable on purpose via `pond_sql_query` (`parent_session_id`). Don't surface them in the main search.
- `search_text` (the column the FTS index, embeddings, and `pond_search` all use, built in `sessions.rs::search_text`) carries ONLY user- and assistant-role conversational text (plus file metadata). Tool calls, tool results, reasoning, and system/tool-role messages are deliberately excluded. This is a deliberate signal-quality choice, NOT a gap or a bug: that excluded content almost never holds anything findable beyond what the conversational text already says, and indexing it would pollute both results and the caller's context. Never "improve" search by folding tool/reasoning/system content into `search_text`. Tool-body archaeology goes through `parts.variant_data` via `pond_sql`.
- Subagent sessions are excluded from default `pond_search` results for the same reason (low unique signal, high pollution); they remain reachable on purpose via `pond_sql` (`parent_session_id`). Don't surface them in the main search.

## MCP surfaces are hard-enforced read-only

- Every pond MCP action is read-only, hard-enforced (durable user constraint) - any new MCP tool that reaches the store must clear the same bar; never expose a write path through MCP. `pond_sql_query` proves it in two layers: (1) a pre-parse gate requiring exactly one `Statement::Query` - this also catches `EXPLAIN ANALYZE` / `DESCRIBE`, which DataFusion's `SQLOptions` alone misses; (2) `sql_with_options` with `allow_ddl` / `allow_dml` / `allow_statements` all false (only a bare `EXPLAIN` of a SELECT flips `allow_statements`). Fresh `SessionContext` per query.
- Every pond MCP action is read-only, hard-enforced (durable user constraint) - any new MCP tool that reaches the store must clear the same bar; never expose a write path through MCP. `pond_sql` proves it in two layers: (1) a pre-parse gate requiring exactly one `Statement::Query` - this also catches `EXPLAIN ANALYZE` / `DESCRIBE`, which DataFusion's `SQLOptions` alone misses; (2) `sql_with_options` with `allow_ddl` / `allow_dml` / `allow_statements` all false (only a bare `EXPLAIN` of a SELECT flips `allow_statements`). Fresh `SessionContext` per query.

## Sync change-detection oracle (S3 perf, measured)

Expand Down Expand Up @@ -83,6 +83,11 @@ The wizard prompts (`pond init`, source discovery, etc.) go through cliclack/dia
## Repo layout

- One flat crate. `src/` holds the `adapter/` module folder alongside top-level files (`handlers.rs`, `sessions.rs`, `substrate.rs`, `transport.rs`, `wire.rs`, `embed.rs`, `config.rs`, `main.rs`, `lib.rs`). Unit tests live in `#[cfg(test)] mod tests` inside the file they test; `tests/` is for cross-module integration only.
- Root `SKILL.md` is compiled into the binary via `include_str!` (`pond skill` prints it; `pond init` installs it to `~/.claude/skills/pond/`), so it MUST stay out of Cargo.toml's `exclude` list - excluding it breaks the crates.io publish build.

## MCP tool routing is deliberate

- Clients (Claude Code, Codex) defer MCP tool descriptions behind tool search; the always-loaded pond-authored surfaces are the server `instructions` (transport.rs `get_info`), the tool NAMES, and the installed skill. The instructions therefore carry all routing: "analyze / review / summarize a session" binds to `pond_get(session_id)`, and `pond_sql` self-describes as the escape hatch (never as the "analytic" tool - that word routes "analyze this session" to SQL). Keep cookbook detail in the `schema://` resources and recovery guidance in error messages (sql.rs `enrich`, the timeout text) - those are the surfaces agents reliably see. Don't fatten tool descriptions back up.

## Documentation

Expand Down
9 changes: 5 additions & 4 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ exclude = [
"moon.yml",
"AGENTS.md",
"CLAUDE.md",
"SKILL.md",
# SKILL.md is NOT excluded: `include_str!` in main.rs/init.rs compiles it
# into the binary, so the published package must carry it.
"flake.nix",
"nix/",
"**/.DS_Store",
Expand Down Expand Up @@ -59,7 +60,7 @@ anyhow = "1.0.102"
# Lock-free read of the loaded `RowKeyMap` on the hot search path; replaced
# atomically when `ensure_rowmap` rebuilds. Already in the tree via lance.
arc-swap = "1"
# ndjson export writer for `pond_sql_query` (already in the tree via the arrow
# ndjson export writer for `pond_sql` (already in the tree via the arrow
# stack; named directly here for the `LineDelimitedWriter`).
arrow-json = "58.3"
# Already in the tree via the arrow stack; named directly for
Expand Down Expand Up @@ -111,7 +112,7 @@ hf-hub = { version = "0.5", default-features = false, features = ["ureq"] }
indicatif = "0.18"
lance = { version = "8.0.0", features = ["protoc"] }
lance-arrow = "8.0.0"
# `pond_sql_query` registers lance's JSON / contains_tokens UDFs on the SQL
# `pond_sql` registers lance's JSON / contains_tokens UDFs on the SQL
# SessionContext via `lance_datafusion::udf::register_functions`. Already in
# the tree transitively; named here to reach that one function.
lance-datafusion = "8.0.0"
Expand All @@ -136,7 +137,7 @@ memmap2 = "0.9"
# drive a conditional put (`PutMode::Create`) through `ObjectStore::inner` and
# classify the typed errors (AlreadyExists / Unauthenticated / NotImplemented).
object_store = { version = "0.13", default-features = false }
# Parquet export writer for `pond_sql_query`. The only genuinely new compiled
# Parquet export writer for `pond_sql`. The only genuinely new compiled
# crate the SQL tool adds (lance reads/writes its own format, not parquet).
# `arrow` feature only; results are written uncompressed via `ArrowWriter`.
parquet = { version = "58.3", default-features = false, features = ["arrow"] }
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ Install, run guided setup, and ingest your local sessions:

```sh
brew install tenequm/tap/pond
pond init # guided setup: storage, adapters, MCP registration, optional schedule
pond init # guided setup: storage, adapters, MCP + agent skill, optional schedule
pond sync # ingest, embed, update indexes - every enabled adapter
```

`pond init` registers pond as an MCP server for detected clients (by hand: `claude mcp add -s user pond -- pond mcp`, `codex mcp add pond -- pond mcp`). Then ask your agent - real prompts from daily use:
`pond init` registers pond as an MCP server for detected clients and installs the bundled pond skill for Claude Code (by hand: `claude mcp add -s user pond -- pond mcp`, `codex mcp add pond -- pond mcp`; skill: save `pond skill` output to `~/.claude/skills/pond/SKILL.md`). Then ask your agent - real prompts from daily use:

```
check in pond how we solved this before, then apply the same fix here
Expand Down Expand Up @@ -123,7 +123,7 @@ pond copy --from snapshot.pond --to local

### Read-only SQL

Ask structured questions with read-only SQL (the same surface as the `pond_sql_query` MCP tool):
Ask structured questions with read-only SQL (the same surface as the `pond_sql` MCP tool):

```sh
pond sql "SELECT project, count(*) FROM messages GROUP BY project ORDER BY 2 DESC"
Expand Down Expand Up @@ -193,7 +193,7 @@ The full contract is in [`docs/spec.md`](docs/spec.md). Key choices:
- **Index lifecycle decoupled from writes.** Writes commit data (embeddings included, computed inline at ingest) without folding the search indexes. `pond sync` runs index maintenance by default, and `pond optimize --only index` runs it on demand; Lance merges index results with a flat scan over unindexed fragments, so reads stay correct.
- **Single-arm retrieval.** Each query runs one retriever - `vector` (cosine, with a gentle recency tiebreaker) or `fts` (BM25) - chosen per query; no server-side fusion. The vector arm falls back to full-text when the store has no embeddings, and `--sort-by recency` returns newest-first. Results group to one summary per session, keyed on `session_root`.
- **Language-neutral full-text.** Word-level `simple` tokenizer with English stemming (ascii-folding on); tokens the stemmer does not recognize pass through unchanged and stay exact-matchable, so pond indexes sessions in any language alike.
- **Two transports, one handler set.** HTTP+JSON (axum) and MCP (rmcp) both dispatch into the same handlers. Wire ops: `pond_search`, `pond_get`, `pond_ingest`. MCP additionally exposes the read-only `pond_sql_query` tool and the `schema://pond`, `schema://pond-sql`, and `stats://pond` resources.
- **Two transports, one handler set.** HTTP+JSON (axum) and MCP (rmcp) both dispatch into the same handlers. Wire ops: `pond_search`, `pond_get`, `pond_ingest`. MCP additionally exposes the read-only `pond_sql` tool and the `schema://pond`, `schema://pond-sql`, and `stats://pond` resources.
- **Opaque-string multi-tenancy.** Each tenant is a `namespace` string the integrator supplies; pond does not authenticate, authorize, or model identity. The object store's IAM is the storage boundary.
- **Encryption is operational.** Bucket SSE plus filesystem encryption; pond holds no keys and adds no application-level crypto.

Expand Down
46 changes: 31 additions & 15 deletions SKILL.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,49 @@
---
name: pond
description: Recall past AI agent sessions (Claude Code, Codex, opencode, and more) - lossless storage with semantic and full-text search, served over MCP.
description: Recall and analyze past AI agent sessions (Claude Code, Codex, opencode, and more). Find prior work and decisions, read/review/summarize a past session transcript, or run SQL analytics over session history. Use whenever the user references past sessions, prior work, "check pond", or asks what was done or decided before.
---

# pond

pond is your memory across every AI coding session you have run - stored
losslessly, searchable. If a task needs context you lack, pond likely has it:
recall first, then answer. Before you say "I don't know" or re-derive something
that sounds prior, search pond.
losslessly, searchable over MCP. If a task needs context you lack, pond likely
has it: recall first, then answer. Before you say "I don't know" or re-derive
something that sounds prior, search pond.

## Which tool

- Past work, by meaning ("what did we decide", "last week", "that other repo")
-> `pond_search` (`mode=vector` default; `mode=fts` for exact words/symbols).
- A known session or message -> `pond_get` (transcript, or one message in context).
- Counts, filters, trends across sessions -> `pond_sql_query` (read-only SQL over
`sessions` / `messages` / `parts`).
- Find past work by meaning ("what did we decide", "have we hit this before")
-> `pond_search` (`mode=vector` default; `mode=fts` for exact whole words).
- Read, analyze, review, or summarize a session -> `pond_get(session_id)` -
one call, full readable transcript. `pond_get(message_id)` expands one
message with its full tool bodies.
- Corpus-wide aggregation, exact strings inside tool bodies, subagent
sessions, bulk export -> `pond_sql` (read-only SQL). Read resource
`schema://pond-sql` first - do not guess columns or JSON paths.

Params and response shapes live in each tool's MCP description and the
`schema://pond`, `schema://pond-sql`, `stats://pond` resources - read them at
call time, don't guess.
## Rules that prevent wrong conclusions

- Long sessions supersede their own early conclusions. For "what did we
decide / latest state", read the end (`pond_get(session_id,
session_from="end")`) or `pond_search` with `sort_by=recency` - relevance
rank favors the early, confident, possibly overturned phrasing.
- Search covers only user/assistant conversational text - tool output is
excluded by design. A weak search result is NOT proof of absence: verify
exact strings with `pond_sql` `contains_tokens(search_text, '...')` before
concluding something never happened.
- Tool bodies in SQL: tool_call is `{call_id, name, params}` (a Bash command
is `json_extract(variant_data, '$.params.command')`); tool_result is
`{call_id, name, is_failure, result}`.
- On a remote store, SQL over `parts` costs seconds per round-trip: scope by
`session_id` / `tool_name`, and raise `timeout_seconds` when a broad scan
is genuinely needed.

## Setup

`brew install tenequm/tap/pond` (or `cargo binstall pond-db`, or `nix profile
add github:tenequm/pond#pond`), then `pond init`, then `claude mcp add -s
user pond -- pond mcp`. Keep current with `pond sync`; `pond --help` for the rest.
Claude.ai chats are not synced automatically - request a data export
add github:tenequm/pond#pond`), then `pond init` - it registers the MCP server
and installs this skill. Keep current with `pond sync`; `pond --help` for the
rest. Claude.ai chats are not synced automatically - request a data export
(claude.ai Settings -> Privacy -> Export data, arrives as an emailed `.zip`),
then `pond sync claude-ai-export --path <export.zip>`.
Docs: https://pond.locker/
6 changes: 3 additions & 3 deletions benches/serve_mem_bench.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
//! - vector_first : first vector query (the cold E5 model-load spike)
//! - vector_steady : N vector-arm queries (default arm; needs the embedder)
//! - get_steady : N `pond_get` hydration calls on prior hits
//! - sql_steady : N `pond_sql_query` calls (the analytic tool)
//! - sql_steady : N `pond_sql` calls (the analytic tool)
//! - idle/drained : resting footprint; drained drops the model (cache stays)
//!
//! Run:
Expand Down Expand Up @@ -85,7 +85,7 @@ const QUERIES: &[&str] = &[
"schema evolution add column",
];

/// `pond_sql_query` workload: one metadata-only count (manifest, no data read),
/// `pond_sql` workload: one metadata-only count (manifest, no data read),
/// two column scans, a token filter (FTS-accelerated), and a group-by. Mirrors
/// the analytic shapes the MCP tool actually serves.
const SQL_QUERIES: &[&str] = &[
Expand Down Expand Up @@ -1201,7 +1201,7 @@ async fn main() -> Result<()> {
.map(|s| phases.push(s))?;
}

// ---- Phase: sql_steady (the pond_sql_query analytic tool) ----
// ---- Phase: sql_steady (the pond_sql analytic tool) ----
run_sql_phase("sql_steady", &store, &sampler, SQL_QUERIES)
.await
.map(|s| phases.push(s))?;
Expand Down
2 changes: 1 addition & 1 deletion benches/sync_oracle_bench.rs
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ async fn fetch_tables(store: &Store) -> Result<Tables> {
}

/// Run a SQL query through pond's own `sql::run` path - so the bench exercises
/// the exact DataFusion + Lance pushdown wiring the MCP `pond_sql_query` tool
/// the exact DataFusion + Lance pushdown wiring the MCP `pond_sql` tool
/// uses. Returns the row count from the result; the inline rendering cost is
/// kept tiny by passing `inline_rows = 0`.
async fn run_sql(tables: &Tables, sql: &str) -> Result<usize> {
Expand Down
10 changes: 5 additions & 5 deletions docs/site/src/pages/get-started/connect-your-agents.mdx
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
# Connect your agents

pond is an MCP server. `pond init` registers it for detected clients. To add it by hand:
pond is an MCP server. `pond init` registers it for detected clients and installs the bundled pond skill into Claude Code's user skills dir. To add the server by hand:

```sh
claude mcp add -s user pond -- pond mcp # Claude Code
codex mcp add pond -- pond mcp # Codex
```

Other clients: use `pond mcp` (alias for `pond serve --transport stdio`) as the server command.
Other clients: use `pond mcp` (alias for `pond serve --transport stdio`) as the server command. The skill installs by hand too: save `pond skill` output to `~/.claude/skills/pond/SKILL.md`.

## Tools the agent gets

- `pond_search` - search across all sessions (vector or full-text)
- `pond_get` - fetch a session or message by id
- `pond_sql_query` - read-only SQL over the corpus
- `pond_search` - find relevant messages across all sessions (vector or full-text)
- `pond_get` - read, analyze, or summarize a session; or one message with its full tool bodies
- `pond_sql` - the escape hatch: read-only SQL over the corpus

## Agent setup prompt

Expand Down
2 changes: 1 addition & 1 deletion docs/site/src/pages/reference/cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ pond <command> --help # full flags for any command
```

## Setup
- `pond init` - guided setup-and-repair: storage, adapters, MCP, schedule
- `pond init` - guided setup-and-repair: storage, adapters, MCP + agent skill, schedule
- `pond adapters list|discover|enable|disable` - manage which adapters sync ingests
- `pond storage check|use` - probe a destination; switch the active store
- `pond creds add|list|delete` - manage URL-scoped credential sets
Expand Down
Loading