feat(proxy): three operating modes — sync, async (cache-safe deferred compaction), and observe (measure without enforcing)

[OsherElhadad]

Problem statement

Compaction is unconditionally synchronous: proxy/proxy.go:364-373 calls apply.BodyFull inline before forwarding, and the caller waits. Measured cost on Terminal-Bench: ~450 ms/req added, ~1,592 s cumulative on one arm, almost all of it extract_llm's model call.

That forces two bad choices on a user. To get savings they must accept latency on every request. And to find out whether context-guru helps their workload, they must enforce it in production and compare against history — there is no way to evaluate it safely first.

Motivation

• Latency. Synchronous compaction is the right default for correctness but the wrong default for a long-lived agent session, where a decision computed once is replayed for many turns. There is no reason the first turn must pay for a compaction that benefits turn 5 onward.
• Adoption. "Run it and see what it would have saved, without touching my requests" is the single most useful thing a compaction proxy can offer, and neither reference implementation has it: headroom has no observe/shadow/dry-run mode at all (its token vs cache modes are both enforcing; the only control arm is a 10% output-shaper holdout). This is a genuine differentiator, not a copy.
• Evidence. Every conclusion in docs/results/ came from running full paired benchmark arms. Observe mode makes that comparison available to any user on their own traffic.

Desired behavior — three modes

Mode 1 — sync (current behavior, remains the default)

Before an LLM request: context-guru compacts, the caller waits, the compacted request is sent. Prioritizes immediate savings; adds compaction latency to the request path.

Mode 2 — async

Compaction happens asynchronously. The agent continues making LLM calls while context-guru computes a new compacted/frozen state; once ready, subsequent turns use it.

The cache requirement is the hard part. By default, while waiting for async compaction, the newest un-compacted steps must not be provider-cached in a way that later forces us to rewrite a cached suffix once the compaction arrives. Design this against real Anthropic prompt-caching semantics: a cache-write costs 11.5× a cache-read (($2.50 − $0.20)/$0.20), so a naive async implementation that lets the uncompacted tail get cached and then replaces it converts a read into a write and is strictly worse than sync. That failure mode is not hypothetical — it is precisely what tripled headroom's cache-write on Terminal-Bench (12.37M vs baseline 4.01M) by rewriting the live zone.

Configuration must allow changing this, but the safe default protects cache-write economics.

Design for: stable prefix · mutable tail · cache breakpoints · compaction generation/version · stale async results arriving after newer turns · newer turns arriving while compaction runs · cancellation and coalescing · at most one useful compaction job per session/version · race conditions · session isolation.

Prior art worth studying rather than reinventing: headroom's BackgroundCompressor is a single bounded asyncio.Queue + one drain task, dedup-by-key with the pending slot claimed before the job becomes observable (so dedup is atomic against a concurrent enqueue), no request-coupled deadline, fail-open on every path ("lost savings, never lost correctness"), and a stats() exposing {queued, pending, processed, dropped, errors}. Its instructive weakness: the dashboard shows only queued, so dropped — the "we silently gave up savings" counter — is invisible. Expose the whole tuple.

Mode 3 — observe

Context-guru asynchronously computes what it would have done and never applies it. The agent receives the untouched original conversation.

Reports: potential tokens saved · potential dollar savings · potential per-component contributions · hypothetical cache implications · hypothetical context-guru overhead · actual baseline usage · projected optimized usage.

The dashboard must make it unmistakable: "You are currently in observe mode. context-guru did not modify requests. Here is what it could have saved."

This requires careful metric semantics so observed/hypothetical numbers can never be confused with actual enforced savings. That is the primary correctness risk of this issue — a mis-labeled hypothetical number is worse than no number.

Current behavior / relevant code locations

• proxy/proxy.go:353-383 — the synchronous block: panic-recover backstop, apply.BodyFull, RecordAddedLatency, expand.Inject, then h.serve.
• apply/apply.go:115-126 — top-level fail-open recover; :127-173 — messages parse, session resolve (session.Resolve), resolveCacheAware, maxCachedIdx = prevLen(st, sessionID) - 1 (:161) and the deferred putLen (:162); :164-173 Ctx construction. putLen running in a defer is a direct async hazard: two concurrent turns of one session would race on the per-session length.
• components/component.go:101-135 — Ctx.CacheAware, MaxCachedIdx, TailOnly (note MaxCachedIdx < 0 ⇒ TailOnly true for all indices, i.e. fail-open into mutating the cached prefix — coordinate with fix(cache): freeze-store lifetime — sliding TTL, longer default, and no cache-destructive regression on a store miss #25).
• components/offload/state.go:47-81 — freeze/reapplyFrozen: the existing "compute once, replay byte-identically" mechanism, which is the natural substrate for async generations.
• store/store.go — TTL/LRU; generation state lives here (fix(cache): freeze-store lifetime — sliding TTL, longer default, and no cache-destructive regression on a store miss #25 governs its lifetime).
• metrics/metrics.go:70 — one mutex-guarded Aggregator; :196-221 Snapshot. Mode must become a dimension here.
• proxy/proxy.go:68-72 — Options.InjectExpand, the precedent for a tri-state option.
• session/ — session identity.

Proposed architecture

1. Mode as an explicit Options field threaded onto Ctx, defaulting to sync. Never inferred.
2. A per-session compaction generation. A request records the generation it was built from. An async result that targets a stale generation is discarded, not applied — this is the single most important invariant, and it must be tested with -race.
3. One worker pool, bounded, owned by the proxy — not per-request goroutines. Enqueue is dedup-by-(session, generation) with the pending slot claimed before the job is observable. Bounded queue; drop rather than block; count drops. Cancel on session end; coalesce supersessions.
4. Async cache policy. Until a compaction lands, the uncompacted tail must be excluded from the cached region — i.e. do not place a breakpoint at or beyond the not-yet-compacted tail, so the eventual replacement rewrites nothing the provider committed to. cacheinject already owns breakpoint placement (components/reformat/cacheinject.go), and apply/prefixsplit.go shows the body-level precedent for this kind of surgery. Make the policy explicit and configurable, with the cache-safe choice as default.
5. Observe mode runs the same pipeline on a copy, discards the output, and records results into a strictly separate metric namespace. The enforced path must be provably untouched: assert byte-identity of the forwarded body in a test.
6. Metrics carry the mode: at minimum sync_enforced, async_enforced, observe_hypothetical, distinct in the payload and filterable in the dashboard.

Alternatives considered

• Async via unmanaged goroutines per request. Rejected explicitly by the requirement, and rightly: goroutine leaks, stale writes over newer generations, no cancellation, unbounded fan-out under load.
• Observe mode as a config flag on the existing sync path (compute, log, discard, still synchronous). Much simpler and worth considering as a first increment since it needs none of the async machinery — but it keeps the latency it was meant to avoid, so it is a stepping stone, not the destination.
• Sampling instead of observe (compact 10% of traffic). Cheaper but answers a different question and perturbs real requests; the output-shaper holdout in headroom shows the pattern's limits.
• A separate replay tool over captured traffic. Zero production risk, and CONTEXT_GURU_CAPTURE already supports offline replay — but it cannot report live projected savings, which is the point.

Observability implications

Mode-tagged everything. Observe mode needs its own vocabulary in the payload (potential_* / projected_*), never sharing a key with an enforced metric. Async needs {queued, pending, processed, dropped, errors, stale_discarded} and a "savings realized on turn N+k" figure so deferred value is attributable rather than invisible. Dashboard (#30) must filter and compare by mode and display the observe-mode banner.

Storage / data-model implications

Per-session generation counters and pending-job state, bounded and session-isolated. Observe-mode results must be stored in a way that cannot be summed into enforced totals — separate columns or an explicit mode column that every aggregate query filters on. Getting this wrong silently inflates the product's headline claim, so it is a correctness requirement, not a nicety.

Backward compatibility

Default sync reproduces today's behavior exactly, byte for byte — assert this with a golden test. /stats keeps its current shape (the harnesses in deploy/harbor/*.py parse it); new mode fields are additive. Existing configs unchanged.

Configuration design

mode: sync            # sync | async | observe
async:
  cache_uncompacted_tail: false   # safe default: protect cache-write economics
  max_queue: 256

One option per real decision. The cache-policy escape hatch exists because a non-caching backend genuinely does not need the protection.

Testing plan

• go test -race throughout — this is a concurrency issue and race tests are mandatory, not optional.
• Unit: sync output byte-identical to pre-change (golden).
• Unit: observe mode leaves the forwarded body byte-identical while still recording hypothetical savings.
• Unit: an async result for a stale generation is discarded, never applied.
• Unit: two concurrent turns of one session do not corrupt prevLen/generation state (today's deferred putLen is the hazard).
• Unit: enqueue dedup is atomic against a concurrent enqueue of the same key.
• Unit: a full queue drops and counts, and never blocks the request path.
• Unit: worker cancellation on session end; no goroutine leak (goleak-style assertion or an explicit count).
• Unit: with cache_uncompacted_tail: false, no breakpoint is placed at/beyond the un-compacted tail.
• Unit: observe-mode metrics cannot be summed into enforced totals (aggregate query test).

Real-world benchmark plan

2–3 SWE-bench and 2–3 Terminal-Bench tasks per mode (9–18 trials), plus a real Claude Code session in each mode. Record per trial: commit SHA, mode, config, model, task id, fresh/cache-read/cache-write/output tokens, total billed cost, context-guru's component cost, steps, reward, latency (added latency per mode is the headline for async), expands, reverts, savings, and the async queue counters.

Specific questions to answer with numbers:

• Does async actually reduce added latency without increasing cache-write? (If cache-write rises, the cache policy is wrong.)
• Does async reach the same steady-state savings as sync, just later?
• Does observe add measurable latency to the enforced path? (It must not.)
• Do observe-mode projected savings match what sync actually achieved on the same tasks? That agreement is the validation of the whole mode — report the discrepancy honestly.

Acceptance criteria

• Three modes implemented; sync is the default and byte-identical to today (golden test).
• Async uses a bounded, owned worker pool with generation-versioned results; stale results discarded; dedup/coalescing atomic; cancellation clean; no goroutine leaks.
• Async's default cache policy provably avoids rewriting a cached suffix; measured cache-write does not exceed sync.
• Observe mode never modifies a request (byte-identity test) and reports the full projected set.
• Observe/hypothetical metrics are namespace-separated and cannot be aggregated into enforced savings.
• Mode is a filterable dimension in /stats and the dashboard; the observe-mode banner is unmistakable.
• Async queue counters exposed in full, including dropped and stale_discarded.
• go test -race green across proxy, apply, components, store.
• Benchmarked in all three modes on real tasks; observe-mode projections compared against sync's actuals, with discrepancies reported.

Documentation updates

New docs/how-to/operating-modes.md (when to use each, the async cache trade-off, how to read observe-mode numbers), docs/reference/config.md, docs/design.md (generations, job lifecycle, cache policy, fail-open behavior per mode), docs/dashboard.md (mode filtering + observe banner), mkdocs.yml, README.

Dependencies

Interacts with #25 (frozen-state lifetime is the substrate for async generations; MaxCachedIdx fail-open must be resolved or async inherits it). #27's xdedup must be lifecycle-compatible with async/observe. #30 must render all three modes with distinct semantics — coordinate the metric namespace early. Consider landing observe mode first: it is the smaller, lower-risk half and delivers the adoption story without the concurrency surface.

[OsherElhadad]

Reopening: observe mode landed in #43, but async mode did not. #35 is deliberately held as a draft pending a measurement, not a code fix.

Why async is held: its headline latency win (1,599 ms → 25 ms) came almost entirely from deferring extract_llm's model call. #34 makes extract_llm hard-decline on caching backends, so on the primary workload there is nothing expensive left to defer — measured directly on Terminal-Bench, async 26.8 ms vs sync 26.9 ms with zero LLM calls. Its cache-write evidence was also withdrawn after review found the tail-protection only stripped context-guru's own breakpoints and never the agent's, so lower cache-write came from writing fewer breakpoints rather than from the policy working.

All eleven review fixes (S1–S11) remain on #35, which is the ready-to-measure branch. What would unblock it: a 50-task paired arm with strip_caller_breakpoints: true showing cache-write no higher than sync's and a real latency reduction on traffic that still makes model calls.

This issue stays open to track async. The observe half is complete and merged.