Skip to content

Commit 6645f97

Browse files
kevincodex1claude
andauthored
perf(agent): cheap compaction — dedicated summarizer model, input cap… (#532)
* perf(agent): cheap compaction — dedicated summarizer model, input caps, free reactive prune, window re-derive Compaction summarization was the single most expensive recurring event in a long run: the full elided middle — verbatim tool bodies included — went to the session's main model at full price (a ~100k-token call near a 200k window, recursively multiplied on overflow). Four changes, each independently safe: - Dedicated summarizer model. Compaction summaries route to a cheap model resolved as ZERO_COMPACTION_MODEL > preferences.compactionModel > a curated default on official endpoints (Haiku 4.5 / Gemini Flash-Lite / gpt-4.1-mini; custom-endpoint catalogs are unknowable so they get none). Built lazily on the first paid compaction; ANY failure permanently falls back to the main provider for the run, so a misconfigured summarizer can never break compaction. "main" at any level forces the old behavior. - Summarizer input caps. renderTranscript clamps tool-result bodies to 2,000 chars and tool-call arguments to 500 (head+tail, rune-safe) — the summary needs what was done and decided, not raw payloads. Cuts summarizer input 5-10x on tool-heavy sessions. - Free reactive prune. recover() now runs the same prune-first stage as the proactive path: a context-limit error is first answered by pruning stale tool bodies at zero cost, without consuming the one-shot reactive budget; the paid summarizer only runs when a later error finds nothing left to prune. - Window re-derivation on model switch. Options.ContextWindowFor (wired to the model registry in exec and the TUI) lets the loop re-derive the compaction threshold after escalate_model, closing the documented KNOWN LIMITATION where a switch to a smaller-window model could overflow and a larger one over-compacted. ## Rebase notes (2026-08-25, onto 2749447) Rebased across 214 commits. main now summarizes through the context planner (#879) and switches models via turn sessions, so the agent side was reworked rather than merged textually: - Summarizer selection lives on compactionState and is resolved lazily per main model through summarizeWithPlanner (planner + trace preserved); a failing dedicated summarizer is marked broken and the same slice is retried once on the main provider. - Options.Summarizer now receives the main model in force: `func(ctx, mainModelID string) (Provider, error)`, and may return (nil, nil) when no dedicated model applies. After a successful escalate_model switch the loop calls compactor.switchModel, which re-derives the window and re-resolves the summarizer — a run that started on the cheap model and escalated regains cheap summaries (review P2). - recover()'s free prune stage runs before the one-shot reactive guard (pruning is free and cannot loop) and mirrors maybeCompact's two prune passes; the free retry is taken only when the pruned history fits the window the provider enforces (with a 5% estimate margin), otherwise the same call falls through to the paid summarizer. Gating on the 70% proactive threshold would have sent histories in the 70-95% band to the summarizer unnecessarily (review P1 and the follow-up question). - ContextWindowFor is wired through the turn-session switch path; the planner's window is updated alongside the compactor's. Tests: the PR's suite is ported to the new seams, plus insufficient-prune fall-through, free prune after the budget is spent, window-vs-threshold gating, and summarizer re-resolution across a model switch. * fix(agent): branch on a dedicated-summarizer flag instead of comparing providers Review follow-ups for #532: - summarizeProvider now reports whether the dedicated summarizer was selected and summarizeClosure branches on that flag; the previous `chosen == provider` compared Provider interface values, which panics when the dynamic type is not comparable. Regression test uses a provider with a slice field for both the summarizer and the main provider. - CompactionModelID's curated-default test covers the official OpenAI profile alongside Anthropic and Google. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012JqvAf7zsP1qYFVZayKXHq --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 2749447 commit 6645f97

12 files changed

Lines changed: 966 additions & 42 deletions

File tree

‎internal/agent/compaction.go‎

Lines changed: 172 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,9 @@ import (
44
"context"
55
"encoding/json"
66
"errors"
7+
"strconv"
78
"strings"
9+
"unicode/utf8"
810

911
"github.com/Gitlawb/zero/internal/trace"
1012
"github.com/Gitlawb/zero/internal/zeroruntime"
@@ -390,6 +392,27 @@ type compactionState struct {
390392
// starts at 1.0 and converges via an EMA as each turn reports actual usage, so
391393
// later turns compact nearer to real capacity. Zero is treated as 1.0.
392394
calibrationRatio float64
395+
396+
// summarizerFactory lazily builds the dedicated (cheap) summarizer provider
397+
// (Options.Summarizer) for the current main model. summarizerProvider
398+
// memoizes the built provider and summarizerResolved records that the
399+
// factory ran (it may legitimately return no provider: the main model is
400+
// already the cheap one). summarizerBroken is the sticky fallback: once a
401+
// build or call fails, the run stays on the main provider until the next
402+
// model switch — a misconfigured cheap model must never break (or
403+
// repeatedly delay) compaction.
404+
summarizerFactory func(ctx context.Context, mainModelID string) (Provider, error)
405+
summarizerProvider Provider
406+
summarizerResolved bool
407+
summarizerBroken bool
408+
// window is the context window the threshold was derived from. The
409+
// reactive free-prune stage compares against it (the limit the provider
410+
// enforces), not against the 70% proactive threshold — a pruned history
411+
// between the two would still succeed on retry.
412+
window int
413+
// mainModel is the model the run currently streams on; the summarizer
414+
// choice is made against it and remade after a mid-run switch.
415+
mainModel string
393416
}
394417

395418
// calibrate folds one turn's (rawEstimate, actualPromptTokens) sample into the
@@ -424,17 +447,88 @@ func (state *compactionState) calibratedTokens(raw int) int {
424447

425448
func newCompactionState(options Options, task *taskState) *compactionState {
426449
state := &compactionState{
427-
enabled: options.ContextWindow > 0,
428-
threshold: compactionThreshold(options.ContextWindow),
429-
preserveLast: options.CompactionPreserveLast,
430-
onUsage: options.OnUsage,
431-
task: task,
432-
planner: newContextPlanner(contextPlannerConfig{contextWindow: options.ContextWindow, serviceTier: options.ServiceTier}),
433-
trace: options.Trace,
450+
enabled: options.ContextWindow > 0,
451+
threshold: compactionThreshold(options.ContextWindow),
452+
window: options.ContextWindow,
453+
mainModel: options.Model,
454+
preserveLast: options.CompactionPreserveLast,
455+
onUsage: options.OnUsage,
456+
task: task,
457+
planner: newContextPlanner(contextPlannerConfig{contextWindow: options.ContextWindow, serviceTier: options.ServiceTier}),
458+
trace: options.Trace,
459+
summarizerFactory: options.Summarizer,
434460
}
435461
return state
436462
}
437463

464+
// updateWindow re-derives the compaction threshold after a mid-run model
465+
// switch. The low-water mark is reset: it was measured against the old
466+
// threshold, and keeping it could suppress a compaction the smaller window
467+
// now requires. <= 0 (unknown model) leaves everything unchanged.
468+
func (state *compactionState) updateWindow(window int) {
469+
if window <= 0 {
470+
return
471+
}
472+
state.enabled = true
473+
state.window = window
474+
state.threshold = compactionThreshold(window)
475+
state.lowWaterMark = 0
476+
if state.planner != nil {
477+
state.planner.config.contextWindow = window
478+
}
479+
}
480+
481+
// switchModel records a mid-run model switch: the window is re-derived
482+
// (window <= 0 keeps the old one) and the summarizer choice is forgotten so
483+
// the next compaction re-resolves it against the new main model. A run that
484+
// started on the cheap model (no dedicated summarizer) and escalated to an
485+
// expensive one therefore regains cheap summaries — the exact workflow a
486+
// dedicated summarizer exists for.
487+
func (state *compactionState) switchModel(modelID string, window int) {
488+
state.mainModel = modelID
489+
state.updateWindow(window)
490+
state.summarizerProvider = nil
491+
state.summarizerResolved = false
492+
state.summarizerBroken = false
493+
}
494+
495+
// summarizeProvider returns the provider summarization calls should use and
496+
// whether it is the dedicated summarizer: that one when it applies to the
497+
// current main model and is healthy, else the main provider. The factory runs
498+
// at most once per main model; a build error marks the summarizer broken for
499+
// that model. Callers branch on the boolean rather than comparing Provider
500+
// values — interface comparison panics on a non-comparable dynamic type.
501+
func (state *compactionState) summarizeProvider(ctx context.Context, main Provider) (Provider, bool) {
502+
if state.summarizerFactory == nil || state.summarizerBroken {
503+
return main, false
504+
}
505+
if !state.summarizerResolved {
506+
state.summarizerResolved = true
507+
built, err := state.summarizerFactory(ctx, state.mainModel)
508+
if err != nil {
509+
state.summarizerBroken = true
510+
return main, false
511+
}
512+
state.summarizerProvider = built
513+
}
514+
if state.summarizerProvider == nil {
515+
return main, false
516+
}
517+
return state.summarizerProvider, true
518+
}
519+
520+
// fitsWindow reports whether a calibrated size is safely under the limit the
521+
// provider enforces. The proactive threshold (70% of the window) is a trigger
522+
// for the paid summarizer, not the hard limit; the reactive path compares
523+
// against the window itself, with a small margin for estimate error, so a
524+
// history pruned to, say, 85% of the window takes the free retry.
525+
func (state *compactionState) fitsWindow(size int) bool {
526+
if state.window <= 0 {
527+
return size <= state.threshold
528+
}
529+
return size <= state.window-state.window/20
530+
}
531+
438532
// maybeCompact runs proactive compaction at the top of a turn. It returns the
439533
// (possibly compacted) message slice. It is a no-op when compaction is disabled,
440534
// when the history is under threshold, or when the history has not grown past
@@ -528,10 +622,34 @@ func (state *compactionState) recover(
528622
// context-limit error never triggers an unexpected summarization call.
529623
return messages, false, nil
530624
}
531-
if state.reactiveAttempted {
625+
if !isContextLimitError(errorMessage) {
532626
return messages, false, nil
533627
}
534-
if !isContextLimitError(errorMessage) {
628+
629+
// CHEAP FIRST STAGE (mirrors maybeCompact): a context-limit failure can
630+
// often be fixed for free by dropping superseded reads and pruning old
631+
// tool-result bodies. It runs regardless of the one-shot reactive budget
632+
// (it costs nothing and cannot loop: a second pass finds nothing left to
633+
// prune). The free retry is taken only when the pruned history fits the
634+
// window the provider enforces — otherwise a retry would fail the same
635+
// way, so fall through to the paid summarizer in this same call.
636+
toolTokens := estimateToolDefTokens(tools)
637+
prunedAny := false
638+
if pruned, reclaimed := pruneSupersededReadResults(messages); reclaimed > 0 {
639+
messages, prunedAny = pruned, true
640+
}
641+
if pruned, reclaimed := pruneStaleToolOutput(messages, state.preserveLast); reclaimed > 0 {
642+
messages, prunedAny = pruned, true
643+
}
644+
if prunedAny {
645+
size := state.calibratedTokens(estimateTokens(messages) + toolTokens)
646+
if state.fitsWindow(size) {
647+
state.lowWaterMark = size
648+
return messages, true, nil
649+
}
650+
}
651+
652+
if state.reactiveAttempted {
535653
return messages, false, nil
536654
}
537655

@@ -571,11 +689,21 @@ func (state *compactionState) recover(
571689
// provider call. The summary stream intentionally does NOT forward OnText (so
572690
// compaction stays invisible on the user-facing surface), but it DOES forward
573691
// OnUsage so the summarizer's token cost is still counted by usage/budgeting.
692+
// The call goes to the dedicated (cheap) summarizer when one applies; if that
693+
// call fails the summarizer is marked broken for the current model and the
694+
// same slice is retried once on the main provider, so the caller sees at most
695+
// one (main-provider-quality) error.
574696
func (state *compactionState) summarizeClosure(ctx context.Context, provider Provider) func([]zeroruntime.Message) (string, error) {
575697
if state.planner == nil {
576698
state.planner = newContextPlanner(contextPlannerConfig{})
577699
}
578700
return func(toSummarize []zeroruntime.Message) (string, error) {
701+
chosen, dedicated := state.summarizeProvider(ctx, provider)
702+
summary, err := summarizeWithPlanner(ctx, chosen, toSummarize, state.onUsage, state.planner, state.trace)
703+
if err == nil || !dedicated {
704+
return summary, err
705+
}
706+
state.summarizerBroken = true
579707
return summarizeWithPlanner(ctx, provider, toSummarize, state.onUsage, state.planner, state.trace)
580708
}
581709
}
@@ -659,8 +787,41 @@ func summarizeMessagesOnce(ctx context.Context, provider Provider, messages []ze
659787
return summary, nil
660788
}
661789

790+
// summaryToolResultClamp bounds each tool-result body in the summarizer input.
791+
// Tool outputs dominate the elided middle's bulk (build logs, file reads,
792+
// search results), yet the summary instructions only need their gist — the
793+
// model that ACTED on the full output already turned it into decisions the
794+
// surrounding messages record. 2,000 chars (~500 tokens) keeps the useful
795+
// head+tail of a result while cutting the summarizer input by 5-10x on
796+
// tool-heavy sessions; the summarizer's own cost is the price being cut.
797+
const summaryToolResultClamp = 2000
798+
799+
// summaryToolArgsClamp bounds each tool-call argument blob in the transcript:
800+
// a write_file call carries the whole file body in its arguments.
801+
const summaryToolArgsClamp = 500
802+
803+
// clampForSummary head+tail-truncates text for the summarizer transcript,
804+
// snapping to rune boundaries so the clamp never splits a UTF-8 sequence.
805+
func clampForSummary(text string, max int) string {
806+
if max <= 0 || len(text) <= max {
807+
return text
808+
}
809+
head := max * 3 / 4
810+
tail := max - head
811+
for head > 0 && !utf8.RuneStart(text[head]) {
812+
head--
813+
}
814+
start := len(text) - tail
815+
for start < len(text) && !utf8.RuneStart(text[start]) {
816+
start++
817+
}
818+
return text[:head] + "\n…[" + strconv.Itoa(len(text)-head-(len(text)-start)) + " chars omitted for summarization]…\n" + text[start:]
819+
}
820+
662821
// renderTranscript flattens messages into a plain-text transcript for the
663822
// summarizer. Secret scrubbing already happened upstream at the tool boundary.
823+
// Tool-result bodies and tool-call arguments are clamped (see the constants
824+
// above): the summary needs what was done and decided, not the raw payloads.
664825
func renderTranscript(messages []zeroruntime.Message) string {
665826
lines := make([]string, 0, len(messages))
666827
for _, message := range messages {
@@ -670,13 +831,13 @@ func renderTranscript(messages []zeroruntime.Message) string {
670831
if len(message.ToolCalls) > 0 {
671832
calls := make([]string, 0, len(message.ToolCalls))
672833
for _, call := range message.ToolCalls {
673-
calls = append(calls, call.Name+"("+call.Arguments+")")
834+
calls = append(calls, call.Name+"("+clampForSummary(call.Arguments, summaryToolArgsClamp)+")")
674835
}
675836
line += "\n[tool calls: " + strings.Join(calls, "; ") + "]"
676837
}
677838
lines = append(lines, line)
678839
case zeroruntime.MessageRoleTool:
679-
lines = append(lines, "tool result: "+message.Content)
840+
lines = append(lines, "tool result: "+clampForSummary(message.Content, summaryToolResultClamp))
680841
default:
681842
lines = append(lines, string(message.Role)+": "+message.Content)
682843
}

0 commit comments

Comments
 (0)