fix(config): housellm configured extract_llm into a no-op - #106
Conversation
Every extraction call this service has made was a COLD one — 132 of 132, `extraction_calls.cold = 1` — so cold_cache.min_tokens is the knob that decides whether the component does anything at all. housellm shipped it at 3000, and at 3000 the sweep refused every candidate it saw: below_output_floor on all 36 sweeping turns, 0 extractions across 3,437 requests, while the component reported itself fully enabled. Lower it to 1000 — which is also defaultColdFloor, so the preset had been overriding a sensible default with a value that broke it. It is the value both accounts that measured net-positive were running (+$2.21 of savings against $1.01 of our own model spend, 63% of calls accepted). min_tokens and trigger.min_request_tokens come back to 1000/3000 from the same configuration; context_messages to 7. Drop cold_cache.max_calls: 20 and take defaultColdMaxCalls (one concurrency round, 4). 20 was never measured. The number that WAS measured is in coldCacheConfig's own comment: one sweep made 27 calls, spent $0.229 and added 76.6s to a turn whose upstream took 33.5s. The sweep draws on no other cap, so this is its only brake, and shipping 5x the documented-safe bound on the one path with a recorded latency pathology is not a default. per_output becomes true, and on a prompt-caching backend that is very nearly a no-op — verified rather than assumed, at zero warm calls over a 28-request replay. It is the correct default for a non-caching backend, and false also suppressed the frozen-replay path for no gain. It is not the lever that turns extraction on; the cold floor is, which the previous comment had backwards. allow_on_caching_backend is REMOVED. It was inert only because per_output was false; with the hot arm live it would lift `val.cached && !allowCached` on every warm turn, a combination our own numbers price at break-even ~30,500 tokens per output against a largest-observed 2,053. Both guards now read the shipped preset instead of a transcription of it: - TestNoDefaultConfigRunsExtractLLMOnCachingBackend fails if the flag returns - TestHousellmColdSweepActuallyFires fails if the cold floor rises again, reproducing the production gate (below_output_floor) in a unit test The replay instrument could not rank the variants and is not cited as if it could: the same config scored 45,458 and 60,835 saved tokens on two identical cold runs, so the case here rests on the production ledger. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
savedTokenValue returned `cached: true` for the whole REQUEST, so a candidate in the uncached TAIL — content being written INTO the cache on this very turn — was valued at the cache-READ rate. Measured on 4,384 warm production requests the tail is billed as cache_creation, not fresh input: 17.2M cache_write against 9.4M fresh_input, ~4,124 written tokens per warm turn. The honest rate is cache-WRITE, 12.5x higher. The `val.cached && !allowCached` decline was therefore refusing tail candidates on the strength of a value computed as though they were already cached, and the ~30,500-token break-even quoted against it is explicitly the CACHED break-even. Together they read as "extraction cannot pay on a caching backend" — true at depth, and never established for the tail. That is why the hot path had never made a single production call. savedTokenValueAt prices per CANDIDATE: write rate in the tail, read rate at depth, unchanged on cold sweeps and on non-caching backends. WHAT IT IS WORTH, measured on real sessions through a local proxy rather than inferred. The tail path now fires — 5 warm calls, 4 accepted, reductions up to 74% — and at a 1000 floor it LOSES money: 8,718 tokens saved for $0.0771, net -$0.036, with every call permitted by the exploration budget (2 per session) rather than by the arithmetic. A call costs ~$0.015-0.018 and the cost is OUTPUT-dominated, so trimming the prompt does not help: context_messages 7 / 2 / 0 gave $0.0149 / $0.0159 / $0.0179 per call. Per ACCEPTED result, including the 1-in-5 rejected outright that pays full price for nothing, a call costs $0.0193 — 4,060 saved tokens at the cache-write rate, which at the observed ~65% reduction needs a ~6,250-token candidate. So housellm gets min_tokens: 8000, derived from that, and context_messages: 2, because a tail call carries the candidate and not the conversation. On the same corpus that floor makes ZERO warm calls and loses nothing, while the cold sweep still returns net +$0.062. The tail path is armed and correctly priced; it is not yet shown to save money on this deployment's rates, and the floor is what keeps "not yet" from costing anything. allow_on_caching_backend stays absent and is now vestigial rather than load-bearing: the check it lifts no longer fires on a warm turn. Guards, all mutation-verified, reading the shipped preset rather than a copy: - TestDefaultConfigsSpendOnlyOnTheUncachedTail — position decides. Replaces TestNoDefaultConfigRunsExtractLLMOnCachingBackend, whose premise this reverses; that test's own setup line said the candidate was in the tail. - TestHousellmDoesNotAttemptTheTailBelowBreakEven — a ~4k tail candidate, the shape that lost money, must be refused by the FLOOR, before exploration can spend on it. - TestTheTailIsStillGatedOnItsOwnEconomics — correcting the value did not remove the gate. - TestSavedTokenValueAtPricesTheTailAtTheWriteRate — the 12.5x, directly. - TestFireOnSizeDemotesTheEconomicGateToAdvisory — retargeted, since the caching guard it used to observe no longer decides anything on a warm turn. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Second commit: the tail was mispriced by 12.5x
That understatement is why the hot path had never made a single production call.
It works. It does not yet save money.Real Claude Code sessions through a local proxy on this build, bash/grep-heavy so the candidates are not all AUTO-skipped file reads:
5 warm calls, all But: 8,718 tokens saved for $0.0771 → net −$0.036 at the cache-write rate. And every call was permitted by the exploration budget (2/session), not by the arithmetic — the corrected price still does not clear the gate on its own. The cost is output-dominated, so trimming the prompt cannot fix it:
So the floor is derived, not chosenPer accepted result — including the 1-in-5 rejected outright that pays full price for nothing — a call costs $0.0193. At $4.75/MTok that needs 4,060 saved tokens; at the observed ~65% reduction that needs a ~6,250-token candidate. Same corpus (35 captured requests), same build:
The tail path is armed and correctly priced, and gated so that "not yet profitable" costs nothing. It will fire on genuinely large tool outputs; on the traffic I could generate, none reached 8,000 tokens in the tail, so it fired zero times and lost zero dollars. I am not claiming a warm saving — the measured warm saving is zero, by design of the floor. Caveat I can't measure awayThe corpora are my own synthetic sessions (35 captured requests, plus 3 live bash-heavy sessions). Production's real workload mix accepted 83/132 on the cold path and was net-positive; my source-read-heavy corpus accepted 0/3 there. Workload dominates, and Verification27 packages pass. Both new decisions mutation-verified:
|
…backwards
expectedReuses returned 4, then dropped to 3 once turnsSoFar >= 20, "because
fewer turns remain to amortize over". Session length on this deployment is
heavy-tailed — 6,744 sessions, median 1 turn, mean 5.5, max 6,024 — and in a
heavy tail the expected REMAINING length grows with the length so far. Measured
median turns still to come:
turns so far 1 3 5 10 20 40 80 160
median remaining 2 16 19 34 59 112 147 242
Monotonically increasing at every step, so the gate was cheapest exactly where
amortization is most real. The realized multiplier agrees:
saved_gross/saved_unique over ALL extract_llm rows is 54.6x in aggregate and
16.6x at the per-session median, against the 5.0x the first-sighting prior
implies. A live 22-message session realized 20x (56,820 gross against 2,841
unique, 18 same-session replays).
The >=20 band now rises 5/8/12 instead of falling to 3. Deliberately narrow:
- the early value stays 4 — a first-message candidate has median remaining 2,
so the measurement does not argue for raising it, and short sessions are
where a wrong prior wastes money fastest;
- seenBefore stays a flat 6, because every published break-even in
docs/components/extract_llm.md is quoted for recurring content and
TestBreakEvenSizesMatchTheDocumentedVerdict pins them;
- 12 caps at the top of the 4.0-12.0 band this file already documents, rather
than extrapolating the 242 median at 160 messages, which would license
churn this change has not measured.
housellm's min_tokens becomes 3000, replacing the 8000 of the previous commit.
8000 was wrong and the previous commit's message overstated it: tool outputs on
this workload top out near 7,399 tokens and only 1 of 132 production candidates
reached 8000, so that floor did not "arm" the tail path, it disabled it. 3000
sits at the corrected break-even (a $0.0193 call needs ~2,073 saved tokens at
reuses 12, ~3,190 candidate tokens at the observed 65% reduction) and reaches
48% of production candidates. The turn-aware prior now does the discriminating
that a blunt floor was standing in for.
TestReusePriorMatchesTheMeasuredLedger asserted the refuted direction outright;
it now pins the monotone shape and the cap, and fails on the old constants
(verified: "1 gave 4.0 but 20 gave 3.0").
NOT a demonstrated saving, and the remaining deficit is not the gate. On long
live sessions the gate allows real work (2 calls, both accepted, 10 candidates
declined on economics, 8 below the floor) and still lands at net -$0.008 — of
which $0.0224 is one duplicate: two parallel sessions extracted byte-identical
content 1.6s apart, both paying, because the cross-session result cache has no
in-flight deduplication. Without that duplicate the same session is +$0.015.
Single-flighting identical concurrent extractions is the next lever and it is
larger than anything left in the gate; colleagues working the same repo through
one proxy is exactly the shape that triggers it.
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Third commit: the amortization prior discounted long transcripts backwardsCorrecting the tail's rate (previous commit) wasn't enough, because the other factor in the numerator was wrong too — and wrong in direction, not just magnitude.
Monotonically increasing at every step. The gate was cheapest exactly where amortization is most real. The realized multiplier agrees: The
Correcting the previous commit: 8000 was wrongI have to walk back a claim in the last comment. I said
Still not a demonstrated saving — and the gate is no longer the problemLong live sessions, corrected prior, floor 3000. The gate is clearly working: 10 candidates declined on economics, 8 below the floor, 2 calls allowed, both accepted. Net: −$0.008. But of the $0.0417 spent, $0.0224 was one duplicate: two parallel sessions extracted byte-identical content 1.6 seconds apart, and both paid, because the cross-session result cache has no in-flight deduplication.
Single-flighting identical concurrent extractions is now the biggest remaining lever, and it is larger than anything left in the gate. Colleagues working the same repo through one shared proxy is precisely the shape that triggers it. Not implemented here — it is a keyed-in-flight-map change, not a gate change, and it deserves its own commit and its own measurement. Verification27 packages pass. Both new decisions mutation-verified:
One fixture bug found and fixed by measuring rather than estimating: the loss-sized candidate was built at 12 tokens/line when the real figure is 23, which had put it above the floor it was meant to sit under. |
The cross-session result cache only helps a request that arrives AFTER the first one finished. Two requests carrying byte-identical content at the same time both missed it and both paid. MEASURED on two live sessions started together: the same 4,577-token candidate was extracted twice 1.6s apart, $0.0224 for a result the system was already deriving — 54% of that run's entire extraction spend, for nothing. Ten colleagues working one repo through one proxy is exactly this shape. extractInflight collapses them, keyed on extract.ResultKey — the same key the persistent cache uses, so the in-flight window and the stored window agree on what "identical" means. singleflight rather than a hand-rolled map because golang.org/x/sync is already in the module graph. `executed` is what keeps the accounting honest: singleflight hands every waiter the leader's value, so without it each waiter would record a ModelCall and its saved tokens, double-counting a saving that happened once. Only the goroutine whose closure ran sets its own flag; a follower takes the result and charges nothing, recording the deduped_inflight_extraction gate. Also stops reporting phantom calls. rep.Calls was assigned the whole per-candidate slice, so a follower's unfilled slot reached the ledger as `cand=0 saved=0 $0.00` — visible in the first live run — inflating the call count with work that by definition did not happen. Only slots that actually called are reported now. TestConcurrentIdenticalExtractionsCollapseToOneCall blocks the model until both goroutines are provably in flight, rather than relying on a sleep to make them race, and asserts one model call across two sessions. Verified by mutation: without the dedup it reports "made 2 model calls". MEASURED END TO END, 7 real Claude Code sessions through a local proxy on this preset, 78 requests, cache_aware on 78 of 78, 4 warm/tail extraction calls (extraction_calls.cold = 0 on all four — this is the last-message path, not the cold sweep): unique tokens removed 9,875 gross incl. replays 211,994 (21.5x amortization) extract_llm cost $0.04827 at sonnet-5 rates (what these runs billed) savings $0.06511 net +$0.01684 ROI 1.35x at opus-5 rates (what the accounts bill) savings $0.12371 net +$0.07544 ROI 2.56x Savings are priced per request as they accrue: the new removal at the cache-write rate on the turn it happens, every later replay at the cache-read rate, which is what those turns would actually have been billed. In the first run the running net crossed zero at request 10 and stayed positive for the remaining 33. The spread between the two runs is the honest caveat: 39x amortization in one and 9.4x in the other, on the same config. What a removal is worth depends on how many turns follow it, so a short session can still lose. n=4 calls. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Fourth commit + the end-to-end measurement you asked forTwo bugs fixed, then 7 real Claude Code sessions on this preset. Bug 1: concurrent identical extractions both paidThe cross-session cache only helps a request arriving after the first finished. Two live sessions started together extracted the same 4,577-token candidate 1.6 s apart and both paid — $0.0224, 54% of that run's entire extraction spend, for a result already being derived. Ten colleagues on one repo through one proxy is exactly that shape. Now single-flighted on Bug 2: phantom ledger rows
Results: 7 sessions, 78 requests, cache-aware on 78 of 78All 4 extraction calls have
Priced per request as it accrues: the new removal at the cache-write rate on the turn it happens, every later replay at the cache-read rate — what those turns would actually have been billed. Cumulative, request by request (run A, opus rates)One call, paid once at request 3; the removal then replays on all 39 later requests. At sonnet rates the running net crossed zero at request 10 and stayed positive for the remaining 33; at opus rates it is positive from the first request. The honest caveatThe two runs amortized 39× and 9.4× on the same config — run B nets +$0.0136 at opus rates and −$0.0084 at sonnet. What a removal is worth depends entirely on how many turns follow it, so a short session can still lose. n = 4 calls. These are ledger figures from live runs, not projections, but they are these sessions. Three further sessions produced no data: the local proxy had been terminated and they hit a dead port. Worth stating rather than quietly reporting 7 as though 10 were attempted. 27 packages pass. Dedup verified by mutation ("made 2 model calls" without it). |
github-project-automation
Bot
moved this from New/ToDo
to Done
in Rossoctl Issue Prioritization
2 participants
What was wrong
housellmreportedextract_llmas fully enabled and it could not fire at all.Every extraction call this service has ever made was a cold one — 132 of 132,
extraction_calls.cold = 1. The hot per-output pass has never produced a call in production, and that is the caching-backend guard working correctly, not a defect: Claude Code is a prompt-caching client, so a warm candidate is priced at the cache-read rate andextract_econ.godeclines it.So
cold_cache.min_tokensis the knob that decides whether the component does anything, and the preset shipped it at 3000:extract_llmper_output_disabledbelow_output_floor3000 also overrode
defaultColdFloor, which is 1000. The preset was overriding a working default with a value that broke it.The change
cold_cache.min_tokens: 3000 → 1000— the value both accounts that measured net-positive were running (+$2.21 of savings against $1.01 of our own model spend, 63% of calls accepted).min_tokens 3000 → 1000,trigger.min_request_tokens 20000 → 3000,context_messages 2 → 7— same configuration.cold_cache.max_calls: 20dropped, takingdefaultColdMaxCalls(one concurrency round, 4). 20 was never measured. The number that was is incoldCacheConfig's own comment: one sweep made 27 calls, spent $0.229 and added 76.6 s to a turn whose upstream took 33.5 s. The sweep draws on no other cap, so this is its only brake.per_output: false → true, andallow_on_caching_backendremoved. The flag was inert only becauseper_outputwas false; with the hot arm live it liftsval.cached && !allowCachedon every warm turn — priced at break-even ~30,500 tokens per output against a largest-observed 2,053.Evidence
Both guards now read the shipped preset rather than a transcription of it, so neither can go quietly stale:
TestNoDefaultConfigRunsExtractLLMOnCachingBackend— fails if the flag comes back (verified by mutation:calls=1).TestHousellmColdSweepActuallyFires— new; fails if the cold floor rises again, reproducing the production gate in a unit test (verified by mutation:below_output_floor:1).Live, on a real Claude Code session through a local proxy on this preset, with a genuine >5 min TTL gap to force a cold turn:
per_output_disabledon every request, 0 extraction calls, on a context reaching 63,879 tokens.skip_file_read,cached_prefix) at zero spend; the cold turn fires — gateallow: cold cache, whole transcript re-billed at the write rate.What this does NOT establish
It is not shown to save money, and the honest numbers cut both ways.
low_yield_content_class:read_with_line_numbersalready says so.skipFRis deliberately false on a sweep, so cold turns do not skip file reads the way warm turns do.The A/B replay instrument could not rank the variants and is not cited as if it could: the same config scored 45,458 and 60,835 saved tokens on two identical cold runs.
Highest-value follow-up: teach the cold sweep to skip source-file reads, the one class where the acceptance check can essentially never pass. Not done here — n=3 is not a mandate for a code change.