fix: Exa wire-format + post-audit hardening (3.29.5 → 3.29.6) - #90
Merged
Conversation
added 2 commits
June 22, 2026 17:32
….29.5)
ExaSearch/ExaAnswer/ExaReadUrls paid USDC, got a valid 200, then dropped
the data and reported "no results". The live gateway returns Exa payloads
at the top level ({ results }, { answer }), but the tool read res.data.*
(the documented-but-stale shape), so every field was undefined.
- exa.ts reads `res.data ?? res` for all three endpoints — parses the live
top-level shape, still tolerates a legacy data-wrapped envelope.
- test/e2e.mjs skipIfRateLimited matched bare payment words (payment
required / verification failed / insufficient). Franklin is a payment
agent and x402 == HTTP 402 "Payment Required", so the correct Exa answer
tripped it — the 3 Exa tests always skipped and hid this regression.
Now matches real signatures only ([Payment]/[PaymentRejected] labels,
failed (402), insufficient balance/usdc).
- test/exa.local.mjs: no-spend regression tests for all three tools +
legacy-envelope backward-compat, wired into npm test.
Verified: 4/4 new unit tests, local suite 458/458, live paid Exa e2e 3/3
(was 3 skipped). Media endpoints (images/videos/audio) use the OpenAI
{ data: [...] } shape and are unaffected.
…recovery, CI gate (3.29.6) Follow-up to the 3.29.5 Exa fix. A multi-agent audit surfaced and adversarially verified these; landing the high-value, well-scoped ones. - Image budget bypass (wallet safety): estimateImageCostUsd returns $0 for models absent from its 5-entry static table, so paid image models charged USDC but counted $0 against the content budget cap. ImageGen now resolves per-image cost from the live gateway catalog (findModel+estimateCostUsd, static fallback) and threads it through checkImageBudget/recordImageAsset/ recordUsage. record-image gains optional caller-resolved cost args. - Crash-safe persistence: new storage/atomic.ts (temp-write + atomic rename + .bak). trading/store.ts and content/store.ts use it and recover from .bak on corruption — a crash mid-write no longer wipes P&L / budget spend. - VoiceCall: 200 with no call_id now returns isError instead of reporting "initiated" after the $0.54 charge (parity with imagegen/videogen). - error-classifier: map context_length_exceeded / context window / context_window to context_limit; loop auto-fallback now also fires on overloaded (529) streaks, not just server 5xx. - CI: ci.yml now runs `npm test` (was type-check + build only) — the gap that let the Exa wire-format regression ship. - test/audit-batch.local.mjs: no-spend regression tests for all of the above. Verified: local suite 464/464, tsc --noEmit clean.
VickyXAI
changed the title
added 5 commits
June 22, 2026 19:45
getAvailableUpdate() opts out under CI / GITHUB_ACTIONS / FRANKLIN_NO_UPDATE_CHECK, so this test passed locally but returned null (→ failed) once `npm test` started running in CI. Save+clear those vars for the duration of the test and restore after. Surfaced by the new CI test gate added in 3.29.6. Verified: full suite 464/464 under CI=1 GITHUB_ACTIONS=true.
… budget (3.29.7) A second multi-agent review of the 3.29.5/3.29.6 branch found that both the Exa wire-format fix and the image-budget fix each missed a path: - intent-prefetch.ts exaAnswerTry() makes its own paid /v1/exa/answer call and still read the dead body.data?.answer legacy shape — so against the live top-level gateway response it paid the USDC, dropped the answer, and reported the spend as $0. Now reads both shapes via readExaAnswer() (mirrors exa.ts) and counts the real costDollars.total (estimate fallback) even on a paid-but-empty reply, so news prefetch works and spend telemetry stops under-reporting. - imagegen size-blind catalog price undercounted large sizes (e.g. gpt-image-1 1536x1024 ~$0.04 booked at the 1024 ~$0.02) by up to 2x. New resolveImageUnitCost() takes max(catalog price, size-aware static estimate). - imagegen findModel() is now wrapped in try/catch so a cold catalog fetch hiccup can't abort an already-consented paid generation; degrades to the static estimate. - atomic.ts fsyncs the temp file before rename + fsyncs the dir after (durable across power loss, not just process kill), and unlinks the orphan .tmp on a failed rename. - loop.ts gateway-error-as-text comment corrected to match the actual break. New no-spend regression tests cover the prefetch wire shape and size-aware cost resolution. Verified: local suite 470/470.
… Exa/ImageGen/Voice e2e Completes the deferred items from the branch review: - loop.ts: a rate-limit/overload error leaked as 200-OK text now classifies and auto-falls-back to a different free provider (mirrors the real-429/529 path) instead of dead-ending the turn. Safe — responseParts is never in history, so no re-poisoning; bounded by turnFailedModels. - e2e Exa tests: previously could green against the wire-format regression they guard (they matched a URL / "x402" the model can fabricate). Now assert on tool-rendered markers (## Exa search —, _Source:, _Cost: $) that only appear when the live payload parsed, surfaced via an env-gated full-tool-output mode in terminal.ts. - Added a paid ImageGen e2e (PNG signature + size) and a no-spend VoiceCall no-call_id unit test (the isError money-safety guard had no coverage). Local suite 471/471; paid e2e additions gated behind RUN_PAID_E2E=1.
…rden loop fallback & readExaAnswer Round-3 adversarial self-review of the prior two fix commits. Confirmed + fixed: - VideoGen/MusicGen budget undercount (the unfixed ImageGen siblings). VideoGen split-sourced cost: budget check + content-budget asset record used the flat $0.05/s estimate while confirm + stats used the catalog price, so a Seedance (per_second $0.15/s) 8s clip recorded ~$0.40 against the per-piece cap on a ~$1.26 charge (~3x under-record). MusicGen was flat-priced + model-blind. Both now resolve a single resolveVideoUnitCost / resolveMusicUnitCost (max of catalog vs flat, findModel in try/catch) threaded through every consumer. - loop.ts leaked-text auto-fallback hardened: overloaded only auto-switches under a routing profile (a user-pinned paid model is respected, matching the real-529 path); the concrete resolvedModel is excluded from the fallback set (no wasted self-retry on the same model); the exhausted-chain terminal error appends the "switch to a paid model" hint (parity with the real-429 path). - readExaAnswer is now type-safe: a non-string answer / non-object body degraded to a thrown TypeError that exaAnswerTry's catch swallowed -> a settled paid call mis-booked as $0. Now coerces defensively and preserves the cost count. - resolveImageUnitCost margin consistency: the size-aware static operand is margin-adjusted before Math.max, so the static-wins path no longer undercounts the realized x402 charge by the 5% gateway margin. - CHANGELOG: narrowed the overstated "spend telemetry" wording (prefetch cost feeds the status line, not the recordFetch sink). New regression tests: video/music cost resolution + malformed-answer. Suite 474/474.
…e hit isn't over-reported as $0.001 The intent-prefetch stock leg hardcoded cost: 0.001 on every success, but the paid /v1/stocks price is cached for 5 min — a repeat ticker within the window pays no USDC yet still reported $0.001 in the prefetch status line. It now diffs blockrun telemetry (blockrunSpendUsdToday) around the call: recordFetch fires only on a fresh paid fetch, never on a cache hit, so a cached lookup books $0. Mirrors the readExaAnswer paid-flag intent. New no-spend test proves a cache hit yields a $0 diff. Suite 475/475.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related fixes, found by running the e2e suite then a multi-agent audit.
1. Exa research suite silently broken (3.29.5) — HIGH, money loss
ExaSearch/ExaAnswer/ExaReadUrlspaid USDC, got a valid200, then discarded the data and reported "no results": the live gateway returns Exa payloads top-level ({ results },{ answer }) but the tool readres.data.*(matched the stale docs). Fix: readres.data ?? res(tolerates both). Hidden because the paid e2e tests false-skipped on payment vocabulary (x402 == HTTP 402 "Payment Required") —skipIfRateLimitednow matches real signatures only.2. Post-audit hardening batch (3.29.6)
A multi-agent audit (18 findings, adversarially verified) surfaced these; landing the high-value, well-scoped ones:
estimateImageCostUsdreturns $0 for models absent from its 5-entry static table → paid image models charged USDC but counted $0 against the content budget cap. ImageGen now resolves cost from the live catalog (static fallback) and threads it through budget check + asset recording + stats.storage/atomic.ts(temp + atomic rename +.bak); portfolio and content library no longer get wiped by a crash mid-write, and recover from.bakon corruption.isErrorwhen the gateway omitscall_id(was reporting "initiated" after a $0.54 charge).context_limit; loop auto-fallback now fires onoverloaded(529) streaks too.npm test(was type-check + build only) — the gap that let the Exa regression ship.Verification
tsc --noEmit+ buildNew tests:
test/exa.local.mjs,test/audit-batch.local.mjs.Deferred follow-ups (flagged in the audit; need broader change / owner input)
nvidia/deepseek-v4-flashis gateway-hidden; ~18 scattered literals).loop.ts) — needs a loop-level test.price.amountaccounting for images; gateway docs vs live Exa shape mismatch (gateway-side).🤖 Generated with Claude Code