Jev for smarter tool retries: Jev-backed failure classification, retry scoring and alternate-tool suggestions #7341

Summary

Add decision-assisted recovery for tool failures: deterministic normalization and trusted failure facts first, Jev classification and bounded alternate-tool advice only when facts do not settle the case. Preserve the existing no-progress ladder, host policy, approvals, and keyword fallback.

The deliverable is fewer futile calls and fewer premature halts, without duplicating external effects, expanding an agent's capabilities, or making recovery depend on a decision-model service. Jev advises; OpenHuman decides what is permitted; the main model issues any subsequent tool call.

Design revision: 2026-10-10. Code review baseline: OpenHuman aa356266a5, including its recorded vendor gitlinks. Proposed filenames/interfaces below are implementation targets, not claims that this feature exists already.

What exists today, and what actually needs changing

Known web-tool timeout allowances and terminal module-fault handling are already present. The historical device findings below motivate evaluation, but must not be used as evidence that those gaps still exist at this baseline. The outstanding work is robust classification of unfamiliar third-party failures, calibrated advice, and end-to-end measurement.

Research that informs this design

These sources inform the proposed constraints; they do not establish that Jev improves OpenHuman's recovery rate.

• MCP tools specification: distinguish protocol failures from tool execution failures (isError). Tool annotations such as read-only/idempotent are hints from a server, not trusted authorization. Consequence: normalize at the owning adapter; host-approved declarations determine retry safety. Do not interpret arbitrary JSON ok:false as a failure globally.
• RFC 9110 §9.2.2: non-idempotent requests need known idempotency or evidence that the original was not applied before automatic retry. §10.2.3 permits both delay-seconds and HTTP-date for Retry-After. Consequence: an unknown write outcome requires reconciliation; parse both header forms from trusted transport metadata.
• AWS: making retries safe with idempotent APIs: explicit request identity and semantic equivalence matter for duplicate suppression. Consequence: never infer idempotency from similar arguments or invent a new key during a retry. Future write retries need a producer-owned contract and the same logical request key.
• AWS: timeouts, retries, and backoff with jitter: layered retries multiply load, timeouts can leave side effects, and bounded backoff/jitter reduces retry storms. Consequence: one recovery ledger; no hidden execution loop in after_tool; account for adapter retries and impose a total decision deadline.
• JSON Schema object reference: additionalProperties:false rejects extra fields; it does not define a repair algorithm. Consequence: dropping fields, coercing values, or inferring missing arguments needs a separate explicit tool contract, not merely successful validation afterward.
• TypeSafe's Jev introduction: describes structured decision models and calibration. Consequence: use typed decisions, but validate calibration on this failure corpus rather than treating vendor calibration claims or a concentrated distribution as proven local accuracy.

Execution model and invariants

model tool call
  -> bounded pre-execution representation normalization (TinyAgents)
  -> schema + host scope/policy + approval + timeout admission
  -> execute ONCE through the existing harness
  -> observe result and normalize trusted failure facts
       -> success: current success path; NO recovery model call
       -> hard fact / safe deterministic advice: existing host policy
       -> unresolved: bounded recovery evaluation, or keyword fallback
  -> one classified-ledger observation + existing exact-repeat guard
  -> ephemeral nudge / existing halt summary
  -> main model chooses its next call, admitted through the same gates

MVP does not automatically re-execute failed tools. This resolves the earlier conflict between “repair goes straight back to execution” and “retries remain model calls.” Safe representation repair happens before the first execution. After execution, corrections and alternate tools are advice. A later automatic read-retry feature requires a separate harness design with admission, cancellation, accounting, and transcript identities; it is not an after_tool middleware side effect.

Hard invariants:

1. Jev never upgrades permission, revives a denied/expired approval, overrides a process-terminal fault, or proves an uncertain external action did not happen.
2. Trusted structured facts outrank model predictions. A server's free text or annotation is not made trusted merely by parsing it as JSON.
3. Every failed invocation counts once, even if the evaluator fails, abstains, or advises against retry. A low score does not bypass accounting. Jev may reduce the existing recovery allowance, never increase it in MVP.
4. Unknown-name/schema rejection is pre-execution only when the harness supplies that fact. Fuzzy similarity suggests a candidate; it never executes a different tool automatically.
5. Alternate tools must be authorized, discoverable in this session, and executable through the existing direct/deferred bridge. Hidden tools and tools excluded by scope are not candidates.
6. Semantic budget keys remain stable across classifier disagreement and superficial argument/name changes. Preserve operation/scope grouping; also retain exact-repeat detection so changing a predicted class cannot reset the blocker indefinitely.
7. Original tool output stays intact. Recovery advice is bounded and separate; nudges do not rewrite sealed transcripts, frozen prompts, or recorded tool declarations.

Stage A: deterministic facts and safe normalization

Representation repair must be bounded by bytes/depth and produce at most one normalized call before admission. No new broad “error text repair engine” in OpenHuman.

Stage B: typed recovery decisions

Add a recovery-specific request/answer contract alongside tool search in tinytools-jev, without changing the existing JevEvaluator semantics or callers.

Proposed interfaces: RecoveryObservation, RecoveryDecision, RecoveryEvaluator (async, transport-neutral). Observation contains the normalized failure kind/phase, tool description, bounded relevant argument shape, trusted effect facts, current attempt counts, and optional authorized candidate summaries. Decision contains a closed class distribution, recoverability Noul, optional Score rubric/distribution, optional alternate candidate key, and provider usage/latency. This is advisory data; it contains no executable command or authorization.

Classification options: transient, rate_limited, authentication, permission, wrong_arguments, wrong_tool, unavailable, unsupported, unknown. Keep abstention (unknown) explicit. Cancellation, denied approval, policy restrictions, terminal module fault, and uncertain side effects are resolved as trusted host facts outside this classifier. The core maps the advisory vocabulary to its existing ledger keys and product status; do not rename public wire enums just to match these options.

Decision semantics:

• Choice: classify the unresolved tool-layer failure. Describe classes precisely; distinguish an unavailable service from an unavailable tool/module.
• Noul: estimate whether a policy-permitted next attempt, using only evidenced corrections or a genuinely transient condition, can recover without an external prerequisite. Not “does this request need a tool.”
• Score: optional ordered rubric for correction feasibility (no actionable correction → speculative correction → evidenced concrete correction). Do not call its numeric position P(success); use Noul for a binary probability question. Only score a concrete candidate correction when one exists. Do not score a hypothetical “corrected retry” with unspecified changes.
• Alternate Choice: only on wrong-tool advice or a bounded repeated blocker; precompute authorized candidates, exclude the failed operation and known-refused services, include none. Similar names alone are insufficient; the capability must address the original intent.

Batch independent class/Noul/Score questions in one request over the same bounded state. Conditional alternate discovery/evaluation may use one additional request inside the same total deadline, never a new full deadline. Questions are independent: Score cannot implicitly depend on a Choice answer not yet available.

Validate missing IDs, wrong answer types, unknown option keys, non-finite/out-of-range values, malformed distributions, and contradictory advice. Invalid, low-confidence, unavailable, or timed-out advice falls back to the existing keyword/exact-repeat behavior. Thresholds are versioned policy inputs selected on held-out data, not unexplained constants.

Classification alone does not require embeddings. Reuse embeddings for alternate discovery when usable; without embeddings, skip that optional enhancement or use the existing bounded lexical discovery path. Missing embeddings must not disable failure classification. Missing a credential/evaluator means immediate keyword fallback.

What goes where

Upstream delivery order: TinyTools contract PR → TinyAgents changes/gitlink PR if needed → owning module metadata PRs if needed → OpenHuman adapter/config/tests PR with available upstream pins → benchmark/rollout evidence. Create a superproject worktree for cross-repo work; open each PR against its canonical upstream. Do not update host gitlinks to unpublished local objects. Keep dependency PRs independently reviewable.

Configuration, latency, and data boundaries

Proposed configuration:

[agent.recovery]
classifier = "keywords" # keywords | compare | jev | off
# jev_route / jev_base_url: absent means inherit existing Jev route settings
# decision_timeout_ms: bounded total recovery decision time
# alternate_tools: opt-in during rollout

• keywords initially: retain existing behavior.
• compare: serve the keyword outcome; compute Jev advice under a bounded deadline solely for scrubbed comparison. It must not change nudge copy, budgets, halts, tools or executions.
• jev: trusted facts first, then advisory Jev for unresolved failures; fallback retains old behavior.
• off: disable the added decision assistance, not approvals, uncertain-effect checks, or no-progress protection.

Validate modes, finite thresholds and limit ranges through the project's configuration path. Explicit routes must not fall through to another credential. Reuse existing environment override precedence and credential fingerprint invalidation without printing secrets.

Proposed initial total decision ceiling: 3 s, further limited by remaining run time; the current bare evaluator also defaults to 3 s. This is a rollout proposal, not a claim about measured p95. Include client retries, retrieval and optional alternate evaluation in that ceiling; cancellation returns immediately. One outstanding evaluation per failure, bounded per-run decision count/concurrency, no recursive recovery of the recovery service. Copy state out of locks before awaiting.

The decision service receives only the minimum bounded, scrubbed failure context needed. Treat tool output as untrusted data, never as decision instructions. Do not send full transcripts, arbitrary file contents, credentials, URLs with secret queries, or raw argument values by default. If scrubbed context is insufficient, abstain. Reuse route credentials without implying permission to leak tool payloads.

Telemetry fields: source (trusted/keywords/jev), class, rubric/question version, confidence/distribution summary, latency, decision attempts, fallback/abstention reason, advice/action, repair rule ID and counts, alternate-present boolean, and eventual recovery outcome. Avoid raw error strings, tool arguments, user text, identifiers, and before→after value diffs. Any local debug capture is explicit and redacted; metadata is not automatically secret-safe merely because model rendering omits it.

Implementation sequence and verification

• 1. Pin baseline and reproduce behavior. Add failing tests for unfamiliar failures and missing advice; retain historical cases below as sanitized fixtures. Inventory what ToolResult.error_kind/metadata and invocation identity preserve across adapters. Record which facts are trusted. No implementation before a reproducing test for each actual defect.
• 2. Land upstream decision contract. Add the recovery types/question builder and fake evaluator tests in TinyTools. Confirm existing tool-search tests remain unchanged. Reuse TinyInference answer validators; extend them upstream only for a proven missing check.
• 3. Land bounded generic hooks where necessary. Reuse existing argument recovery and unknown-tool suggestions. Test any new normalization against semantic/collision cases. Prove retry-not-before enforcement with a fake clock and cancellation rather than real sleeps; retain one ledger observation per call.
• 4. Add TinyHumans recovery provider. Factor reusable client/route setup, implement batched recovery questions and remaining-deadline alternate lookup, install through the core/embedding seam. Test route inheritance, explicit overrides, credential changes, missing credentials, malformed answers and deadline/cancellation with local mocks.
• 5. Integrate host policy in keyword/compare mode. Existing driver calls the helper after trusted facts and before its single accounting step. Preserve optional-service steer-once behavior, command/task report exemptions, and uncertain-effect reconciliation. Add config/docs; keep default keywords.
• 6. Enable opt-in Jev and alternate advice. Use only filtered session candidates; render a bounded ephemeral nudge. A deferred suggestion must use the existing discovery/execution mechanism and must not silently enlarge the frozen tool catalogue.
• 7. Evaluate and roll out. Run benchmark replay, opt-in compare for at least one release cycle, then a separately reviewed default-flip PR only after the gates below pass. Roll back by changing mode; no session migration is necessary.

Required automated cases (colocated sibling *_tests.rs, no inline Rust test modules):

Run existing middleware_classified_failure_tests, command-exit/loop-guard/nudge tests and upstream arg-recovery/unknown-tool suites. Add host integration under the existing Rust E2E framework; any new root test requires an explicit [[test]] in crates/openhuman-cli/Cargo.toml. Run Rust layout/crate-chain and feature-forwarding gates, appropriate enabled/disabled builds, and changed-line coverage ≥80%. Long checks use scripts/ci-cancel-aware.sh; do not export CARGO_TARGET_DIR. Unit/integration tests use mocks, never real providers.

Acceptance criteria and default-flip gate

• No change on successful tool calls; no added execution retry loop.
• Every hard safety/accounting/session invariant above is proven by tests.
• Keywords mode passes existing suites unchanged; compare preserves model-visible behavior.
• Classification works without embeddings; alternate discovery degrades independently.
• Held-out benchmark reports per-class precision/recall, abstention/coverage, confusion matrix, Noul Brier score/reliability, ordinal Score performance, latency p50/p95/p99, extra cost and decision attempts.
• Hold out entire tool families/error templates and separate rewritten calls from identical reissues. Consecutive same-tool success alone is not evidence of causality.
• Live comparison includes eventual task recovery, futile repeat count, premature halts, uncertain-effect violations and alternate suggestion usefulness. Shadow outcomes are observational; they do not prove the counterfactual action would succeed.
• Proposed release gate: zero safety-invariant violations in automated/adversarial suites; materially fewer futile repeats without an increase in premature halts on the reviewed slice; recovery decision p95 within the agreed 3 s ceiling. Publish sample counts/confidence intervals and choose accuracy/coverage thresholds before evaluating the held-out slice.
• Default stays keywords until benchmark + release-cycle comparison justify a separate flip. Do not flip solely because overall classification accuracy is higher.

Scope decisions

• Side-effecting retry scoring is diagnostic only in MVP; it never authorizes a repeat. Reconcile uncertain outcomes first.
• Alternate suggestions use the existing ephemeral nudge channel; no new UI or steering transport is required.
• Keep deterministic representation normalization; defer broad schema coercion, inferred missing fields, fuzzy execution, shell rewrites and stale-ID replacement.
• Do not add other conditional-model backends just because TinyInference supports them. The generic trait permits a later evaluated provider without expanding this implementation.
• Model-call RetryMiddleware, emitted-stream retry, provider routing errors, UI redesign and automatic failed-tool re-execution are outside scope.

Historical device evidence (preserved from the original issue)

The following is the original issue author's 2026-10-10 local snapshot, not re-collected during this design enrichment. It is workload-specific, mixes error templates and versions, and should seed sanitized fixtures rather than be treated as a production-wide benchmark. No user IDs or raw local payloads are included here.

• Reported journal: 1,056 runs, 952 completed tool calls, 203 tool errors; known test-fixture families excluded from the described real-error analysis.
• Reported real examples: shell non-zero exits (63 plus SIGPIPE 141); web-answer 30 s timeouts (4); module install/load permission failures (7 tool errors, 32 log lines); missing arguments, edit misses, invalid handles and missing thread goals.
• Reported logs: 13 unclassified search-tool failure lines and 9 breaker halts in six days. Model endpoint “reasoning mandatory” failures and stream-retry delta loss were observed but are outside scope.

These are associations, not controlled identical-retry experiments. A successful next shell call may have changed code, arguments or state. Reproduce against the pinned baseline before claiming a missing timeout allowance or module-fault guard; those safeguards already exist in current code.