Skip to content

fix: report model failures returned as data and use single-token E2E markers - #749

Merged
AlexanderZ-Band merged 3 commits into
mainfrom
fix/fixpython-port-ts-live-baseline-fixes-that-apply-t-INT-1641
Oct 7, 2026
Merged

AlexanderZ-Band merged 3 commits into
mainfrom
fix/fixpython-port-ts-live-baseline-fixes-that-apply-t-INT-1641

Conversation

@AlexanderZ-Band

@AlexanderZ-Band AlexanderZ-Band commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Resolves INT-1641. Plan: https://linear.app/thenvoi/issue/INT-1641

Model failures returned as data

Agno, Google ADK and Gemini return a failed model run as a value instead of raising, so a blocked call could end a turn silently. If the turn had already done real work, it was even marked complete.

  • ProviderRunError(code, detail) (band.core.exceptions) and generic_provider_failure(provider, error) (band.core.protocols) are the one rule. The room sees GENERIC_PROVIDER_FAILURE_MESSAGE plus the coarse code. detail (provider text) only goes to the log.
  • Agno: AgnoRunError now subclasses ProviderRunError. Behavior is unchanged.
  • Google ADK: a non-STOP event.error_code is recorded and raised once the run drains. Draining avoids leaving ADK's generators open. ADK ≤1.10 reports an empty normal finish as "STOP", which stays a normal end of turn.
  • Gemini: _model_failure(response) reads the response the way ADK's LlmResponse.create does. A reply with content (MAX_TOKENS included) or an empty STOP is not a failure.
  • docs/turn-outcome.md documents the rule.

E2E markers

  • unique_marker(label) returns one uppercase token (NOTE0A832EC4) and rejects non-alphanumeric labels. Models dropped the word-like note- prefix.
  • Hyphenated labels are renamed. The prefix coaching in the cross-framework rehydration recall is removed, and test_copilot_sdk uses unique_marker instead of inline markers.

Diagnostic

  • test_invites_messages_and_removes_a_peer now prints the agent's tool calls and failed results when the add is missing.

Tests

  • ADK runs a real InMemoryRunner against a scripted local BaseLlm registered in LLMRegistry. Responses go through ADK's own LlmResponse.create.
  • Gemini gets real GenerateContentResponses at the client boundary.
  • Both judge turns through on_event. Cases covered:
    • SAFETY, a blocked prompt, and (ADK only) no candidates
    • SAFETY after real work
    • empty STOP gives exactly the missing-reply failure
    • Gemini: MAX_TOKENS after a reply completes
    • usage is still emitted on failure
  • Every regression case fails with its adapter fix reverted. Verified on ADK 1.1.1 (py3.14) and 1.10.0 (py3.12).
  • Full unit suite, ruff, pyrefly and the markdown docs pass.

Departures from the plan: the "real work" tool in the tests is band_create_chatroom, which is advertised by default (the plan named band_create_task, which needs the tasks capability). Gemini's check returns the error itself rather than only a code, so the provider message stays attached for the log.

Live E2E

All runs are on c03d192 with first_attempt_diagnostics=true (no retry rescue).

Run Result
core, test_rehydration_cross_framework or test_participant_management, 1/3 ✅ 37437871435
same, 2/3 ❌ 37438460290: test_rehydrates_foreign_peer_message[pydantic_ai], core / ubuntu. The recall turn ended with no message. It failed the same way on main with the old prompt (37094178741), so the marker change didn't cause it. Tracked in INT-1698.
same, 3/3 ✅ 37439206975
core, full ✅ 37439751401
google, full ✅ 37437855994
backends, full ❌ 37437852493: ubuntu ✅. Windows failed only test_usage_not_cumulative_across_turns[opencode], which timed out mid-turn. This PR doesn't touch that cell, and it has failed on main before (36517841221, 35558057110, 35486844106). Tracked in INT-1699.

core runs were dispatched one at a time, because a newer core run on the same branch and OS cancels the one in progress.

🤖 Generated with Claude Code

https://claude.ai/code/session_017jhEdA7VR9ng3nZpKW2Z2X

…markers

Agno, Google ADK and Gemini return a failed model run as a value instead of
raising. A shared ProviderRunError and generic_provider_failure fail the turn
with the coarse code (SAFETY, PROHIBITED_CONTENT, ...) even after the turn did
real work; provider text stays in the agent log.

E2E markers become one uppercase alphanumeric token, since models dropped the
word-like "note-" prefix. The add-participant smoke now reports the agent's
tool calls and failed results when the add is missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jhEdA7VR9ng3nZpKW2Z2X
@linear-code

linear-code Bot commented Oct 6, 2026

Copy link
Copy Markdown

INT-1641

…rror detail

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jhEdA7VR9ng3nZpKW2Z2X
@AlexanderZ-Band
AlexanderZ-Band requested a review from a team October 6, 2026 10:28

@amit-gazal-band amit-gazal-band left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. One small fix worth making before merge:

Gemini _model_failure misses ADK's no-candidates case (src/band/adapters/gemini.py, _model_failure)

The docstring says this reads the response the way google-adk's LlmResponse.create does. For a response with candidates=[] and no prompt_feedback, ADK returns error_code='UNKNOWN_ERROR', but _model_failure returns None. So the ADK adapter fails that turn (as test_no_candidates_fails_with_adks_own_code checks), while the Gemini adapter treats it as a normal finish. If the turn already did real work, for example a tool round, it gets judged completed and the failure is lost. That is the silent-failure case this PR is meant to fix.

Suggested fix: in the no-candidates branch, return ProviderRunError("UNKNOWN_ERROR") when there is no block_reason, and add the matching Gemini test case (the shared Google model-failure cases already include no-candidates for ADK).

@AlexanderZ-Band
AlexanderZ-Band added this pull request to the merge queue Oct 7, 2026
Merged via the queue into main with commit 7dd65d8 Oct 7, 2026
28 checks passed
@AlexanderZ-Band
AlexanderZ-Band deleted the fix/fixpython-port-ts-live-baseline-fixes-that-apply-t-INT-1641 branch October 7, 2026 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants