Repository navigation
fix: report model failures returned as data and use single-token E2E markers - #749
Conversation
…markers Agno, Google ADK and Gemini return a failed model run as a value instead of raising. A shared ProviderRunError and generic_provider_failure fail the turn with the coarse code (SAFETY, PROHIBITED_CONTENT, ...) even after the turn did real work; provider text stays in the agent log. E2E markers become one uppercase alphanumeric token, since models dropped the word-like "note-" prefix. The add-participant smoke now reports the agent's tool calls and failed results when the add is missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017jhEdA7VR9ng3nZpKW2Z2X
…rror detail Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017jhEdA7VR9ng3nZpKW2Z2X
…at-apply-t-INT-1641
amit-gazal-band
left a comment
There was a problem hiding this comment.
Approving. One small fix worth making before merge:
Gemini _model_failure misses ADK's no-candidates case (src/band/adapters/gemini.py, _model_failure)
The docstring says this reads the response the way google-adk's LlmResponse.create does. For a response with candidates=[] and no prompt_feedback, ADK returns error_code='UNKNOWN_ERROR', but _model_failure returns None. So the ADK adapter fails that turn (as test_no_candidates_fails_with_adks_own_code checks), while the Gemini adapter treats it as a normal finish. If the turn already did real work, for example a tool round, it gets judged completed and the failure is lost. That is the silent-failure case this PR is meant to fix.
Suggested fix: in the no-candidates branch, return ProviderRunError("UNKNOWN_ERROR") when there is no block_reason, and add the matching Gemini test case (the shared Google model-failure cases already include no-candidates for ADK).
Resolves INT-1641. Plan: https://linear.app/thenvoi/issue/INT-1641
Model failures returned as data
Agno, Google ADK and Gemini return a failed model run as a value instead of raising, so a blocked call could end a turn silently. If the turn had already done real work, it was even marked complete.
ProviderRunError(code, detail)(band.core.exceptions) andgeneric_provider_failure(provider, error)(band.core.protocols) are the one rule. The room seesGENERIC_PROVIDER_FAILURE_MESSAGEplus the coarsecode.detail(provider text) only goes to the log.AgnoRunErrornow subclassesProviderRunError. Behavior is unchanged.event.error_codeis recorded and raised once the run drains. Draining avoids leaving ADK's generators open. ADK ≤1.10 reports an empty normal finish as"STOP", which stays a normal end of turn._model_failure(response)reads the response the way ADK'sLlmResponse.createdoes. A reply with content (MAX_TOKENS included) or an empty STOP is not a failure.docs/turn-outcome.mddocuments the rule.E2E markers
unique_marker(label)returns one uppercase token (NOTE0A832EC4) and rejects non-alphanumeric labels. Models dropped the word-likenote-prefix.test_copilot_sdkusesunique_markerinstead of inline markers.Diagnostic
test_invites_messages_and_removes_a_peernow prints the agent's tool calls and failed results when the add is missing.Tests
InMemoryRunneragainst a scripted localBaseLlmregistered inLLMRegistry. Responses go through ADK's ownLlmResponse.create.GenerateContentResponses at the client boundary.on_event. Cases covered:Departures from the plan: the "real work" tool in the tests is
band_create_chatroom, which is advertised by default (the plan namedband_create_task, which needs the tasks capability). Gemini's check returns the error itself rather than only a code, so the provider message stays attached for the log.Live E2E
All runs are on c03d192 with
first_attempt_diagnostics=true(no retry rescue).test_rehydration_cross_framework or test_participant_management, 1/3test_rehydrates_foreign_peer_message[pydantic_ai], core / ubuntu. The recall turn ended with no message. It failed the same way onmainwith the old prompt (37094178741), so the marker change didn't cause it. Tracked in INT-1698.test_usage_not_cumulative_across_turns[opencode], which timed out mid-turn. This PR doesn't touch that cell, and it has failed onmainbefore (36517841221, 35558057110, 35486844106). Tracked in INT-1699.coreruns were dispatched one at a time, because a newercorerun on the same branch and OS cancels the one in progress.🤖 Generated with Claude Code
https://claude.ai/code/session_017jhEdA7VR9ng3nZpKW2Z2X