Skip to content

Reliable interaction between the ARM API Reviewer and Critic - #43782

Closed
Ravi Eda (ravimeda) wants to merge 9 commits into
mainfrom
raeda/reviewer-critic
Closed

Ravi Eda (ravimeda) wants to merge 9 commits into
mainfrom
raeda/reviewer-critic

Conversation

@ravimeda

@ravimeda Ravi Eda (ravimeda) commented Jun 5, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR improves reliability of the Reviewer <--> Critic loop for ARM API reviews by
tightening protocol contracts and fallback behavior. The Critic was introduced in
#43494

What Changed

  1. Added a non-empty response invariant for Critic dispatch: empty / "Agent
    completed with no output" returns are treated as host-side failures, retried up
    to 3 times with a reduced/compact-mode payload, then auto-fall back to
    critic-mode=unavailable.
  2. Added compact-mode iteration support for iterations 2 and 3 (changed-findings +
    carry-over verdicts), with session-SHA re-fetch pinning and file-drift
    (carry-over-stale) checks.
  3. Replaced the rigid fenced-YAML Critic input schema with tolerant prose parsing
    and sensible defaults; simplified the graph-handling flag to Graphs: true|false
    and folded prior-iteration carryover inline into the report.
  4. Removed the manual session-handoff fallback path in favor of automatic
    unavailable-mode disclosure after bounded retries (no user action required).
  5. Shrank the non-overridable FAIL catalog to 6 reasons; downstream-CI and
    reconciliation FAILs are now overridable with justification.
  6. Updated supporting documentation (graph reasoning, downstream-CI impact,
    reviewer/critic workflow).
  7. Updated the evaluation suite: new critic-subagent-failure-auto-unavailable
    protocol-safety test, refreshed fast-path / suppressions-yaml / typespec-required
    evals, and a hardened run-evals.ps1 with infrastructure-only auto-retry.

Impact

  1. Reduces false-pass outcomes from empty or malformed Critic returns.
  2. Makes retry and fallback behavior deterministic and auditable, and removes a
    brittle manual-paste handoff step.
  3. Improves consistency between protocol docs, templates, agent guidance, and the
    eval suite.

Scope

Agent protocol, documentation, and evaluation-suite (test tooling) updates only.
No REST API spec surface changes. No SDK generation changes.

@github-actions

github-actions Bot commented Jun 5, 2026 •

Copy link
Copy Markdown

Next Steps to Merge

Next steps that must be taken to merge this PR:
  • ❌ The required check named Protected Files has failed. Refer to the check in the PR's 'Checks' tab for details on how to fix it and consult the aka.ms/ci-fix guide


Comment generated by summarize-checks workflow run.

@ravimeda Ravi Eda (ravimeda) changed the title Raeda/reviewer critic Reliable interaction between the ARM API Reviewer and Critic Jun 5, 2026
@ravimeda

Copy link
Copy Markdown
Member Author

Thanks for the contribution, Sandip Shahane (@sandipsh)

@ravimeda

Copy link
Copy Markdown
Member Author

Suhas Rao (ARM) (@raosuhas) please could you take a look

@ravimeda
Ravi Eda (ravimeda) enabled auto-merge (squash) June 16, 2026 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants