Skip to content

Add ARM API Reviewer issue-submission skill - #47395

Open
Ravi Eda (ravimeda) wants to merge 2 commits into
mainfrom
raeda/submit-an-issue
Open

Ravi Eda (ravimeda) wants to merge 2 commits into
mainfrom
raeda/submit-an-issue

Conversation

@ravimeda

@ravimeda Ravi Eda (ravimeda) commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

Summary

  1. Add /arm-api-reviewer-issue for natural-language concerns about the ARM API Reviewer, including API guidance, review process, documentation, and repository infrastructure.
  2. Research public resources and existing issues before asking focused questions. Users do not need rule IDs or detailed contract knowledge.
  3. Provide an evidence-backed issue template, contributor guidance, and 21 Waza cases across three suites.

Tracking issue: #47396.

This changes engineering tooling and documentation only. API specifications, SDK configuration, reviewer rules, and production workflows are unchanged, so the API and SDK-specific templates do not apply.

Behavior and safeguards

  1. Show the complete title, body, destination, and labels in the main chat before a separate approval prompt. Clearly announce when the draft is ready and awaiting a decision.
  2. Accept clear conversational approval without requiring a special phrase. Pause without repeated prompts when approval is unavailable. Material revisions require a complete new preview and renewed approval.
  3. Check all open issues with the exact ARM API Reviewer agent label, including pagination. Exact duplicates return the existing link without creating or changing an issue. Recheck before publication.
  4. Publish only in public Azure/azure-rest-api-specs. Exclude private evidence, secrets, internal URLs, and local-only references.
  5. Use exactly Summary, Issue, Evidence, Solution, and References. Summary contains 2-3 sentences. Solution includes design, implementation, validation, assumptions, and limitations.
  6. Distinguish approval, blocked prerequisites, creation, pending verification, and verified publication. A known creation URL with failed read-back leads to retrying verification, not creating a duplicate.
  7. Use plain language without emoji or em dashes, including reference descriptions and approval messages.

Validation

Reviewed the complete 29-file branch scope for consistency and accuracy. Final fresh Waza v0.38.10 runs passed all 21 cases: 20 conversation cases and one deterministic MCP lookup case, with no failures or errors. Verification checked the exact source hashes, 74 grader results, and actual model tool calls.

Commands below are shown relative to the repository or isolated evaluation area described in the evaluation README.

Command Result
waza --no-update-check check .github\skills\arm-api-reviewer-issue --format json Passed: High readiness, 497/500 tokens, and 14 valid links.
foreach ($suite in 'routing','workflow','mocked') { waza --no-update-check check ".\issue-skill-schema-checks\$suite\arm-api-reviewer-issue" --format json } Passed: all three suite definitions and all 21 task schemas, using isolated package copies.
waza --no-update-check run .\complete.eval.yaml --parallel --workers 4 --output .\complete-final-results.json Passed: 20/20 real-model conversation cases.
waza --no-update-check run .\mocked.eval.yaml --output .\mocked-final-results.json Passed: 1/1 local fixture case, with exactly two successful issue-list calls for pages 1 and 2.
.\node_modules\.bin\oxfmt.cmd --check .github\skills\arm-api-reviewer-issue Passed: all 28 skill-package files.
git diff --check and git diff --cached --check Passed.
  1. Temporary suites use a source-identical skill copy and a narrowly scoped harness. The model has no shell, network, filesystem-write, or live GitHub capability. The MCP fixture exposes only a deterministic local issue-listing tool.
  2. Coverage includes exact draft sections, Summary length, output style, visible approval handoff, ordinary consent, revised drafts, cancellation, duplicate races, privacy, unavailable authentication/discovery, uncertain creation, and pending or mismatched verification.
  3. The fixture places the duplicate on page 2. Graders and result verification check repository, open state, exact label, pagination, and the absence of creation.
  4. The suites now default to auto rather than copying an unavailable legacy model. Shared repository configuration is unchanged; --model remains available for selecting a supported model.

Limitations

  1. Automated tests do not publish or modify live GitHub issues. A model response alone is not proof that a real API operation succeeded.
  2. Main-chat preview text is graded, but the host's actual approval-dialog ordering and activity indicator are not integration-tested. A skill cannot directly change the IDE's "Working" indicator.
  3. Approval and privacy instructions are behavioral safeguards, not a new technical security boundary. Duplicate checks cannot make concurrent creation atomic.
  4. Automatic model selection is not a reproducibility guarantee. The full repository build/test suites were not run because no TypeSpec or executable workflow implementation changed.

Documentation

  1. Skill workflow.
  2. Five-section issue template.
  3. Evaluation setup, coverage, and limits.
  4. Contributor entry point.

Add research-first concern intake, mandatory duplicate checks and draft approval, a five-section issue template, focused Waza evaluations, and contributor guidance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Next Steps to Merge

✅ All automated merging requirements have been met! To get your PR merged, see aka.ms/azsdk/specreview/merge.

Comment generated by summarize-checks workflow run.

Display the complete draft in main chat before a separate decision prompt, and distinguish approval, publication checks, creation, and verification states.

Expand Waza coverage to 21 cases with explicit safety and output checks plus a deterministic paginated issue-lookup fixture. Keep all automated tests isolated from live GitHub writes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant