[Jiaqi Zhang (JiaqiZhang-Dev)]

Resolved issues: https://github.com/Azure/azure-sdk-pr/issues/3115, https://github.com/Azure/azure-sdk-pr/issues/3182

Problems

• Incomplete PR diagnosis: troubleshooting can miss failed-job details, access issues, and merge blockers.
• Noisy review history: superseded Copilot reviews and expandable details consume context when only the latest summary is needed.
• Missing Q&A answers: knowledge search can return a historical question without its answer, while tenant routing and filters hide relevant guidance.
• Incomplete responses: answers can omit important conditions, exceptions, or required steps from retrieved evidence.

What changed

• PR investigation: add a shared workflow with bounded diagnostic rounds; fix GitHub log retrieval, pagination guidance, and oversized-result handling.
• Review post-processing: retain the latest Copilot review per returned page and omit all expandable bodies without matching their content. Preserve visible verdicts, summary labels, human feedback, and pagination caveats; retrieve inline findings only when needed. No new API or service.
• Paired-answer retrieval: automatically retrieve Q&A answers in code and prioritize them over long question/log text.
• Access to relevant guidance: remove permission-based rerouting and API Spec Review's restrictive document-title filter so each tenant can diagnose access issues with the current request context. Retain service-plane filters and verify historical advice against current documentation.
• GitHub file evidence: preserve embedded file text rather than returning only a download-success message. Share the existing 8,000-character budget across content blocks, retain URI/MIME and coverage metadata, and explicitly identify truncated, binary, or link-only resources. Preserve trusted-author redaction and review compaction.
• Answer coverage: improve evidence-driven searches and instructions to retain conditions, exceptions, and required steps. The latest evidence-handling follow-up makes no additional instruction edits.
• Documentation-aligned evaluation expectations: correct seven ground truths covering published preview versions, API/SDK release lifecycle, TypeSpec suppression scope, Key Vault control-plane migration, automatic TypeSpec management SDK generation, LRO header modeling, and Batch custom error handling. Include authoritative references; preserve questions, row identities, and provenance. Runtime fixes and expectation corrections are separate commits so their effects remain distinguishable.

Preserved: existing read-only safeguards and response limits.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
70 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

Required sequential PR diagnostics conflict with the retained single-batch and limited-follow-up constraints.

1 open issue

• Allow bounded PR diagnostic rounds under shared batching rules · New

What changed in this PR

Extracts shared chatbot diagnostics and listener changes from #17266 without adding the API Stewardship Board tenant. Production listeners remain team-specific.

Changes:

• Centralizes evidence-based PR investigation and updates tenant routing and retrieval guidance.
• Enables GitHub job-log retrieval with bounded diagnostic excerpts and parser tests.
• Shares Azure SDK listeners across dev/test team selections and removes redundant MCP parameter files.

Paths below are relative to tools/sdk-ai-bots/azure-sdk-qa-bot-agent/.

File	Description
tools/​knowledge_tools.py	Targets search queries to current evidence gaps.
tools/​github_mcp_tools.py	Enables job logs and adds bounded overflow diagnostics.
tests/​tenant_config_test.py	Tests shared workflow guidance and tenant scope.
tests/​knowledge_tools_test.py	Tests updated search-query guidance.
tests/​github_mcp_parser_test.py	Covers log excerpts, output limits, and redaction.
skills/​tenant_skills.py	Removes permission-driven skill routing.
prompts/​tenants/​api_spec_review.md	Delegates PR diagnosis to shared guidance.
pipelines/​logicapp/​parameters.azure_sdk.test.json	Adds the shared MCP test-channel subscription.
pipelines/​logicapp/​parameters.azure_sdk.dev.json	Adds the shared MCP dev-channel subscription.
pipelines/​logicapp/​parameters.azure_mcp_server.test.json	Removes dedicated MCP test parameters.
pipelines/​logicapp/​parameters.azure_mcp_server.dev.json	Removes dedicated MCP dev parameters.
pipelines/​logicapp-cd.yml	Resolves dev/test deployments to the shared listener.
config/​tenant_config.py	Removes permission routing and broadens review engineering-doc retrieval.
agents/​chat_agent/​instruction.md	Centralizes PR diagnosis and conditional, evidence-based answers.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Diagnostic budgeting can exceed the response cap and suppress later actionable records, while the promised Python Q&A source remains unregistered and unavailable to default searches.

3 open issues

• Register the Python-only static Q&A source · New
• Skip oversized diagnostics and continue selecting later records · New
• Budget the complete response envelope, including failure excerpts · New

1 resolved since last review

• Allow bounded PR diagnostic rounds under shared batching rules

_{Review effort: Balanced}

[Jiaqi Zhang (JiaqiZhang-Dev)]

Evaluation completed against the dev agent for this branch (089a977b, agent v303, GPT-5.4 with medium reasoning).

The run covered 289 cases. 166 cases were graded; the other 123 TypeSpec cases returned answers but could not be graded because the Foundry result upload exceeded its size limit.

Metric	Main	This PR	Change
Composite	139/166 (83.73%)	141/166 (84.94%)	+2 / +1.20 pp
Similarity	146/166 (87.95%)	147/166 (88.55%)	+1 / +0.60 pp
Response completeness	140/166 (84.34%)	142/166 (85.54%)	+2 / +1.20 pp
Groundedness	130/166 (78.31%)	125/166 (75.30%)	-5 / -3.01 pp
Relevance	166/166 (100%)	166/166 (100%)	No change
Coherence	166/166 (100%)	166/166 (100%)	No change
Fluency	166/166 (100%)	166/166 (100%)	No change

I also manually compared paired bot answers. The PR shows real, user-visible improvements in several cases:

Case	Main answer	This PR	Score
test-proxy URI mismatch in playback	Suggested several speculative sanitizer/configuration causes without giving the expected direct fix.	Identified the two conflicting canonical endpoints and gave the exact Sanitized.cognitiveservices.azure.com preparer change with a code example.	2.6 → 5.0
main → RPSaaSMaster merge process	Implied that a regular sync presented documented risk of changing private specification files.	More accurately separated automatic repository-infrastructure mirroring from service-owner-managed specification synchronization, while avoiding an unsupported zero-risk guarantee.	2.8 → 4.0
Changes made out-of-band to azure-rest-api-specs	Explained the TypeSpec source-of-truth shift, but the operational remedy was less complete.	Clearly explained why public TypeSpec changes collide with an ADO-first workflow and recommended an immediate back-sync plus targeted reconciliation before promotion.	3.0 → 4.0
Go spec to consumable SDK workflow	Gave general release-plan guidance but was less direct about where the Go work appears.	Directly identified the generated Go SDK PR and Release Plan Dashboard as the artifacts to review, with a recovery path if generation is missing.	3.6 → 4.2
Python/Go/Java SDK release pipeline	Correct, but mixed the normal automatic flow with manual recovery details.	Led directly with “do not manually queue another pipeline,” then kept manual steps clearly scoped to recovery.	4.4 → 5.0

Conclusion: this PR demonstrates meaningful improvements in the targeted retrieval and process-answering cases, especially exact remediation and source-of-truth guidance. The overall composite pass rate improved by 1.20 percentage points, with gains in similarity and response completeness while maintaining 100% relevance, coherence, and fluency.

[Yuchao Yan (msyyc)]

These comments focus on making the Python code easier to read, change, and test. They are not reports of runtime bugs. I skipped the type-modeling concern that Renhe already raised.