[Jiaqi Zhang (JiaqiZhang-Dev)]

Purpose

Bring the existing Azure SDK chatbot to the API Stewardship Board channel to help with data-plane API design, review processes, compatibility questions, and Board review guidance. Reuse azure-sdk-chat-agent rather than introduce a separate agent.

Related to Azure/azure-sdk-pr#3031.

PR stack and scope

Depends on #17398. This PR temporarily targets jiaqizhang-dev-shared-chatbot-fixes, so its diff contains only Stewardship onboarding and its required configuration. Merge #17398 first, then synchronize this branch with current main as needed and return this PR's base to main.

Shared PR/CI investigation, GitHub log/overflow handling, retrieval guidance, fixes to existing tenants, and generic dev/test listener consolidation have moved to #17398. They are not part of this PR's onboarding diff.

Changes

• Add the Stewardship tenant, data-plane advisory skill, source selection, and Board-specific regression coverage. Keep the shared agent and read-only tools; the bot does not perform approvals or claim Board decisions.
• Add 1 basic and 34 perf evaluation cases based on channel questions, with scenario-to-channel routing and expert-answer provenance checks.
• Add maintained knowledge for Board questions: Avocado troubleshooting and package-name approval documentation, with source registration and configuration/citation coverage.
• Add the Board team option and dev/test channel subscriptions on top of [Teams Chatbot] Add PR review workflow, improve GitHub MCP result analyzer #17398's shared-listener behavior. For the separate production Teams team, add a dedicated Board listener; for dev/test, test through channels in the existing SDK dev/test teams and their shared azure_sdk listeners, rather than creating dedicated Board dev/test listeners or parameter files.
• Keep the dedicated Board production listener explicitly Disabled. Parameterize workflow state for that safety requirement while preserving Enabled as the existing listeners' default. No listener was deployed or activated.

Split verification

The shared branch was incorporated with a normal merge commit, preserving the original onboarding history without a force-push. The combined tracked tree exactly matches the original feature state at aac2cabc4 plus the upstream main changes; the onboarding-only diff is 18 files.

Focused combined agent coverage passed 128 tests; retained evaluation pure-logic coverage passed 34 tests, and knowledge configuration coverage passed 5 tests. All 9 team/environment combinations compiled in preview-only Logic App checks at 1fb73656b9ab7066130a0305c60827fc193a45c1: dev/test resolve to the shared SDK listener, production to the selected team's listener. No deployments or new hosted evaluations were run for the split. The evaluation below is the existing historical run, not a new result.

Evaluation

Perf CI 6992928 evaluated 34 cases against dev agent v288, from commit a564eaa72.

Composite pass rate: 85.3% (29/34). The pipeline failed its quality gate on five cases.

Before rerunning, we audited the dataset against the 44-thread channel export and restored reasonable expert conclusions and necessary question context, while removing unrelated expectations. These remain curated scenarios, not verbatim transcripts.

Metric	Mean / 5	Passed
Composite	4.30	29/34
Similarity	4.50	32/34
Response completeness	4.00	30/34
Groundedness*	5.00	34/34
Relevance	5.00	34/34
Coherence	5.00	34/34
Fluency	4.65	34/34

All 34 cases used the updated references and routed to the correct agent version and Board tenant; none failed answer collection.

* All 34 grader context fields remain empty, so the groundedness scores do not establish factual accuracy.

Example questions and bot answers

Two passing cases and one below-threshold case are shown below. Questions and bot answers are copied verbatim from this run.

Previously approved break repeats in new preview (PASS; composite 5.0/5)

Question

title: Breaking-change report in my next preview

question: Our previous preview renamed a resource compared with the latest GA and received breaking-change approval. Our next preview keeps that rename, but reports the same break against GA again. Why is it still failing, and do we need another approval?

Bot answer

Why it still fails

• The breaking-change check is expected to fire again because each new API version — including a new preview — is validated against the latest previous GA, not against your previous preview.
• The guidance is explicit on both points:
  • a new preview or GA version must be backward compatible with the latest GA; and
  • a new preview does not have to be compatible with the previous preview.
• So if preview N renamed a resource relative to GA, then preview N+1 that keeps the same rename is still breaking relative to GA, and the tooling will report it again.

Do you need another approval?

• I could not verify a doc that says a prior breaking-change approval automatically carries forward to a later preview.
• Because this is a new API version / new PR that is again breaking against the latest GA baseline, the safe guidance is: do not assume the earlier approval transfers automatically.
• Best next step: submit the current case through the documented breaking-change workflow, and include the earlier approved preview/change as supporting context so reviewers can judge continuity rather than treat it as a brand-new rationale. See the process guidance here: Deep-dive into breaking changes on spec PRs.

Adjacent distinction

• This is different from “preview-to-preview” compatibility. The policy allows a new preview to differ from the previous preview, but that does not remove the separate requirement that the new preview be compatible with the latest GA.

Resource-specific LRO monitor is allowed (PASS; composite 5.0/5)

Question

title: Resource-scoped status monitor for our LRO action

question: We have an LRO action on a resource. Can its status monitor live beneath that resource, with an operation ID in the URL, and can we use separate monitor endpoints for different resource types?

Bot answer

Yes — for an LRO action on a specific resource, a resource-scoped status monitor under that resource is an acceptable pattern, and using an operation ID as the final path segment is consistent with Azure guidance and Azure.Core modeling.

• Resource-scoped monitor: Azure TypeSpec guidance explicitly supports defining the status monitor as a child resource of the target resource, so it appears beneath that resource in the route hierarchy.
• Operation ID in the URL: Azure LRO guidance expects the status monitor to have its own URL, and the operation ID is commonly the last URL path segment. Azure.Core guidance also says the status-monitor URL can be derived from the resource key properties plus an additional operationId.
• Different monitor endpoints per resource type: Yes. That is often cleaner. If you instead use one shared monitor endpoint for multiple operation kinds or resource types with different result shapes, the status monitor must be polymorphic and include a required kind property.

Practical recommendation

• For POST /resources/{id}:action, a monitor shape like .../resources/{id}/operations/{operationId} is fine.
• If Widget and Gadget actions return different monitor/result shapes, separate per-resource-type monitor endpoints are preferable to a single mixed endpoint.

Important distinction

• This answer is about an LRO tied to a specific resource.
• If the operation is not related to a specific resource (for example, a batch job), guidance prefers a different pattern: a top-level operation URL created with PUT and a user-specified operation ID.

Unknown InternalARMContracts synchronization rule (BELOW THRESHOLD; composite 2.0/5)

Question

title: Updating our InternalARMContracts branch

question: We need to add an internal API to InternalARMContracts. Do we have to merge all of main first, or can we update only the relevant files?

Bot answer

No — you do not have to merge all of main first just to add an internal API to InternalARMContracts.

• What the docs support: InternalARMContracts is a private branch used for ARM-internal APIs, never published and with no SDK generation. It is also treated as a non-standard branch with only team-member approval required for merge.
• Practical guidance: update only the relevant files for your internal API unless you specifically need changes from main.
• Merge from main only when needed, for example if:
  • you need shared file changes already made in main,
  • repo infrastructure changed and your branch is stale enough that tooling/checks start misbehaving, or
  • the branch owner wants the branch resynced.

A useful distinction:

• If you are updating the InternalARMContracts branch itself: just change the files you need.
• If you are working in a topic branch that targets InternalARMContracts: don’t merge main into that topic branch just because it exists; the release-branch guidance says to bring over only the specific changes you need, typically by cherry-pick or manual copy.

If EngSys or branch behavior is acting up on this non-standard branch, the maintained guidance says the branch owner is the right contact, and the remediation is to bring the branch state back in line with main as needed.

[azure-pipelines]

Azure Pipelines:
Successfully started running 2 pipeline(s).
70 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

Missing channel mappings, unsupported dev/test deployment selections, and incomplete check retrieval leave routing and diagnosis gaps.

3 open issues

• Add API Stewardship Board channel mappings to tenant configuration · New
• Read GitHub Actions check runs during PR preflight · New
• Fix missing Board parameter files for dev and test environments · New

What changed in this PR

Onboards the API Stewardship Board to the existing Azure SDK chat agent, with advisory-only guidance, scoped knowledge retrieval, and evaluation coverage.

Changes:

• Registers the Board tenant and guidance, adds dev/test listener channels, and configures a disabled production listener.
• Improves shared PR diagnosis, policy applicability checks, and oversized GitHub log handling.
• Adds maintained knowledge sources, 35 evaluation records, and regression coverage.

File	Description
tools/​sdk-ai-bots/​perf-evaluation.yml	Adds the Board performance scenario.
tools/​sdk-ai-bots/​offline-evaluation.yml	Adds the Board offline scenario.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-knowledge-sync/​tests/​configurationLoader.test.ts	Tests the new documentation sources.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-knowledge-sync/​config/​knowledge-config.json	Adds Avocado and package-name approval documentation.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​unit_tests/​test_pure_logic.py	Tests datasets, tenant selection, and pipeline invocation.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​perf/​apistewardshipboard.jsonl	Adds 34 Board evaluation cases.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​basic/​apistewardshipboard.jsonl	Adds a basic review-request case.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​_evals_runner.py	Maps the Board scenario to its channel name.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​tools/​knowledge_tools.py	Targets retrieval at current evidence gaps.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​tools/​github_mcp_tools.py	Enables job-log retrieval and bounded diagnostic excerpts.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​tests/​tenant_config_test.py	Covers Board configuration and shared guidance.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​tests/​source_scoping_test.py	Tests Board knowledge-source filtering.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​tests/​knowledge_tools_test.py	Checks updated search-query guidance.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​tests/​github_mcp_parser_test.py	Tests log excerpts, output limits, and redaction.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​skills/​tenant_skills.py	Advertises Board guidance and adjusts specialist routing.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​prompts/​tenants/​api_stewardship_board.md	Defines advisory scope and review guidance.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​prompts/​tenants/​api_spec_review.md	Delegates PR diagnosis to shared guidance.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​pipelines/​logicapp/​template.json	Makes listener state configurable.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​pipelines/​logicapp/​parameters.azure_sdk.test.json	Adds the Board test channel.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​pipelines/​logicapp/​parameters.azure_sdk.dev.json	Adds the Board development channel.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​pipelines/​logicapp/​parameters.api_stewardship_board.prod.json	Configures the dedicated, disabled production listener.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​pipelines/​logicapp-cd.yml	Adds the Board deployment selection.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​config/​tenant_config.py	Registers the tenant, sources, and scope.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​agents/​chat_agent/​instruction.md	Refines shared PR investigation and evidence-based answers.

_{Review effort: Balanced}

[Renhe Li (lirenhe)]

Is this the general flow or does it only impact API Stewardship Board channel?

[Renhe Li (lirenhe)]

As this PR is about API Stewardship Board, shall we split this change in a separate PR?

[Jiaqi Zhang (JiaqiZhang-Dev)]

Agreed, this is shared behavior, not specific to the Board channel. Moved the shared PR-review workflow, GitHub log/overflow fixes, retrieval and existing-tenant fixes, and generic dev/test listener consolidation to #17398.

#17266 now temporarily targets that branch, so its diff contains only Board onboarding and the required configuration. The combined code is preserved without rewriting history. After #17398 merges, we'll return this PR's base to main. Board production remains disabled.

[Jialin Huang (JialinHuang803)]

Just curious, why do we update the api-spec-review and azure-sdk-onboarding descriptions? Is it because repository permissions might relate to spec-review?

[Jiaqi Zhang (JiaqiZhang-Dev)]

Yes—repository/workflow access can be the cause of a spec PR's checks being stuck or awaiting approval. Those edits remove the blanket reroute to SDK Onboarding so the shared PR workflow can diagnose the access/trigger evidence without switching tenants just for permissions.

That is an existing-tenant behavior change, not a Board onboarding requirement, so I've moved the description/configuration changes and their tests to #17398. This PR's skill-description diff now only adds the Board skill.

The base branch was changed.