Skip to content

Live release-plan eval starts without configured Azure SDK MCP tools #17363

Closed as not planned

Description

The existing live release-plan eval on main starts successfully with gpt-6.1-sol, but no MCP servers are registered, so the workflow cannot run.

Evidence

  • Pipeline run 6983285 — main commit ece99a752a0d43d0e6d72c294aedd60021bf12e4.
  • Eval task log: model gpt-6.1-sol, 3 tool calls, score 20% (80% required).
  • Run artifact eval-debug-area_release_plan-1, file otel-spans.jsonl: all four invoke_agent spans report github.copilot.context.mcp_server_names: [].
  • events.jsonl contains only three skill calls. The agent reports: “I can’t create the release plan because the required azure-sdk-mcp server isn’t available.”

Expected: the configured azsdk-mcp-live environment exposes the real Azure SDK MCP tools to the agent. No release plans, generation pipelines, or SDK PRs were created. The cause of the missing MCP registration is not yet established.

Activity

  1. added
    needs-triageWorkflow: This is a new issue that needs to be triaged to the appropriate team.
    on Oct 21, 2026
  2. weidongxu-microsoft commented on Oct 21, 2026

    @weidongxu-microsoft
    MemberAuthor

    Likely addressed by #17350, which upgrades Vally to restore MCP evals with scoped GitHub App tokens. Closing in favor of that fix; resolution of this specific live-run failure has not yet been verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageWorkflow: This is a new issue that needs to be triaged to the appropriate team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions