Repository navigation
Upgrade Vally to 0.19.0 for scoped App-token MCP evals - #17350
Conversation
|
Azure Pipelines: Successfully started running 4 pipeline(s). 68 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
The following pipelines have been queued for testing: |
|
The following pipelines have been queued for testing: |
|
The following pipelines have been queued for testing: |
|
The following pipelines have been queued for testing: |
|
The following pipelines have been queued for testing: |
There was a problem hiding this comment.
🟡 Changes recommended
Both evaluator lockfiles need public npmjs URLs and SHA-512 integrity metadata restored to preserve portable installs and the documented update convention.
1 open issue
What changed in this PR
Upgrades both Vally evaluators to 0.19.0 to restore mock MCP calls with scoped GitHub App tokens and adopt upstream reliability improvements.
Changes:
- Updates Vally dependencies and raises the Node.js minimum to 22.14.
- Migrates eval settings and skill assertions, adding explicit skill mounts.
- Aligns release-plan prompts with mock IDs, uses relative creation dates, and corrects the successful setup response.
| File | Description |
|---|---|
tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/Verify/VerifySetupHandler.cs |
Returns no requirement failures for successful setup. |
evals/workflows/mock/release-planner-workflows.eval.yaml |
Updates mock IDs and creation timelines. |
evals/workflows/mock/multi-turn-release-workflows.eval.yaml |
Aligns multi-turn release prompts with mock data. |
evals/workflows/mock/multi-turn-pipeline-workflows.eval.yaml |
Migrates execution settings to defaults. |
evals/workflows/live/release-planner.eval.yaml |
Removes unsupported repository metadata. |
evals/tools/prompt-to-tool-releaseplan.eval.yaml |
Updates release-plan IDs, dates, and prompt context. |
eng/skill-eval/package.json |
Pins Vally 0.19.0 and Node.js 22.14+. |
eng/skill-eval/package-lock.json |
Refreshes the skill evaluator dependency graph. |
eng/common/scripts/eval/package.json |
Updates shared evaluator and runtime requirements. |
.github/skills/skill-authoring/evals/eval.yaml |
Migrates defaults and adds explicit skill invocation checks. |
.github/skills/markdown-token-optimizer/evals/eval.yaml |
Migrates defaults and adds explicit skill invocation checks. |
.github/skills/azsdk-common-sdk-release/evals/eval.yaml |
Migrates defaults and adds explicit skill invocation checks. |
.github/skills/azsdk-common-sdk-breaking-change/evals/trigger.eval.yaml |
Explicitly mounts the skill under test. |
.github/skills/azsdk-common-prepare-release-plan/evals/schedule-risk.eval.yaml |
Uses a relative future creation target. |
.github/skills/azsdk-common-prepare-release-plan/evals/eval.yaml |
Migrates execution settings to defaults. |
.github/skills/azsdk-common-generate-sdk-locally/evals/trigger.eval.yaml |
Mounts competing skills and asserts breaking-change routing. |
.github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml |
Migrates execution settings to defaults. |
.github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml |
Migrates execution settings to defaults. |
Review effort: Balanced
|
The following pipelines have been queued for testing: |
Co-authored-by: helen229 <19517014+helen229@users.noreply.github.com>
|
The following pipelines have been queued for testing: |
|
The following pipelines have been queued for testing: |
Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#17350 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow) --------- Co-authored-by: helen229 <gaoh@microsoft.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: helen229 <19517014+helen229@users.noreply.github.com> Co-authored-by: azure-sdk <azuresdk@microsoft.com>
Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#17350 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow) --------- Co-authored-by: helen229 <gaoh@microsoft.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: helen229 <19517014+helen229@users.noreply.github.com> Co-authored-by: azure-sdk <azuresdk@microsoft.com>
Sync .github/skills directory with azure-sdk-tools for PR Azure/azure-sdk-tools#17350 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow) --------- Co-authored-by: helen229 <gaoh@microsoft.com>
Summary
Upgrade both evaluators from Vally 0.14.0 to 0.19.0. This restores mock MCP tool calls with the existing scoped GitHub App token and brings documented reliability and diagnostic improvements not just a version bump.
Upgrade benefits
runIdto JSONL trial results and summaries, making trajectories and repeated/sharded runs easier to correlate (#1234).These are documented in the Vally changelog. Vally 0.18 also offers an optional
--in-placemode that avoids expensive workspace copies (#1268); this PR keeps isolated execution and does not claim measured speed or token savings.Scope
Validation
177aa4308, with all 19 shard artifacts inspected and dashboard publication disabled:Complete pipeline-consumer audit
Related: #17238 and #17323.