Repository navigation
Update GPT-5.4 evals to GPT-6.1 Sol - #17323
Merged
Merged
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 71 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Helen Gao (helen229)
marked this pull request as ready for review
October 19, 2026 23:25
Helen Gao (helen229)
requested review from
a team and
Praveen Kuttappan (praveenkuttappan)
as code owners
October 19, 2026 23:25
Sameeksha Vaity (samvaity)
approved these changes
Oct 19, 2026
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Required executor and prompt-judge inference validation using the pipeline’s App token remains outstanding.
No open issues 🎉
What changed in this PR
Addresses #17322 by replacing unavailable gpt-5.4 settings with gpt-6.1-sol across the evaluation suites.
Changes:
- Updates executor and judge models in 52 eval files.
- Updates matching documentation examples and comments.
- Leaves TypeSpec authoring, existing
gpt-5.6-solsettings, and non-model configuration unchanged.
| File | Description |
|---|---|
| plugins/azure-sdk-tools/skills/skill-authoring/evals/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/skill-authoring/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/sdk-release/evals/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/sdk-release/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/prepare-release-plan/evals/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/prepare-release-plan/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/pipeline-troubleshooting/evals/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/pipeline-troubleshooting/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/markdown-token-optimizer/evals/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/markdown-token-optimizer/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/live-and-recorded-tests/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/generate-sdk-locally/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/apiview-feedback-resolution/evals/eval.yaml | Updates plugin eval model. |
| plugins/azure-sdk-tools/skills/apiview-feedback-resolution/eval.yaml | Updates plugin eval model. |
| evals/workflows/mock/typespec-generation-step02.eval.yaml | Updates workflow executor model. |
| evals/workflows/mock/rename-client-property.eval.yaml | Updates workflow executor model. |
| evals/workflows/mock/release-planner-workflows.eval.yaml | Updates workflow executor model. |
| evals/workflows/mock/multi-turn-release-workflows.eval.yaml | Updates workflow executor model. |
| evals/workflows/mock/multi-turn-pipeline-workflows.eval.yaml | Updates workflow executor model. |
| evals/workflows/mock/fix-pipeline.eval.yaml | Updates executor and prompt-judge models. |
| evals/workflows/mock/check-public-repo-then-validate.eval.yaml | Updates workflow executor model. |
| evals/workflows/mock/analyze-failed-pipeline.eval.yaml | Updates executor and prompt-judge models. |
| evals/workflows/live/release-planner.eval.yaml | Updates executor, judge, and explanatory comment. |
| evals/tools/prompt-to-tool-verify.eval.yaml | Updates tool-routing executor model. |
| evals/tools/prompt-to-tool-typespec.eval.yaml | Updates tool-routing executor model. |
| evals/tools/prompt-to-tool-releaseplan.eval.yaml | Updates tool-routing executor model. |
| evals/tools/prompt-to-tool-pipeline.eval.yaml | Updates tool-routing executor model. |
| evals/tools/prompt-to-tool-package.eval.yaml | Updates tool-routing executor model. |
| evals/tools/prompt-to-tool-github.eval.yaml | Updates tool-routing executor model. |
| evals/tools/prompt-to-tool-engsys.eval.yaml | Updates tool-routing executor model. |
| evals/tools/prompt-to-tool-config.eval.yaml | Updates tool-routing executor model. |
| evals/tools/prompt-to-tool-apiview.eval.yaml | Updates tool-routing executor model. |
| evals/tools/add-arm-resource.eval.yaml | Updates tool eval executor model. |
| evals/README.md | Updates CLI model example. |
| .github/skills/skill-authoring/evals/trigger.eval.yaml | Updates trigger eval model. |
| .github/skills/skill-authoring/evals/eval.yaml | Updates skill eval model. |
| .github/skills/markdown-token-optimizer/evals/trigger.eval.yaml | Updates trigger eval model. |
| .github/skills/markdown-token-optimizer/evals/eval.yaml | Updates skill eval model. |
| .github/skills/eval-authoring/README.md | Updates model documentation examples. |
| .github/skills/eval-authoring-workflow/evals/eval.yaml | Updates skill eval model. |
| .github/skills/eval-authoring-tool/evals/eval.yaml | Updates skill eval model. |
| .github/skills/eval-authoring-skill/evals/eval.yaml | Updates skill eval model. |
| .github/skills/code-review/evals/eval.yaml | Updates skill eval model. |
| .github/skills/azure-typespec-assessment/evals/assessment.eval.yaml | Updates executor and judge defaults. |
| .github/skills/azsdk-common-sdk-release/evals/trigger.eval.yaml | Updates trigger eval model. |
| .github/skills/azsdk-common-sdk-release/evals/eval.yaml | Updates skill eval model. |
| .github/skills/azsdk-common-prepare-release-plan/evals/trigger.eval.yaml | Updates trigger eval model. |
| .github/skills/azsdk-common-prepare-release-plan/evals/schedule-risk.eval.yaml | Updates schedule-risk eval model. |
| .github/skills/azsdk-common-prepare-release-plan/evals/eval.yaml | Updates skill eval model. |
| .github/skills/azsdk-common-pipeline-fixer/evals/trigger.eval.yaml | Updates model and timing comment. |
| .github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml | Updates trigger eval model. |
| .github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml | Updates skill eval model. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml | Updates trigger eval model. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml | Updates skill eval model. |
Review effort: Balanced
This was referenced Oct 20, 2026
Merged
Merged
Merged
Helen Gao (helen229)
added a commit
to Azure/azure-sdk-for-js
that referenced
this pull request
Oct 20, 2026
Sync .github/skills directory with azure-sdk-tools for PR Azure/azure-sdk-tools#17323 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow) Co-authored-by: helen229 <gaoh@microsoft.com>
Member
Author
|
/check-enforcer override |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #17322
Changes
gpt-5.4executor/judge settings withgpt-6.1-solin 52 eval files, and update the matching documentation examples.gpt-5.6-solmodel settings untouched.Why
Run 6976132 passed all seven App-token logins but failed all 426 trials because
gpt-5.4was unavailable. The preceding pre-merge run had the same failure. The pinned SDK/runtime's model catalog listsgpt-6.1-sol.Checks
gpt-5.4literals.npm testis blocked by the separate publisher package's missingfflatedependency in this local setup; no unrelated dependency fix is included.Real pipeline validation
Tested this exact commit with the shared GitHub App token; all 17 login steps passed. Dashboard publishing was disabled. No live evals or TypeSpec-authoring benchmarks were run.
gpt-6.1-soland reported token usage; one session-idle timeout. Grading: 8 passed, 418 failed.gpt-6.1-solfiles account for 81/84 passed, and the unchangedgpt-5.6-solfiles account for 1/3.gpt-6.1-solprompt judges returned valid verdicts: one pass and one content-based fail. No model-unavailable or judge-authentication error occurred.