Skip to content

Commit 6eeeef0

Browse files
Sync .github/skills directory with azure-sdk-tools for PR 17350 (#50945)
* Fix Vally 0.19 schema loading outside TypeSpec benchmarks * Fix hermetic eval fixtures and mock prerequisites * Trim eval upgrade to minimal configuration fixes --------- Co-authored-by: helen229 <gaoh@microsoft.com>
1 parent a917737 commit 6eeeef0

7 files changed

Lines changed: 25 additions & 7 deletions

File tree

‎.github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ tags:
88
area: azsdk-common-apiview-feedback-resolution
99
type: ci-gate
1010

11-
config:
11+
defaults:
1212
runs: 1
1313
timeout: "90s"
1414
model: gpt-6.1-sol

‎.github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ tags:
88
area: azsdk-common-generate-sdk-locally
99
type: ci-gate
1010

11-
config:
11+
defaults:
1212
runs: 1
1313
timeout: "90s"
1414
model: gpt-6.1-sol

‎.github/skills/azsdk-common-generate-sdk-locally/evals/trigger.eval.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,10 @@ scoring:
2020
stimuli:
2121
- name: trigger-generate-python-sdk-locally
2222
prompt: "Generate the Python SDK locally from my TypeSpec project at specification/compute/resource-manager/Microsoft.Compute/ComputeRP/tspconfig.yaml."
23+
environment:
24+
skills:
25+
- ..
26+
- ../../azsdk-common-sdk-breaking-change
2327
graders:
2428
- type: skill-invocation
2529
config:
@@ -28,7 +32,12 @@ stimuli:
2832
# Should NOT trigger
2933
- name: anti-trigger-mitigate-breaking-change
3034
prompt: "Fix the SDK breaking changes"
35+
environment:
36+
skills:
37+
- ..
38+
- ../../azsdk-common-sdk-breaking-change
3139
graders:
3240
- type: skill-invocation
3341
config:
42+
required: ["azsdk-common-sdk-breaking-change"]
3443
disallowed: ["azsdk-common-generate-sdk-locally"]

‎.github/skills/azsdk-common-prepare-release-plan/evals/eval.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ tags:
88
area: azsdk-common-prepare-release-plan
99
type: ci-gate
1010

11-
config:
11+
defaults:
1212
runs: 1
1313
timeout: "90s"
1414
model: gpt-6.1-sol

‎.github/skills/azsdk-common-prepare-release-plan/evals/schedule-risk.eval.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,14 +58,16 @@ stimuli:
5858
config:
5959
substring: "https://github.com/Azure/azure-rest-api-specs/pull/12345"
6060

61+
# The new target stays future-proof; the mock's older plan supplies the warning.
6162
- name: create-relays-past-due-guidance
6263
environment:
6364
skills:
6465
- ..
6566
prompt: |
6667
Create a public preview release plan for TypeSpec project
6768
specification/contosowidgetmanager/Contoso.WidgetManager with target
68-
release month June 2026, spec PR
69+
release month next calendar month (resolve to an explicit month and year
70+
using the current date), spec PR
6971
https://github.com/Azure/azure-rest-api-specs/pull/38387, service tree ID
7072
a7f2b8e4-9c1d-4a3e-b6f9-2d8e5a7c3b1f, and product tree ID
7173
f1a8c5d2-6e4b-4f7a-9c2d-8b5e1f3a6c9e. Use the Azure SDK MCP tools.

‎.github/skills/azsdk-common-sdk-breaking-change/evals/trigger.eval.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,9 @@ scoring:
2020
stimuli:
2121
- name: trigger-detect-sdk-breaking-changes-for-service
2222
prompt: "Detect and mitigate SDK breaking changes for the service webpubsub"
23+
environment:
24+
skills:
25+
- ..
2326
graders:
2427
- type: skill-invocation
2528
config:

‎.github/skills/azsdk-common-sdk-release/evals/eval.yaml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ tags:
88
area: azsdk-common-sdk-release
99
type: ci-gate
1010

11-
config:
11+
defaults:
1212
runs: 1
1313
timeout: "90s"
1414
model: gpt-6.1-sol
@@ -60,12 +60,16 @@ stimuli:
6060

6161
- name: basic-trigger
6262
prompt: "I need to release a new version of my Azure SDK package. Walk me through the process."
63+
environment:
64+
skills:
65+
- ..
6366
graders:
6467
- type: output-contains
6568
config:
6669
substring: "release"
67-
constraints:
68-
expect_skills: ["azsdk-common-sdk-release"]
70+
- type: skill-invocation
71+
config:
72+
required: ["azsdk-common-sdk-release"]
6973

7074
- name: anti-trigger
7175
prompt: "How do I write unit tests for my Azure SDK client?"

0 commit comments

Comments
 (0)