File tree Expand file tree Collapse file tree
azsdk-common-apiview-feedback-resolution/evals
azsdk-common-generate-sdk-locally/evals
azsdk-common-prepare-release-plan/evals
azsdk-common-sdk-breaking-change/evals
azsdk-common-sdk-release/evals Expand file tree Collapse file tree Original file line number Diff line number Diff line change 88 area : azsdk-common-apiview-feedback-resolution
99 type : ci-gate
1010
11- config :
11+ defaults :
1212 runs : 1
1313 timeout : " 90s"
1414 model : gpt-6.1-sol
Original file line number Diff line number Diff line change 88 area : azsdk-common-generate-sdk-locally
99 type : ci-gate
1010
11- config :
11+ defaults :
1212 runs : 1
1313 timeout : " 90s"
1414 model : gpt-6.1-sol
Original file line number Diff line number Diff line change @@ -20,6 +20,10 @@ scoring:
2020stimuli :
2121 - name : trigger-generate-python-sdk-locally
2222 prompt : " Generate the Python SDK locally from my TypeSpec project at specification/compute/resource-manager/Microsoft.Compute/ComputeRP/tspconfig.yaml."
23+ environment :
24+ skills :
25+ - ..
26+ - ../../azsdk-common-sdk-breaking-change
2327 graders :
2428 - type : skill-invocation
2529 config :
@@ -28,7 +32,12 @@ stimuli:
2832 # Should NOT trigger
2933 - name : anti-trigger-mitigate-breaking-change
3034 prompt : " Fix the SDK breaking changes"
35+ environment :
36+ skills :
37+ - ..
38+ - ../../azsdk-common-sdk-breaking-change
3139 graders :
3240 - type : skill-invocation
3341 config :
42+ required : ["azsdk-common-sdk-breaking-change"]
3443 disallowed : ["azsdk-common-generate-sdk-locally"]
Original file line number Diff line number Diff line change 88 area : azsdk-common-prepare-release-plan
99 type : ci-gate
1010
11- config :
11+ defaults :
1212 runs : 1
1313 timeout : " 90s"
1414 model : gpt-6.1-sol
Original file line number Diff line number Diff line change @@ -58,14 +58,16 @@ stimuli:
5858 config :
5959 substring : " https://github.com/Azure/azure-rest-api-specs/pull/12345"
6060
61+ # The new target stays future-proof; the mock's older plan supplies the warning.
6162 - name : create-relays-past-due-guidance
6263 environment :
6364 skills :
6465 - ..
6566 prompt : |
6667 Create a public preview release plan for TypeSpec project
6768 specification/contosowidgetmanager/Contoso.WidgetManager with target
68- release month June 2026, spec PR
69+ release month next calendar month (resolve to an explicit month and year
70+ using the current date), spec PR
6971 https://github.com/Azure/azure-rest-api-specs/pull/38387, service tree ID
7072 a7f2b8e4-9c1d-4a3e-b6f9-2d8e5a7c3b1f, and product tree ID
7173 f1a8c5d2-6e4b-4f7a-9c2d-8b5e1f3a6c9e. Use the Azure SDK MCP tools.
Original file line number Diff line number Diff line change @@ -20,6 +20,9 @@ scoring:
2020stimuli :
2121 - name : trigger-detect-sdk-breaking-changes-for-service
2222 prompt : " Detect and mitigate SDK breaking changes for the service webpubsub"
23+ environment :
24+ skills :
25+ - ..
2326 graders :
2427 - type : skill-invocation
2528 config :
Original file line number Diff line number Diff line change 88 area : azsdk-common-sdk-release
99 type : ci-gate
1010
11- config :
11+ defaults :
1212 runs : 1
1313 timeout : " 90s"
1414 model : gpt-6.1-sol
@@ -60,12 +60,16 @@ stimuli:
6060
6161 - name : basic-trigger
6262 prompt : " I need to release a new version of my Azure SDK package. Walk me through the process."
63+ environment :
64+ skills :
65+ - ..
6366 graders :
6467 - type : output-contains
6568 config :
6669 substring : " release"
67- constraints :
68- expect_skills : ["azsdk-common-sdk-release"]
70+ - type : skill-invocation
71+ config :
72+ required : ["azsdk-common-sdk-release"]
6973
7074 - name : anti-trigger
7175 prompt : " How do I write unit tests for my Azure SDK client?"
You can’t perform that action at this time.
0 commit comments