Skip to content

Add release-plan next-step process-awareness eval coverage - #16926

Open
Helen Gao (helen229) wants to merge 7 commits into
mainfrom
feat/release-plan-next-step-eval
Open

Helen Gao (helen229) wants to merge 7 commits into
mainfrom
feat/release-plan-next-step-eval

Conversation

@helen229

@helen229 Helen Gao (helen229) commented Sep 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds process-awareness coverage for #15328: answer an open "what is next?" question from the release plan's current lifecycle state instead of guessing or starting an action.

Changes

  • Four mock workflows cover pending API review, approved specs without SDK generation, generated SDKs awaiting review/release, and completed releases.
  • All four competing release-plan, generation, and release skills remain mounted. Every case requires reading the plan and forbids generation, release, and plan-update calls.
  • Plan-status guidance and its plugin mirror now keep next-step questions read-only. Linked SDK PRs are not treated as proof of merge or release readiness.
  • Lifecycle fixtures coexist with main's API-version selector validation. The pending-review fixture no longer contradicts the readiness mock.
  • Serialization regressions verify that cached PR status is omitted. Rubrics use the fields agents actually receive rather than requiring that hidden field.
  • Merged current main, including the landed abandonment guard, normally and preserved published changelog history. Pending notes use 0.6.56 (Unreleased). The whole PR contains twelve files.

Sync Validation Fix

The skill edit was introduced in merge commits. The old sync step used git format-patch, which skipped those commits and produced no patch despite the real skill diff. A focused helper now creates a mail patch from the final directory delta using a temporary Git object. Source refs and the existing downstream patch, merge, and approval behavior are unchanged.

  • Five Git regressions pass, including the formerly empty merge-only case, applying the generated patch to a clean consumer, pattern filtering, and source-ref preservation.
  • The actual PR skill delta produces a nonempty one-file patch matching the final skill content.
  • The regressions are discovered by the existing common PowerShell CI suite; scoped script analysis has no warnings or errors.
  • Hosted sync run 7007002 has passed patch creation and CreateSync. Downstream sync PR verification and its normal approval gates are not claimed complete.
  • The hosted Windows common-test run caught a missing YAML module in the new wiring assertion. The test now uses the repository's CompatibleConvertFrom-Yaml bootstrap helper; all five focused regressions pass again. The corrective test-only follow-up is published, and its fresh hosted result remains a required gate.
  • The CLI source tree is unchanged by the sync fix. The CLI and workflow results below were run on the preceding source-identical CLI revision; they were not redundantly rerun for the PowerShell-only fix.

Validation

  • Current pinned Vally 0.19.0, Node 24.14.1, and gpt-6.1-sol: 4/4 workflow cases passed, with all skill, tool-call, and judgment graders passing. The explicit 1.0 threshold is unchanged. Each trial only loaded the plan skill and called the plan lookup.
  • 20/20 focused lifecycle tests passed.
  • Full non-integration CLI suite: 3,620 passed, 32 not executed; mock suite: 93 passed; zero failures.
  • Strict production/mock Release builds, normal Release solution build, package creation, and validation of the changelog inside the produced package passed.
  • Canonical skill and workflow strict lint and pinned Prettier 3.8.4 checks passed. The added guidance is identical in the plugin mirror.

The first current-model run exposed contradictory mock readiness, obsolete cached-status expectations, and unnecessary readiness checks. Those issues were repaired before the passing runs. Safety tool-call restrictions and the 1.0 threshold are unchanged; state rubrics now use observable fields. These are local mock validations, not a live release or a claim that hosted CI has finished.

The existing plugin directory/name mismatch and three async-without-await warnings in main's onboarding mocks are unchanged. Strict solution-wide warning-as-error compilation is not claimed; normal solution compilation and strict production compilation passed.

Fixes #15328

Adds mock fixtures for 4 release-plan lifecycle stages (spec pending review, spec approved/no SDK, SDK generated/pending release, fully released) and stimuli asking an open what's-next question against each, graded on whether the agent gives the state-correct answer without jumping straight to an action tool.

Fixes #15328
Copilot AI balanced review requested due to automatic review settings September 3, 2026 21:08
@github-actions github-actions Bot added the azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli label Sep 3, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
61 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several fixtures diverge from production release-plan states, and one scenario permits the action it intends to prohibit.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds workflow evaluation coverage for release-plan lifecycle guidance.

Changes:

  • Adds four release-plan lifecycle fixtures.
  • Adds Vally scenarios validating state-aware next-step responses.
File summaries
File Description
GetReleasePlanHandler.cs Provides lifecycle-specific mock release plans.
release-plan-next-step.eval.yaml Tests next-step guidance across four lifecycle stages.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 6
  • Review effort level: Balanced

@helen229
Helen Gao (helen229) enabled auto-merge (squash) October 27, 2026 21:23

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a SDK agent testing for workflow related prompt

3 participants