Repository navigation
Add release-plan next-step process-awareness eval coverage - #16926
Open
Helen Gao (helen229) wants to merge 7 commits into
Open
Helen Gao (helen229) wants to merge 7 commits into
Helen Gao (helen229) wants to merge 7 commits into
Conversation
Adds mock fixtures for 4 release-plan lifecycle stages (spec pending review, spec approved/no SDK, SDK generated/pending release, fully released) and stimuli asking an open what's-next question against each, graded on whether the agent gives the state-correct answer without jumping straight to an action tool. Fixes #15328
Helen Gao (helen229)
requested review from
a team and
Praveen Kuttappan (praveenkuttappan)
as code owners
September 3, 2026 21:08
|
Azure Pipelines: Successfully started running 1 pipeline(s). 61 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Several fixtures diverge from production release-plan states, and one scenario permits the action it intends to prohibit.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds workflow evaluation coverage for release-plan lifecycle guidance.
Changes:
- Adds four release-plan lifecycle fixtures.
- Adds Vally scenarios validating state-aware next-step responses.
File summaries
| File | Description |
|---|---|
GetReleasePlanHandler.cs |
Provides lifecycle-specific mock release plans. |
release-plan-next-step.eval.yaml |
Tests next-step guidance across four lifecycle stages. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 6
- Review effort level: Balanced
Praveen Kuttappan (praveenkuttappan)
approved these changes
Oct 6, 2026
Helen Gao (helen229)
requested a review
from Praveen Kuttappan (praveenkuttappan)
October 27, 2026 21:03
Praveen Kuttappan (praveenkuttappan)
approved these changes
Oct 27, 2026
Helen Gao (helen229)
enabled auto-merge (squash)
October 27, 2026 21:23
Helen Gao (helen229)
requested a review
from Praveen Kuttappan (praveenkuttappan)
October 28, 2026 18:01
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds process-awareness coverage for #15328: answer an open "what is next?" question from the release plan's current lifecycle state instead of guessing or starting an action.
Changes
Sync Validation Fix
The skill edit was introduced in merge commits. The old sync step used
git format-patch, which skipped those commits and produced no patch despite the real skill diff. A focused helper now creates a mail patch from the final directory delta using a temporary Git object. Source refs and the existing downstream patch, merge, and approval behavior are unchanged.CreateSync. Downstream sync PR verification and its normal approval gates are not claimed complete.CompatibleConvertFrom-Yamlbootstrap helper; all five focused regressions pass again. The corrective test-only follow-up is published, and its fresh hosted result remains a required gate.Validation
The first current-model run exposed contradictory mock readiness, obsolete cached-status expectations, and unnecessary readiness checks. Those issues were repaired before the passing runs. Safety tool-call restrictions and the 1.0 threshold are unchanged; state rubrics now use observable fields. These are local mock validations, not a live release or a claim that hosted CI has finished.
The existing plugin directory/name mismatch and three async-without-await warnings in main's onboarding mocks are unchanged. Strict solution-wide warning-as-error compilation is not claimed; normal solution compilation and strict production compilation passed.
Fixes #15328