[DRAFT] feat(evals): onboard planners into multi-agent BEval

[jkim323]

Pull Request

Description

This change expanded BEval from DT Coach-only execution to a six-agent matrix and onboarded the Accessibility, Privacy, Responsible AI, Security, and Software Supply Chain Security planners.

The workflow now derives each evaluation configuration, case directory, agent definition, output path, and uploaded artifact name from the matrix agent slug. Existing permissions, checkout hardening, timeout, ACP startup, and token mappings remain unchanged.

Each onboarded planner received:

• An ACP launch definition in agent.yaml
• A planner-specific validation and judge configuration in eval.config.yaml
• Four session-phase behavioral cases
• Six deeper behavioral regression cases

The resulting suites contain ten independent, prompt-conditioned cases per planner. They cover startup behavior, phase interaction, domain reasoning, proportional assessment, evidence handling, confirmation gates, qualified-human boundaries, and handoff behavior.

Related Issue(s)

None.

Type of Change

Select all that apply:

Code & Documentation:

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality)
• Breaking change (fix or feature causing existing functionality to change)
• Documentation update

Infrastructure & Configuration:

• GitHub Actions workflow
• Linting configuration (markdown, PowerShell, etc.)
• Security configuration
• DevContainer configuration
• Dependency update

AI Artifacts:

• Reviewed contribution with prompt-builder agent and addressed all feedback
• Copilot instructions (.github/instructions/*.instructions.md)
• Copilot prompt (.github/prompts/*.prompt.md)
• Copilot agent (.github/agents/*.agent.md)
• Copilot skill (.github/skills/*/SKILL.md)
• Copilot hook (.github/hooks/*/*.json)
• Eval spec added/updated for changed AI artifacts (evals/)

Note for AI Artifact Contributors:

• Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review .github/agents/ before creating new ones.
• Skills: Must include both bash and PowerShell scripts. See Skills.
• Model Versions: Only contributions targeting the latest Anthropic and OpenAI models will be accepted. Older model versions (e.g., GPT-3.5, Claude 3) will be rejected.
• See Agents Not Accepted and Model Version Requirements.

Other:

• Script/automation (.ps1, .sh, .py)
• Other (please describe):

Sample Prompts (for AI Artifact Contributions)

User Request:

Run BEval behavioral regression coverage for the onboarded planners.

Execution Flow:

1. The BEval workflow expands the evaluation job across six agent slugs.
2. Each matrix job starts the Copilot ACP agent and ACP judge services.
3. BEval loads the selected planner's eval.config.yaml, agent.yaml, and cases/ directory.
4. The planner cases submit independent scenarios and grade response timing, length, and semantic behavior.
5. Each matrix job uploads its planner-specific result directory as a uniquely named artifact.

Output Artifacts:

Each planner writes results.json below evals/beval/<agent-slug>/results/. GitHub Actions uploads the directory as beval-results-<agent-slug>-<run-id>.

Success Indicators:

• Every configured matrix job starts its planner and judge through ACP.
• Each planner discovers ten cases across its session and behavior-regression suites.
• Results are written and uploaded under the matching planner slug.
• One planner failure does not cancel the remaining matrix jobs.

For detailed contribution requirements, see:

• Common Standards: docs/contributing/ai-artifacts-common.md - Shared standards for XML blocks, markdown quality, RFC 2119, validation, and testing
• Agents: docs/contributing/custom-agents.md - Agent configurations with tools and behavior patterns
• Prompts: docs/contributing/prompts.md - Workflow-specific guidance with template variables
• Instructions: docs/contributing/instructions.md - Technology-specific standards with glob patterns
• Skills: docs/contributing/skills.md - Task execution utilities with cross-platform scripts

Testing

Validation completed during implementation:

• Passed git diff --check.
• Passed workflow dependency-pinning validation.
• Passed action-version-consistency validation.
• Passed workflow-permissions validation.
• Parsed all five new behavior-regression YAML files successfully.
• Verified six unique regression case IDs per planner.
• Verified ten total cases per planner across both case files.

Checklist

Required Checks

• Documentation is updated (if applicable)
• Files follow existing naming conventions
• Changes are backwards compatible (if applicable)
• Tests added for new functionality (if applicable)

AI Artifact Contributions

• Used /prompt-analyze to review contribution
• Addressed all feedback from prompt-builder review
• Verified contribution follows common standards and type-specific requirements

Required Automated Checks

The following validation commands must pass before merging:

• Markdown linting: npm run lint:md
• Spell checking: npm run spell-check
• Frontmatter validation: npm run lint:frontmatter
• Skill structure validation: npm run validate:skills
• Link validation: npm run lint:md-links
• PowerShell analysis: npm run lint:ps
• Eval spec schema and coverage (if AI artifacts changed): npm run eval:lint:schema
• Plugin freshness: npm run plugin:generate
• Docusaurus tests: npm run docs:test

Security Considerations

• This PR does not contain any sensitive or NDA information
• Any new dependencies have been reviewed for security issues
• Security-related scripts follow the principle of least privilege

Additional Notes

A follow-up PR will implement Phase 2 of the BEval automation plan. It will add automatic pull_request triggers for opened, synchronized, and reopened PRs targeting main or develop; guard fork and Dependabot contexts from protected-token execution; and complete the associated workflow and repository validation. This PR remains focused on planner onboarding, behavioral regression coverage, and multi-agent matrix execution.

[github-actions]

Eval Execution

✅ Status: Passed

• Artifacts evaluated: 0
• Specs run: 0
• Assertions passed: 0
• Assertions failed (blocking): 0
• Assertions failed (advisory): 0
• Failed specs (merge-blocking): 0

No changed AI artifacts required evaluation.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.66%. Comparing base (05cd2e1) to head (030b382).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2471      +/-   ##
==========================================
- Coverage   82.72%   82.66%   -0.06%     
==========================================
  Files         154      142      -12     
  Lines       20782    20693      -89     
  Branches       13        0      -13     
==========================================
- Hits        17191    17106      -85     
+ Misses       3589     3587       -2     
+ Partials        2        0       -2     

Flag	Coverage Δ
docusaurus	?
pester	86.20% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 13 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.