feat(agents): add Copilot Studio Agent Builder (agent + safety instructions + pac skill) by Dongbumlee · Pull Request #2411 · microsoft/hve-core

Dongbumlee · 2026-07-07T21:13:18Z

Pull Request

Description

Adds a Copilot Studio Agent Builder contribution for the Power Platform surface — a net-new agent that takes design inputs (ADRs / architecture / requirements) and produces a deployable, safety-gated Microsoft Copilot Studio agent, together with its safety-floor instruction file and a supporting skill.

Three coordinated AI artifacts plus their registrations and evals:

Agent — .github/agents/power-platform/copilot-studio-agent-builder.agent.md: a ten-phase builder (discovery → design → scaffold → gates → attended/designated/gated deploy → post-deploy behavior verification). Safety invariants are fail-closed: evalPassed / raiApproved are set only post-deploy against the live agent by independent owners, never self-certified; Production is gated on both green flags plus explicit, environment-named approval.
Instructions — .github/instructions/power-platform/copilot-studio.instructions.md: the companion safety floor (data-isolation, no-advice / fair-lending guardrails, execution boundary, RAI-controls taxonomy).
Skill — .github/skills/power-platform/copilot-studio-pac/: pac-workflow guidance plus an executable topic-integrity validator (scripts/validate-topics.ps1 + .sh launcher) with a Pester test suite (59/59, 93.97% coverage) that catches latent Copilot Studio pack defects (e.g. a custom topic silently collapsing onto a system trigger).

Generated registrations (collections/*.collection.md, plugins/*/README.md) and eval specs are regenerated from source, not hand-edited.

Related Issue(s)

None — net-new contribution.

Type of Change

Code & Documentation:

New feature (non-breaking change adding functionality)

AI Artifacts:

Reviewed contribution with prompt-builder agent and addressed all feedback
Copilot instructions (.github/instructions/*.instructions.md)
Copilot agent (.github/agents/*.agent.md)
Copilot skill (.github/skills/*/SKILL.md)
Eval spec added/updated for changed AI artifacts (evals/)

Other:

Script/automation (.ps1, .sh, .py) — topic-integrity validator (PowerShell + bash launcher)

Sample Prompts (for AI Artifact Contributions)

User Request:

"Here are my ADRs, architecture doc, and requirements for a retail-bank customer assistant. Build me a deployable Copilot Studio agent from them."

Execution Flow:

Reads the design inputs and derives a capability profile, refusal taxonomy, and grounding plan.
Scaffolds an authentic pac copilot workspace (*.mcs.yml agent + settings + topics), running the topic-integrity validator to fail-closed on any construct that would collapse on pac copilot pack.
Runs the design-adequacy governance checklist (Phase 8), then ALM (Phase 9).
On explicit, environment-named approval, performs an attended deploy to a designated Dev/Test environment (Phase 10), then runs the behavior eval and an independent RAI assessment against the live agent before any Production gate.

Output Artifacts:

A pac-importable Copilot Studio workspace (agent.mcs.yml, settings.mcs.yml, topics/, knowledge/, connectionreferences/).
Design specs, deployment settings (dev/test/prod), and a deploy-and-iterate runbook.
An evals/agent-behavior stimulus + expectations spec for the builder itself.

Success Indicators:

npm run validate:skills, lint:frontmatter, lint:ps, test:ps, and eval:lint:schema pass.
The topic-integrity validator exits 0 on the scaffold; pac copilot pack produces no silently-dropped topics.
Post-deploy safety rows pass against the live agent, and evalPassed / raiApproved are set only by their independent owners.

Testing

Skill unit tests: Pester 5.7.1 — 59/59 pass, 93.97% line coverage of validate-topics.ps1.
Validator ground truth: exercised against a retail-bank topic set (7/7 topics pass, fraud-topic collision correctly avoided) and against the known-defective pattern (custom topic re-declaring a system trigger → correctly flagged, exit 1).
Behavior eval: the builder's own evals/agent-behavior spec discriminates treatment vs control; eval:lint:schema reports full coverage.
Live pipeline (own Dev sandbox, out of tree): the produced scaffold was pac copilot pack → pac solution import --publish-changes → pac copilot clone round-trip verified; a 12-row safety probe (cross-subject isolation, typed-account isolation, money-movement no-execution, fraud/scam escalation, investment/credit refusals) passed 11/11, with one grounded-citation row documented as a safe deferred gap (no fabrication).

Checklist

Required Checks

Documentation is updated (if applicable)
Files follow existing naming conventions
Changes are backwards compatible (if applicable)
Tests added for new functionality (if applicable)

AI Artifact Contributions

Used /prompt-analyze to review contribution
Addressed all feedback from prompt-builder review
Verified contribution follows common standards and type-specific requirements

Required Automated Checks

Markdown linting: npm run lint:md
Spell checking: npm run spell-check
Frontmatter validation: npm run lint:frontmatter
Skill structure validation: npm run validate:skills
Link validation: npm run lint:md-links — no new external links added by this PR; local run is network-bound (only pre-existing dead links elsewhere in the repo). Deferred to CI.
PowerShell analysis: npm run lint:ps
Eval spec schema and coverage (if AI artifacts changed): npm run eval:lint:schema
Plugin freshness: npm run plugin:generate
Docusaurus tests: npm run docs:test

Security Considerations

This PR does not contain any sensitive or NDA information
Any new dependencies have been reviewed for security issues — no new package dependencies added
Security-related scripts follow the principle of least privilege — the validator is read-only (no writes, no network, no deploy)

Additional Notes

Draft for maintainer feedback on placement: the artifacts currently live under power-platform/ and are registered in the experimental + hve-core-all collections — happy to relocate (e.g. agents/experimental/) per maintainer preference.
Model versions target the latest Anthropic and OpenAI models; the builder deploys to Copilot Studio's current GPT-5 / Claude Opus 4.x model options.
All safety flags are fail-closed and never self-certified — see the ten-phase gate ordering in the agent and the safety floor in the instructions.

…-code Adds a standalone Power Platform contribution to HVE Core: - .github/agents/power-platform/copilot-studio-agent-builder.agent.md: a phased agent that guides authors through designing and building a Microsoft Copilot Studio agent (capability profiling, purpose/success, system instructions, grounding via Knowledge/Web Search/Work Context, actions/tools/orchestration, testing, and pac-CLI deployment guidance). - .github/instructions/power-platform/copilot-studio.instructions.md: companion safety-floor instructions (R1-R4 risk taxonomy, DLP guidance, and Section 9 capability-module guardrails). capabilities.tools covers first-party actions/connectors only; enabling any MCP tool, connected agent, or trigger sets capabilities.orchestration, crosses a trust boundary, and requires the Section 9 guardrail plus an orchestration-boundary test. - TRANSPARENCY-NOTE.md: adds an Intended-uses entry and Appendix 6 (Copilot Studio Agent Builder) documenting the external Copilot Studio / Power Platform target and decision-shaping behavior. - .cspell/general-technical.txt: adds pac CLI terms msapp and packagetype. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…kill, evals, registrations Extend the experimental Copilot Studio Agent Builder with a grounded pac CLI skill, behavior evals, and QA-hardened deploy/iteration recipes; register it across the experimental and hve-core-all collections and plugins. Agent + instructions (QA hardening): - Phase 10 deploy recipe: correct Flow 1 (net-new: pack --publisher-prefix -> solution import --publish-changes) vs Flow 2 (update: clone -> push --project-dir -> publish --bot); fix invalid flags (drop --schemaName on push, add --bot-id to status) that contradicted the cited skill. - Add discover-then-confirm deploy-target handling (pac auth list / pac org who -> propose-and-confirm or ask) so an undesignated env is never turned into an "I can't run pac" refusal; preserve the never-guessed invariant. - Add a bounded verify-then-remediate deploy loop: trust pac copilot list + clone round-trip over the cosmetic publish/status verb exit; re-run import once, then HALT + escalate. - Phase 7/8 bounded auto-fix on failure; Phase 8 automated-RAI evidence tiers (platform-native config-present + opt-in pro-dev behavior-verified) that INFORM but never self-certify evalPassed/raiApproved (fail-closed S6 held). - prompt-analyze fixes: disable-model-invocation frontmatter; de-bold structural title bullets to match reference agents. New: .github/skills/power-platform/copilot-studio-pac (verified pac recipes, deploy flows, verb reference); behavior stimulus + expectations and eval.yaml/ AGENTS.yml coverage; experimental + hve-core-all collection/plugin entries; cspell terms; TRANSPARENCY-NOTE appendix update. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…acy + post-deploy behavior gate The Phase 8 'Responsible AI Gate' ran the heavy independent RAI assessment and the eval pre-deploy, so for any safety-oriented agent they could only ever return a conditional 'not yet' (nothing live to behavior-verify), collapsing genuine-HALT and routine proceed-to-Dev/Test into the same red REJECTED surface (crying-wolf gate) and blocking deploy before the very behavior the gate needs could exist. F5-v2 redesign: - Phase 8 becomes a 'Governance & RAI Design-Adequacy Review': a cheap Builder-run checklist that renders CLEARED-FOR-DEV/TEST (new flag phases.governance.raiDevTestCleared) or HALT (genuine unmitigable design-level risk only). It never sets raiApproved and escalates to @rai-planner only on a flagged design risk. - phases.testPlan.evalPassed and phases.governance.raiApproved are set only post-deploy in Phase 10, against the live agent. One authoritative independent @rai-planner behavior assessment runs once, after evalPassed. Prod at Phase 8 is DEFERRED, never rejected. - Phase 10 documents the eval execution channel (test canvas / Direct Line; secret never routed through the model; operator-attended, fail-closed) and the eval->RAI->Prod sequence. Mirrored across companion instructions, copilot-studio-pac SKILL, and the behavior expectations. Safety invariant preserved: the Builder still cannot self-certify the Production flags. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ion + verify live tool execution post-deploy (F8/F9) F8: under generative orchestration the planner ignores classic topic routing-priority, so a top-priority Escalate topic is not a reliable safety control. Move the R1-R4 escalation/refusal floor into system instructions, bar tool-calls for refusal-class/safety intents, and forbid empty/blank turns on safety intents (Phases 2/3 + instructions section 2). Add paraphrase-robust escalation coverage and a no-empty-turn check to the Phase 7 test plan. F9: a [cite:...] provenance token proves knowledge-grounding, not live tool execution. Add a Phase 5 connector-action schema-completeness check (required inputs bound + connection reference minted before 'live'), a mandatory Phase 10 post-deploy tool-invocation smoke probe (with a write/destructive-action carve-out honoring guardrail microsoft#3/R2), a Phase 7 tool-execution grader, a Phase 9 bindings tool-invocation-result column, and an instructions section 6 bind-before-push rule. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…y tiers (F10/F11) Add a topic-file integrity contract and a complexity-tier model to the Copilot Studio Agent Builder harness so structural topic defects are caught at design time and fail-closed before pack/import, rather than surfacing as runtime errors on a live agent. Instructions (copilot-studio.instructions.md): - Section 3: Topic-integrity invariants (i-v) - componentName uniqueness with system-topic exemption; custom topics may not redefine a system trigger kind (redirecting to a system topic is legal, redefining it is not); no undeclared {...} tokens (declared System./Topic./Global. tokens allowed); load-bearing skeleton with a per-tier terminal node; at most one topic per system trigger. - Section 3: Topic complexity tier (T1-T4) with a Learn node-vocab reference and the tier>=2-collapsed-to-one-SendActivity anti-pattern. - Section 7: Topic-tree reconciliation (workspace <-> out/*-src <-> pack; topicCount = custom-topic source-file count reconciled to the packed count). Agent (copilot-studio-agent-builder.agent.md): - Phase 2 objective + gate: assert the integrity invariants and record each topic's complexity tier (node graph required for tier >= 2) at design time. - Phase 9: fail-closed pre-pack topic-integrity gate (schema-valid skeleton, componentName rules, no undeclared tokens, tree reconciliation, topicCount). Additive only; no existing safety rule weakened. Validators green (spell-check, lint:frontmatter, validate:skills). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Ship validate-topics.mjs, a fail-closed static validator that enforces the Copilot Studio Agent Builder Phase 9 pre-pack topic-integrity gate: schema skeleton (mcs.metadata, kind: AdaptiveDialog, beginDialog.id: main), undeclared {token} detection, system-trigger collision, at-most-one-per-trigger, componentName uniqueness + filename match, and topicCount reconciliation. Zero third-party deps (js-yaml resolved from the hve-core workspace); reads topic source only, never writes to an environment. Exit 0 all pass / 1 a topic FAILs / 2 usage-or-parse error. Add validate-topics.ps1 and validate-topics.sh launchers (satisfy the non-Python scripts/ .ps1+.sh structure rule; forward args and propagate the engine exit code). Wire the gate into the harness: agent.md Phase 9 gate now references the executable validator, and copilot-studio-pac/SKILL.md documents it (usage, exit codes, checks). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- eval-presence: add stimulus backlinks for the copilot-studio instruction and copilot-studio-pac skill so the eval-presence gate reports them covered (evals/behavior-conformance/instructions.eval.yaml + skill-behavior.eval.yaml) - plugins: repair 6 plugin stubs from regular files to 120000 symlinks that point at the canonical .github/ artifacts (broken on Windows checkout) - agent: reword two retext-profanities-flagged terms (traps->pitfalls, 'drug name'->'record ID') to satisfy eval:lint:text Gates verified green locally: eval:presence (76 covered / 0 missing vs upstream/main), eval:lint:schema (10 specs, 49/49 agent coverage), eval:lint:text (0 errors). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…es, eval regen) Resolves three compliance findings from the full CONTRIBUTING.md pre-submit sweep: - TRANSPARENCY-NOTE.md: remove 8 em dashes (no-em-dash rule) and wrap 2 lines over the 500-char MD013 limit in the Appendix 6 / intended-uses copy; lint:md now exits 0. Meaning preserved (punctuation + hard-wraps only). - format:tables: apply the repo table formatter to the agent file and 3 skill reference docs (cell-alignment whitespace only; no content change). - evals/agent-behavior/eval.yaml: regenerate from source stimulus via Build-AgentBehaviorSpec.ps1 -Force to clear eval:lint:vally drift (do-not-edit-by-hand generated file). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ester Replace Node.js validate-topics.mjs with a native PowerShell implementation and add a Pester suite, per docs/contributing/skills.md (supported languages = Bash/PowerShell/Python; scripts MUST have unit tests >=80% coverage). Harden the system-topic legitimacy check to require BOTH filename and componentName to be canonical, closing a fail-closed-gate false pass. Fix the missing-dependency guard to exit 2 (not 1) and correct SKILL.md exit-code docs. 69/69 Pester (93.48% cov), validate:skills + lint:ps green. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Restructure the Phase 2 and Phase 8 Objective paragraphs in the Copilot Studio Agent Builder from ~300-400-word run-on sentences into scannable bulleted invariants (lossless: every rule, trigger kind, and state variable preserved). Add the Phase-10 deploy-and-iterate.md runbook to the Output-Artifacts tree and a ## Scope section to the companion instructions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Normalize the Parameters table to the aligned form the repo postprocess formatter (markdown-table-formatter) emits, matching every other skill in .github/skills and making the file idempotent under plugin:generate so the plugin-freshness CI gate stays green. Whitespace/alignment only; no content change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

… descriptions Sync the generated experimental + hve-core-all collection docs and plugin READMEs with the current source frontmatter: the Copilot Studio Agent Builder description now reads "ten phases" (Phase 10 deploy-and-iterate) and the copilot-studio-pac skill description was shortened. Keeps npm run plugin:generate idempotent so the plugin-freshness CI gate passes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Dongbumlee and others added 12 commits July 1, 2026 11:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(agents): add Copilot Studio Agent Builder (agent + safety instructions + pac skill)#2411

feat(agents): add Copilot Studio Agent Builder (agent + safety instructions + pac skill)#2411
Dongbumlee wants to merge 12 commits into
microsoft:mainfrom
Dongbumlee:feat/copilot-studio-agent-builder

Dongbumlee commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Dongbumlee commented Jul 7, 2026

Pull Request

Description

Related Issue(s)

Type of Change

Sample Prompts (for AI Artifact Contributions)

Testing

Checklist

Required Checks

AI Artifact Contributions

Required Automated Checks

Security Considerations

Additional Notes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant