Skip to content

feat: add guidelines-security-analysis skill (architecture, questionnaire, threat model, enforcement mapping) - #43

Closed
iamsreec wants to merge 8 commits into
mainfrom
feat/guidelines-security-analysis
Closed

feat: add guidelines-security-analysis skill (architecture, questionnaire, threat model, enforcement mapping)#43
iamsreec wants to merge 8 commits into
mainfrom
feat/guidelines-security-analysis

Conversation

@iamsreec

@iamsreec iamsreec commented Aug 19, 2026

Copy link
Copy Markdown
Member

Summary

Adds the guidelines-security-analysis skill: a 4-step pipeline (architecture analysis -> policy guidance questionnaire -> threat model -> enforcement mapping) that turns a target MCP server's source and OWASP Top 10 for Agentic AI Security (ASI01–ASI10) catalog into a scoped set of security guidelines for OPA policy.

Closes: #issue-number

Changes

  • opa_policy/guidelines-security-analysis/guidelines-security-analysis.md — new orchestrator doc that sequences the 4 steps
  • opa_policy/guidelines-security-analysis/owasp/architecture_analysis.md — maps an MCP server's layers, trust boundaries, and OPA enforcement points into architecture.md
  • opa_policy/guidelines-security-analysis/owasp/policy_guidance_questionnaire.md — derives a 22-question policy-intent questionnaire from architecture.md + guidance.txt/system_vars.json/tool_definitions.json where present
  • opa_policy/guidelines-security-analysis/owasp/threat_model.md — evaluates all 10 ASI categories against the tool using a three-question applicability test, sourced live from the catalog rather than hardcoded
  • opa_policy/guidelines-security-analysis/owasp/enforcement_mapping.md — maps each threat instance to an enforcement layer (OPA vs. agent/tool/infra) and produces plain-English policy-rule requirements grounded in the catalog's mitigations
  • src/smith/data/owasp_10_ai_catalog.json — new repo-relative data file: the OWASP Top 10 for Agentic AI Security catalog (ASI01–ASI10), single source of truth for category names, descriptions, impact, mitigations, and attack scenarios

All four step docs share a consistent "Authoritative Paths" contract: inputs are read from ONLY the named files under <TARGET_AGENT_PATH>/smith/guidelines-security-analysis/ (or the repo-relative catalog path), with an explicit stop-and-ask instruction if a required file is missing.

Checks

  • make ci passes (lint, Rego lint, license headers, build smoke)
  • make test passes (policy scorecard — needed if policy behavior changed) — not applicable, no assets/policy.rego changes in this PR
  • CHANGELOG.md updated under ## [Unreleased] (if user-facing)
  • Commits are signed off for the DCO (git commit -s) — verified via git log -1 --format='%B', Signed-off-by present

Notes (optional)

Base branch should be hl/promptfoo_config_auto, not main — this skill was built on top of unreleased work already on that branch. Confirmed clean: git diff --stat hl/promptfoo_config_auto..feat/guidelines-security-analysis shows exactly these 6 files (1392 insertions, 0 deletions), no policy_build/CLI/example files leaking in.

@iamsreec iamsreec self-assigned this Aug 19, 2026
@araujof
araujof changed the base branch from main to hl/promptfoo_config_auto August 20, 2026 20:04
Base automatically changed from hl/promptfoo_config_auto to main August 21, 2026 18:05
Sai Sree Laya Chukkappalli and others added 7 commits August 21, 2026 14:13
…aire, threat model, enforcement mapping)

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-104-106.watson.ibm.com>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
…tion after each step) or Autonomous (all four steps back-to-back, one final review at the end

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
… threat vectors

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
…t existing

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
…d list against the target's existing

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
@araujof
araujof force-pushed the feat/guidelines-security-analysis branch from 640bb95 to 9e5c8c8 Compare August 21, 2026 18:13

@araujof araujof left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

Can you please look into the following issues, then should be good to go. Additional tests, examples, etc. can be open in a separate PR.

  • Make the workflow discoverable: opa_policy/guidelines-security-analysis/guidelines-security-
    analysis.md:1 is not named SKILL.md, and the root SKILL.md:16 never routes security-analysis requests to it. Installed agents therefore won’t invoke this workflow.

  • Fix the OPA input path: opa_policy/guidelines-security-analysis/owasp/enforcement_mapping.md:42
    consistently uses input.arguments.*, while Smith’s current canonical representation and policy workflow use input.args.*. Generated specifications can target nonexistent fields. Please check this with @dhl123 to confirm.

  • It looks like opa_policy/guidelines-security-analysis/owasp/
    enforcement_mapping.md:334 considers overlapping value sets covered. If guidance blocks {exe} and the candidate blocks {exe, dll}, dll is silently omitted. Shuld coverage require the existing rule to contain the candidate’s full triggering set?

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
@iamsreec iamsreec closed this Aug 21, 2026
@iamsreec
iamsreec deleted the feat/guidelines-security-analysis branch August 21, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants