feat: add guidelines-security-analysis skill (architecture, questionnaire, threat model, enforcement mapping) - #43
Conversation
…aire, threat model, enforcement mapping) Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-104-106.watson.ibm.com>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
…tion after each step) or Autonomous (all four steps back-to-back, one final review at the end Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
… threat vectors Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
…t existing Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
…d list against the target's existing Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
640bb95 to
9e5c8c8
Compare
There was a problem hiding this comment.
Nice work!
Can you please look into the following issues, then should be good to go. Additional tests, examples, etc. can be open in a separate PR.
-
Make the workflow discoverable: opa_policy/guidelines-security-analysis/guidelines-security-
analysis.md:1 is not named SKILL.md, and the root SKILL.md:16 never routes security-analysis requests to it. Installed agents therefore won’t invoke this workflow. -
Fix the OPA input path: opa_policy/guidelines-security-analysis/owasp/enforcement_mapping.md:42
consistently usesinput.arguments.*, while Smith’s current canonical representation and policy workflow useinput.args.*. Generated specifications can target nonexistent fields. Please check this with @dhl123 to confirm. -
It looks like opa_policy/guidelines-security-analysis/owasp/
enforcement_mapping.md:334 considers overlapping value sets covered. If guidance blocks {exe} and the candidate blocks {exe, dll}, dll is silently omitted. Shuld coverage require the existing rule to contain the candidate’s full triggering set?
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Summary
Adds the guidelines-security-analysis skill: a 4-step pipeline (architecture analysis -> policy guidance questionnaire -> threat model -> enforcement mapping) that turns a target MCP server's source and OWASP Top 10 for Agentic AI Security (ASI01–ASI10) catalog into a scoped set of security guidelines for OPA policy.
Closes: #issue-number
Changes
opa_policy/guidelines-security-analysis/guidelines-security-analysis.md— new orchestrator doc that sequences the 4 stepsopa_policy/guidelines-security-analysis/owasp/architecture_analysis.md— maps an MCP server's layers, trust boundaries, and OPA enforcement points intoarchitecture.mdopa_policy/guidelines-security-analysis/owasp/policy_guidance_questionnaire.md— derives a 22-question policy-intent questionnaire fromarchitecture.md+guidance.txt/system_vars.json/tool_definitions.jsonwhere presentopa_policy/guidelines-security-analysis/owasp/threat_model.md— evaluates all 10 ASI categories against the tool using a three-question applicability test, sourced live from the catalog rather than hardcodedopa_policy/guidelines-security-analysis/owasp/enforcement_mapping.md— maps each threat instance to an enforcement layer (OPA vs. agent/tool/infra) and produces plain-English policy-rule requirements grounded in the catalog'smitigationssrc/smith/data/owasp_10_ai_catalog.json— new repo-relative data file: the OWASP Top 10 for Agentic AI Security catalog (ASI01–ASI10), single source of truth for category names, descriptions, impact, mitigations, and attack scenariosAll four step docs share a consistent "Authoritative Paths" contract: inputs are read from ONLY the named files under
<TARGET_AGENT_PATH>/smith/guidelines-security-analysis/(or the repo-relative catalog path), with an explicit stop-and-ask instruction if a required file is missing.Checks
make cipasses (lint, Rego lint, license headers, build smoke)make testpasses (policy scorecard — needed if policy behavior changed) — not applicable, noassets/policy.regochanges in this PRCHANGELOG.mdupdated under## [Unreleased](if user-facing)git commit -s) — verified viagit log -1 --format='%B',Signed-off-bypresentNotes (optional)
Base branch should be
hl/promptfoo_config_auto, notmain— this skill was built on top of unreleased work already on that branch. Confirmed clean:git diff --stat hl/promptfoo_config_auto..feat/guidelines-security-analysisshows exactly these 6 files (1392 insertions, 0 deletions), nopolicy_build/CLI/example files leaking in.