Skip to content

feat: security analysis guidelines - #44

Open
iamsreec wants to merge 31 commits into
mainfrom
sc/security-analysis
Open

feat: security analysis guidelines#44
iamsreec wants to merge 31 commits into
mainfrom
sc/security-analysis

Conversation

@iamsreec

@iamsreec iamsreec commented Aug 21, 2026

Copy link
Copy Markdown
Member

Summary

Adds the guidelines-security-analysis skill: a 4-step pipeline (architecture analysis -> policy guidance questionnaire -> threat model -> enforcement mapping) that turns a target MCP server's source and OWASP Top 10 for Agentic AI Security (ASI01–ASI10) catalog into a scoped set of security guidelines for OPA policy.

Closes: #issue-number

Changes

  • opa_policy/guidelines-security-analysis/guidelines-security-analysis.md — new orchestrator doc that sequences the 4 steps
  • opa_policy/guidelines-security-analysis/owasp/architecture_analysis.md — maps an MCP server's layers, trust boundaries, and OPA enforcement points into architecture.md
  • opa_policy/guidelines-security-analysis/owasp/policy_guidance_questionnaire.md — derives a 22-question policy-intent questionnaire from architecture.md + guidance.txt/system_vars.json/tool_definitions.json where present
  • opa_policy/guidelines-security-analysis/owasp/threat_model.md — evaluates all 10 ASI categories against the tool using a three-question applicability test, sourced live from the catalog rather than hardcoded
  • opa_policy/guidelines-security-analysis/owasp/enforcement_mapping.md — maps each threat instance to an enforcement layer (OPA vs. agent/tool/infra) and produces plain-English policy-rule requirements grounded in the catalog's mitigations
  • src/smith/data/owasp_10_ai_catalog.json — new repo-relative data file: the OWASP Top 10 for Agentic AI Security catalog (ASI01–ASI10), single source of truth for category names, descriptions, impact, mitigations, and attack scenarios

All four step docs share a consistent "Authoritative Paths" contract: inputs are read from ONLY the named files under <TARGET_AGENT_PATH>/smith/guidelines-security-analysis/ (or the repo-relative catalog path), with an explicit stop-and-ask instruction if a required file is missing.

Checks

  • make ci passes (lint, Rego lint, license headers, build smoke)
  • make test passes (policy scorecard — needed if policy behavior changed) — not applicable, no assets/policy.rego changes in this PR
  • CHANGELOG.md updated under ## [Unreleased] (if user-facing)
  • Commits are signed off for the DCO (git commit -s) — verified via git log -1 --format='%B', Signed-off-by present

Notes (optional)

Base branch should be hl/promptfoo_config_auto, not main — this skill was built on top of unreleased work already on that branch. Confirmed clean: git diff --stat hl/promptfoo_config_auto..feat/guidelines-security-analysis shows exactly these 6 files (1392 insertions, 0 deletions), no policy_build/CLI/example files leaking in.

Sai Sree Laya Chukkappalli and others added 17 commits August 19, 2026 11:41
…aire, threat model, enforcement mapping)

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-104-106.watson.ibm.com>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
…tion after each step) or Autonomous (all four steps back-to-back, one final review at the end

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
… threat vectors

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
…t existing

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
…d list against the target's existing

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
…y analysis steps

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
@iamsreec iamsreec changed the title Sc/security analysis feat: security analysis guidelines Aug 21, 2026

@araujof araujof left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sai Sree Laya Chukkappalli and others added 9 commits August 25, 2026 13:35
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@dhcp-9-31-107-214.watson.ibm.com>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Sai Sree Laya Chukkappalli added 3 commits August 26, 2026 11:01
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
@araujof
araujof self-requested a review August 26, 2026 18:34

@araujof araujof left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

Can you please make the following changes before merging?

  • opa_policy/guidelines-security-analysis/steps/enforcement_mapping.md:42: Smith uses input.args.*, not input.arguments.*. Using the wrong path here could produce rules for fields that do not exist. Please update the workflow for consistency.

  • opa_policy/guidelines-security-analysis/steps/enforcement_mapping.md:319: Step 8 assumes guidance contains numbered rules, but most bundled examples use headings and bullets. Since Step E overwrites guidance.txt, this could potentially discard user-written context. What if we preserved the original file and appended only missing rules?

  • opa_policy/guidelines-security-analysis/guidelines-security-analysis.md:149: This says to include non-OPA gaps in guidance_updated.txt, while Step 8 explicitly forbids it. Please choose one behavior; keeping those gaps only in owasp_policy_guidelines.md seems safest.

…erwrite) guidance.txt, drop gap register from guidance_updated.txt

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
@iamsreec

Copy link
Copy Markdown
Member Author

Updated based on the comments.

@araujof

araujof commented Aug 26, 2026

Copy link
Copy Markdown
Member

Updated based on the comments.

Thanks! I just checked, and it looks like some of the issues have some residuals:

  • Guidance preservation: Step E now appends, but SKILL.md still says to overwrite, Step 8 still assumes numbered rules, and bundled guidance_updated.txt examples still contain full replacements.

  • Gap-register conflict: fixed in the core workflow, but the root and example READMEs still say non-OPA findings are added to guidance_updated.txt.

…KILL/READMEs, teach STEP 8 to handle prose-with-headers guidance.txt

Signed-off-by: Sai Sree Laya Chukkappalli <saisree@Sais-MacBook-Pro.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants