Skip to content

feat(schema): optional verification_method field + Agentic_AIUC1 pilot - #186

Open
a-moskvin wants to merge 2 commits into
GenAI-Security-Project:mainfrom
a-moskvin:feat/verification-method
Open

a-moskvin wants to merge 2 commits into
GenAI-Security-Project:mainfrom
a-moskvin:feat/verification-method

Conversation

@a-moskvin

Copy link
Copy Markdown

What this PR changes

Refs #101. Adds an optional verification_method field to the schema v2 row format and pilots it in one file.

  • scripts/generate.js: parses a "Verification method" column as a v2 metadata column (stored as verification_method, excluded from notes)
  • data/schema.json, src/index.ts: optional string field
  • docs/SCHEMA_V2_MIGRATION.md: field documented
  • scripts/generate.test.mjs: regression test — fails if method text lands in notes
  • agentic-top10/Agentic_AIUC1.md: column added to all 10 tables (ASI01–ASI10); 22 of 26 requirement-level rows filled, all DRAFT
  • Regenerated: data/entries/ASI*.json (9), docs/data.js

Type of change

  • New mapping file
  • Update to existing mapping (content, controls, CVE refs)
  • Bug fix (broken link, typo, incorrect cross-ref)
  • New recipe (shared/RECIPES.md)
  • New tool (shared/TOOLS.md)
  • Infrastructure (scripts, CI, templates)
  • Translation (i18n/)

Source / evidence

Methods adapted from the NPW Agentic AI Control Catalogue v2.4.0 (CC BY-SA 4.0): https://www.newpacificway.com/ai-controls.

Checklist

Content

  • Follows the file template structure (header comment, H1, Why section, quick-reference table, audience
    tags, detailed per-entry mappings, references, changelog)
  • Severity ratings consistent with AIVSS / OWASP definitions in shared/SEVERITY.md
  • Cross-references are bidirectional — if this file mentions Agentic_X.md, that file mentions this one back
  • All referenced vulnerability IDs are valid (LLM01–LLM10, ASI01–ASI10, DSGAI01–DSGAI21)
  • License header present: CC BY-SA 4.0

Links & data

  • All internal .md links resolve to real files
  • All external URLs return 200 (checked manually or via lychee)
  • data/schema.json compatible (if adding a new entry type)

Project hygiene

  • Changelog entry added at bottom of every modified file (YYYY-MM-DD format)
  • CHANGELOG.md updated if this is a new mapping file (include in correct version section)
  • README.md counts updated if file count changed (badge + summary table + section heading + repo tree)
  • Ran node scripts/validate.js --file <path> locally and it passes

For new mapping files only

  • Added to correct section in README.md mapping table with "Standout content" description
  • Added to CROSSREF.md primary frameworks column where relevant
  • Framework coverage matrix in README.md updated (✅ for the new cell)
  • File named correctly: SourceList_Framework.md (e.g., Agentic_SAMM.md)

Notes for reviewers

  • Non-breaking: with no file carrying the column, the schema commit regenerates nothing.

  • DRAFT status: every value awaits SME review, like relationship/confidence.

  • Intentional gaps: 12 domain-level rows (A, C, D, E, F) are out of scope as too generic. 4 rows (ASI04/B003, ASI05/B005, ASI09/B009, and ASI10/B006) have no defensible method. Contributions welcome.

  • Attribution: text after the (NPW …) citation is contributor-authored.

  • One source: all methods come from one catalogue. Methods from other sources, each with its own citation, are welcome.

  • OLIR: the field is not exported; OLIR has no corresponding field.

  • Open design question: I suggest a many-to-many model of mapping controls to verification methods. Happy to propose it as a follow-up issue if this pilot is accepted.

@a-moskvin
a-moskvin requested a review from emmanuelgjr as a code owner October 1, 2026 07:48
@emmanuelgjr

Copy link
Copy Markdown
Contributor

Thanks, @a-moskvin. This matches the pilot shape from #101: optional, one file, cited, and every value DRAFT.

Verification of this branch (ca89494), local runs:

  • Against current main: validate 0 errors (93 warnings, same as main), stats:check current, 90/90 unit tests including your new one, and a second generate.js produces no drift.
  • Exports are unaffected. compliance-report.js copies notes into the CSV and OSCAL output, so keeping the column out of notes matters, and your test guards it. The AIUC-1 OSCAL and CSV exports from this branch contain no verification text.
  • Citations: I checked every method phrase against the live catalogue page. 36 of 37 phrase uses appear verbatim, and every cited control id (C02, H08, E09, …) is the control the phrase sits under. The one I couldn't find is the ASI04/B001 method ("Confirm tools, connectors and MCP servers are in the scope of…"). It carries no NPW citation, so I read it as your own text, which is fine as a DRAFT.

Nits, non-blocking:

  • git diff --check flags trailing whitespace on three added lines (data/schema.json:170 and scripts/generate.js:426 and :623), and two added lines in v2HeaderIndex are indented with tabs where the file uses spaces.
  • The stored value keeps the DRAFT — prefix in verification_method. That's consistent with the cell, and it means consumers can see the review state, so I'd leave it. Just noting that the prefix is part of the data.

On the many-to-many follow-up: please open it as an issue once this pilot settles, so it's discussed separately from this PR.

Merge-order note. This PR, #185 and #187 all regenerate shared files. I simulated main + #184 + #185 + this PR + #187. This PR merges cleanly in that stack. The only conflict is data/stats.json, between #184 and #187. After regenerating: 0 errors, 93/93, no drift.

CI hasn't run here. Workflows on fork PRs wait for a maintainer to approve them, so every result above is from a local run, not a GitHub check.

@emmanuelgjr

Copy link
Copy Markdown
Contributor

Following up after a closer look at where this fits long term. My first comment only checked correctness. This one is about direction, and it changes my earlier "merge-ready" read. The maintainer has decided to hold this PR in its current shape. The idea isn't being turned down: a "how do I check it" layer is a real gap. SCF ships Assessment Objectives and an Evidence Request List, and CSA's AI Controls Matrix has auditing guidelines. We want this to be done in a shape that lasts. Three things stand in the way:

1. AIUC-1 already publishes its own verification layer. The standard has official evidence for each requirement (standard.aiuc-1.com/evidence, e.g. "B001.1 Report: …"). It's versioned quarterly and audited by accredited firms. An AIUC-1 assessor will work from that list, so methods from another catalogue would sit beside the canonical ones as competing guidance. The pilot should either:

  • cite AIUC-1's own evidence ids as the primary source, with NPW as supplementary, or
  • move to a framework that has no verification layer of its own. That's where it adds the most.

(While checking this we found that the repo's AIUC-1 registry names the wrong publisher, licence and URL: #189. That one's ours to fix, not yours.)

2. The method mostly belongs to the control, not the risk–control pair. B001's text repeats across five entries with small variations. Free text on each row will duplicate and drift. The review state (DRAFT — ) and the citation ((NPW C02)) are also inside the prose, so tools can't read them, and no field records who reviewed it. Your many-to-many suggestion points to the better model:

  • methods held once, on the control in the framework registry (data/frameworks/<fw>.json);
  • structured, e.g. { "text": …, "source": "NPW", "source_id": "C02", "status": "draft", "reviewed_by": [] };
  • an optional per-row override for the risk-specific part (e.g. "repeat for several users with different permissions").

3. The rows underneath aren't reviewed yet. Agentic_AIUC1.md is still a legacy table (no relationship or confidence columns), and all 147 AIUC-1 rows are unreviewed. A second DRAFT judgment on top of them doubles what a reviewer has to clear. A schema v2 file is the better home for a schema v2 field.

Proposed next step: open the many-to-many follow-up as an issue now, with the data model above (or your counter-proposal), and settle it there first. Once it's agreed, a reworked PR can land against the agreed shape. Most of your work carries over: the parser and the test, and the 36 method phrases that trace verbatim to NPW controls all port directly.

One more thing, for transparency: since NPW is your catalogue, please say so in the file header and the PR when the reworked version lands. That's normal open-source practice and not a concern in itself; we just apply it to everything sourced from a single author.

Leaving this open so the history stays in one place. Thanks for the careful work and for raising #101.

@a-moskvin

Copy link
Copy Markdown
Author

Hi @emmanuelgjr ,

Thank you for the feedback, the direction makes sense.

Agreed on all three points:

  • AIUC-1's evidence layer is a fair catch: I should have checked it. Will pick another framework without its own verification layer, so our work has more value.
  • I will open a dedicated issue on the data model for verification methods linked to controls. Will keep feat(schema): optional verification_method field + Agentic_AIUC1 pilot #186 open as history until the new PR lands.
  • Will add a note about NPW catalogue ownership.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants