Skip to content

data_validation: OWASP 2026 lists, CVSS v4.0, format checks, 2020-12 schemas - #93

Merged
emmanuelgjr merged 2 commits into
mainfrom
data-validation/standards-refresh
Oct 9, 2026
Merged

emmanuelgjr merged 2 commits into
mainfrom
data-validation/standards-refresh

Conversation

@emmanuelgjr

Copy link
Copy Markdown
Contributor

Summary

This PR implements items 1–6 and 14 of the data-validation review: the places where the validation lagged current standards or silently skipped a check.

# Change
1 OWASP Top 10 for LLM Applications 2026. The 2026 edition (published 2026-08-04, canonical repo) renumbered the list; for example, LLM03 changed from Supply Chain to Excessive Agency. Schemas now accept LLMnn:2025 and LLMnn:2026, always with the year. reference_data/owasp_top10.csv holds both editions plus a 2026→2025 crosswalk, which I used to add the 2026 equivalent to every existing mapping (111 files, 163 IDs; nothing removed).
2 OWASP Top 10 for Agentic Applications 2026. New optional owasp_agentic_top10_mapping field (ASI01:2026…ASI10:2026) in the exploit, vulnerability and agent-trace schemas. OWASP writes these IDs without a year; the suffix keeps a future edition from colliding with this one, as just happened with the LLM list.
3 CVSS v4.0. Adds cvss_v4_score and cvss_v4_vector. The qualitative rating "Informational" becomes FIRST's "None" (no entry used it). A new severity_check.py recomputes every score from its vector (v3.0/3.1/4.0) and checks that the qualitative rating matches the score's FIRST band. Data fixes found by the check, against NVD:
• CVE-2024-37032: 8.8 is High, not Critical.
• CVE-2025-0845: the published vector gives 6.4, not 6.5.
• Vectors added for CVE-2024-34359, CVE-2025-0845 and CVE-2025-32711 (the CNA vector, which matches the stored 9.3).
4 Format enforcement. The exploit, vulnerability and risk-assessment validators built their validator without a format checker, so "format": "date" and "uri" were never checked. They now check both, and jsonschema[format-nongpl] supplies the URI checker. All current data passes.
5 JSON Schema 2020-12 for incident, rag, promptinj_testcase and crossframework, matching the other datasets. The Turkish pairs validator still passes against the migrated promptinj schema.
6 Python 3.11+, because 3.10 reaches end of life this month. CI runs the tests on 3.11, 3.12 and 3.13.
14 reference-data-refresh workflow. Runs on the 2nd of each month (and on demand) and regenerates the ATLAS and CWE tables. If anything changed, it pushes automation/reference-data-YYYY-MM and opens an issue with the compare link: this repo doesn't let workflows open PRs (can_approve_pull_request_reviews: false), and I didn't loosen that setting.

The new tests/test_standards.py fails if a schema drifts from 2020-12, if the OWASP enums stop matching the reference table, or if a dataset validator drops its format checker.

Verification

  • run_all_checks.py: 7 validators pass. The shared checks find 0 errors across 591 files; the only 2 warnings are the intentional retired-ID notes from exploit_dataset: move retired ATLAS IDs to AML.T0115 sub-techniques #90.
  • 122 tests pass on Python 3.12 and 3.13 (1 skipped: the contributor-maintained contrastive schema). agentdataflow's own 18 tests pass.
  • actionlint is clean on both workflows.

🤖 Generated with Claude Code

…schemas

Brings the validation onto current standards:

- OWASP Top 10 for LLM Applications 2026 (renumbered) and the Agentic Top
  10 (ASI01-ASI10:2026). Schemas accept LLMnn:2025 and LLMnn:2026 and a new
  owasp_agentic_top10_mapping; reference_data/owasp_top10.csv lists all
  three with a 2026-to-2025 crosswalk, used to add the 2026 equivalent to
  every existing LLM mapping (111 files). crossref_validator rejects
  unknown OWASP IDs and warns on IDs without an edition year.
- CVSS v4.0: cvss_v4_score/cvss_v4_vector in the vulnerability schema; the
  qualitative "Informational" becomes the FIRST rating "None". New
  severity_check.py recomputes every score from its vector and checks the
  qualitative rating against the FIRST band. Fixes from it, against NVD:
  CVE-2024-37032 8.8 is High, not Critical; CVE-2025-0845's vector gives
  6.4, not 6.5; vectors added for CVE-2024-34359, CVE-2025-0845 and
  CVE-2025-32711.
- The exploit, vulnerability and risk-assessment validators now enforce
  "format" (dates, URIs); jsonschema[format-nongpl] provides the URI
  checker. All current data passes.
- incident, rag, promptinj_testcase and crossframework schemas move from
  draft-07 to 2020-12, matching the rest.
- Python 3.11+ (3.10 reaches end of life this month); CI runs the tests on
  3.11, 3.12 and 3.13.
- reference-data-refresh workflow: monthly ATLAS/CWE regeneration; on a
  change it pushes a branch and opens an issue (workflows here may not open
  PRs).
- test_standards.py guards the above: schemas are 2020-12, OWASP enums
  match the reference table, dataset validators enforce format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ❌ 1 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ✅ 0 package(s) with unknown licenses.
See the Details below.

License Issues

data_validation/requirements.txt

PackageVersionLicenseIssue Type
cvss>= 3.2LGPL-3.0 AND LGPL-3.0-only AND LGPL-3.0-or-laterIncompatible License
Allowed Licenses: Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, MIT, MPL-2.0, CC-BY-4.0, CC0-1.0, Unlicense

OpenSSF Scorecard

PackageVersionScoreDetails
actions/actions/checkout 3d3c42e5aac5ba805825da76410c181273ba90b1 🟢 6.3
Details
CheckScoreReason
Maintained🟢 45 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 4
Code-Review🟢 10all changesets reviewed
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Pinned-Dependencies🟢 3dependency not pinned by hash detected -- score normalized to 3
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 9security policy file detected
Packaging⚠️ -1packaging workflow not detected
SAST🟢 10SAST tool is run on all commits
Branch-Protection🟢 6branch protection is not maximal on development and all release branches
actions/actions/setup-python 5fda3b95a4ea91299a34e894583c3862153e4b97 🟢 6.6
Details
CheckScoreReason
Maintained🟢 1017 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Binary-Artifacts🟢 10no binaries found in the repo
Code-Review🟢 10all changesets reviewed
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies🟢 7dependency not pinned by hash detected -- score normalized to 7
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 9security policy file detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
SAST🟢 9SAST tool is not run on all commits -- score normalized to 9
pip/cvss >= 3.2 UnknownUnknown

Scanned Files

  • .github/workflows/reference-data-refresh.yml
  • data_validation/requirements.txt

# Contributor-maintained schema for one sub-collection; its own validator owns it.
CONTRIBUTOR_SCHEMAS = {"contrastive_testcase.schema.json"}

OWASP = list(csv.DictReader(open(REFERENCE_DIR / "owasp_top10.csv", newline="", encoding="utf-8")))
Dependency Review rejected cvss (LGPL-3.0); the repository allows only
permissive licences. cvss_score.py computes base scores itself: v3.0/v3.1
from the FIRST specification formulas, v4.0 as a port of cvss40.js, the
FIRST reference calculator (BSD-2-Clause, licence and lookup tables
included). The port matches the reference on 500 vectors with base,
threat and environmental metrics; the v3 scorer matches the NVD/CNA scores
of the dataset's CVEs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@emmanuelgjr

Copy link
Copy Markdown
Contributor Author

Dependency Review flagged the cvss package as LGPL-3.0, which the repo's licence policy doesn't allow. I left the policy alone and replaced the package with validators/cvss_score.py:

  • v3.0/v3.1 use the FIRST specification formulas.
  • v4.0 is a port of FIRST's BSD-2-Clause reference calculator (cvss40.js); its licence and lookup tables are included.
  • The port matches the reference on 500 vectors (tests/test_cvss_score.py), and the v3 scorer reproduces the NVD/CNA scores of the dataset's CVEs.

The tooling is back to one runtime dependency (jsonschema).

@emmanuelgjr
emmanuelgjr merged commit 4df1fe5 into main Oct 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant