Skip to content

Commit 4df1fe5

Browse files
emmanuelgjrclaude
andauthored
data_validation: OWASP 2026 lists, CVSS v4.0, format checks, 2020-12 schemas (#93)
* data_validation: OWASP 2026 lists, CVSS v4.0, format checks, 2020-12 schemas Brings the validation onto current standards: - OWASP Top 10 for LLM Applications 2026 (renumbered) and the Agentic Top 10 (ASI01-ASI10:2026). Schemas accept LLMnn:2025 and LLMnn:2026 and a new owasp_agentic_top10_mapping; reference_data/owasp_top10.csv lists all three with a 2026-to-2025 crosswalk, used to add the 2026 equivalent to every existing LLM mapping (111 files). crossref_validator rejects unknown OWASP IDs and warns on IDs without an edition year. - CVSS v4.0: cvss_v4_score/cvss_v4_vector in the vulnerability schema; the qualitative "Informational" becomes the FIRST rating "None". New severity_check.py recomputes every score from its vector and checks the qualitative rating against the FIRST band. Fixes from it, against NVD: CVE-2024-37032 8.8 is High, not Critical; CVE-2025-0845's vector gives 6.4, not 6.5; vectors added for CVE-2024-34359, CVE-2025-0845 and CVE-2025-32711. - The exploit, vulnerability and risk-assessment validators now enforce "format" (dates, URIs); jsonschema[format-nongpl] provides the URI checker. All current data passes. - incident, rag, promptinj_testcase and crossframework schemas move from draft-07 to 2020-12, matching the rest. - Python 3.11+ (3.10 reaches end of life this month); CI runs the tests on 3.11, 3.12 and 3.13. - reference-data-refresh workflow: monthly ATLAS/CWE regeneration; on a change it pushes a branch and opens an issue (workflows here may not open PRs). - test_standards.py guards the above: schemas are 2020-12, OWASP enums match the reference table, dataset validators enforce format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * data_validation: score CVSS in-house instead of the LGPL cvss package Dependency Review rejected cvss (LGPL-3.0); the repository allows only permissive licences. cvss_score.py computes base scores itself: v3.0/v3.1 from the FIRST specification formulas, v4.0 as a port of cvss40.js, the FIRST reference calculator (BSD-2-Clause, licence and lookup tables included). The port matches the reference on 500 vectors with base, threat and environmental metrics; the v3 scorer matches the NVD/CNA scores of the dataset's CVEs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent d257ec2 commit 4df1fe5

147 files changed

Lines changed: 1704 additions & 271 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/datasets-validate-all.yml‎

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,11 @@ name: datasets-validate-all
22

33
# Runs data_validation/run_all_checks.py: every dataset's own validate.py
44
# (including ones in sub-collections), then the shared checks in
5-
# data_validation/validators/ on every data file. Then the data_validation
6-
# test suite, and the cross-dataset consistency check. The coverage report
7-
# and anomaly list go to the job summary for reviewers; they never fail
8-
# the job.
5+
# data_validation/validators/ on every data file, and the cross-dataset
6+
# consistency check. The tests job runs the data_validation test suite on
7+
# each supported Python. The coverage report and anomaly list go to the job
8+
# summary for reviewers; they never fail the job.
9+
910
on:
1011
push:
1112
branches: [main]
@@ -43,9 +44,6 @@ jobs:
4344
- name: Cross-dataset consistency
4445
run: python data_validation/qc_tools/consistency_check.py
4546

46-
- name: data_validation tests
47-
run: python -m pytest data_validation/tests -v
48-
4947
- name: Coverage report and anomalies (job summary)
5048
if: always()
5149
run: |
@@ -58,3 +56,25 @@ jobs:
5856
python data_validation/qc_tools/anomaly_detector.py
5957
echo '```'
6058
} >> "$GITHUB_STEP_SUMMARY"
59+
60+
# The tooling's own tests on every Python release it supports.
61+
tests:
62+
runs-on: ubuntu-latest
63+
strategy:
64+
fail-fast: false
65+
matrix:
66+
python-version: ['3.11', '3.12', '3.13']
67+
steps:
68+
- name: Checkout
69+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
70+
71+
- name: Set up Python
72+
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
73+
with:
74+
python-version: ${{ matrix.python-version }}
75+
76+
- name: Install requirements
77+
run: python -m pip install --quiet -r data_validation/requirements.txt
78+
79+
- name: data_validation tests
80+
run: python -m pytest data_validation/tests -v
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
name: reference-data-refresh
2+
3+
# Monthly: regenerate the MITRE ATLAS and CWE tables in
4+
# data_validation/reference_data/ from MITRE's latest releases. If anything
5+
# changed, push the result to a branch and open an issue so a maintainer can
6+
# open the PR (this repository does not let workflows open pull requests).
7+
on:
8+
schedule:
9+
- cron: '17 6 2 * *' # 06:17 UTC on the 2nd of each month
10+
workflow_dispatch:
11+
12+
permissions:
13+
contents: write
14+
issues: write
15+
16+
jobs:
17+
refresh:
18+
runs-on: ubuntu-latest
19+
steps:
20+
- name: Checkout
21+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
22+
23+
- name: Set up Python
24+
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
25+
with:
26+
python-version: '3.11'
27+
28+
- name: Install requirements
29+
run: python -m pip install --quiet -r data_validation/requirements.txt
30+
31+
- name: Regenerate ATLAS and CWE tables
32+
run: python data_validation/reference_data/update_reference_data.py
33+
34+
- name: Run the checks against the new tables
35+
id: checks
36+
continue-on-error: true
37+
run: python data_validation/run_all_checks.py --no-shared && python data_validation/validators/crossref_validator.py
38+
39+
- name: Push a branch and open an issue if anything changed
40+
env:
41+
GH_TOKEN: ${{ github.token }}
42+
CHECKS: ${{ steps.checks.outcome }}
43+
run: |
44+
if git diff --quiet -- data_validation/reference_data; then
45+
echo "Reference data is current."
46+
exit 0
47+
fi
48+
branch="automation/reference-data-$(date -u +%Y-%m)"
49+
git config user.name "github-actions[bot]"
50+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
51+
git switch -c "$branch"
52+
git add data_validation/reference_data
53+
git commit -m "reference_data: refresh MITRE ATLAS and CWE tables"
54+
git push --force origin "$branch"
55+
versions=$(grep -E '^\| (MITRE ATLAS|CWE) ' data_validation/reference_data/SOURCES.md | cut -d'|' -f2,4)
56+
{
57+
echo "The monthly refresh regenerated the MITRE ATLAS and CWE tables."
58+
echo
59+
echo "Versions: $versions"
60+
echo
61+
echo "Identifier check against the existing data: **$CHECKS**. A failure usually means MITRE retired or renamed an ID the datasets cite; the crossref output in the run log names the entries."
62+
echo
63+
echo "Open the PR: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/compare/main...${branch}?expand=1"
64+
echo
65+
echo "Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
66+
} > body.md
67+
if gh issue list --state open --search "in:title Reference data refresh $(date -u +%Y-%m)" --json number --jq 'length' | grep -q '^0$'; then
68+
gh issue create --title "Reference data refresh $(date -u +%Y-%m)" --body-file body.md
69+
fi

‎data_validation/README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,8 @@ The tooling has two jobs: keep the research data behind OWASP deliverables (the
4040
|---|---|---|
4141
| `schema_validator.py` | Schema violations, for files no dataset validator covers | File with no schema to check against |
4242
| `dsgai_mapping_check.py` | DSGAI ID not in the taxonomy; a mapping value that is not a DSGAI ID | Same ID twice in one mapping; more than 6 mappings |
43-
| `crossref_validator.py` | Malformed CVE/GHSA/CWE/ATLAS ID; CVE year in the future; unknown CWE or ATLAS ID | Deprecated CWE; retired ATLAS ID; malformed ID in free text (often a placeholder) |
43+
| `crossref_validator.py` | Malformed CVE/GHSA/CWE/ATLAS ID; CVE year in the future; unknown CWE, ATLAS or OWASP Top 10 ID (`LLM01:2026`, `ASI01:2026`) | Deprecated CWE; retired ATLAS ID; OWASP ID without its edition year; malformed ID in free text (often a placeholder) |
44+
| `severity_check.py` | CVSS v3.x/v4.0 vector that doesn't parse; score that doesn't match its vector; qualitative rating outside the score's FIRST band | Score with no vector |
4445
| `anonymization_scanner.py` | — | Email addresses, keys and tokens, private or (in a network context) public IPs, internal hostnames, home paths, SSNs |
4546
| `dedup_checker.py` | Two records in one dataset with the same ID | Records with identical or ≥ 80% similar text |
4647

@@ -75,7 +76,7 @@ After the automated checks pass, contributions are reviewed by cybersecurity and
7576

7677
## Running the checks
7778

78-
Python 3.10 or higher.
79+
Python 3.11 or higher (CI tests 3.11, 3.12 and 3.13).
7980

8081
```bash
8182
pip install -r data_validation/requirements.txt
@@ -111,13 +112,13 @@ data_validation/
111112
├── qc_tools/ Reports for reviewers
112113
├── schemas/ Schemas for datasets without their own schema.json
113114
│ (the rest are pointers to the dataset-local schemas)
114-
├── reference_data/ DSGAI, MITRE ATLAS and CWE lookup tables; see SOURCES.md
115+
├── reference_data/ DSGAI, MITRE ATLAS, CWE and OWASP Top 10 lookup tables; see SOURCES.md
115116
└── tests/ Tests and deliberately valid/invalid fixtures
116117
```
117118

118119
Dataset-local schemas (`datasets/<name>/schema.json`) are authoritative. `schemas/exploit`, `vulnerability`, `riskassessment` and `agentdataflow_trace` are kept only as `$ref` pointers to them; `schemas/incident`, `rag`, `crossframework_mapping` and `promptinj_testcase` are the only schemas for their data.
119120

120-
The ATLAS and CWE tables are generated from MITRE's releases by `reference_data/update_reference_data.py`; don't edit them by hand.
121+
The ATLAS and CWE tables are generated from MITRE's releases by `reference_data/update_reference_data.py`; don't edit them by hand. The `reference-data-refresh` workflow runs it on the 2nd of each month and, when MITRE has published something new, pushes a branch and opens an issue for a maintainer to turn into a PR. `owasp_top10.csv` is maintained by hand when OWASP publishes a new edition.
121122

122123
---
123124

‎data_validation/SETUP.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,17 +17,17 @@ The data validation pipeline is a set of Python scripts that check contributed d
1717

1818
## Prerequisites
1919

20-
### Python 3.10+
20+
### Python 3.11+
2121

22-
The scripts require Python 3.10 or higher. Most modern systems have Python pre-installed.
22+
The scripts require Python 3.11 or higher. Most modern systems have Python pre-installed.
2323

2424
**Check if you have it:**
2525

2626
```bash
2727
python3 --version
2828
```
2929

30-
If you see `Python 3.10.x` or higher, you're good. If not:
30+
If you see `Python 3.11.x` or higher, you're good. If not:
3131

3232
| Platform | How to install |
3333
|---|---|
@@ -95,7 +95,7 @@ You'll know it's active when your terminal prompt changes to show `(venv)` at th
9595
pip install -r requirements.txt
9696
```
9797

98-
> **What this does:** Reads the `requirements.txt` file and installs all the Python libraries the validation scripts need. That is `jsonschema` (schema validation) and `pytest` (the tests), plus `pyyaml`, which only the reference-data refresh script uses.
98+
> **What this does:** Reads the `requirements.txt` file and installs all the Python libraries the validation scripts need. That is `jsonschema` with its format checkers (schema validation, including dates and URIs) and `pytest` (the tests), plus `pyyaml`, which only the reference-data refresh script uses.
9999
100100
**If you get a permissions error:** Make sure your virtual environment is activated (Step 2). If you're not using a virtual environment, add `--user` to the command: `pip install --user -r requirements.txt`
101101

@@ -169,7 +169,7 @@ Python might be installed as `python` instead of `python3` on your system (commo
169169
python --version
170170
```
171171

172-
If that shows 3.10+, use `python` everywhere this guide says `python3`.
172+
If that shows 3.11+, use `python` everywhere this guide says `python3`.
173173

174174
### "No module named 'jsonschema'" (or any other module)
175175

‎data_validation/reference_data/SOURCES.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,11 @@ Lookup tables the validators check identifiers against.
66
|---|---|---|---|
77
| DSGAI | `dsgai_entries.json` | 1.0 | OWASP GenAI Data Security Risks and Mitigations 2026 (kept identical to `datasets/_shared/dsgai_taxonomy.json`; a test enforces it) |
88
| MITRE ATLAS | `mitre_atlas_techniques.csv` | 2026.09 | latest release listed in `https://raw.githubusercontent.com/mitre-atlas/atlas-data/main/dist/manifest.yaml` |
9+
| OWASP Top 10s | `owasp_top10.csv` | LLM 2025, LLM 2026, Agentic 2026 | LLM: `https://github.com/GenAI-Security-Project/GenAI-LLM-Top10` (2026 published 2026-08-04). Agentic: `https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/` (published 2025-12-09) |
910
| CWE | `cwe_ids.csv` | 4.20 | `https://cwe.mitre.org/data/xml/cwec_latest.xml.zip` (weaknesses, categories and views) |
1011

12+
`owasp_top10.csv` is maintained by hand. The 2026 LLM list renumbered the entries (LLM03 was Supply Chain in 2025 and is Excessive Agency in 2026), so IDs always carry the edition year; `equivalent_2025` gives the 2025 ID each 2026 entry continues (LLM08:2026 Hidden Context Exposure broadens LLM07:2025 System Prompt Leakage). OWASP writes the Agentic IDs without a year (ASI01); this repository adds `:2026` so a future edition cannot collide.
13+
1114
Regenerate the ATLAS and CWE tables with `python update_reference_data.py`; it fills in the Version column. Do not edit those two CSVs by hand.
1215

1316
`datasets/_shared/dsgai_taxonomy.json` is the DSGAI list the validators use. `dsgai_entries.json` is kept because the Turkish prompt-injection validator reads it.
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
id,list,edition,name,equivalent_2025
2+
LLM01:2025,LLM,2025,Prompt Injection,
3+
LLM02:2025,LLM,2025,Sensitive Information Disclosure,
4+
LLM03:2025,LLM,2025,Supply Chain,
5+
LLM04:2025,LLM,2025,Data and Model Poisoning,
6+
LLM05:2025,LLM,2025,Improper Output Handling,
7+
LLM06:2025,LLM,2025,Excessive Agency,
8+
LLM07:2025,LLM,2025,System Prompt Leakage,
9+
LLM08:2025,LLM,2025,Vector and Embedding Weaknesses,
10+
LLM09:2025,LLM,2025,Misinformation,
11+
LLM10:2025,LLM,2025,Unbounded Consumption,
12+
LLM01:2026,LLM,2026,Prompt Injection,LLM01:2025
13+
LLM02:2026,LLM,2026,Sensitive Information Disclosure,LLM02:2025
14+
LLM03:2026,LLM,2026,Excessive Agency,LLM06:2025
15+
LLM04:2026,LLM,2026,Supply Chain,LLM03:2025
16+
LLM05:2026,LLM,2026,Data and Model Poisoning,LLM04:2025
17+
LLM06:2026,LLM,2026,Unbounded Consumption,LLM10:2025
18+
LLM07:2026,LLM,2026,Misinformation,LLM09:2025
19+
LLM08:2026,LLM,2026,Hidden Context Exposure,LLM07:2025
20+
LLM09:2026,LLM,2026,Vector and Embedding Weaknesses,LLM08:2025
21+
LLM10:2026,LLM,2026,Improper Output Handling,LLM05:2025
22+
ASI01:2026,ASI,2026,Agent Goal Hijack,
23+
ASI02:2026,ASI,2026,Tool Misuse and Exploitation,
24+
ASI03:2026,ASI,2026,Identity and Privilege Abuse,
25+
ASI04:2026,ASI,2026,Agentic Supply Chain Vulnerabilities,
26+
ASI05:2026,ASI,2026,Unexpected Code Execution,
27+
ASI06:2026,ASI,2026,Memory and Context Poisoning,
28+
ASI07:2026,ASI,2026,Insecure Inter-Agent Communication,
29+
ASI08:2026,ASI,2026,Cascading Failures,
30+
ASI09:2026,ASI,2026,Human-Agent Trust Exploitation,
31+
ASI10:2026,ASI,2026,Rogue Agents,

‎data_validation/requirements.txt‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Validators, QC tools and their tests
2-
jsonschema>=4.18.0
2+
# [format-nongpl] adds the checkers for "uri" and other formats; without it
3+
# jsonschema treats those formats as annotations and never fails them.
4+
jsonschema[format-nongpl]>=4.18.0
35
pytest>=7.0.0
46
# Only for reference_data/update_reference_data.py (refreshing the ATLAS table)
57
pyyaml>=6.0

‎data_validation/run_all_checks.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,8 @@
99
silently passing.
1010
2. Shared checks (data_validation/validators/) on every data file:
1111
schema (for files no dataset validator covers), DSGAI IDs, CVE/CWE/ATLAS
12-
IDs, anonymization patterns and duplicates. Errors fail the run;
12+
and OWASP Top 10 IDs, CVSS scores and ratings, anonymization patterns
13+
and duplicates. Errors fail the run;
1314
heuristic warnings (anonymization, near-duplicates, retired IDs) are
1415
counted, and printed with --verbose.
1516
@@ -34,6 +35,7 @@
3435

3536
from validators import ( # noqa: E402
3637
anonymization_scanner, crossref_validator, dedup_checker, dsgai_mapping_check, schema_validator,
38+
severity_check,
3739
)
3840
from validators._common import ( # noqa: E402
3941
ERROR, SKIP_DIRS, Finding, ensure_utf8_stdout, iter_data_files, load_json, load_taxonomy_ids,
@@ -100,6 +102,7 @@ def shared_checks(dataset: Path, datasets_root: Path) -> tuple[int, list[Finding
100102
findings += schema_validator.validate_file(path, dataset_root=dataset)
101103
findings += dsgai_mapping_check.check_data(path, data, taxonomy)
102104
findings += crossref_validator.check_data(path, data)
105+
findings += severity_check.check_data(path, data)
103106
findings += anonymization_scanner.check_data(path, data)
104107
records, load_errors = dedup_checker.load_records(files)
105108
findings += load_errors + dedup_checker.check_records(records)
@@ -162,7 +165,7 @@ def main(argv: list[str] | None = None) -> int:
162165

163166
checked_files = errors = warnings = 0
164167
if not args.no_shared:
165-
print("\nShared checks (schema, DSGAI IDs, CVE/CWE/ATLAS IDs, anonymization, duplicates):")
168+
print("\nShared checks (schema, DSGAI, CVE/CWE/ATLAS/OWASP IDs, CVSS, anonymization, duplicates):")
166169
for dataset in datasets:
167170
n, findings = shared_checks(dataset, datasets_root)
168171
if not n:

‎data_validation/schemas/crossframework_mapping.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"$schema": "http://json-schema.org/draft-07/schema#",
2+
"$schema": "https://json-schema.org/draft/2020-12/schema",
33
"title": "Cross-Framework Control Mapping Entry",
44
"type": "object",
55
"required": ["dsgai_id", "dsgai_name", "framework", "control_id", "control_name", "relationship"],

‎data_validation/schemas/incident.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"$schema": "http://json-schema.org/draft-07/schema#",
2+
"$schema": "https://json-schema.org/draft/2020-12/schema",
33
"title": "Incident Dataset Entry",
44
"type": "object",
55
"required": ["incident_id", "date", "summary", "dsgai_mapping", "attack_vector", "data_types_affected", "impact"],

0 commit comments

Comments
 (0)