data_validation: implement the validators and QC tools, fix reference data

[emmanuelgjr]

Summary

Builds on #87. Before this PR, data_validation/ mostly described checks that didn't exist:

• All six validators/ scripts and all three qc_tools/ scripts were stubs that printed "contributions welcome".
• The tests passed without testing anything.
• reference_data/mitre_atlas_techniques.csv had the wrong name for 21 of its 28 IDs. For example, AML.T0046 was listed as "Prompt Injection - Direct" but is officially "Spamming AI System with Chaff Data".
• The runner from data_validation: make run_all_checks.py run every dataset validator #87 missed the Turkish prompt-injection validator because it sits in a sub-folder.

Runner (run_all_checks.py)

• Also finds a validate.py inside a dataset's sub-folders, so promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs now runs in CI.
• Each dataset validator has a timeout (default 300 s, --timeout).
• After the dataset validators, it runs the shared checks on every data file. Errors fail the run. Warnings are heuristic hits; they're counted, listed with --verbose, and fail the run only with --strict.
• data_validation: make run_all_checks.py run every dataset validator #87's output lines, summary line and exit codes are unchanged, and its 5 tests still pass.

Shared checks (validators/)

Check	ERROR	WARN
schema_validator	Schema violations, for files no dataset validator covers (rag, altaysec cases). Each file's schema is found from its $schema key, then the nearest schema.json, then data_validation/schemas/	File with no schema
dsgai_mapping_check	DSGAI ID not in _shared/dsgai_taxonomy.json	Same ID twice in one mapping; more than 6 mappings
crossref_validator	Malformed CVE/GHSA/CWE/ATLAS ID; unknown CWE or ATLAS ID	Deprecated CWE; retired ATLAS ID; malformed ID in free text
anonymization_scanner	—	Emails, keys and tokens, IP addresses, internal hosts (same patterns as the agentdataflow validator)
dedup_checker	Two records with the same ID	Identical or ≥ 80% similar text

completeness_check.py is removed: required fields and allowed values are exactly what schema validation checks.

QC tools (qc_tools/)

• bias_report: DSGAI × dataset coverage matrix and distributions of severity and category fields.
• anomaly_detector: one DSGAI entry dominating a dataset, records with far more mappings than usual, skewed fields.
• consistency_check: fails on dangling internal references (DSGAI-VULN-…, DSGAI-EXP-…, DSGAI-RA-…).
• CI writes the coverage report and the anomaly list to the job summary.

Reference data

• update_reference_data.py regenerates the ATLAS table (release 2026.09, 208 techniques, with 3 retired IDs marked) and the CWE table (4.20, 1,450 entries) from MITRE's published data. Versions are recorded in reference_data/SOURCES.md.
• The five header-only framework_control_ids/*.csv files and the 11-row NIST CSF sample are removed. Nothing read them, and complete ISO and CIS control lists can't be redistributed.
• datasets/_shared/dsgai_taxonomy.json: DSGAI07 and DSGAI12 now use the paper's full titles, matching the main README and dsgai_entries.json. A test keeps the two DSGAI lists in sync.

Tests, CI and docs

• 82 real tests (they were placeholders before), and CI now runs the whole data_validation/tests suite. The fixtures are updated to the current schemas.
• README and SETUP describe what actually exists. Python requirement is now 3.10+, and the unused pandas and regex dependencies are dropped.
• venv/ is now gitignored (SETUP already claimed it was).

Results on current data

• 5 dataset validators pass; 3 datasets still have no validator (rag, incident, crossframework).
• Shared checks over 587 files: 0 errors, 9 warnings.
  • 8 warnings: exploit entries citing ATLAS IDs that ATLAS 2026.07 retired. AML.T0019, AML.T0058 and AML.T0104 were merged into AML.T0115.000–.002 ("Publish Poisoned AI Artifacts"). Affected entries: AML.T0019.json, AML.T0058.json, AML.T0110.json, DSGAI-EXP-rome-targeted-fact-editing.json.
  • 1 warning: tr_altayduel_testcases.json has no schema. It's an aggregate file, and its 5 cases don't match promptinj_testcase.schema.json.
• Consistency check: no dangling references, and every CVE cited by an exploit entry has its own vulnerability entry.
• Verified on Python 3.12 and 3.13. agentdataflow's own tests (18) still pass.

Not in this PR

• Remapping the exploit entries that cite retired ATLAS IDs: renaming exploit_id renames files, so it should be a separate dataset PR.
• Validators for rag, incident and crossframework (the shared schema check covers their data for now).

Checklist

• Validation scripts pass (python data_validation/run_all_checks.py)
• Tests: python -m pytest data_validation/tests (82 passed)

🤖 Generated with Claude Code

[emmanuelgjr]

Updated to match #92: rag and incident now have their own validate.py, crossframework is superseded by the crosswalk repo, and the example paths use rag_dataset/entries/RAG-0001.json. Suggested merge order: #90, #91, #92, then this PR. Once all four are in, run_all_checks.py should report no shared-check warnings: #90 removes the retired ATLAS IDs and #91 replaces the schema-less AltayDuel file. The only exceptions are the two intentional old-ID mentions in #90's notes fields.