Repository navigation
datasets: validators for rag and incident; point crossframework at the crosswalk - #92
Merged
Merged
Conversation
…e crosswalk
- rag_dataset/validate.py and incident_dataset/validate.py check every
entries/*.json against data_validation/schemas/{rag,incident}.schema.json
(draft-07, with date formats checked), DSGAI IDs against the shared
taxonomy, ID-matches-filename and duplicate IDs. An empty entries/ passes,
so incident_dataset is green until its first entry.
- RAG-0001.json moves to rag_dataset/entries/ and gains a $schema line, like
the other datasets' entries; its content is unchanged.
- rag and incident READMEs: the "TODO: define schema" placeholders now point
at the schema, the entries/ layout and validate.py; incident entries are
JSON only, since Markdown cannot be validated.
- crossframework_mapping_dataset never held data and framework mappings now
live in the GenAI Crosswalk repo, so its README and the datasets table
point there instead of inviting contributions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
emmanuelgjr
added a commit
that referenced
this pull request
Oct 9, 2026
…path) rag_dataset and incident_dataset get their own validate.py in #92, crossframework_mapping_dataset is superseded by the crosswalk repo, and RAG-0001.json moves to rag_dataset/entries/. Update the README dataset table, the example paths in README, SETUP and the validator docstrings, and the suggested first contribution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 tasks done
emmanuelgjr
added a commit
that referenced
this pull request
Oct 9, 2026
… data (#89) * data_validation: implement the validators and QC tools, fix reference data The validators/ and qc_tools/ scripts were stubs that printed "contributions welcome", the tests passed without testing anything, and the MITRE ATLAS reference table had the wrong name for 21 of its 28 IDs. - run_all_checks.py also finds validate.py in dataset sub-collections (the Turkish pairs validator now runs in CI), gives each a timeout, and then runs the shared checks on every data file. Errors fail the run; heuristic hits are warnings (--strict makes them fail). #87's output format and exit codes are kept. - validators/: schema (finds each file's schema; covers datasets with no validate.py, folding in completeness_check), DSGAI IDs, CVE/GHSA/CWE/ATLAS IDs, anonymization patterns, duplicates. Shared helpers in _common.py. - qc_tools/: coverage report, anomaly list, cross-dataset reference check. CI posts the report and anomalies to the job summary. - reference_data/: ATLAS (2026.09, with retired IDs marked) and CWE (4.20) tables regenerated from MITRE by update_reference_data.py; sources in SOURCES.md. Header-only framework control CSVs removed. - datasets/_shared/dsgai_taxonomy.json: DSGAI07 and DSGAI12 use the paper's full titles, matching the README and dsgai_entries.json. - Real tests for every check (82); CI runs the whole suite. - README and SETUP describe what exists; Python 3.10+; unused pandas and regex dependencies dropped; venv/ gitignored as SETUP already claimed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * data_validation: docs follow #92 (rag/incident validators, RAG entry path) rag_dataset and incident_dataset get their own validate.py in #92, crossframework_mapping_dataset is superseded by the crosswalk repo, and RAG-0001.json moves to rag_dataset/entries/. Update the README dataset table, the example paths in README, SETUP and the validator docstrings, and the suggested first contribution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three datasets showed
NO VALIDATORinrun_all_checks.py. This PR closes that gap for two of them and retires the third.rag_dataset and incident_dataset: new
validate.pyBoth follow the pattern of the existing dataset validators. Each one checks every
entries/*.json:data_validation/schemas/(rag.schema.json,incident.schema.json), which per Central schemas in data_validation/schemas/ have drifted from the datasets: decide per file (point, retire, or keep) #76 is the only schema for these datasets; date formats are checked toodatasets/_shared/dsgai_taxonomy.jsonAn empty
entries/passes ("no entries yet"), soincident_datasetstays green until its first contribution. A schema violation, bad date, unknown DSGAI ID, ID/file-name mismatch or duplicate ID each fail (tested on a scratch copy).RAG-0001.json(contributed in #58) moves torag_dataset/entries/and gains a$schemaline, like entries in the other datasets. Its content is unchanged.The two READMEs replace their "TODO: Define schema" placeholders with the schema link, the
entries/layout and thevalidate.pycommand. Incident entries are JSON only now; the README used to also allow Markdown, which can't be validated.crossframework_mapping_dataset: superseded
This folder never held data. Framework mappings live in the GenAI Crosswalk (26 frameworks, 3,771 mappings), and new mapping proposals go there. Its README (which still pointed at a
/mappingsdirectory that no longer exists) and the datasets table now say so. The folder and its schema stay, so existing links still resolve.Verification
python validate.pypasses in both datasets (rag: 1 entry; incident: no entries yet).python data_validation/run_all_checks.py: 6 validators pass. Onlycrossframework(superseded) andpromptinjremain without a top-level validator; data_validation: implement the validators and QC tools, fix reference data #89 runs the promptinj sub-collection validator.Merge order
Merge this before #89: #89's README and docstring examples will be updated to the new
entries/RAG-0001.jsonpath.🤖 Generated with Claude Code