datasets: define the agent-trace schema, validator and format docs

agentdataflow_toolexchange_traces/ has been open for contributions with a
TODO where its data format should be. This defines the format and ships the
tooling that enforces it. Entries follow in a separate PR, so the format can
be reviewed without the data riding along.

provenance.tier records how a trace was obtained - sanitized production, lab
or CTF, derived from a documented public fact, or hypothetical - because the
people who hold real agent traces mostly cannot publish them. Every tier
above hypothetical requires evidence a reader can check without the
contributor, enforced by validate.py, so the kinds of contribution never
pretend to be one another. A contributor-controlled source does not
establish a precondition at any tier; that rule cannot be checked
mechanically and is stated as a review-time obligation in the README and
schema, not implied to be automated.

The category enum matches the values already used by the repo's central
data_validation/schemas/agentdataflow_trace.schema.json. Where this schema
diverges from that one, the divergence is deliberate and set out in the pull
request description.

validate.py checks schema conformance, the span graph (including that a
span cannot name itself or a later span as its parent), citation form per
evidence type, and ISO dates on date_added/date_observed and every span
timestamp in every environment, and scans every string in an entry - not
only span payloads - for credential-, secret-, and internal-hostname-shaped
values and for clusters of addresses that look like real routable public
IPv4 addresses, independently of the contributor's sanitization attestation.
It passes with entries/ empty, validating example.json on its own.

A GitHub Actions workflow runs validate.py and checks index.csv is current,
path-filtered to this dataset, so the checks above run on every PR rather
than depending on a contributor having run them locally.

Author: gautamgb

.github/workflows/dataset-agentdataflow-validate.yml

@@ -0,0 +1,46 @@
+name: dataset-agentdataflow-validate
+
+# Runs the dataset-local validator for agentdataflow_toolexchange_traces on
+# every change to it, so "enforced by validate.py" in that dataset's README
+# is true of the merged tree and not only of whoever remembered to run it
+# locally. Path-filtered to the dataset, same pattern as scanner-selftest.
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'datasets/agentdataflow_toolexchange_traces/**'
+      - 'datasets/_shared/**'
+      - '.github/workflows/dataset-agentdataflow-validate.yml'
+  pull_request:
+    paths:
+      - 'datasets/agentdataflow_toolexchange_traces/**'
+      - 'datasets/_shared/**'
+      - '.github/workflows/dataset-agentdataflow-validate.yml'
+
+permissions:
+  contents: read
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - name: Set up Python
+        uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
+        with:
+          python-version: '3.11'
+
+      - name: Install jsonschema
+        run: python -m pip install --quiet jsonschema
+
+      - name: Validate entries and example against schema.json
+        run: python validate.py
+        working-directory: datasets/agentdataflow_toolexchange_traces
+
+      - name: index.csv is up to date with entries/
+        run: |
+          python build_index.py
+          git diff --exit-code index.csv
+        working-directory: datasets/agentdataflow_toolexchange_traces

datasets/agentdataflow_toolexchange_traces/README.md

@@ -10,39 +10,113 @@ Structured traces capturing how data flows through agentic AI systems — betwee
 
 Categories include:
 
-- **Tool call traces** — Sequences of tool invocations showing what data is sent to and received from external tools (APIs, databases, file systems, code execution environments)
-- **Multi-agent delegation chains** — Traces showing how tasks and data are passed between agents, including what context is forwarded, filtered, or accumulated
-- **Plugin data exchange logs** — Records of data shared with third-party plugins, including request/response payloads and metadata
-- **Credential and permission flows** — How credentials, tokens, and permissions are scoped, delegated, and consumed across agent workflows
-- **Context accumulation patterns** — Traces showing how agent context windows grow over multi-step tasks, what data persists across steps, and where over-sharing occurs
-- **Memory read/write traces** — How agents interact with short-term and long-term memory stores, including what data is persisted and retrieved
+- **Tool call traces** (`tool_call`) — Sequences of tool invocations showing what data is sent to and received from external tools (APIs, databases, file systems, code execution environments)
+- **Multi-agent delegation chains** (`multi_agent_delegation`) — Traces showing how tasks and data are passed between agents, including what context is forwarded, filtered, or accumulated
+- **Plugin data exchange logs** (`plugin_data_exchange`) — Records of data shared with third-party plugins, including request/response payloads and metadata
+- **Credential and permission flows** (`credential_flow`) — How credentials, tokens, and permissions are scoped, delegated, and consumed across agent workflows
+- **Context accumulation patterns** (`context_accumulation`) — Traces showing how agent context windows grow over multi-step tasks, what data persists across steps, and where over-sharing occurs
+- **Memory read/write traces** (`memory_read_write`) — How agents interact with short-term and long-term memory stores, including what data is persisted and retrieved
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `schema.json` | JSON Schema (draft 2020-12) every entry validates against |
+| `example.json` | Worked example. Read it with the schema before writing an entry |
+| `entries/` | One trace per file, named `<trace_id>.json`. Empty until the first entry merges; `validate.py` passes against `example.json` alone in the meantime. |
+| `validate.py` | `python validate.py` - schema, provenance, span graph, and secret-scan checks |
+| `build_index.py` | `python build_index.py` - regenerates `index.csv` |
+| `index.csv` | Flat index of all entries, for filtering and citation. Generated once `entries/` is non-empty. |
 
 ## Data Format
 
-<!-- TODO: Define schema once initial data is contributed -->
+One trace per file in `entries/`, named after its `trace_id` (`DSGAI-TRACE-<slug>.json`). The authoritative definition is `schema.json`; the summary below is orientation.
 
-Contributions should include:
+Every entry carries:
 
-- **Trace ID**
-- **Category** — From the list above
-- **DSGAI mapping** — Primary DSGAI entries relevant to this trace
-- **Agent framework** — LangGraph, AutoGPT, CrewAI, custom, etc. (if disclosable)
-- **Trace data** — The sequence of events, tool calls, and data exchanges in structured format (JSON, JSONL, or OpenTelemetry-compatible spans)
-- **Data sensitivity annotations** — Flag any steps where sensitive data is present, over-shared, or inadequately scoped
-- **Security observations** — What data security risks this trace illustrates
-- **Benign / adversarial** — Whether this is a normal workflow trace or one demonstrating a security failure
+- **`trace_id`, `title`, `description`** - what flows, and what makes it security-relevant
+- **`category`** - one of the six above
+- **`disposition`** - `benign`, `adversarial`, or `unintentional_failure`
+- **`dsgai_mapping`** - DSGAI entries this trace illustrates, primary risk first
+- **`provenance`** - where the trace came from, and what backs it (see below)
+- **`sanitization`** - a hard attestation plus the techniques applied
+- **`spans`** - the trace itself: an ordered sequence of events
+- **`security_observations`** - what the trace demonstrates, one falsifiable claim per item
+
+Optional: `agent` (framework, protocol, topology, autonomy), `owasp_llm_top10_mapping`, `mitre_atlas_mapping`, `mitigations`, `contributor`, `tags`, `notes`.
+
+### Spans
+
+A span is one event. The shape is OpenTelemetry-compatible without requiring an OTel pipeline to produce it, so a trace can be exported from a real system or written by hand and still validate the same way.
+
+```json
+{
+  "span_id": "s7",
+  "parent_span_id": "s6",
+  "t_offset_ms": 176400910,
+  "actor": "tool",
+  "actor_id": "tool://server-b/lookup_record",
+  "operation": "tool.result",
+  "summary": "Tool returns the full record, three fields wider than the schema the agent bound to.",
+  "payload": { "contact_email": "<synthetic:email>", "internal_notes": "<shape:string,len=1840>" },
+  "data_classes": ["tool_output", "pii"],
+  "sensitivity": "high",
+  "finding": {
+    "dsgai_id": "DSGAI01",
+    "note": "Entitlement was enforced once, against a schema that has since changed.",
+    "severity": "High"
+  }
+}
+```
+
+Two conventions carry most of the weight:
+
+**Payloads are shapes, not content.** String values in `payload` should be typed placeholders - `<synthetic:email>`, `<redacted:bearer-token>`, `<shape:string,len=1840>`. Literal content belongs in a trace only where the literal *is* the published artifact, such as a prompt-injection string from a paper. This is not only a privacy rule: it is what makes traces from real systems shareable at all, because the finding almost always lives in the shape.
+
+**Findings hang off spans.** Attaching a risk to the span where it materializes lets a reader see exactly where a flow stops being safe, instead of inferring it from prose. `validate.py` requires every span-level `finding.dsgai_id` to appear in the trace's top-level `dsgai_mapping`.
+
+Prefer `t_offset_ms` over wall-clock timestamps. Relative offsets carry the ordering that matters without inventing precision or disclosing when a system was running. Use `timestamp` only where the absolute date is part of the finding.
+
+### Provenance tiers
+
+Most people who hold real agent traces cannot publish them, because production telemetry contains everything that must not be published. A dataset that accepts only production telemetry stays empty; a dataset that accepts anything fills with plausible fiction. The `provenance.tier` field is how this dataset takes both contributions without confusing them:
+
+| Tier | Meaning |
+|---|---|
+| `observed_production` | Sanitized telemetry from a real deployed system |
+| `observed_lab` | Sandbox, test environment, CTF, or red-team exercise |
+| `derived_from_public_documentation` | The mechanism is rendered as a trace, and its precondition is a documented public fact carried in `evidence` - a specification, an advisory, a CVE, third-party research. The source may be normative (what a protocol permits) or empirical (what was observed); either is fine, as long as it is checkable without the contributor. |
+| `hypothetical` | Illustrative only, no empirical backing |
+
+Every tier except `hypothetical` requires at least one `provenance.evidence` item, enforced by `validate.py`, and every item has to say what it supports, narrowly. Acceptable backing is anything a reader can check without the contributor: a normative statement in a protocol specification, a published advisory or CVE, a vendor changelog, third-party research, a DOI. A source the contributor owns or controls - their own site, their own dataset, their own publication - does not establish a precondition here, whatever it says. This rule is not machine-checkable and is not enforced by `validate.py`; it is a review-time obligation, the same as the sanitization judgment call a reviewer makes on every entry. Where a contributor-controlled source is the only backing available, the honest tier is `hypothetical`. Statistics computed over this dataset should be reported per tier, never pooled across them.
+
+Choose the tier honestly. `derived_from_public_documentation` is not a lesser contribution; it is the tier that lets a documented phenomenon be studied as a data flow without anyone publishing their logs. A specification saying what a protocol permits and telemetry saying what happened are different claims with different weight, which is exactly why the field is named "documentation" and not "observation."
+
+### MITRE ATLAS mappings
+
+`mitre_atlas_mapping` is release-pinned by construction: an entry records the ATLAS release it was verified against and the name each identifier carried in that release. ATLAS identifiers and their names have both moved between releases, so an unpinned pair silently rots. Omit the property rather than guess.
 
 ## Sanitization Requirements
 
 All traces **must** be sanitized before submission:
 
-- No real API keys, tokens, credentials, or secrets — replace with placeholder values (e.g., `sk-REDACTED`, `Bearer EXAMPLE_TOKEN`)
+- No real API keys, tokens, credentials, or secrets — replace with placeholder values (e.g. `<redacted:bearer-token>`)
 - No real PII, PHI, or proprietary data — use synthetic equivalents
 - No internal hostnames, IP addresses, or infrastructure details
 - Generalize organization-specific tool names if they could identify the source
 
 Traces from test environments, sandboxes, or CTF exercises are ideal. Production traces must be thoroughly sanitized.
 
+Two things back this up rather than leaving it to good intentions. `sanitization.attestation` must be a literal `true`, so an entry cannot merge with the box quietly unticked. And `validate.py` scans every string in the entry, independently of that attestation: payloads, summaries, notes, evidence locators. Placeholders are exempt; values matching one of its patterns fail the run - common credential shapes, RFC1918 addresses, `.internal`/`.corp`/`.lan`/`.intra` hostnames, home directory paths, credential-shaped `key=value` assignments, and two or more addresses in one string that look like real routable public IPv4 addresses. The scan is deliberately wider than the payloads, because prose written while looking at a real trace is where a real hostname actually gets typed. It is pattern matching, not a guarantee: a contributor is still the one certifying `sanitization.attestation`, and a reviewer is still the backstop for anything the patterns miss.
+
+Naming a third party as the subject of a security failure is a disclosure act, not a dataset contribution. Generalize the counterparty unless you have gone through disclosure and are prepared to say so in `notes`.
+
 ## Contributing
 
-Add traces as individual JSON, JSONL, or YAML files and submit a pull request. See the [main datasets README](../README.md) for general contribution guidelines.
+1. Copy `example.json` and edit it, or export spans from your own system into the same shape
+2. Save as `entries/<trace_id>.json`
+3. Run `python validate.py` - it must print OK
+4. Run `python build_index.py` to refresh `index.csv`
+5. Open a pull request describing what the trace demonstrates and which DSGAI entries it maps to
+
+See the [main datasets README](../README.md) for general contribution guidelines, and `#team-genai-data-security-initiative` on the [OWASP Slack workspace](https://owasp.slack.com) for anything that needs a conversation first.

datasets/agentdataflow_toolexchange_traces/build_index.py

@@ -0,0 +1,100 @@
+"""Generate index.csv from every entry in ./entries/.
+
+Usage: python build_index.py
+Writes ./index.csv with key flat columns; arrays are joined with '|'.
+Stdlib only.
+"""
+from __future__ import annotations
+
+import csv
+import json
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent
+ENTRIES_DIR = ROOT / "entries"
+INDEX_PATH = ROOT / "index.csv"
+
+SENSITIVITY_ORDER = ["none", "low", "moderate", "high"]
+
+COLUMNS = [
+    "trace_id",
+    "title",
+    "category",
+    "disposition",
+    "provenance_tier",
+    "primary_dsgai",
+    "dsgai_mapping",
+    "owasp_llm_top10_mapping",
+    "agent_framework",
+    "tool_protocol",
+    "span_count",
+    "finding_count",
+    "max_sensitivity",
+    "data_classes",
+    "evidence_count",
+    "primary_evidence",
+    "date_added",
+    "tags",
+]
+
+
+def max_sensitivity(spans: list[dict]) -> str:
+    best = -1
+    for span in spans:
+        value = span.get("sensitivity")
+        if value in SENSITIVITY_ORDER:
+            best = max(best, SENSITIVITY_ORDER.index(value))
+    return SENSITIVITY_ORDER[best] if best >= 0 else ""
+
+
+def row_for(entry: dict) -> dict[str, str]:
+    spans = entry.get("spans", []) or []
+    prov = entry.get("provenance", {}) or {}
+    agent = entry.get("agent", {}) or {}
+    evidence = prov.get("evidence", []) or []
+    dsgai = entry.get("dsgai_mapping", []) or []
+
+    data_classes: list[str] = []
+    for span in spans:
+        for cls in span.get("data_classes", []) or []:
+            if cls not in data_classes:
+                data_classes.append(cls)
+
+    return {
+        "trace_id": entry.get("trace_id", ""),
+        "title": entry.get("title", ""),
+        "category": entry.get("category", ""),
+        "disposition": entry.get("disposition", ""),
+        "provenance_tier": prov.get("tier", ""),
+        "primary_dsgai": dsgai[0] if dsgai else "",
+        "dsgai_mapping": "|".join(dsgai),
+        "owasp_llm_top10_mapping": "|".join(entry.get("owasp_llm_top10_mapping", []) or []),
+        "agent_framework": agent.get("framework", ""),
+        "tool_protocol": agent.get("tool_protocol", ""),
+        "span_count": str(len(spans)),
+        "finding_count": str(sum(1 for s in spans if s.get("finding"))),
+        "max_sensitivity": max_sensitivity(spans),
+        "data_classes": "|".join(sorted(data_classes)),
+        "evidence_count": str(len(evidence)),
+        "primary_evidence": evidence[0].get("citation", "") if evidence else "",
+        "date_added": entry.get("date_added", ""),
+        "tags": "|".join(entry.get("tags", []) or []),
+    }
+
+
+def main() -> None:
+    rows = []
+    for path in sorted(ENTRIES_DIR.glob("*.json")):
+        entry = json.loads(path.read_text(encoding="utf-8"))
+        rows.append(row_for(entry))
+
+    with INDEX_PATH.open("w", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(f, fieldnames=COLUMNS)
+        writer.writeheader()
+        writer.writerows(rows)
+
+    print(f"Wrote {INDEX_PATH.name} with {len(rows)} entries.")
+
+
+if __name__ == "__main__":
+    main()