Fix(symbolicai,#20628): re-execution des carnets Argument_Analysis apres resync des taxonomies + organe de detection

.pre-commit-config.yaml

@@ -223,3 +223,24 @@ repos:
         pass_filenames: true
         files: '^MyIA\.AI\.Notebooks/SymbolicAI/Lean/KNOTS/.*\.ipynb$'
         stages: [pre-commit]
+
+      - id: vendored-data-consumers
+        name: '#20628 — list notebooks reading a changed vendored taxonomy (advisory)'
+        description: |
+          A vendored data file (Argumentum OWL/CSV taxonomy) can change without
+          any notebook being re-executed: committed outputs then describe the
+          data as it was before, and the prose that comments on them too. The
+          founding case is #20628 (Obs-04 measured: 1513 classes / 7825 IRI
+          annotations on the old blob versus 1536 / 6632 on the current one).
+          This hook runs only when such a file is staged, prints every
+          notebook whose source cites that file, and marks the ones absent
+          from the same change. Advisory on purpose: it consumes no CI queue
+          slot (#13097 famine) and never blocks a legitimate commit; the
+          blocking verdict is available by running the script without
+          ``--advisory`` (a PR reviewer, or the author who wants the check).
+          Acceptance path chosen for #20628: the "organ" one, this hook.
+        entry: python scripts/notebook_tools/list_data_consumers.py --advisory
+        language: system
+        pass_filenames: false
+        files: '^MyIA\.AI\.Notebooks/SymbolicAI/Argument_Analysis/(ontologies/.*\.owl|data/.*taxonomy.*\.csv)$'
+        stages: [pre-commit]

docs/reference/scripts-reference.md

@@ -106,6 +106,7 @@ python scripts/notebook_tools/notebook_tools.py execute <notebook> --cell-by-cel
 | `detect_bare_cross_dir_load.py` | Détecte un `#load "X.cs"` **bare** (nom nu, sans séparateur) dont le `.cs` n'existe PAS dans le dossier du notebook — anti-pattern du rollout SVG inline (#6927) où le kernel résout un `#load` relatif et échoue silencieusement / charge le mauvais fichier |
 | `check_notebook_navlinks.py` | Vérifie les liens de navigation relatifs **cassés dans les cellules markdown des `.ipynb`** (« précédent / suivant »). `check_docs_links.py` couvre uniquement les fichiers markdown (CLAUDE.md, docs/, README) ; cet outil couvre le markdown *intérieur* des notebooks |
 | `check_plotly_static_risk.py` | Détecte les cellules Plotly-CDN dans les `.ipynb` = **blanc en static rendering** (cf #6927). Le pattern canonique `record PlotlyHtml + Formatter.Register` émet un `<script src="https://cdn.plot.ly/...">` externe qui rend en kernel live mais pas sur GitHub/nbviewer |
+| `list_data_consumers.py` | **Donnée vendorée modifiée → carnets consommateurs** (#20628). Sur un changement qui touche `Argument_Analysis/ontologies/*.owl` ou `data/*taxonomy*.csv`, liste les carnets dont la source cite le fichier (index git, `_archive/` et traductions `-EN` exclus) et marque ceux qui **ne sont pas dans le même changement**. Mode indexé par défaut, `--base <ref>` pour un changement de PR, `--advisory` (rc 0 forcé, mode hook), `--json`, `--self-test`. Câblé en pre-commit **advisory** (aucun emplacement de file CI consommé, #13097) : la version bloquante est l'appel sans `--advisory`, pour l'auteur ou le reviewer |
 
 ### Render-suite SVG inline (rollout #6927)
 La migration Plotly-CDN → SVG inline `text/html` (canon ai-01 `svg-6927-canon.md`) est gardée par 5 détecteurs couvrant tous les cas de figure cassé. `detect_svg_decimal_commas.py` (dans la section Tests ci-dessous) traite le cas virgule-décimale fr-FR.

scripts/notebook_tools/list_data_consumers.py

@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+"""Liste les carnets qui lisent une donnee vendoree modifiee, et signale ceux
+qui ne font pas partie du meme changement.
+
+Une resynchronisation de taxonomie (OWL/CSV vendoree) ne declenche aucun
+carnet : les sorties commitees continuent de decrire les donnees d'avant,
+et la prose qui les commente aussi (#20628, cas mesure Obs-04).
+
+Modes :
+  (defaut)        fichiers indexes (`git diff --cached --name-only`)
+  --base REF      fichiers du changement depuis REF (`git diff REF...HEAD`)
+  --advisory      toujours rc 0 : affiche la liste sans bloquer
+  --json          sortie machine
+  --self-test     controle positif + negatif
+
+rc 1 quand un carnet consommateur d'une donnee modifiee n'est pas lui-meme
+dans le changement. En mode pre-commit, le hook tourne en --advisory : la
+liste est imprimee, aucun commit legitime n'est bloque (penurie de runners
+CI, #13097 -- le hook ne consomme aucun emplacement de file).
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+# Donnees vendorees dont la mise a jour doit propager une re-execution.
+DATA_PATTERNS = (
+    re.compile(r"^MyIA\.AI\.Notebooks/SymbolicAI/Argument_Analysis/ontologies/.*\.owl$"),
+    re.compile(r"^MyIA\.AI\.Notebooks/SymbolicAI/Argument_Analysis/data/.*taxonomy.*\.csv$"),
+)
+# Un carnet de traduction ou archive ne se re-execute pas avec la source.
+EXCLUDED_CONSUMER = re.compile(r"(^|/)_archive/|-EN\.ipynb$")
+
+
+def git(*args: str, cwd: Path | None = None) -> str:
+    proc = subprocess.run(
+        ["git", *args],
+        capture_output=True,
+        text=True,
+        encoding="utf-8",
+        errors="replace",
+        cwd=str(cwd) if cwd else None,
+    )
+    if proc.returncode != 0:
+        raise RuntimeError(f"git {' '.join(args)}: {proc.stderr.strip()}")
+    return proc.stdout
+
+
+def changed_files(base: str | None, cwd: Path | None = None) -> list[str]:
+    if base:
+        raw = git("diff", "--name-only", "--diff-filter=ACMR", f"{base}...HEAD", cwd=cwd)
+    else:
+        raw = git("diff", "--cached", "--name-only", "--diff-filter=ACMR", cwd=cwd)
+    return [line.strip() for line in raw.splitlines() if line.strip()]
+
+
+def is_vendored_data(path: str) -> bool:
+    return any(p.match(path) for p in DATA_PATTERNS)
+
+
+def consumers_of(basename: str, cwd: Path | None = None) -> list[str]:
+    """Carnets dont le source cite ce nom de fichier (index git, pas le disque)."""
+    try:
+        raw = git("grep", "-l", "-F", basename, "--", "*.ipynb", cwd=cwd)
+    except RuntimeError:
+        return []  # aucun consommateur : git grep rend 1 sur zero resultat
+    out = []
+    for line in raw.splitlines():
+        path = line.split(":", 1)[1] if ":" in line else line
+        path = path.strip()
+        if path and not EXCLUDED_CONSUMER.search(path):
+            out.append(path)
+    return sorted(set(out))
+
+
+def analyse(files: list[str], cwd: Path | None = None) -> dict:
+    data = [f for f in files if is_vendored_data(f)]
+    changed = set(files)
+    report = {"data_files": [], "missing": []}
+    for path in data:
+        basename = Path(path).name
+        cons = consumers_of(basename, cwd=cwd)
+        entries = [{"notebook": c, "in_change": c in changed} for c in cons]
+        report["data_files"].append({"data": path, "consumers": entries})
+        report["missing"].extend(e["notebook"] for e in entries if not e["in_change"])
+    report["missing"] = sorted(set(report["missing"]))
+    return report
+
+
+def render(report: dict) -> str:
+    if not report["data_files"]:
+        return ""
+    lines = ["Donnee vendoree modifiee -- carnets consommateurs :"]
+    for entry in report["data_files"]:
+        lines.append(f"  {entry['data']}")
+        if not entry["consumers"]:
+            lines.append("    (aucun carnet ne la lit)")
+        for c in entry["consumers"]:
+            flag = "dans le changement" if c["in_change"] else "A RE-EXECUTER"
+            lines.append(f"    [{flag}] {c['notebook']}")
+    if report["missing"]:
+        lines.append(
+            "  -> re-executer les carnets marques A RE-EXECUTER dans le meme changement, "
+            "ou ouvrir une issue nommee qui les liste."
+        )
+    return "\n".join(lines)
+
+
+def self_test(cwd: Path | None = None) -> int:
+    ok = True
+    positive = analyse(
+        ["MyIA.AI.Notebooks/SymbolicAI/Argument_Analysis/ontologies/argumentum_fallacies.owl"],
+        cwd=cwd,
+    )
+    found = [c["notebook"] for e in positive["data_files"] for c in e["consumers"]]
+    if not found:
+        print("[self-test] ECHEC : aucun consommateur pour l'OWL des sophismes")
+        ok = False
+    else:
+        print(f"[self-test] controle positif : {len(found)} consommateur(s)")
+    if not positive["missing"]:
+        print("[self-test] ECHEC : le controle positif ne signale aucun carnet manquant")
+        ok = False
+    negative = analyse(["docs/reference/common-commands.md"], cwd=cwd)
+    if negative["data_files"] or negative["missing"]:
+        print("[self-test] ECHEC : un fichier non vendore a produit un rapport")
+        ok = False
+    else:
+        print("[self-test] controle negatif : aucune donnee vendoree, rc 0")
+    # Deduplication : le meme carnet citant deux donnees ne compte qu'une fois.
+    both = analyse(
+        [
+            "MyIA.AI.Notebooks/SymbolicAI/Argument_Analysis/ontologies/argumentum_fallacies.owl",
+            "MyIA.AI.Notebooks/SymbolicAI/Argument_Analysis/data/argumentum_fallacies_taxonomy.csv",
+        ],
+        cwd=cwd,
+    )
+    if len(both["missing"]) != len(set(both["missing"])):
+        print("[self-test] ECHEC : doublon dans la liste des carnets manquants")
+        ok = False
+    print("[self-test] " + ("OK" if ok else "ECHEC"))
+    return 0 if ok else 1
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--base", default=None, help="Ref de base (defaut : fichiers indexes)")
+    parser.add_argument("--advisory", action="store_true", help="Toujours rc 0 (mode hook)")
+    parser.add_argument("--json", action="store_true", help="Sortie machine")
+    parser.add_argument("--self-test", action="store_true")
+    parser.add_argument("--repo", default=None, help="Racine du depot (defaut : cwd)")
+    args = parser.parse_args(argv)
+
+    cwd = Path(args.repo) if args.repo else None
+    if args.self_test:
+        return self_test(cwd=cwd)
+
+    report = analyse(changed_files(args.base, cwd=cwd), cwd=cwd)
+    if args.json:
+        print(json.dumps(report, indent=2, ensure_ascii=False))
+    else:
+        text = render(report)
+        if text:
+            print(text)
+    if not report["missing"]:
+        return 0
+    return 0 if args.advisory else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/notebook_tools/tests/test_list_data_consumers.py

@@ -0,0 +1,108 @@
+"""Tests for scripts/notebook_tools/list_data_consumers.py.
+
+The organ answers one question: when a vendored taxonomy file changes, which
+notebooks read it, and which of those are absent from the same change set?
+Tests cover the predicate, the exclusion of translation/archive consumers,
+deduplication, and the two controls (a real data file finds consumers; a
+non-vendored file finds none).
+"""
+
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+import list_data_consumers as organ
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+
+OWL = "MyIA.AI.Notebooks/SymbolicAI/Argument_Analysis/ontologies/argumentum_fallacies.owl"
+CSV = "MyIA.AI.Notebooks/SymbolicAI/Argument_Analysis/data/argumentum_fallacies_taxonomy.csv"
+CARDS = "MyIA.AI.Notebooks/SymbolicAI/Argument_Analysis/Argumentation-02b-Argumentum-Cards-Python.ipynb"
+
+
+class TestVendoredDataPredicate:
+    def test_owl_is_vendored(self):
+        assert organ.is_vendored_data(OWL)
+
+    def test_taxonomy_csv_is_vendored(self):
+        assert organ.is_vendored_data(CSV)
+
+    def test_other_csv_is_not_vendored(self):
+        assert not organ.is_vendored_data(
+            "MyIA.AI.Notebooks/SymbolicAI/Argument_Analysis/data/other.csv"
+        )
+
+    def test_docs_are_not_vendored(self):
+        assert not organ.is_vendored_data("docs/reference/common-commands.md")
+
+
+class TestConsumerExclusions:
+    def test_archive_consumer_is_excluded(self):
+        assert organ.EXCLUDED_CONSUMER.search(
+            "MyIA.AI.Notebooks/SymbolicAI/Argument_Analysis/_archive/Old.ipynb"
+        )
+
+    def test_translation_consumer_is_excluded(self):
+        assert organ.EXCLUDED_CONSUMER.search(
+            "MyIA.AI.Notebooks/SymbolicAI/Argument_Analysis/Argumentation-02b-EN.ipynb"
+        )
+
+    def test_source_consumer_is_kept(self):
+        assert not organ.EXCLUDED_CONSUMER.search(CARDS)
+
+
+@pytest.mark.skipif(not REPO_ROOT.joinpath(".git").exists(), reason="needs the git index")
+class TestPositiveControl:
+    def test_owl_finds_consumers(self):
+        cons = organ.consumers_of(Path(OWL).name, cwd=REPO_ROOT)
+        assert cons, "aucun carnet ne lit l'OWL des sophismes"
+        assert any("Argumentation-" in c for c in cons)
+
+    def test_changed_consumer_is_not_reported_missing(self):
+        report = organ.analyse([CSV, CARDS], cwd=REPO_ROOT)
+        assert CARDS not in report["missing"]
+
+    def test_unchanged_consumer_is_reported_missing(self):
+        report = organ.analyse([OWL], cwd=REPO_ROOT)
+        assert report["missing"], "un changement de donnee seule devrait signaler des carnets"
+
+    def test_render_mentions_the_action(self):
+        text = organ.render(organ.analyse([OWL], cwd=REPO_ROOT))
+        assert "A RE-EXECUTER" in text
+
+
+@pytest.mark.skipif(not REPO_ROOT.joinpath(".git").exists(), reason="needs the git index")
+class TestNegativeControl:
+    def test_non_vendored_file_is_silent(self):
+        report = organ.analyse(["docs/reference/common-commands.md"], cwd=REPO_ROOT)
+        assert report["data_files"] == []
+        assert report["missing"] == []
+        assert organ.render(report) == ""
+
+
+@pytest.mark.skipif(not REPO_ROOT.joinpath(".git").exists(), reason="needs the git index")
+class TestDeduplication:
+    def test_notebook_citing_two_data_files_appears_once(self):
+        report = organ.analyse([OWL, CSV], cwd=REPO_ROOT)
+        assert len(report["missing"]) == len(set(report["missing"]))
+
+
+class TestExitCodes:
+    @pytest.mark.skipif(not REPO_ROOT.joinpath(".git").exists(), reason="needs the git index")
+    def test_advisory_never_fails(self, monkeypatch):
+        monkeypatch.setattr(
+            organ, "changed_files", lambda base, cwd=None: [OWL]
+        )
+        assert organ.main(["--advisory", "--repo", str(REPO_ROOT)]) == 0
+
+    @pytest.mark.skipif(not REPO_ROOT.joinpath(".git").exists(), reason="needs the git index")
+    def test_strict_mode_fails_on_missing_consumer(self, monkeypatch):
+        monkeypatch.setattr(
+            organ, "changed_files", lambda base, cwd=None: [OWL]
+        )
+        assert organ.main(["--repo", str(REPO_ROOT)]) == 1
+
+    def test_self_test_passes(self):
+        assert organ.self_test(cwd=REPO_ROOT) == 0