From 95b49d35beac1b3dc53d2c0ffea67fffcef8b7b4 Mon Sep 17 00:00:00 2001 From: jsboige Date: Thu, 10 Sep 2026 04:16:19 +0200 Subject: [PATCH 1/2] =?UTF-8?q?feat(ci,#15327):=20output-collapse=20ratche?= =?UTF-8?q?t=20(advisory)=20=E2=80=94=20mirror=20of=20the=20flood=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A re-execution must not silently collapse a notebook's output volume. Founding case #15209 (Lean-7b-Examples 6b327a9bf -> 56d98429a): 11 -> 11 code cells, 0 errors, real execution_counts, and 10637 -> 2985 output chars — every gate green while the substance vanished (graceful degradation "if api_ok:" printing "Execution sautee (API non configuree)"). Discriminating design per the ai-01 measurement on the issue (aggregate contraction alone = ~100% FP over 14 days): SIGNATURE (substantial base replaced by a graceful-degradation motif) + MAGNITUDE (per-cell order-of-magnitude loss), exempted by the two mechanically detectable legitimate causes — content moved to a notebook created by the same diff, and diagnostic-text purge (CS####/warning lines, char-weighted). Wired advisory (blocking=False) in fast-lane TRANCHE2 with dispatch-only stub workflow; pr-review-discipline.md section D gains point 7. Calibration: reference replay fires (3 signature findings, factors 14.9/67.6/49.4); 18/18 recently merged notebook PRs clean; of the three known-legitimate contractions, split and CS-purge pass silently, declared lightening labels (advisory, body justification — by design). Co-Authored-By: Claude Sonnet 5 --- .claude/rules/pr-review-discipline.md | 2 + .../notebook-output-collapse-ratchet.yml | 82 +++ scripts/ci/fast_lane_registry.py | 38 +- .../notebook_tools/check_output_collapse.py | 480 ++++++++++++++++++ .../tests/test_check_output_collapse.py | 186 +++++++ scripts/tests/test_fast_lane.py | 10 +- 6 files changed, 791 insertions(+), 7 deletions(-) create mode 100644 .github/workflows/notebook-output-collapse-ratchet.yml create mode 100644 scripts/notebook_tools/check_output_collapse.py create mode 100644 scripts/notebook_tools/tests/test_check_output_collapse.py diff --git a/.claude/rules/pr-review-discipline.md b/.claude/rules/pr-review-discipline.md index 746035b35f..27e537451a 100644 --- a/.claude/rules/pr-review-discipline.md +++ b/.claude/rules/pr-review-discipline.md @@ -61,6 +61,8 @@ Single-seed ou single-fold = **CHANGES_REQUESTED** sauf flag explicite `[POC]` d 6. **PRs notebook : vérifier le verdict du check-run `Output-failure ratchet (base vs PR)`** (organe `scripts/notebook_tools/check_output_failure_text.py`, enregistré **bloquant** dans `scripts/ci/fast_lane_registry.py`). Le check-run DOIT être `success`. `TOOL_FAILURE` (bannières « `program is not installed` » ou rendu d'échec) ou `MACHINE_PATH` (chemin machine dans une sortie) qui **augmente** sur la PR (N → N+k, type `0 → 21`) = **régression → `CHANGES_REQUESTED`**, même si les points 1-3 passent : les bannières d'échec ne sont **pas** des exceptions Python, elles ne déclenchent ni `exec_count` nul ni `grep -nE "raise NotImplementedError|assert False|1/0"` — une PR qui **remplace** un rendu SVG de factor-graph par une bannière est le dégât exact de #3473/#11685, pas un « ça tourne ». Récurrence : #13517 (PR #13036 LDA — bannières 0→21, `MACHINE_PATH` 0→14, approuvée par Hermes alors que le garde rend rc=1) ; juin → #3473 (~15 filles) ; 18/08 → #11693. +7. **PRs notebook : lire le check-run ADVISORY `Output-collapse ratchet (base vs PR, advisory)`** (organe `scripts/notebook_tools/check_output_collapse.py`, enregistré `blocking=False` dans `scripts/ci/fast_lane_registry.py`, #15327). Conclusion neutre par design — le signal vit dans le détail du check-run. Un finding `SIGNATURE` (sortie base substantielle remplacée par « `Execution sautee (API non configuree)` » et consœurs) = **re-exécution sans les clés → `CHANGES_REQUESTED`** : les cellules se sont « exécutées avec succès » en dégradation gracieuse (`if api_ok:`), c'est le contournement de C.2 par la porte de secours. Contre-exemple mesuré (fondateur) : #15209, `Lean-7b-Examples.ipynb` `6b327a9bf` → `56d98429a` — 11 → 11 cellules, 0 erreur, `execution_count` réels partout, et **10637 → 2985** caractères de sortie (cellules `2195 → 147`, `2568 → 38`, `2074 → 42`) : tous les organes verts, la perte réelle. Un finding `MAGNITUDE` (perte d'un ordre de grandeur par cellule, non couvert par les exemptions automatiques contenu-déplacé/purge-diagnostic) exige une **justification dans le body** (allègement déclaré, au même titre que les autres ratchets) — sans elle : `CHANGES_REQUESTED`. + **Advisory `.NET execution_count` ≠ outputs vides autorisés (#5214).** L'advisory autorise à sauter la ré-exécution **CI** (pas de kernel .NET en CI), **pas** à committer des sorties vides : `.NET Interactive` s'exécute **localement** sur chaque worker → une cellule .NET committée **DOIT** porter `execution_count != null`. `validate_pr_notebooks.py` FAIL sur `.NET` + `null`, et ne tolère `null` que là où l'exécution locale est aussi impossible (QC Cloud, Lean). Verdict attendu dans le body : `EXEC_PROVED` vs `STRUCTURAL_ONLY` (refus). ### E. Documentation / Admin : groupement obligatoire diff --git a/.github/workflows/notebook-output-collapse-ratchet.yml b/.github/workflows/notebook-output-collapse-ratchet.yml new file mode 100644 index 0000000000..a40ab122ae --- /dev/null +++ b/.github/workflows/notebook-output-collapse-ratchet.yml @@ -0,0 +1,82 @@ +name: Notebook Output Collapse Ratchet + +# #15327 / réserve user sur #15209 (2026-09-08) — une ré-exécution ne doit +# pas laisser s'effondrer silencieusement le VOLUME de sortie des cellules +# code. Miroir de notebook-output-flood-ratchet.yml, qui lui attrape +# l'inflation. +# +# Cas fondateur : Lean-7b-Examples.ipynb (6b327a9bf -> 56d98429a), 11 -> 11 +# cellules code exécutées « avec succès », 0 erreur, execution_counts +# réels — et 10637 -> 2985 caractères de sortie (-71,9 %). Les gardes +# if api_ok: du notebook ont fait leur travail de dégradation gracieuse : +# trois cellules ont imprimé « Execution sautee (API non configuree) » au +# lieu de leur vraie sortie (2195 -> 147, 2568 -> 38, 2074 -> 42 chars). +# +# Contrainte de design (mesure ai-01 sur #15327, 2026-09-09) : la +# contraction AGREGÉE seule est un signal à ~100 % de faux positifs (3/3 +# contractions > 50 % sur 14 jours étaient des PRs légitimes). L'organe +# discrimine donc par la CAUSE : signature de dégradation gracieuse et +# perte d'un ordre de grandeur PAR CELLULE, avec exemption des deux causes +# légitimes mécaniquement détectables (contenu déplacé vers un notebook +# créé dans le même diff ; purge de texte diagnostique CS####/warning). +# +# Placement ADVISORY (point 3 de l'issue) : le verdict est publié sous le +# check-run `Output-collapse ratchet (base vs PR, advisory)` avec une +# conclusion neutre — jamais bloquant tant que le seuil n'est pas calibré +# davantage sur l'historique. + +# Livré directement dans la voie rapide (#12567) : le verdict par-PR est +# rendu par `fast-lane-shadow.yml` via l'entrée TRANCHE2 de +# `scripts/ci/fast_lane_registry.py`. Ce fichier ne garde que +# `workflow_dispatch` (relance manuelle). +on: + workflow_dispatch: + inputs: + base_ref: + description: 'Base ref to ratchet against (default origin/main)' + required: false + default: 'origin/main' + +permissions: + contents: read + +concurrency: + group: notebook-output-collapse-ratchet-${{ github.ref }} + cancel-in-progress: true + +jobs: + ratchet: + name: Output-collapse ratchet (base vs PR, advisory) + # Même jambe que les siblings (#13378 tranche 5) : garde Python pur, + # sans secret. La garde same-repo ci-dessous saute les PRs de fork : + # aucun code de fork n'atteint le runner. + runs-on: [self-hosted, coursia-ephemeral, coursia-linux] + if: github.event.pull_request.head.repo.full_name == null || github.event.pull_request.head.repo.full_name == github.repository + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + filter: blob:none + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + # Le self-test passe FIRST et gate le garde : contrôles positifs et + # négatifs + replay du cas de référence (#15209, 3 findings signature). + - name: Detector self-test (positive + negative control) + run: python scripts/notebook_tools/check_output_collapse.py --self-test + + - name: Output-collapse axis unit tests + run: python -m pytest scripts/notebook_tools/tests/test_check_output_collapse.py -q + + # Refs reach the script via env, never inline in the run script + # (script-injection guard, same as the siblings). + - name: Ratchet check (advisory) + env: + BASE_REF: ${{ github.event.inputs.base_ref }} + PR_BASE: ${{ github.base_ref }} + run: | + BASE="${BASE_REF:-origin/${PR_BASE:-main}}" + python scripts/notebook_tools/check_output_collapse.py "$BASE" diff --git a/scripts/ci/fast_lane_registry.py b/scripts/ci/fast_lane_registry.py index c5d47ef05f..8f0acfc4a0 100644 --- a/scripts/ci/fast_lane_registry.py +++ b/scripts/ci/fast_lane_registry.py @@ -392,8 +392,8 @@ class Guard: # TRANCHE 2 d'absorption (#12567) -- meme contrat que la tranche 1 (nom # canonique, conclusion reelle, workflow d'origine retire de pull_request), # trois formes moteur nouvelles par rapport a la tranche 1, portees par -# QUATRE gardes (deux instances du ratchet autonome : failure-text puis -# output-flood, #14959) : +# CINQ gardes (trois instances du ratchet autonome : failure-text, +# output-flood puis output-collapse advisory, #14959/#15327) : # # - ratchet AUTONOME : le script fait lui-meme son diff base...HEAD, la lane # ne fournit que {base_ref}. Son self-test est un PRE-CONTROLE (`pre_argv`) @@ -457,6 +457,40 @@ class Guard: needs_base=True, absorbed=True, ), + # Output-volume COLLAPSE ratchet, advisory (#15327): the two siblings + # above watch output GROWTH (flood) and failure banners; nothing watched + # contraction -- #15209 lost 72 % of its output chars (11 code cells + # executed "successfully", graceful-degradation guards if api_ok:) while + # every gate stayed green. Design constraint (ai-01 measurement on the + # issue, 2026-09-09): AGGREGATE contraction alone is ~100 % FP (3/3 + # contractions > 50 % over 14 days were legitimate), so this ratchet + # flags per-CELL order-of-magnitude loss and graceful-degradation + # signatures (execution sautee / non configure / mode simulation / + # skipped), exempting the two mechanically detectable legitimate causes: + # content moved to a notebook created by the same diff, and + # diagnostic-text purge (CS####/warning lines). Advisory until the + # threshold is calibrated further on history (issue point 3). + # Source : notebook-output-collapse-ratchet.yml (stub dispatch-only). + Guard( + name="Output-collapse ratchet (base vs PR, advisory)", + source="notebook-output-collapse-ratchet.yml", + paths=[ + "**.ipynb", + "scripts/notebook_tools/check_output_collapse.py", + ".github/workflows/notebook-output-collapse-ratchet.yml", + ], + pre_argv=[ + "python", "scripts/notebook_tools/check_output_collapse.py", + "--self-test", + ], + argv=[ + "python", "scripts/notebook_tools/check_output_collapse.py", + "{base_ref}", + ], + blocking=False, + needs_base=True, + absorbed=True, + ), # Forme 2 : iter par notebook change, rc=1 defaut / rc=2 illisible. # Source : fabricated-output-gate.yml (job `fabricated-output`). Guard( diff --git a/scripts/notebook_tools/check_output_collapse.py b/scripts/notebook_tools/check_output_collapse.py new file mode 100644 index 0000000000..96999dd99a --- /dev/null +++ b/scripts/notebook_tools/check_output_collapse.py @@ -0,0 +1,480 @@ +"""Advisory ratchet: a re-execution must not silently collapse a notebook's +OUTPUT VOLUME -- the mirror image of check_output_flood.py. + +Issue #15327 / user reserve on #15209 (2026-09-08: « Il n'est pas normal que +ni le CI ni la review l'ait detecte. Gros rouge et investigation necessaire ! »). +Measured firsthand on the reference case (Lean-7b-Examples.ipynb, +6b327a9bf -> 56d98429a): 11 -> 11 code cells, 0 error cells, real +execution_counts everywhere, and output_chars 10637 -> 2985 (-71.9 %). The +graceful-degradation guards (``if api_ok:``) did their job: three cells ran +"successfully" and printed ``Execution sautee (API non configuree)`` instead +of their real output (2195 -> 147, 2568 -> 38, 2074 -> 42 chars). + +Every existing surface stayed green on that diff: H.1 (no ``error`` output), +H.3/pre-commit (cells executed with outputs), the failure-text ratchet (a +skip notice is not a failure banner), the output-flood ratchet (asymmetric: +it catches INFLATION, not contraction), markdown-content-loss (it measures +markdown, and the markdown GREW). The repo had a ratchet against output +inflation and none against output collapse. This file closes that hole. + +Design constraint (ai-01 measurement on #15327, 2026-09-09): the AGGREGATE +notebook contraction ratio alone is a ~100 % false-positive signal -- over +14 days / 162 commits, every contraction > 50 % (3 of them) was a legitimate +PR (declared lightening, content split into a notebook created in the same +commit, compiler-warning purge), and the one real degradation (#15209) never +crossed a 50 % aggregate threshold that would have separated it from noise +anyway. A useful instrument therefore needs a CAUSE DISCRIMINANT, not a +finer aggregate threshold. Two signals, two mechanical exemptions: + + SIGNATURE a cell whose base output is substantial and whose head output + matches a graceful-degradation motif (``execution sautee``, + ``non configure``, ``mode simulation``, ``skipped``, + ``not available``) while contracting at all -- catches the + whole "re-executed without the keys" class, not just #15209. + Narrow by construction; not subject to the exemptions below + (a split or a warning purge does not leave skip notices behind). + MAGNITUDE a cell whose output loses an ORDER OF MAGNITUDE (factor >= 10 + over a noise floor), subject to the two legitimate causes that + are mechanically detectable: + - MOVED CONTENT: a notebook CREATED by the same diff gains a + comparable output volume (>= MOVED_GAIN_FRACTION of the + loss) -- compare at diff scale, not file scale. + - DIAGNOSTIC PURGE: the removed text is predominantly + diagnostic-shaped (CS#### / warning / DeprecationWarning + lines) -- classify the removed lines, not just count them. + The third legitimate cause (declared lightening) is + human-side: the advisory label appears and the PR body + justifies it, same contract as the other ratchets. + +Placement (issue point 3): ADVISORY first, blocking later once the threshold +is calibrated on history. Wired as ``blocking=False`` in the fast lane: the +verdict is published under a neutral conclusion, never gate. + +Cells are matched BASE -> HEAD by nbformat cell ``id`` when both sides carry +one, falling back to positional index (same contract as the flood sibling). +A cell DELETED by the PR has no head counterpart and produces no finding (a +notebook that deletes code cells loses volume legitimately); a notebook +ADDED by the branch has no baseline and is never judged, though its output +volume counts toward the moved-content exemption of the same diff. + +Usage: + python check_output_collapse.py [--json] + python check_output_collapse.py --self-test + +Base ref is resolved to merge-base(base-ref, HEAD), same as the siblings. + +Exit 1 iff at least one changed notebook carries a non-exempt finding. With +``--json`` the full row structure (including exempted findings and their +reasons) is emitted for calibration. +""" + +import argparse +import json +import re +import sys +import unicodedata +from collections import Counter + +# Reuse the siblings' git plumbing so all three ratchets share one +# implementation of "what changed between the merge base and HEAD". +from check_output_failure_text import ( + EXCLUDE_MARKERS, + changed_notebooks, + resolve_base, + read_notebook_at, +) +from check_output_flood import _cell_key + +# A base output below this many chars is noise (a shorter tqdm bar, a +# different seed): never judged. The reference case's guilty cells sit at +# 2074-2568 chars; ai-01's calibration puts the p99 per-cell output object +# count at 1, so 500 chars of text is already a substantial cell. +BASE_FLOOR = 500 + +# Order-of-magnitude contraction (reference case: factors 14.9 / 67.6 / 49.4). +MAGNITUDE_FACTOR = 10 + +# The signature signal needs SOME real contraction, not sub-noise jitter, but +# deliberately far below the magnitude factor: the motif itself is the +# discriminator, the ratio only rules out a coincidental substring. +SIGNATURE_FACTOR = 2 + +# Moved-content exemption: an added notebook's output volume counts against +# the magnitude losses of the same diff from this fraction up ("un volume +# comparable", ai-01 on #14795's split). +MOVED_GAIN_FRACTION = 0.5 + +# Diagnostic-purge exemption: this fraction of the REMOVED lines must be +# diagnostic-shaped for the cell to be exempt (Sudoku-06 #15144 purged 21 +# CS8632 warnings -- the warnings WERE the output). +DIAGNOSTIC_LINE_FRACTION = 0.8 + +# Graceful-degradation motifs, matched on accent-stripped lowercase text +# (the reference notebook prints "configuree" without its accent; other +# notebooks print "configuré"). List from the issue's design point 2 plus +# the English twins of the observed forms. +DEGRADATION_RE = re.compile( + r"execution sautee" + r"|comparaison sautee" + r"|non configure" + r"|not configured" + r"|mode simulation" + r"|\bskipped\b" + r"|not available" +) + +# Diagnostic-shaped lines (removed-text classification). CS#### is the C# +# compiler; the warning family covers Python (DeprecationWarning and +# friends) and both spellings of the French "warning : CS####". +DIAGNOSTIC_RE = re.compile( + r"\bCS\d{4}\b" + r"|warning" + r"|deprecationwarning" + r"|futurewarning" + r"|userwarning" + r"|runtimewarning" + r"|pendingdeprecationwarning" +) + +# Reference case (#15209), replayed by --self-test. 56d98429a is the +# measured PR head; its parent-side base is the main commit 6b327a9bf. +SELF_TEST_BASE = "6b327a9bf" +SELF_TEST_HEAD = "56d98429a" +SELF_TEST_NOTEBOOK = "MyIA.AI.Notebooks/SymbolicAI/Lean/Lean-7b-Examples.ipynb" +SELF_TEST_MIN_FINDINGS = 3 + + +def _normalize(text): + """Lowercase, accents stripped: motif matching is accent-blind.""" + decomposed = unicodedata.normalize("NFD", text) + stripped = "".join(c for c in decomposed + if unicodedata.category(c) != "Mn") + return stripped.lower() + + +def _text_of(value): + """nbformat stores output text as str or list[str]; flatten both.""" + if isinstance(value, list): + return "".join(str(x) for x in value) + return str(value or "") + + +def _cell_output_text(cell): + """Concatenated `text` + `data["text/plain"]` of a cell's outputs. + + This is the exact quantity the issue defines as the output volume. + """ + parts = [] + for o in cell.get("outputs", []) or []: + parts.append(_text_of(o.get("text"))) + data = o.get("data") or {} + parts.append(_text_of(data.get("text/plain"))) + return "".join(parts) + + +def _cell_chars(cell): + return len(_cell_output_text(cell)) + + +def _code_cells(nb): + """{(id-or-positional-key): cell} for every code cell.""" + if not nb: + return {} + out = {} + for i, c in enumerate(nb.get("cells", []) or []): + if c.get("cell_type") != "code": + continue + out[_cell_key(c, i)] = c + return out + + +def _diagnostic_fraction(base_text, head_text): + """Fraction (in chars) of the REMOVED text that is diagnostic-shaped. + + Removed lines are a multiset difference (duplicate warnings count), so a + purge of 21 identical CS8632 lines is fully accounted. The fraction is + weighted by CHARS, not lines: the ratchet guards output VOLUME, so a + purge that also swallows a large real output line must not ride along a + handful of short warnings. + """ + removed = Counter(base_text.splitlines()) - Counter(head_text.splitlines()) + total = sum(len(line) * n for line, n in removed.items()) + if total == 0: + return 0.0 + diag = sum(len(line) * n for line, n in removed.items() + if DIAGNOSTIC_RE.search(_normalize(line))) + return diag / total + + +def analyze(base_nb, head_nb): + """Pure per-notebook collapse analysis on two parsed notebooks. + + ``base_nb`` is None for a notebook ADDED by the branch (never judged). + Returns one row dict; the moved-content exemption is applied later, at + diff scale, by ``compare`` (it needs the other rows of the same diff). + """ + added = base_nb is None + b_cells, h_cells = _code_cells(base_nb), _code_cells(head_nb) + + b_total = sum(_cell_chars(c) for c in b_cells.values()) + h_total = sum(_cell_chars(c) for c in h_cells.values()) + + findings = [] + for key, h_cell in h_cells.items(): + b_cell = b_cells.get(key) + if b_cell is None: + continue # cell added by the branch: no baseline, no judgement + b_chars, h_chars = _cell_chars(b_cell), _cell_chars(h_cell) + if b_chars < BASE_FLOOR or h_chars >= b_chars: + continue + factor = b_chars / max(h_chars, 1) + h_text = _cell_output_text(h_cell) + if factor >= SIGNATURE_FACTOR and DEGRADATION_RE.search( + _normalize(h_text)): + findings.append({ + "cell": key, "kind": "signature", + "base": b_chars, "head": h_chars, + "factor": round(factor, 1), + }) + continue + if factor >= MAGNITUDE_FACTOR: + b_text = _cell_output_text(b_cell) + diag_frac = _diagnostic_fraction(b_text, h_text) + if diag_frac >= DIAGNOSTIC_LINE_FRACTION: + findings.append({ + "cell": key, "kind": "exempt-diagnostic", + "base": b_chars, "head": h_chars, + "factor": round(factor, 1), + "diagnostic_fraction": round(diag_frac, 2), + }) + else: + findings.append({ + "cell": key, "kind": "magnitude", + "base": b_chars, "head": h_chars, + "factor": round(factor, 1), + }) + + regressed = (not added) and any( + f["kind"] in ("signature", "magnitude") for f in findings) + return { + "added": added, + "base_total": b_total, + "head_total": h_total, + "cells": findings, + "regressed": regressed, + } + + +def _apply_moved_content(rows): + """Diff-scale exemption: volume that reappears in an ADDED notebook of + the same diff is content that moved, not content that vanished.""" + moved_gain = sum(r["head_total"] for r in rows if r["added"]) + magnitude_loss = sum(f["base"] - f["head"] for r in rows + for f in r["cells"] if f["kind"] == "magnitude") + if magnitude_loss <= 0 or moved_gain < MOVED_GAIN_FRACTION * magnitude_loss: + return rows + for r in rows: + for f in r["cells"]: + if f["kind"] == "magnitude": + f["kind"] = "exempt-moved" + f["moved_gain"] = moved_gain + r["regressed"] = any(f["kind"] == "signature" for f in r["cells"]) + return rows + + +def compare(base_ref, head_ref, paths, cwd=None): + """Per-notebook collapse between two refs, then the diff-scale exemption. + + ``head_ref`` is None for the working tree (same contract as the siblings). + """ + rows = [] + for path in paths: + base_nb = read_notebook_at(base_ref, path, cwd=cwd) + head_nb = read_notebook_at(head_ref, path, cwd=cwd) + if head_nb is None: + continue + row = analyze(base_nb, head_nb) + row["notebook"] = path + rows.append(row) + return _apply_moved_content(rows) + + +def self_test(cwd=None): + """Positive and negative control, then the reference-case replay. + + A detector that cannot be shown to fire is indistinguishable from one + that is unplugged -- the same contract that keeps the siblings honest. + """ + failures = [] + + def _nb(cells): + """Build a synthetic notebook from [(id, output_text)].""" + return {"cells": [ + {"cell_type": "code", "id": k, "outputs": [ + {"output_type": "stream", "text": t}]} + for k, t in cells]} + + def _analyze(base_cells, head_cells, added=False): + return analyze(None if added else _nb(base_cells), _nb(head_cells)) + + long_text = "x" * 1000 + skip_notice = ("[INFO] API non configuree - execution sautee\n" + "Configurez OPENAI_API_KEY pour executer les exemples\n") + + # 1. signature fires: substantial base, skip-notice head, real contraction + r = _analyze([("a", long_text)], [("a", skip_notice)]) + if not (r["cells"] and r["cells"][0]["kind"] == "signature"): + failures.append("signature (1000 -> skip notice) not flagged") + # 2. magnitude fires: order-of-magnitude contraction without a motif + r = _analyze([("a", long_text)], [("a", "y" * 40)]) + if not (r["cells"] and r["cells"][0]["kind"] == "magnitude"): + failures.append("magnitude (1000 -> 40) not flagged") + # 3. benign variance stays silent (30 % churn is not a collapse) + if _analyze([("a", long_text)], [("a", "x" * 700)])["cells"]: + failures.append("benign contraction (1000 -> 700) flagged") + # 4. small base stays silent (below the noise floor, even x1000) + if _analyze([("a", "x" * 400)], [("a", "y")])["cells"]: + failures.append("sub-floor base (400 -> 1) flagged") + # 5. growth stays silent + if _analyze([("a", long_text)], [("a", "x" * 5000)])["cells"]: + failures.append("growth (1000 -> 5000) flagged") + # 6. diagnostic purge is exempt: base is 30 warning lines, head is clean + warnings = "".join( + "warning CS8632: Une reference nullable est requise %d\n" % i + for i in range(30)) + r = _analyze([("a", warnings)], [("a", "Build succeeded.\n")]) + if not (r["cells"] and r["cells"][0]["kind"] == "exempt-diagnostic"): + failures.append("diagnostic purge not exempted") + if r["regressed"]: + failures.append("diagnostic purge exempted but still regressed") + # 7. mixed cell is NOT exempt: the purge swallowed a large real output + # line together with the warnings (char-weighted fraction ~0.65) + mixed = warnings + "real result: " + "z" * 900 + "\n" + r = _analyze([("a", mixed)], [("a", "real result: " + "z" * 50 + "\n")]) + if not (r["cells"] and r["cells"][0]["kind"] == "magnitude"): + failures.append("purge swallowing real output exempted (should flag)") + # 8. accent-blind motif: "configuré" with accent still matches + r = _analyze([("a", long_text)], + [("a", "Execution sautée (API non configurée)\n")]) + if not (r["cells"] and r["cells"][0]["kind"] == "signature"): + failures.append("accented motif (sautée/configurée) not matched") + # 9. added notebook is never judged + if _analyze([], [("a", skip_notice)], added=True)["regressed"]: + failures.append("added notebook judged instead of skipped") + # 10. deleted cell loses volume legitimately: no head key, no finding + if _analyze([("a", long_text), ("b", long_text)], [("a", long_text)])["cells"]: + failures.append("deleted cell produced a finding") + # 11. index-shift is not misattributed (same contract as the flood) + if any(f["cell"] == "a" for f in + _analyze([("a", long_text)], + [("zz", "y"), ("a", long_text)])["cells"]): + failures.append("unchanged cell flagged after benign insert") + + # 12. moved-content exemption at diff scale: notebook A collapses a cell, + # notebook B is CREATED by the same diff with comparable volume. + rows = [ + analyze(_nb([("a", long_text * 3)]), _nb([("a", "y" * 30)])), + analyze(None, _nb([("b", "z" * 1500)])), + ] + _apply_moved_content(rows) + if rows[0]["cells"] and rows[0]["cells"][0]["kind"] != "exempt-moved": + failures.append("moved content not exempted at diff scale") + if rows[0]["regressed"]: + failures.append("moved content exempted but still regressed") + # 13. ...but a moved gain BELOW the fraction does not exempt + rows = [ + analyze(_nb([("a", long_text * 3)]), _nb([("a", "y" * 30)])), + analyze(None, _nb([("b", "z" * 300)])), + ] + _apply_moved_content(rows) + if not rows[0]["regressed"]: + failures.append("insufficient moved gain exempted the collapse") + + # Replay the reference case (#15209). + from check_output_failure_text import git + if (git("cat-file", "-e", SELF_TEST_HEAD, cwd=cwd) is None + or git("cat-file", "-e", SELF_TEST_BASE, cwd=cwd) is None): + print("SKIP replay: reference commits not in this clone") + else: + rrows = compare(SELF_TEST_BASE, SELF_TEST_HEAD, + [SELF_TEST_NOTEBOOK], cwd=cwd) + r0 = rrows[0] if rrows else {} + n_sig = sum(1 for f in r0.get("cells", []) + if f["kind"] == "signature") + print("replay " + SELF_TEST_HEAD[:12] + " on " + + SELF_TEST_NOTEBOOK.split("/")[-1] + ": total " + + str(r0.get("base_total")) + " -> " + str(r0.get("head_total")) + + ", signature findings " + str(n_sig) + + " " + str([f["cell"] + " x" + str(f["factor"]) + for f in r0.get("cells", [])])) + if n_sig < SELF_TEST_MIN_FINDINGS: + failures.append("replay signature findings " + str(n_sig) + + " < " + str(SELF_TEST_MIN_FINDINGS) + " expected") + + for f in failures: + print("SELF-TEST FAIL: " + f) + if failures: + return 1 + print("SELF-TEST OK: witnesses fired, benign churn silent, exemptions " + "hold, reference replay fires") + return 0 + + +def main(argv=None): + ap = argparse.ArgumentParser( + description="Advisory ratchet: no silent per-cell collapse of output" + " volume between the merge base and HEAD (mirror of" + " check_output_flood.py).") + ap.add_argument("base", nargs="?", + help="Base git ref (CI: origin/)") + ap.add_argument("--self-test", action="store_true", + help="Positive + negative control, then replay the" + " reference case (#15209)") + ap.add_argument("--json", action="store_true", dest="as_json") + args = ap.parse_args(argv) + + if args.self_test: + return self_test() + + if not args.base: + ap.error("base ref required (or --self-test)") + + base = resolve_base(args.base) + paths = changed_notebooks(base) + rows = compare(base, None, paths) + bad = [r for r in rows if r["regressed"]] + + if args.as_json: + print(json.dumps({"base_ref": args.base, "merge_base": base, + "changed": len(paths), "flagged": len(bad), + "rows": rows}, indent=2, ensure_ascii=False)) + else: + print("base " + str(args.base) + " -> merge-base " + base[:12] + + " | " + str(len(paths)) + " changed notebooks | " + + str(len(bad)) + " flagged (advisory)") + for r in rows: + for f in r["cells"]: + if f["kind"].startswith("exempt-"): + print(" EXEMPT (" + f["kind"][7:] + ") " + + r["notebook"] + " cell " + str(f["cell"]) + " " + + str(f["base"]) + " -> " + str(f["head"]) + + " (x" + str(f["factor"]) + ")") + for r in bad: + print("\nFLAGGED (advisory) " + r["notebook"] + " total " + + str(r["base_total"]) + " -> " + str(r["head_total"])) + for f in r["cells"]: + if f["kind"] in ("signature", "magnitude"): + print(" " + f["kind"].upper() + ": cell " + + str(f["cell"]) + " " + str(f["base"]) + " -> " + + str(f["head"]) + " (x" + str(f["factor"]) + ")") + if bad: + print("\nAdvisory, not a gate: justify the contraction in the PR" + " body (declared lightening, moved content, warning purge)" + " or fix the cause and RE-EXECUTE. The signature signal" + " usually means a re-execution without API keys --" + " re-execute with the keys configured (Stop & Repair).") + return 1 if bad else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/notebook_tools/tests/test_check_output_collapse.py b/scripts/notebook_tools/tests/test_check_output_collapse.py new file mode 100644 index 0000000000..73f0ccdd60 --- /dev/null +++ b/scripts/notebook_tools/tests/test_check_output_collapse.py @@ -0,0 +1,186 @@ +"""Tests for check_output_collapse.py — output-volume collapse axis (#15327). + +Pins the discriminating design (ai-01 measurement on the issue: aggregate +contraction alone is ~100 % FP): the SIGNATURE signal fires on a +graceful-degradation motif over a substantial base, the MAGNITUDE signal on +a per-cell order-of-magnitude loss, and the two mechanically detectable +legitimate causes (content moved to a notebook created by the same diff, +diagnostic-text purge) exempt the magnitude signal only. Benign variance, +sub-floor bases, growth, deleted cells and added notebooks are silent. +The logic under test is the pure ``analyze`` / ``_apply_moved_content`` — +no git, no kernel. +""" +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from check_output_collapse import ( + BASE_FLOOR, + DIAGNOSTIC_LINE_FRACTION, + MAGNITUDE_FACTOR, + MOVED_GAIN_FRACTION, + SIGNATURE_FACTOR, + _apply_moved_content, + _diagnostic_fraction, + _normalize, + analyze, +) + + +def code_cell(cid, text): + return {"cell_type": "code", "id": cid, + "outputs": [{"output_type": "stream", "text": text}]} + + +def nb(code_cells): + return {"cells": [code_cell(k, t) for k, t in code_cells], + "metadata": {}, "nbformat": 4, "nbformat_minor": 5} + + +def base(code_cells, added=False): + """None when the notebook is ADDED by the branch (no baseline).""" + return None if added else nb(code_cells) + + +LONG = "x" * 1000 +SKIP_NOTICE = ("[INFO] API non configuree - execution sautee\n" + "Configurez OPENAI_API_KEY pour executer les exemples\n") +WARNINGS = "".join( + "warning CS8632: Une reference nullable est requise %d\n" % i + for i in range(30)) + + +def test_signature_fires_on_skip_notice(): + got = analyze(base([("a", LONG)]), nb([("a", SKIP_NOTICE)])) + assert got["regressed"] + assert got["cells"][0]["kind"] == "signature" + assert got["cells"][0]["factor"] >= SIGNATURE_FACTOR + + +def test_signature_is_accent_blind(): + # "sautée" / "configurée" with accents must match the accent-stripped + # motifs (both spellings exist in the wild). + got = analyze(base([("a", LONG)]), + nb([("a", "Execution sautée (API non configurée)\n")])) + assert got["cells"] and got["cells"][0]["kind"] == "signature" + + +def test_signature_needs_real_contraction(): + # A motif substring in a GROWN output is not degradation. + got = analyze(base([("a", LONG)]), + nb([("a", "skipped 0 tests\n" + "x" * 1500)])) + assert got["cells"] == [] + + +def test_magnitude_fires_on_order_of_magnitude_loss(): + got = analyze(base([("a", LONG)]), nb([("a", "y" * 40)])) + assert got["regressed"] + assert got["cells"][0]["kind"] == "magnitude" + assert got["cells"][0]["factor"] >= MAGNITUDE_FACTOR + + +def test_benign_variance_silent(): + got = analyze(base([("a", LONG)]), nb([("a", "x" * 700)])) + assert got["cells"] == [] + assert not got["regressed"] + + +def test_subfloor_base_silent(): + # Below the noise floor a base is jitter, even at x400 contraction. + assert len("x" * 400) < BASE_FLOOR + got = analyze(base([("a", "x" * 400)]), nb([("a", "y")])) + assert got["cells"] == [] + + +def test_growth_silent(): + got = analyze(base([("a", LONG)]), nb([("a", "x" * 5000)])) + assert got["cells"] == [] + + +def test_diagnostic_purge_exempt(): + got = analyze(base([("a", WARNINGS)]), nb([("a", "Build succeeded.\n")])) + assert got["cells"][0]["kind"] == "exempt-diagnostic" + assert got["cells"][0]["diagnostic_fraction"] >= DIAGNOSTIC_LINE_FRACTION + assert not got["regressed"] + + +def test_purge_swallowing_real_output_still_fires(): + # Char-weighted classification: a purge that also swallows a large real + # output line must not ride along the short warning lines. + got = analyze(base([("a", WARNINGS + "real: " + "z" * 900 + "\n")]), + nb([("a", "real: " + "z" * 50 + "\n")])) + assert got["cells"][0]["kind"] == "magnitude" + assert got["regressed"] + + +def test_deleted_cell_silent(): + # A PR that deletes code cells loses volume legitimately: the deleted + # cell has no head counterpart, so no finding is produced. + got = analyze(base([("a", LONG), ("b", LONG)]), nb([("a", LONG)])) + assert got["cells"] == [] + assert not got["regressed"] + + +def test_added_notebook_never_regressed(): + got = analyze(base([], added=True), nb([("a", SKIP_NOTICE)])) + assert not got["regressed"] + assert got["added"] + + +def test_index_shift_not_misattributed(): + got = analyze(base([("a", LONG)]), nb([("zz", "y"), ("a", LONG)])) + assert got["cells"] == [] + + +def test_moved_content_exempts_at_diff_scale(): + # Split pattern (#14795): notebook A collapses a cell, notebook B is + # CREATED by the same diff with comparable volume. + rows = [analyze(base([("a", LONG * 3)]), nb([("a", "y" * 30)])), + analyze(base([], added=True), nb([("b", "z" * 1500)]))] + _apply_moved_content(rows) + assert rows[0]["cells"][0]["kind"] == "exempt-moved" + assert not rows[0]["regressed"] + + +def test_insufficient_moved_gain_does_not_exempt(): + rows = [analyze(base([("a", LONG * 3)]), nb([("a", "y" * 30)])), + analyze(base([], added=True), nb([("b", "z" * 300)]))] + _apply_moved_content(rows) + assert rows[0]["cells"][0]["kind"] == "magnitude" + assert rows[0]["regressed"] + + +def test_moved_content_never_exempts_signature(): + rows = [analyze(base([("a", LONG)]), nb([("a", SKIP_NOTICE)])), + analyze(base([], added=True), nb([("b", "z" * 5000)]))] + _apply_moved_content(rows) + assert rows[0]["cells"][0]["kind"] == "signature" + assert rows[0]["regressed"] + + +def test_normalize_strips_accents_and_lowercases(): + assert _normalize("Exécution Sautée (non configuré)") == \ + "execution sautee (non configure)" + + +def test_diagnostic_fraction_char_weighted(): + removed_real = "z" * 900 + got = _diagnostic_fraction(WARNINGS + removed_real + "\n", + removed_real + "\n") + # All removed chars are warnings here: fraction 1.0. + assert got == 1.0 + got = _diagnostic_fraction(WARNINGS + removed_real + "\n", "\n") + # The real 900-char line dominates the removed volume: fraction < 0.8. + assert got < DIAGNOSTIC_LINE_FRACTION + + +def test_constants_pinned(): + # Calibration provenance: reference case factors 14.9/67.6/49.4 (all + # above 10), noise floor 500 chars, moved-gain fraction from ai-01's + # "un volume comparable" on #14795's split. + assert BASE_FLOOR == 500 + assert MAGNITUDE_FACTOR == 10 + assert SIGNATURE_FACTOR == 2 + assert MOVED_GAIN_FRACTION == 0.5 + assert DIAGNOSTIC_LINE_FRACTION == 0.8 diff --git a/scripts/tests/test_fast_lane.py b/scripts/tests/test_fast_lane.py index 31729410d4..61d3e712b3 100644 --- a/scripts/tests/test_fast_lane.py +++ b/scripts/tests/test_fast_lane.py @@ -671,11 +671,11 @@ def test_tranche2_guards_are_absorbed_and_declare_their_warn_rc(): d'etre plus strict que ce qu'il remplace. Le ratchet, lui, porte un pre-contrôle self-test.""" from fast_lane_registry import TRANCHE2 - assert len(TRANCHE2) == 4, ( - "la tranche 2 documente trois formes moteur portees par quatre " - "gardes (deux instances du ratchet autonome : failure-text + " - "output-flood) ; si le nombre change, le commentaire du registre " - "et ce test suivent") + assert len(TRANCHE2) == 5, ( + "la tranche 2 documente trois formes moteur portees par cinq " + "gardes (trois instances du ratchet autonome : failure-text, " + "output-flood puis output-collapse advisory, #14959/#15327) ; si le " + "nombre change, le commentaire du registre et ce test suivent") for guard in TRANCHE2: assert guard.absorbed, f"{guard.name} doit porter absorbed=True" warners = {g.name for g in TRANCHE2 if g.warn_rc} From a1696f975b3b3dddd2d908cbad3b91ac465e5c2a Mon Sep 17 00:00:00 2001 From: jsboige Date: Thu, 10 Sep 2026 12:06:37 +0200 Subject: [PATCH 2/2] =?UTF-8?q?ci(guards,#15327):=20allowlist=20notebook-o?= =?UTF-8?q?utput-collapse-ratchet.yml=20=E2=80=94=20voie=202=20review=20ai?= =?UTF-8?q?-01?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Le workflow de l'organe déclare [self-hosted, coursia-ephemeral, coursia-linux] mais n'était pas dans SELF_HOSTED_WORKFLOW_ALLOWLIST : Scripts Tests échouait sur test_current_repository_self_hosted_jobs_ satisfy_isolation_policy (verdict WORKFLOW_NOT_ALLOWED). Ajout de l'entrée avec commentaire de décision ; scanner local (151 workflows / 191 jobs / 124 self-hosted) exit 0 + 57 tests de politique passés. Co-Authored-By: Claude Sonnet 5 --- scripts/ci/check_self_hosted_runner_policy.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/scripts/ci/check_self_hosted_runner_policy.py b/scripts/ci/check_self_hosted_runner_policy.py index ab7d4f3954..d69e61d7f3 100644 --- a/scripts/ci/check_self_hosted_runner_policy.py +++ b/scripts/ci/check_self_hosted_runner_policy.py @@ -218,6 +218,18 @@ # ci-dessus : garde Python pur stdlib, garde same-repo au niveau job, # shell workflow_dispatch-only absorbe par fast-lane TRANCHE2). "notebook-output-flood-ratchet.yml", + # jumeau output-collapse (#15327, owner myia-po-2023:CoursIA) : verdict + # ADVISORY rendu par fast-lane TRANCHE2 (entree 5, check-run + # `Output-collapse ratchet (base vs PR, advisory)`), shell + # workflow_dispatch-ONLY pour re-run manuel -- meme profil que les deux + # jumeaux ci-dessus. Detecteur check_output_collapse.py, stdlib-only, + # garde same-repo au niveau job, aucun GITHUB_TOKEN cote job. L'organe a + # recu l'approval design (Hermes 02:29Z) ; la review ai-01 03:33Z a + # demande la voie 2 : allowlist canonique + test de politique rejoue + # (test_current_repository_self_hosted_jobs_satisfy_isolation_policy, + # scan du repo courant -- aucune autre liste a synchroniser). + # Rollback = revert de la PR (l'entree disparait de l'allowlist). + "notebook-output-collapse-ratchet.yml", "notebook-validation.yml", "owui-playwright-check.yml", "perimeter-review-guard.yml",