Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
225 changes: 118 additions & 107 deletions .github/workflows/catalog-drift.yml
Original file line number Diff line number Diff line change
@@ -1,107 +1,118 @@
name: Catalog Drift Check

# Purpose: DETECT (non-blocking, read-only) when COURSE_CATALOG.generated.json
# and the README CATALOG-STATUS markers drift out of sync with the notebooks.
#
# IMPORTANT — read-only since #2744 (2026-06-09):
# This workflow DETECTS drift and reports it as a notice annotation, but it
# NO LONGER COMMITS anything back to PR branches. The earlier design
# auto-committed a regenerated catalog onto every PR matching **/*.ipynb or
# **/README.md. Because those commits were bot-authored, catalog-pr-guard.yml
# (retiré depuis, #11012) exempted them — so they landed even when they mass-reverted curated git
# fields (issue_pr_associee, last_validation) via the branch git-history
# heuristic. That is the stale-catalog-silent-revert pattern (513 poisoned
# entries observed on branches, 0 on main — source-verified by po-2025,
# see #2744). It poisoned EVERY notebook/README PR and created a worker
# treadmill (strip -> bot re-commits on the next synchronize).
#
# Per decision #2632 (catalog cron-only, off-branch), the ONLY legitimate
# catalog writer is catalog-cron.yml on main (daily regen + push HEAD:main).
# The blocking guard catalog-pr-guard.yml was removed (#11012: phantom entity,
# 0 run on pull_request). This workflow stays as the read-only detector so
# reviewers see drift, without ever writing to a branch.
#
# This job is intentionally NON-BLOCKING and READ-ONLY. The REQUIRED notebook
# gate remains "Static validation (H.1/H.3/C.1)" in notebook-execution-required.yml.

on:
pull_request:
branches: [ main ]
paths:
- 'MyIA.AI.Notebooks/**/*.ipynb'
- 'MyIA.AI.Notebooks/**/README.md'
- 'scripts/notebook_tools/generate_catalog.py'
- 'scripts/notebook_tools/expand_catalog_markers.py'
- 'COURSE_CATALOG.generated.json'
workflow_dispatch:

permissions:
contents: read

concurrency:
group: catalog-drift-${{ github.ref }}
cancel-in-progress: true

jobs:
catalog-drift:
name: "Notebook catalog drift (read-only)"
# Routage #14283 tranche 3 (decision ai-01 2026-09-02) : jambe Linux
# auto-hebergee. Le `if:` ci-dessous est la garde anti-fork exigee par
# scripts/ci/check_self_hosted_runner_policy.py ; un `runs-on` STATIQUE la
# rend auditable (une expression dynamique leve DYNAMIC_RUNS_ON).
# Les PRs de fork sautent le job -- pr_gate.py compte `skipped` comme OK.
# Retour arriere = remettre `runs-on: ubuntu-latest` + retrait de l'allowlist.
runs-on: [self-hosted, coursia-ephemeral, coursia-linux]
if: github.event.pull_request.head.repo.full_name == null || github.event.pull_request.head.repo.full_name == github.repository

steps:
# Read-only checkout of the PR merge ref. We only generate the catalog and
# diff it against the committed copy to detect drift; we never commit or
# push back to a branch (see #2744). The default merge-ref checkout works
# for both same-repo and fork PRs.
- name: Checkout PR
uses: actions/checkout@v4
with:
fetch-depth: 0
filter: blob:none

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

# PyYAML is required by generate_catalog.py's editorial-review registry
# loader (_load_editorial_registry, added in #8345). Without it the
# generator crashes: ModuleNotFoundError: No module named 'yaml'.
- name: Install PyYAML
run: pip install pyyaml

# Canonical regeneration (local to the runner only — nothing is written
# back to the branch). git-tracked notebooks only, for CI determinism.
- name: Regenerate catalog + README markers
run: |
git fetch origin main --depth=1 || true
python scripts/notebook_tools/generate_catalog.py --json-only --git-tracked-only
python scripts/notebook_tools/expand_catalog_markers.py

- name: Detect drift
id: diff
run: |
# Only the catalog JSON and the README marker files are in scope.
git add COURSE_CATALOG.generated.json MyIA.AI.Notebooks/README.md 'MyIA.AI.Notebooks/*/README.md' 2>/dev/null || true
if git diff --cached --quiet; then
echo "drift=false" >> "$GITHUB_OUTPUT"
echo "Catalog and README markers are already in sync. No drift."
else
echo "drift=true" >> "$GITHUB_OUTPUT"
echo "Catalog drift detected in:"
git diff --cached --name-only | sed 's/^/ - /'
echo "::notice title=Catalog drift (informational)::The catalog/README markers are out of sync with the notebooks. This does NOT block the PR. The catalog is regenerated daily on main by catalog-cron.yml; no manual action is required on this branch."
fi

# This job is always green. Drift is surfaced as a notice annotation only.
# catalog-cron.yml (main) is the sole writer; the blocking guard
# catalog-pr-guard.yml was removed (#11012) — this read-only detector and
# the catalog-cron.yml regen PR are the sole organs now (see #2632, #2744).
- name: Done (non-blocking, read-only)
run: echo "Catalog drift check complete (non-blocking, read-only)."
name: Catalog Drift Check

# Purpose: DETECT (non-blocking, read-only) when COURSE_CATALOG.generated.json
# and the README CATALOG-STATUS markers drift out of sync with the notebooks.
#
# IMPORTANT — read-only since #2744 (2026-06-09):
# This workflow DETECTS drift and reports it as a notice annotation, but it
# NO LONGER COMMITS anything back to PR branches. The earlier design
# auto-committed a regenerated catalog onto every PR matching **/*.ipynb or
# **/README.md. Because those commits were bot-authored, catalog-pr-guard.yml
# (retiré depuis, #11012) exempted them — so they landed even when they mass-reverted curated git
# fields (issue_pr_associee, last_validation) via the branch git-history
# heuristic. That is the stale-catalog-silent-revert pattern (513 poisoned
# entries observed on branches, 0 on main — source-verified by po-2025,
# see #2744). It poisoned EVERY notebook/README PR and created a worker
# treadmill (strip -> bot re-commits on the next synchronize).
#
# Per decision #2632 (catalog cron-only, off-branch), the ONLY legitimate
# catalog writer is catalog-cron.yml on main (daily regen + push HEAD:main).
# The blocking guard catalog-pr-guard.yml was removed (#11012: phantom entity,
# 0 run on pull_request). This workflow stays as the read-only detector so
# reviewers see drift, without ever writing to a branch.
#
# This job is intentionally NON-BLOCKING and READ-ONLY. The REQUIRED notebook
# gate remains "Static validation (H.1/H.3/C.1)" in notebook-execution-required.yml.

on:
pull_request:
branches: [ main ]
paths:
- 'MyIA.AI.Notebooks/**/*.ipynb'
- 'MyIA.AI.Notebooks/**/README.md'
- 'scripts/notebook_tools/generate_catalog.py'
- 'scripts/notebook_tools/expand_catalog_markers.py'
- 'COURSE_CATALOG.generated.json'
workflow_dispatch:

permissions:
contents: read

concurrency:
group: catalog-drift-${{ github.ref }}
cancel-in-progress: true

jobs:
catalog-drift:
# Le marqueur `advisory` dans le NOM du job EST le contrat (#15998) : pr_gate.py
# classe par nom (`ADVISORY_MARKER`, regle 6) et ne lit PAS fast_lane_registry.py.
# Sans ce mot, la promesse "non-bloquant" de l'en-tete restait sans effet sur le
# gate : un echec d'infrastructure (runner, pip, generate_catalog) comptait comme
# un check requis et bloquait toute PR notebook/README -- observe sur #15996.
# Ne pas retirer le mot sans relire scripts/pr_gate.py (rule 6) et #15998.
name: "Notebook catalog drift (read-only, advisory)"
# Routage #14283 tranche 3 (decision ai-01 2026-09-02) : jambe Linux
# auto-hebergee. Le `if:` ci-dessous est la garde anti-fork exigee par
# scripts/ci/check_self_hosted_runner_policy.py ; un `runs-on` STATIQUE la
# rend auditable (une expression dynamique leve DYNAMIC_RUNS_ON).
# Les PRs de fork sautent le job -- pr_gate.py compte `skipped` comme OK.
# Retour arriere = remettre `runs-on: ubuntu-latest` + retrait de l'allowlist.
runs-on: [self-hosted, coursia-ephemeral, coursia-linux]
if: github.event.pull_request.head.repo.full_name == null || github.event.pull_request.head.repo.full_name == github.repository

steps:
# Read-only checkout of the PR merge ref. We only generate the catalog and
# diff it against the committed copy to detect drift; we never commit or
# push back to a branch (see #2744). The default merge-ref checkout works
# for both same-repo and fork PRs.
- name: Checkout PR
uses: actions/checkout@v4
with:
fetch-depth: 0
filter: blob:none

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

# PyYAML is required by generate_catalog.py's editorial-review registry
# loader (_load_editorial_registry, added in #8345). Without it the
# generator crashes: ModuleNotFoundError: No module named 'yaml'.
- name: Install PyYAML
run: pip install pyyaml

# Canonical regeneration (local to the runner only — nothing is written
# back to the branch). git-tracked notebooks only, for CI determinism.
- name: Regenerate catalog + README markers
run: |
git fetch origin main --depth=1 || true
python scripts/notebook_tools/generate_catalog.py --json-only --git-tracked-only
python scripts/notebook_tools/expand_catalog_markers.py

- name: Detect drift
id: diff
run: |
# Only the catalog JSON and the README marker files are in scope.
git add COURSE_CATALOG.generated.json MyIA.AI.Notebooks/README.md 'MyIA.AI.Notebooks/*/README.md' 2>/dev/null || true
if git diff --cached --quiet; then
echo "drift=false" >> "$GITHUB_OUTPUT"
echo "Catalog and README markers are already in sync. No drift."
else
echo "drift=true" >> "$GITHUB_OUTPUT"
echo "Catalog drift detected in:"
git diff --cached --name-only | sed 's/^/ - /'
echo "::notice title=Catalog drift (informational)::The catalog/README markers are out of sync with the notebooks. This does NOT block the PR. The catalog is regenerated daily on main by catalog-cron.yml; no manual action is required on this branch."
fi

# This job is always green. Drift is surfaced as a notice annotation only.
# catalog-cron.yml (main) is the sole writer; the blocking guard
# catalog-pr-guard.yml was removed (#11012) — this read-only detector and
# the catalog-cron.yml regen PR are the sole organs now (see #2632, #2744).
# CONTROL #15998 -- etape d'echec forcee, branche jetable (PR de controle).
- name: "CONTROL #15998 -- forced failure"
run: |
echo "Echec force pour observer la classification du check par PR gate (#15998)."
exit 1
- name: Done (non-blocking, read-only)
run: echo "Catalog drift check complete (non-blocking, read-only)."
2 changes: 2 additions & 0 deletions MyIA.AI.Notebooks/Probas/DecisionTheory/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,3 +108,5 @@ Chaque sous-série démarre par ses fondations (`DecInfer-01` / `DecPyMC-1`) et
---

*Cet arc fait partie de la série [Probas](../README.md) du dépôt [CoursIA](https://github.com/jsboige/CoursIA). Documentation détaillée de chaque notebook dans les README des sous-séries.*

<!-- #15998 control: forced catalog-drift trigger (throwaway PR, closed after observation) -->
2 changes: 1 addition & 1 deletion docs/reference/ci-aggregator-rollout.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ GitHub = code pas rapport), §G.6 (audit avant merge cascade).
| `ci / No fabricated text output in changed notebooks` | required | C.4 doc-honesty (#8052) |
| `ci / No degenerate figure in changed notebooks` | required | GenAI rendering (cf #6541) |
| `ci / No bare cross-dir #load in changed notebooks` | required | coupling cellule cross-répertoire |
| `ci / Notebook catalog drift (read-only)` | advisory | catalog-pr-hygiene R1 — détecteur read-only ; le garde bloquant `catalog-pr-guard.yml` est retiré (#11012) |
| `ci / Notebook catalog drift (read-only, advisory)` | advisory | catalog-pr-hygiene R1 — détecteur read-only ; le garde bloquant `catalog-pr-guard.yml` est retiré (#11012). Le marqueur `advisory` du nom est ce qui rend le gate conforme (#15998 : sans lui, un échec d'infra comptait comme requis) |
| `ci / Gitleaks secret scanner` | required | secrets-hygiene |
| `hooks-parity gate` | required | #8782 — gate qui ne peut plus rougir |

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/procedures-recurrentes.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ Toute PR touchant `COURSE_CATALOG.generated.json` rend les autres PRs catalog-to
- Merger les PRs **non-catalogue d'abord**, puis les catalog-touchers **un-par-un**.
- Conflit catalogue = l'auteur **rebase + régénère** : `python scripts/notebook_tools/generate_catalog.py --json --git-tracked-only` (parité CI = N entrées git-tracked ; `--json` nu inclut les `_output.ipynb` locaux = drift), puis `expand_catalog_markers.py`.
- **JAMAIS** force-resolve le conflit côté coord ; **JAMAIS** force-push la branche de l'auteur.
- Check CI **rouge "Notebook catalog drift" = NON mergeable** (propagerait le drift) → bounce à l'auteur pour re-régénérer.
- Check CI **"Notebook catalog drift (read-only, advisory)"** : **advisory, jamais bloquant** (#15998) — le catalogue est réécrit par `catalog-cron.yml` sur `main` (#2632/#2744), une PR n'a rien à régénérer. Un drift signalé = notice informative, pas un bounce.
- **Cascade-independence** : une PR dont le drift-check = SUCCESS **et** qui ne touche pas le catalogue est **indépendante** — ne pas la hold à tort dans la file cascade.

### Trap "APPROVED"
Expand Down
38 changes: 38 additions & 0 deletions scripts/tests/test_pr_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -906,6 +906,44 @@ def test_green_advisory_counts_as_a_normal_pass():
assert ok == ["Large blob advisory (>= 10 MiB)"]


def test_catalog_drift_job_name_carries_the_advisory_marker():
"""#15998 -- the catalog-drift job declared itself NON-BLOCKING twice in its
own header (and in catalog-pr-hygiene / ci-aggregator docs), but
`pr_gate.py` classifies advisory by NAME (rule 6, ADVISORY_MARKER) and never
reads `fast_lane_registry.py`. Named "Notebook catalog drift (read-only)",
the job carried no marker, so any infrastructure failure (runner, pip
install, `generate_catalog` exit 2 on missing git metadata, cf #14831) was
counted as a REQUIRED check and reddened every notebook/README PR --
observed firsthand on #15996: "PR gate: FAIL -- failing checks: Notebook
catalog drift (read-only) (failure)" while every other check passed.

The repair is the marker in the job name, not a registry entry: the
registry is consumed by the fast lane (which would then also RUN the
catalog generation on every PR), whereas `is_advisory` reads the emitted
check-run name. Asserted over whatever the job is called today -- so a
future rename that keeps the marker passes, and one that drops it fails.
"""
if pr_gate.yaml is None: # pragma: no cover - PyYAML is a CI dependency
pytest.skip("PyYAML unavailable: cannot read the workflow")
wf_path = Path(pr_gate.DEFAULT_WORKFLOWS_DIR) / "catalog-drift.yml"
data = pr_gate.yaml.safe_load(wf_path.read_text(encoding="utf-8"))
job_names = pr_gate._workflow_job_names(data)
assert job_names, "catalog-drift.yml must declare at least one job"
for name in job_names:
assert pr_gate.is_advisory(name), (
"the catalog-drift job name must carry the `advisory` marker "
"(#15998): pr_gate.py reads the check-run name, not the header "
"comment, so dropping the marker silently re-arms a hard gate "
"against every PR touching a notebook or a series README"
)
# The defect the marker fixes must stay measurable: with the workflow name
# as it stands, the historical spelling is still classified BLOCKING. If
# this ever flips, the workflow name has absorbed the marker and the
# job-name invariant above stopped being the load-bearing surface.
wf_name = data.get("name") or ""
assert not pr_gate.is_advisory("Notebook catalog drift (read-only)", wf_name)


def test_non_advisory_failure_still_blocks():
"""Guard against the fix becoming a blanket amnesty."""
checks = [
Expand Down
Loading