Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/workflows/catalog-drift.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,13 @@ concurrency:

jobs:
catalog-drift:
name: "Notebook catalog drift (read-only)"
# Le marqueur `advisory` dans le NOM du job EST le contrat (#15998) : pr_gate.py
# classe par nom (`ADVISORY_MARKER`, regle 6) et ne lit PAS fast_lane_registry.py.
# Sans ce mot, la promesse "non-bloquant" de l'en-tete restait sans effet sur le
# gate : un echec d'infrastructure (runner, pip, generate_catalog) comptait comme
# un check requis et bloquait toute PR notebook/README -- observe sur #15996.
# Ne pas retirer le mot sans relire scripts/pr_gate.py (rule 6) et #15998.
name: "Notebook catalog drift (read-only, advisory)"
# Routage #14283 tranche 3 (decision ai-01 2026-09-02) : jambe Linux
# auto-hebergee. Le `if:` ci-dessous est la garde anti-fork exigee par
# scripts/ci/check_self_hosted_runner_policy.py ; un `runs-on` STATIQUE la
Expand Down
31 changes: 23 additions & 8 deletions docs/reference/catalog_markers.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Catalog Markers - README Auto-Update System

Source-of-truth counts driven by `COURSE_CATALOG.generated.json`. Markers in README files are expanded by `scripts/notebook_tools/expand_catalog_markers.py` and verified by CI on every PR.
Source-of-truth counts driven by `COURSE_CATALOG.generated.json`. Markers in README files are expanded by `scripts/notebook_tools/expand_catalog_markers.py` and verified by CI on PRs that touch notebooks, series READMEs, or the catalog — the `catalog-drift.yml` workflow is filtered by `paths:`, so it does not run on every PR.

## Overview

Expand Down Expand Up @@ -75,7 +75,7 @@ python scripts/notebook_tools/expand_catalog_markers.py
# Dry-run (show what would change)
python scripts/notebook_tools/expand_catalog_markers.py --dry-run

# Check for drift (exit 1 if stale, used by CI)
# Check for drift (exit 1 if stale; local tool -- CI regenerates instead and never calls --check)
python scripts/notebook_tools/expand_catalog_markers.py --check

# Expand a specific file
Expand All @@ -100,12 +100,27 @@ on:
- 'COURSE_CATALOG.generated.json'
```

Two checks run in sequence:

1. **CATALOG-STATUS marker drift** — `expand_catalog_markers.py --check` verifies all markers match the catalog
2. **Notebook catalog drift** — `verify_catalog_readme.py` checks declared counts vs actual notebooks on disk

If either check fails, the PR is blocked until markers are updated.
Le job régénère le catalogue et les marqueurs **sur le runner** (rien n'est réécrit sur
la branche), puis compare le résultat aux fichiers commités par un unique test
`git diff --cached`. La dérive est remontée en **annotation `notice` uniquement**.

**Ce check est advisory, non bloquant** (#15998). Le marqueur `advisory` dans le **nom**
du job — `Notebook catalog drift (read-only, advisory)` — **est** le contrat :
`pr_gate.py` classe les checks par nom et ne lit pas `fast_lane_registry.py`. Le job **peut rougir** : seule l'indisponibilité des métadonnées git (`rc=2` de
`generate_catalog.py`) est absorbée en annotation `notice` ; tout autre échec (runner, checkout,
`pip`, `generate_catalog.py` hors `rc=2`) exécute `exit "$rc"` et rend le job rouge. Mais ce
rouge est **exclu des causes bloquantes** : le marqueur `advisory` du nom fait que `PR gate`
le signale sans bloquer — une panne d'infrastructure est remontée, jamais bloquante pour une PR
notebook/README (contrôle positif #16015). Le catalogue est régénéré quotidiennement
sur `main` par `catalog-cron.yml` ; **aucune action manuelle n'est requise sur une branche
de feature** (cf [catalog-pr-hygiene.md](../../.claude/rules/catalog-pr-hygiene.md), #2632).

> Correction factuelle (2026-09-15) : cette section décrivait deux checks en séquence
> (`expand_catalog_markers.py --check`, `verify_catalog_readme.py`) et concluait qu'un
> échec **bloquait** la PR jusqu'à mise à jour des marqueurs. Les deux affirmations
> étaient fausses : le workflow n'utilise pas `--check` (il régénère), ne fait appel à
> `verify_catalog_readme.py` dans **aucun** workflow, et son job est advisory depuis
> #15998.

## Adding Markers to a New README

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/ci-aggregator-rollout.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ GitHub = code pas rapport), §G.6 (audit avant merge cascade).
| `ci / No fabricated text output in changed notebooks` | required | C.4 doc-honesty (#8052) |
| `ci / No degenerate figure in changed notebooks` | required | GenAI rendering (cf #6541) |
| `ci / No bare cross-dir #load in changed notebooks` | required | coupling cellule cross-répertoire |
| `ci / Notebook catalog drift (read-only)` | advisory | catalog-pr-hygiene R1 — détecteur read-only ; le garde bloquant `catalog-pr-guard.yml` est retiré (#11012) |
| `ci / Notebook catalog drift (read-only, advisory)` | advisory | catalog-pr-hygiene R1 — détecteur read-only ; le garde bloquant `catalog-pr-guard.yml` est retiré (#11012). Le marqueur `advisory` du nom est ce qui rend le gate conforme (#15998 : sans lui, un échec d'infra comptait comme requis) |
| `ci / Gitleaks secret scanner` | required | secrets-hygiene |
| `hooks-parity gate` | required | #8782 — gate qui ne peut plus rougir |

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/procedures-recurrentes.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ Toute PR touchant `COURSE_CATALOG.generated.json` rend les autres PRs catalog-to
- Merger les PRs **non-catalogue d'abord**, puis les catalog-touchers **un-par-un**.
- Conflit catalogue = l'auteur **rebase + régénère** : `python scripts/notebook_tools/generate_catalog.py --json --git-tracked-only` (parité CI = N entrées git-tracked ; `--json` nu inclut les `_output.ipynb` locaux = drift), puis `expand_catalog_markers.py`.
- **JAMAIS** force-resolve le conflit côté coord ; **JAMAIS** force-push la branche de l'auteur.
- Check CI **rouge "Notebook catalog drift" = NON mergeable** (propagerait le drift) → bounce à l'auteur pour re-régénérer.
- Check CI **"Notebook catalog drift (read-only, advisory)"** : **advisory, jamais bloquant** (#15998) — le catalogue est réécrit par `catalog-cron.yml` sur `main` (#2632/#2744), une PR n'a rien à régénérer. Un drift signalé = notice informative, pas un bounce.
- **Cascade-independence** : une PR dont le drift-check = SUCCESS **et** qui ne touche pas le catalogue est **indépendante** — ne pas la hold à tort dans la file cascade.

### Trap "APPROVED"
Expand Down
8 changes: 6 additions & 2 deletions scripts/notebook_tools/fix_catalog_drift.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
"""One-shot fix: swap `breakdown: projects=48, Python=48` → `breakdown: Python=48, projects=48`
in MyIA.AI.Notebooks/QuantConnect/README.md, preserving CRLF/LF unchanged.

Used to unblock UNSTABLE PRs whose CI fails on `Notebook catalog drift` due to
non-deterministic Counter.most_common() tie-break (Windows vs Linux sort order).
Historique : ecrit pour debloquer des PRs rendues UNSTABLE par le check
`Notebook catalog drift`, a l'epoque ou `PR gate` le comptait comme un check
requis. Ce check est **advisory** depuis #15998 -- un drift ne bloque plus
aucune PR, puisque le catalogue est regenere par catalog-cron.yml sur main
(#2632/#2744). Le script reste utile comme reparation locale d'un drift de
tie-break Counter.most_common() non deterministe (tri Windows vs Linux).

Usage: python scripts/notebook_tools/fix_catalog_drift.py
"""
Expand Down
38 changes: 38 additions & 0 deletions scripts/tests/test_pr_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -1085,6 +1085,44 @@ def test_green_advisory_counts_as_a_normal_pass():
assert ok == ["Large blob advisory (>= 10 MiB)"]


def test_catalog_drift_job_name_carries_the_advisory_marker():
"""#15998 -- the catalog-drift job declared itself NON-BLOCKING twice in its
own header (and in catalog-pr-hygiene / ci-aggregator docs), but
`pr_gate.py` classifies advisory by NAME (rule 6, ADVISORY_MARKER) and never
reads `fast_lane_registry.py`. Named "Notebook catalog drift (read-only)",
the job carried no marker, so any infrastructure failure (runner, pip
install, `generate_catalog` exit 2 on missing git metadata, cf #14831) was
counted as a REQUIRED check and reddened every notebook/README PR --
observed firsthand on #15996: "PR gate: FAIL -- failing checks: Notebook
catalog drift (read-only) (failure)" while every other check passed.

The repair is the marker in the job name, not a registry entry: the
registry is consumed by the fast lane (which would then also RUN the
catalog generation on every PR), whereas `is_advisory` reads the emitted
check-run name. Asserted over whatever the job is called today -- so a
future rename that keeps the marker passes, and one that drops it fails.
"""
if pr_gate.yaml is None: # pragma: no cover - PyYAML is a CI dependency
pytest.skip("PyYAML unavailable: cannot read the workflow")
wf_path = Path(pr_gate.DEFAULT_WORKFLOWS_DIR) / "catalog-drift.yml"
data = pr_gate.yaml.safe_load(wf_path.read_text(encoding="utf-8"))
job_names = pr_gate._workflow_job_names(data)
assert job_names, "catalog-drift.yml must declare at least one job"
for name in job_names:
assert pr_gate.is_advisory(name), (
"the catalog-drift job name must carry the `advisory` marker "
"(#15998): pr_gate.py reads the check-run name, not the header "
"comment, so dropping the marker silently re-arms a hard gate "
"against every PR touching a notebook or a series README"
)
# The defect the marker fixes must stay measurable: with the workflow name
# as it stands, the historical spelling is still classified BLOCKING. If
# this ever flips, the workflow name has absorbed the marker and the
# job-name invariant above stopped being the load-bearing surface.
wf_name = data.get("name") or ""
assert not pr_gate.is_advisory("Notebook catalog drift (read-only)", wf_name)


def test_non_advisory_failure_still_blocks():
"""Guard against the fix becoming a blanket amnesty."""
checks = [
Expand Down
Loading