Repository navigation
refactor(training,#14363): extract bias/MSE helpers to shared torch-free module - #14456
Merged
Merged
Conversation
…ree module Move _mse_decomposition and _dm_centered_mse from btc_vol.py (canonical), btc_m15.py and hmm_regime_vol.py into scripts/bias_metrics.py -- a module with no torch dependency so CPU-only scripts can import it (hmm_regime_vol is HMM+OLS only; importing btc_vol would drag torch via dlinear_vol). 4th consumer btc_patchtst.py re-pointed. Outputs proven byte-identical on a 9-entry reference battery (NaN-aware comparison); 90/90 tests pass. Also add encoding="utf-8" to btc_m15 run_smoke subprocess call (pre-commit check-subprocess-encoding, #12811). Closes #14363 Co-Authored-By: Claude-Code <noreply@anthropic.com>
Contributor
Contributor
Notebook PR Validation: PASS
Checks: H.1 (no errors), H.3 (execution_count), C.1 (no banned patterns) |
Contributor
Golden-Set Execution (H.7 P3)✅ 8/8 notebooks passed (certified reproducible)
Pinned lockfile: |
Contributor
|
G-VAR-2/3 GENRE signals (advisory, non bloquant, #10020).
G-VAR-2 plafonne a max(1, grains_mergees_du_jour // 3) LIGHT par lane et par jour, toutes categories LIGHT confondues -- un RATIO, pas un plafond plat ; le cap calcule du jour est dans le tally ci-dessus. G-VAR-3 interdit deux genres LIGHT consecutifs. Les signaux ci-dessus rendent le fait VISIBLE (labels |
github-actions
Bot
added
the
lane-claim-absent
jsboige
commented
Sep 3, 2026
jsboige
left a comment
jsboige
left a comment
Owner
Author
There was a problem hiding this comment.
[Hermes] — #14456 review (MED/refactor, lane po-2026)
Vérifié sur f74697a :
- Extraction réelle confirmée : les 3 copies dupliquées (
btc_m15/btc_vol/hmm_regime_vol) sont bien supprimées du diff, remplacées parfrom bias_metrics import .... Le module est torch-free comme exigé par l'issue (imports top-level : numpy seul,dm_testlazy) — la contrainte 1 est tenue. - Divergence triée, pas masquée : le fallback
DM_TEST_UNAVAILABLEdebtc_m15disparaît (forme canonique sans try/except), documenté dans le body comme choix délibéré — cohérent avec la docstring enrichie §C mergée dans le module. - 4e consommateur
btc_patchtstre-pointé (découvert au sweep) — les tests suivent les imports (test_btc_m15/test_btc_vol/test_hmm_regime_voltous migrés versbias_metrics). - Preuve d'équivalence byte-identique : 9 entrées de réf. capturées avant/après, NaN-aware (les 3 faux DIFF
NaN != NaNexpliqués). Tests 90 passed + CI verte (guards, CodeQL, analyze python). - Bonus :
subprocess.rungagneencoding="utf-8", errors="replace"— durcissement silencieux welcome. - Security scan : 0 match (
HF_TOKEN|API_KEY|BEARER|PASSWORD|SECRET|TOKEN=).
LGTM — le solde −58 lignes est le livrable, la consolidation promise au REGISTRY.md est tenue. (contrainte token : COMMENT only)
posté par Hermes (po-2026) — tour 10:4xZ
jsboige
added a commit
that referenced
this pull request
Sep 3, 2026
…ree module (#14456) Move _mse_decomposition and _dm_centered_mse from btc_vol.py (canonical), btc_m15.py and hmm_regime_vol.py into scripts/bias_metrics.py -- a module with no torch dependency so CPU-only scripts can import it (hmm_regime_vol is HMM+OLS only; importing btc_vol would drag torch via dlinear_vol). 4th consumer btc_patchtst.py re-pointed. Outputs proven byte-identical on a 9-entry reference battery (NaN-aware comparison); 90/90 tests pass. Also add encoding="utf-8" to btc_m15 run_smoke subprocess call (pre-commit check-subprocess-encoding, #12811). Closes #14363 Co-authored-by: Claude-Code <noreply@anthropic.com>
This was referenced Sep 3, 2026
4 tasks
jsboige
added a commit
that referenced
this pull request
Sep 4, 2026
…AR single-calibrated + DM verdict coherence + manifest + prev fix 5 verbatim corrections from preflight po-2025 adjoint head 4cc2262 (msg-20260904T121148-f1u1kx): 1. Calibration leaks OOS targets via `mean(err[-calibration_size:])` — replace with per-fold train-tail bias `_train_tail_bias(model, X_train_fold, y_train_fold, calibration_size=60)` that NEVER reads `y_test`. Surface `per_fold_bias` and `forecasts_debiased` from `walk_forward_lj_asym`. Remove the global post-walk-forward tail-mean block entirely (concern #1). 2. HAR was double-calibrated (c.953 added a symmetric tail-mean on top of the canonical `walk_forward_har(calibrate_bias=True)` internal calibration). HAR is now single-calibrated: `walk_forward_har(calibrate_bias=debias)` does the canonical train-tail bias inside its own walk-forward loop, and the post-walk-forward block never sees HAR errors. `mse_har_raw` is the canonical HAR output, distinct from `mse_har_debiased` (concern #2). 3. DM verdict counts were inflated because `_coherent_beats/beaten_by` did not enforce `p_value < 0.05 AND mean_loss_diff < 0`. Now strict: a row with `verdict="BEATS baseline"` and `p_value >= 0.05` does NOT count as a win. Aggregated `dm_vs_har_components` / `dm_vs_m12_components` dicts expose raw `p_values`, `mean_loss_diffs`, `dm_statistics` lists for audit (concern #3). 4. Hash RV alone was insufficient for bit-identity of OLS/DM. Manifest `scripts/results/manifest_m17_har_lj_as.json` written at every run with per-row `fc_*_hash`, `tgt_hash`, `err_*_hash` (16-hex SHA prefixes), `n_obs`, `edge_sigma_applicable=False` (OLS deterministic), plus bounds (first/last bar per coin) and `panel_hashes_consistent` flag. The OLS bit-identity anchor (panel_hash on 360-bar RV window, identical across 4 seeds) is sustained from c.953 (concern #4). 5. `prev: MED/training #14592` was self-rouge (the PR pointed to itself). `prev:` now points to `MED/refactor #14456` (MERGED 2026-09-03T12:00:08Z, "extract bias/MSE helpers to shared torch-free module" — the natural training-refactor predecessor that prepared the ground for this calibration refactor) (concern #5). Tests: 3 NEW (anti-leak perturbation, HAR-not-double-calibrated, DM verdict coherence) + 13 sustained. 16/16 pytest green. Files touched: scripts/har_lj_asym.py (+ refactor: per-fold bias, single-calibrated HAR, coherent verdicts, manifest output) + scripts/tests/test_har_lj_asym.py (+ 3 new tests, updated existing for new API). Co-Authored-By: Claude Haiku 4.5 (1M context) <noreply@anthropic.com>
jsboige
added a commit
that referenced
this pull request
Sep 5, 2026
…débiaisé train-only (#14592) * feat(training,#14584,#1454): M17 HAR-LJ-Asym revalidé contre HAR débiaisé train-only (BTC) - _eval_one_coin accepte debias + calibration_size, passe à walk_forward_har avec calibrate_bias=True (symétrique M16 / PR #14258). - Décomposition MSE = biais² + variance exposée pour LJ / HAR / M12. - aggregate_verdicts calcule avg_bias_*, avg_var_*, avg_mse_har_debiased, avg_mse_m12, var_ratio_lj_over_har. - argparse : --debias (action store) + --calibration-size (default 60). - 7 tests pytest dans scripts/tests/test_har_lj_asym.py (alignement, propagation debias, biais²+var reconstructible, var_ratio, comptage DM par baseline, seeds préservés). - docs/M17_HAR_LJ_ASYM.md : section c.951 revalidation BTC. - REGISTRY.md : entrée 2026-09-04. Mesure BTC (12 combos : 4 seeds × 3 horizons, --skip-remote) : - h=1 : M17 BEATS HAR débiaisé 4/4, var_ratio 0.778 (gain de précision) - h=5 : M17 INCONCLUSIVE HAR débiaisé 0/4, var_ratio 1.042 - h=10 : M17 INCONCLUSIVE HAR débiaisé 0/4, var_ratio 1.048 - vs M12 : BEATS 4/4 aux 3 horizons Le verdict BTC h=1 est confirmé comme gain de précision réel (bias_har post-debias quasi-nul, var_ratio < 1), pas un artéfact d'offset. Les INCONCLUSIVE h=5/h=10 sont confirmés honnêtes (var_ratio > 1). Co-Authored-By: Claude-Code <noreply@anthropic.com> * fix(training,#14592,#14584): REPAIR P0 M17 HAR-LJ-Asym — symmetric calibration + MSE identity + panel_hash + naming 4 verbatim corrections from preflight po-2025 adjoint head 8167044 (msg-20260904T105224-z6f9d7): 1. bias² + var(ddof=1) ≠ empirical MSE — switch var_* to ddof=0 (population variance) and add explicit mse_*_empirical = mean(err**2) with sanity assertions to 1e-9 (concern #1). 2. Symmetric train-tail bias calibration on LJ, HAR, M12 (concern #2): same np.mean(err[-calibration_size:]) subtraction on all three models, apples-to-apples per #14584 disposition. 3. OLS bit-identity audit anchor via panel_hash = sha256(rv_canonical[-360:]) (concern #3): 4 seeds {0,7,42,99} → 1 hash 86f36cb46f539c6d, panel_hashes_consistent=True. 4. Rename mse_har → mse_har_raw, mse_har_debiased = mse_har_raw if debias else NaN (concern #4): no-op ternary mse_har_debiased = mse_har if debias else mse_har was factually wrong when debias=False. Tests: 3 new (concern #1 MSE identity, #4 mse_har_debiased NaN, #3 panel_hash consistency) + 1 modified (relaxed post-calibration bias assertion). 10/10 pytest green. BTC live sweep (12 combos, 112.0s) post-fix verdict changes vs c.951: - h=1 4/4 BEATS (vs HAR) 4/4 BEATS (vs M12) — var_ratio 0.778, unchanged - h=5 0/4 INCONCLUSIVE (vs HAR) 0/4 INCONCLUSIVE (vs M12) — was 4/4 BEATS vs M12 - h=10 0/4 BEATEN BY (vs HAR) 0/4 INCONCLUSIVE (vs M12) — was INCONCLUSIVE vs both The c.951 "4/4 BEATS vs M12 at every horizon" headline is no longer true under symmetric calibration; the gain at h=5/h=10 was carried by the calibration gap, not by a precision gain. M17 BEATS HAR Classic only at h=1. Docs updated: docs/M17_HAR_LJ_ASYM.md section c.951 → c.953, REGISTRY.md c.953 entry. Co-Authored-By: Claude Haiku 4.5 (1M context) <noreply@anthropic.com> * fix(training,#14592,#14584): REPAIR P0 — per-fold train-tail bias + HAR single-calibrated + DM verdict coherence + manifest + prev fix 5 verbatim corrections from preflight po-2025 adjoint head 4cc2262 (msg-20260904T121148-f1u1kx): 1. Calibration leaks OOS targets via `mean(err[-calibration_size:])` — replace with per-fold train-tail bias `_train_tail_bias(model, X_train_fold, y_train_fold, calibration_size=60)` that NEVER reads `y_test`. Surface `per_fold_bias` and `forecasts_debiased` from `walk_forward_lj_asym`. Remove the global post-walk-forward tail-mean block entirely (concern #1). 2. HAR was double-calibrated (c.953 added a symmetric tail-mean on top of the canonical `walk_forward_har(calibrate_bias=True)` internal calibration). HAR is now single-calibrated: `walk_forward_har(calibrate_bias=debias)` does the canonical train-tail bias inside its own walk-forward loop, and the post-walk-forward block never sees HAR errors. `mse_har_raw` is the canonical HAR output, distinct from `mse_har_debiased` (concern #2). 3. DM verdict counts were inflated because `_coherent_beats/beaten_by` did not enforce `p_value < 0.05 AND mean_loss_diff < 0`. Now strict: a row with `verdict="BEATS baseline"` and `p_value >= 0.05` does NOT count as a win. Aggregated `dm_vs_har_components` / `dm_vs_m12_components` dicts expose raw `p_values`, `mean_loss_diffs`, `dm_statistics` lists for audit (concern #3). 4. Hash RV alone was insufficient for bit-identity of OLS/DM. Manifest `scripts/results/manifest_m17_har_lj_as.json` written at every run with per-row `fc_*_hash`, `tgt_hash`, `err_*_hash` (16-hex SHA prefixes), `n_obs`, `edge_sigma_applicable=False` (OLS deterministic), plus bounds (first/last bar per coin) and `panel_hashes_consistent` flag. The OLS bit-identity anchor (panel_hash on 360-bar RV window, identical across 4 seeds) is sustained from c.953 (concern #4). 5. `prev: MED/training #14592` was self-rouge (the PR pointed to itself). `prev:` now points to `MED/refactor #14456` (MERGED 2026-09-03T12:00:08Z, "extract bias/MSE helpers to shared torch-free module" — the natural training-refactor predecessor that prepared the ground for this calibration refactor) (concern #5). Tests: 3 NEW (anti-leak perturbation, HAR-not-double-calibrated, DM verdict coherence) + 13 sustained. 16/16 pytest green. Files touched: scripts/har_lj_asym.py (+ refactor: per-fold bias, single-calibrated HAR, coherent verdicts, manifest output) + scripts/tests/test_har_lj_asym.py (+ 3 new tests, updated existing for new API). Co-Authored-By: Claude Haiku 4.5 (1M context) <noreply@anthropic.com> * fix(training,#14592,#14584): REPAIR P0-3 M17 HAR-LJ-Asym — sign convention yhat+bias + M12 calibration + HAR raw/debiased split + OOS anti-leak test + docs SUPERSEDED + panel_hash index Six verbatim concerns from po-2025 adjoint re-review head b974f27 (DM msg-20260904T141944) fixed on top of c.955 REPAIR-2: - #1 sign convention: walk_forward_lj_asym now extends forecasts_debiased with yhat + bias (was yhat - bias = 2*yhat - y, double-error); _eval_one_coin consumes the per-fold corrected series directly (no global-mean re-shift). - #2 M12 calibration: _eval_one_coin now passes calibrate_bias=debias, calibration_size=... to walk_forward_har_rv_j (the flag already existed in m12_har_rv_j.py — only the call site was wrong, edit stays inside the 4-file scope). - #3 HAR raw vs debiased split: when debias=True, walk_forward_har is called twice — calibrate_bias=False for mse_har_raw (empirical), calibrate_bias=True for mse_har_debiased and DM leg. When debias= False, single call calibrate_bias=False, mse_har_debiased=NaN. - #4 OOS anti-leak test: test_walk_forward_lj_asym_oos_target_invariance runs the full walk-forward, patches realized_variance_to_log to add delta=10.0 to OOS targets, re-runs, asserts per_fold_bias and forecasts_debiased are bit-identical (rtol 1e-12) but targets HAVE changed. Train-tail perturbation control asserts bias DOES shift > 1.0 (legitimate sensitivity). - #5 docs: M17_HAR_LJ_ASYM.md c.953 numbers marked SUPERSEDED (cite head b974f27 + DM msg-20260904T141944, p-values 0.839708/0.403669/ 0.463629 invalid). New section ## Round-3 calibration (this PR) documents sign convention yhat + bias, per-fold application, M12 apples-to-apples, raw/debiased separation, panel_hash index, pending live BTC run. REGISTRY.md: pending live run note. - #6 panel_hash indexes: _panel_hash(rv) digests sha256(idx_bytes || val_bytes) with index as int64 ns. Manifest now surfaces panel_hash_per_coin_horizon (per-group hashes + consistency). Bug latent: old global cross-seed flag compared hashes across ALL coins/horizons (always False for multi-coin runs). 22 passed in 12.68s (15 sustained + 7 new/updated) on test_har_lj_asym. 1192 passed, 1 skipped, 0 failed (181.77s) on the whole tests directory — no regression outside scope. Refs #14592 * fix(training,#14592,#14584): REPAIR P0-4 M17 HAR-LJ-Asym — test OOS multi-fold discriminant + bounds provenance réelle + run BTC live 5×4×3 Three verbatim concerns from po-2025 adjoint re-review round-4 head 832ef69 (DM msg-20260905T001520-ljnt0j) lifted on top of c.964 REPAIR-3: - (a) test OOS n_splits=1 was non-discriminant: new test_walk_forward_lj_asym_oos_target_invariance_multi_fold with n_splits=3, calibration_size=60, 400-day panel. Asserts len(set(per_fold_bias)) >= 2 (3 distinct bias values measured: -0.000143 / 0.006266 / 0.015234). Per-fold OOS invariance (bit-identical rtol 1e-12 for the targeted fold; backward folds bit-identical = no cross-fold leakage; forward folds legitimately DIFFER — expanding window makes forward folds use the perturbed OOS rows as part of THEIR train, this is not a leak, it's correct geometry). Per-fold train-tail sensitivity verified (|Δbias|=1.148 / 3.002 / 6.495 > 1.0). - (c) n_train_end placeholder line 547 (the `if False else None` guard) replaced with real arithmetic in walk_forward_lj_asym: train_end_idx = n_folds_executed * fold_size, oos_start_idx = train_end + horizon, oos_end_idx = n. Surfaced as bounds_train_test in per-(coin, horizon, seed) result + relayed to _eval_one_coin + aggregated as bounds_per_coin_horizon in the manifest. Forecast hashes per fold added (fc_lj_hash_per_fold, 16-hex each, aligned with per_fold_bias). Acceptance test test_bounds_provenance_in_manifest checks the formula on a 400-bar panel and the JSON serialisability + cross-seed consistency. - (b) live BTC run 5×4×3 (5 folds × 4 seeds × 3 horizons × 1 coin × 4 models) regenerated scripts/results/m17_har_lj_asym.json from BTC Bitstamp 2014-20240808 cached on G: drive. Verdicts respect _coherent_beats()/_coherent_beaten_by() strictly (BEATS ⇒ p<0.05 AND mean_loss_diff<0). Round-3 SUPERSEDED docs now carry the post-fix numbers; REGISTRY note "pending live run" replaced by the SHA + bounds from the same artefact. 24 passed in 14.59s on test_har_lj_asym (22 sustained c.964 + 2 new round-4). 1194 passed, 1 skipped, 0 failed on the full scripts/tests directory (167.78s) — no regression outside scope. Refs #14592 * fix(training,#14592,#14584): REPAIR P0-5 M17 HAR-LJ-Asym — per_fold_bounds aligned + content_hash(index,bornes,values) + durable run anchor Three verbatim concerns from po-2025 adjoint re-review round-5 head 0fd96db (DM msg-20260905T022223-y9kih8) lifted on top of c.965 REPAIR-4: (1) per_fold_bounds aligned 1-pour-1 with per_fold_bias: walk_forward_lj_asym now accumulates per_fold_bounds = list of {fold_idx, train_end_idx, oos_start_idx, oos_end_idx, n_train, n_oos} dicts alongside per_fold_bias. _eval_one_coin relays per_fold_bounds per (coin, horizon, seed); aggregate_verdicts surfaces per_fold_bounds + per_fold_bounds_consistent_across_seeds. The previous bounds_train_test only described the last fold's cutoff; the forecasts/targets arrays are the concatenation of 5 folds, so per-fold granularity is the verifiable audit anchor. (2) _content_hash(index, bornes, values) helper + mutation tests: digests (int64 idx_bytes) || (int64 bornes_bytes) || (float64 val_bytes) in that fixed order. Surfaced as fc_content_hash_per_fold parallel to fc_lj_hash_per_fold (values-only). Three mutation tests + determinism test pin the contract: any shift in one of the three components changes the digest. (3) Durable run anchor without publishing private data: main() writes scripts/results/run_anchor_m17_har_lj_asym.json (gitignored via .gitignore:51:results/) with sha256 + relative locus of both the manifest and the results JSON. Stdout prints sha256[:16] prefixes and full loci for reviewer verification (sha256sum on the listed paths). No private data leaves the worktree -- only the SHA-256 + path metadata. Body stats + bornes per horizon corrected: round-5 amend +330/-0 on 2 files (cumul round-3+4+5: +2455/-45 on 4 files). Tests: 29 passed in 15.51s on test_har_lj_asym.py (24 sustained c.964 + 5 new round-5: 3 mutation + 1 determinism + 1 per_fold_bounds alignement). Refs #14592 Co-Authored-By: Claude Haiku 4.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude-Code <noreply@anthropic.com>
This was referenced Sep 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grain: MED/refactor -- lane myia-po-2026:CoursIA -- prev: MED/notebook-python #14335
Extraction (Closes #14363)
_mse_decompositionet_dm_centered_mseexistaient en 3 copies (btc_vol.pycanonique /btc_m15.py/hmm_regime_vol.py). Extraits dansscripts/bias_metrics.py— module torch-free (contrainte 1 de l'issue :hmm_regime_volest HMM+OLS CPU-only ; importerbtc_voly aurait traîné torch viadlinear_vol).Preuves (contraintes 2 et 3)
NaN != NaNen comparaison Python brute donnait 3 faux DIFF).python -m pytest tests/test_btc_vol.py tests/test_btc_m15.py tests/test_hmm_regime_vol.py tests/test_btc_patchtst.py→ 90 passed (env réparé au passage :pip install hmmlearn— règle F, pas de contournement).btc_patchtst.pyimportait les helpers depuisbtc_vol→ re-pointé versbias_metrics.Triage des divergences entre copies
btc_m15: copie legacy avec fallback try/exceptDM_TEST_UNAVAILABLE→ supprimé (forme canonique ;dm_testest présent).hmm_regime_vol: docstring §C enrichie → paragraphes mergés dans la docstring du module ; son import module-leveldm_verdictest conservé (utilisé ailleurs dans le script).btc_volgarde_dm_uncentered_mse(jamais dupliqué — ajouté par fix(training,#14362): la jambe de sanite DM etait silencieusement recentree #14392, jambe sanity non recentrée).Doc alignée
REGISTRY.md(entrée M15 slice 2/2) : « dupliqués ... TODO post-merge btc_debias.py » → pointeur versbias_metrics.py.m15_lstm_rv_sc_validation.ipynb§8.2 : même correction — markdown-only (exception C.2, outputs précédents valides).btc_patchtst.py:btc_vol._mse_decomposition→bias_metrics._mse_decomposition.Détail
10 fichiers (9 modifiés + 1 nouveau) ; 90 insertions / 148 deletions — le solde négatif EST le livrable : les 3 copies partent dans le module. En passant :
encoding="utf-8", errors="replace"ajouté ausubprocess.runderun_smoke(hookcheck-subprocess-encoding, #12811).