refactor(training,#14363): extract bias/MSE helpers to shared torch-free module

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/REGISTRY.md

@@ -539,7 +539,7 @@ Slice 2/2 du ticket #12734 (slice 1/2 = M4 DLinear-vol, livré via PR #12742). L
 
 **Patch persistance** : `evaluate_one_combo` calcule désormais `har_bias_oos = mean(har_pred - target)` OOS et persiste **les arrays `har_errors`/`lstm_errors`** (même guard `len >= 10 and isfinite`) par combo, pour que `analyze_one_combo` (wrapper `btc_m15.py`) puisse appliquer la décomposition `MSE = biais² + variance` et le DM recentré post-hoc. Rétro-compatible : les anciens champs `sharpe_*`, `mse_*`, `dm_*` sont préservés. Le JSON `results.json` de chaque run M15 porte donc `har_bias_oos` + les erreurs brutes par combo — vérifiable depuis git sans relancer le sweep.
 
-**Wrapper `scripts/btc_m15.py`** : pendant BTC-only de `scripts/btc_vol.py`. Orchestre le run M15 BTC + applique la décomposition `MSE = biais² + variance` et le DM recentré (`loss_fn="mse"` sur erreurs centrees). Helpers `_mse_decomposition` et `_dm_centered_mse` **dupliqués** depuis `btc_vol.py` (TODO post-merge : consolider dans un module partage `btc_debias.py`).
+**Wrapper `scripts/btc_m15.py`** : pendant BTC-only de `scripts/btc_vol.py`. Orchestre le run M15 BTC + applique la décomposition `MSE = biais² + variance` et le DM recentré (`loss_fn="mse"` sur erreurs centrees). Helpers `_mse_decomposition` et `_dm_centered_mse` dans le module partage `scripts/bias_metrics.py`, extrait de `btc_vol.py` (issue #14363).
 
 **Section notebook** : section 8 ajoutée à `m15_lstm_rv_sc_validation.ipynb` (méthodologie + commande de run complet + verdict attendu). Markdown-only — la cellule code de lecture du JSON viendra au prochain cycle avec le rerun.
 

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/m15_lstm_rv_sc_validation.ipynb

@@ -726,7 +726,7 @@
     "\n",
     "### 8.2 Wrapper `btc_m15.py`\n",
     "\n",
-    "`scripts/btc_m15.py` est le pendant BTC-only de `scripts/btc_vol.py` (slice 1/2 M4). Il orchestre le run M15 BTC + applique la decomposition biais²+variance et le DM recentre. Les helpers `_mse_decomposition` et `_dm_centered_mse` sont dupliques depuis `btc_vol.py` pour decoupler ce PR de PR #12742 (consolidation dans un module partage est une TODO post-merge).\n",
+    "`scripts/btc_m15.py` est le pendant BTC-only de `scripts/btc_vol.py` (slice 1/2 M4). Il orchestre le run M15 BTC + applique la decomposition biais²+variance et le DM recentre. Les helpers `_mse_decomposition` et `_dm_centered_mse` vivent dans le module partage `bias_metrics.py` (extrait de `btc_vol.py`, issue #14363).\n",
     "\n",
     "### 8.3 Commande de run complet (slice 2/2 dispatche au prochain cycle)\n",
     "\n",

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/scripts/bias_metrics.py

@@ -0,0 +1,59 @@
+"""Shared bias/MSE decomposition and centered-DM helpers (#14363).
+
+Extracted from `btc_vol.py` (PR #12742, issue #12734) once a third copy
+appeared in `hmm_regime_vol.py` (PR #14359). Deliberately torch-free: the
+module must stay importable from CPU-only scripts (`hmm_regime_vol` is HMM +
+OLS and needs no torch). The identical-upto-unification docstring below is
+the canonical, corrected version (#14362); the three consumers' extra
+copies were removed.
+"""
+from __future__ import annotations
+
+import numpy as np
+
+
+def _mse_decomposition(errors: np.ndarray) -> dict:
+    """Decompose MSE of a forecast into bias^2 + variance on the error support."""
+    if errors is None or len(errors) == 0:
+        return {"mse": float("nan"), "bias_sq": float("nan"), "variance": float("nan")}
+    bias = float(np.mean(errors))
+    variance = float(np.var(errors, ddof=0))
+    return {
+        "mse": float(np.mean(errors ** 2)),
+        "bias_sq": bias ** 2,
+        "variance": variance,
+    }
+
+
+def _dm_centered_mse(
+    errors_a: np.ndarray, errors_b: np.ndarray, horizon: int
+) -> dict:
+    """DM test on errors centered by their own mean, with loss_fn='mse'.
+
+    Centering annihilates the bias component (`mean(e_a - mean(e_a)) = 0`),
+    so the resulting `d_mean` measures only the variance differential. The
+    "DM on precision" jambe that #10961 documents is exactly this.
+
+    `loss_fn` stays "mse" on purpose: section C forbids `linear` as the
+    conjunction leg, because on raw signed errors `d_mean = bias_a - bias_b`
+    is blind to dispersion -- it measures the very quantity centering removes.
+    """
+    from dm_test import dm_verdict as dm_verdict_fn
+
+    e_a = np.asarray(errors_a, dtype=float)
+    e_b = np.asarray(errors_b, dtype=float)
+    if e_a.shape != e_b.shape:
+        return {"dm_stat": float("nan"), "dm_pvalue": float("nan"), "dm_verdict": "SHAPE_MISMATCH"}
+    n = len(e_a)
+    if n < 10:
+        return {"dm_stat": float("nan"), "dm_pvalue": float("nan"), "dm_verdict": "INSUFFICIENT_DATA"}
+
+    centered_a = e_a - np.mean(e_a)
+    centered_b = e_b - np.mean(e_b)
+    res = dm_verdict_fn(centered_a, centered_b, horizon=horizon, loss_fn="mse")
+    return {
+        "dm_stat": float(res["dm_statistic"]),
+        "dm_pvalue": float(res["p_value"]),
+        "dm_verdict": str(res["verdict"]),
+        "mean_loss_diff": float(res["mean_loss_diff"]),
+    }

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/scripts/btc_m15.py

@@ -14,10 +14,8 @@
 seed) — to validate the pipeline end-to-end. The full re-run is
 dispatched to the next cycle (see PR body for the command + acceptance).
 
-Helpers `_mse_decomposition` and `_dm_centered_mse` are duplicated from
-`btc_vol.py` (PR #12742) for the same reason PR #12742 needed them in
-isolation: keep this PR self-contained until a shared module is
-extracted (TODO post-merge).
+Helpers `_mse_decomposition` and `_dm_centered_mse` now live in
+`bias_metrics.py`, extracted from `btc_vol.py` (issue #14363).
 """
 from __future__ import annotations
 
@@ -31,49 +29,7 @@
 import numpy as np
 import pandas as pd
 
-# --- helpers duplicated from btc_vol.py (slice 1/2 PR #12742) -----------
-
-
-def _mse_decomposition(errors: np.ndarray) -> dict:
-    """Decompose MSE of a forecast into bias^2 + variance on the error support."""
-    if errors is None or len(errors) == 0:
-        return {"mse": float("nan"), "bias_sq": float("nan"), "variance": float("nan")}
-    bias = float(np.mean(errors))
-    variance = float(np.var(errors, ddof=0))
-    return {
-        "mse": float(np.mean(errors ** 2)),
-        "bias_sq": bias ** 2,
-        "variance": variance,
-    }
-
-
-def _dm_centered_mse(
-    errors_a: np.ndarray, errors_b: np.ndarray, horizon: int
-) -> dict:
-    """DM test on errors centered by their own mean, with loss_fn='mse'."""
-    try:
-        from dm_test import dm_verdict as dm_verdict_fn
-    except ImportError:
-        return {"dm_stat": float("nan"), "dm_pvalue": float("nan"),
-                "dm_verdict": "DM_TEST_UNAVAILABLE", "mean_loss_diff": float("nan")}
-
-    e_a = np.asarray(errors_a, dtype=float)
-    e_b = np.asarray(errors_b, dtype=float)
-    if e_a.shape != e_b.shape:
-        return {"dm_stat": float("nan"), "dm_pvalue": float("nan"), "dm_verdict": "SHAPE_MISMATCH"}
-    n = len(e_a)
-    if n < 10:
-        return {"dm_stat": float("nan"), "dm_pvalue": float("nan"), "dm_verdict": "INSUFFICIENT_DATA"}
-
-    centered_a = e_a - np.mean(e_a)
-    centered_b = e_b - np.mean(e_b)
-    res = dm_verdict_fn(centered_a, centered_b, horizon=horizon, loss_fn="mse")
-    return {
-        "dm_stat": float(res["dm_statistic"]),
-        "dm_pvalue": float(res["p_value"]),
-        "dm_verdict": str(res["verdict"]),
-        "mean_loss_diff": float(res["mean_loss_diff"]),
-    }
+from bias_metrics import _dm_centered_mse, _mse_decomposition  # noqa: E402
 
 
 # --- main analysis --------------------------------------------------------
@@ -227,7 +183,7 @@ def run_smoke(
         "--loss-fn", loss_fn,
     ]
     print(f"[btc_m15] running: {' '.join(cmd)}")
-    proc = subprocess.run(cmd, capture_output=True, text=True)
+    proc = subprocess.run(cmd, capture_output=True, text=True, encoding="utf-8", errors="replace")
     print(proc.stdout[-2000:] if proc.stdout else "")
     if proc.returncode != 0:
         print(f"[btc_m15] FAILED (rc={proc.returncode}): {proc.stderr[-1000:]}")

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/scripts/btc_patchtst.py

@@ -12,7 +12,8 @@
   ``calibrate_bias=True`` (see the debiasing paragraph below).
 - ``dlinear_vol.create_sequences`` for the (input, target) windowing --
   same convention as the DLinear-vol keeper.
-- ``btc_vol._mse_decomposition`` / ``btc_vol._dm_centered_mse`` for the
+- ``bias_metrics._mse_decomposition`` / ``bias_metrics._dm_centered_mse``
+  (shared torch-free module, issue #14363) for the
   MSE = bias^2 + variance decomposition and the DM test on mean-centered
   errors (loss_fn="mse" -> variance differential, the precision leg of
   pr-review section C).
@@ -89,7 +90,7 @@
 from har_model import _make_split_indices as make_expanding_splits  # noqa: E402
 from har_model import walk_forward_har  # noqa: E402
 from dlinear_vol import create_sequences  # noqa: E402
-from btc_vol import _dm_centered_mse, _mse_decomposition  # noqa: E402
+from bias_metrics import _dm_centered_mse, _mse_decomposition  # noqa: E402
 from intraday_loader import hourly_log_returns, load_bitstamp_btc  # noqa: E402
 from realized_variance import (  # noqa: E402
     daily_realized_variance,

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/scripts/btc_vol.py

@@ -49,49 +49,7 @@
     walk_forward_har,
     walk_forward_dlinear,
 )
-
-
-def _mse_decomposition(errors: np.ndarray) -> dict:
-    """Decompose MSE of a forecast into bias^2 + variance on the error support."""
-    if errors is None or len(errors) == 0:
-        return {"mse": float("nan"), "bias_sq": float("nan"), "variance": float("nan")}
-    bias = float(np.mean(errors))
-    variance = float(np.var(errors, ddof=0))
-    return {
-        "mse": float(np.mean(errors ** 2)),
-        "bias_sq": bias ** 2,
-        "variance": variance,
-    }
-
-
-def _dm_centered_mse(
-    errors_a: np.ndarray, errors_b: np.ndarray, horizon: int
-) -> dict:
-    """DM test on errors centered by their own mean, with loss_fn='mse'.
-
-    Centering annihilates the bias component (`mean(e_a - mean(e_a)) = 0`),
-    so the resulting `d_mean` measures only the variance differential. The
-    "DM on precision" jambe that #10961 documents is exactly this.
-    """
-    from dm_test import dm_verdict as dm_verdict_fn
-
-    e_a = np.asarray(errors_a, dtype=float)
-    e_b = np.asarray(errors_b, dtype=float)
-    if e_a.shape != e_b.shape:
-        return {"dm_stat": float("nan"), "dm_pvalue": float("nan"), "dm_verdict": "SHAPE_MISMATCH"}
-    n = len(e_a)
-    if n < 10:
-        return {"dm_stat": float("nan"), "dm_pvalue": float("nan"), "dm_verdict": "INSUFFICIENT_DATA"}
-
-    centered_a = e_a - np.mean(e_a)
-    centered_b = e_b - np.mean(e_b)
-    res = dm_verdict_fn(centered_a, centered_b, horizon=horizon, loss_fn="mse")
-    return {
-        "dm_stat": float(res["dm_statistic"]),
-        "dm_pvalue": float(res["p_value"]),
-        "dm_verdict": str(res["verdict"]),
-        "mean_loss_diff": float(res["mean_loss_diff"]),
-    }
+from bias_metrics import _dm_centered_mse, _mse_decomposition  # noqa: E402
 
 
 def _dm_uncentered_mse(

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/scripts/hmm_regime_vol.py

@@ -37,6 +37,7 @@
 sys.path.insert(0, str(SCRIPTS_DIR))
 
 from dm_test import dm_verdict
+from bias_metrics import _dm_centered_mse, _mse_decomposition  # noqa: E402
 from har_model import HARModel, _make_split_indices
 from intraday_loader import load_binance_eth, load_bitstamp_btc
 from realized_variance import daily_realized_variance, har_lag_features, realized_variance_to_log
@@ -148,48 +149,6 @@ def _aggregate_debiased_state(
     return "INCONCLUSIVE"
 
 
-def _mse_decomposition(errors: np.ndarray) -> dict:
-    """Decompose MSE of a forecast into bias^2 + variance on the error support."""
-    if errors is None or len(errors) == 0:
-        return {"mse": float("nan"), "bias_sq": float("nan"), "variance": float("nan")}
-    bias = float(np.mean(errors))
-    variance = float(np.var(errors, ddof=0))
-    return {
-        "mse": float(np.mean(errors ** 2)),
-        "bias_sq": bias ** 2,
-        "variance": variance,
-    }
-
-
-def _dm_centered_mse(errors_a: np.ndarray, errors_b: np.ndarray, horizon: int) -> dict:
-    """DM test on errors centered by their own mean, with loss_fn='mse'.
-
-    Centering annihilates the bias component (`mean(e - mean(e)) = 0`), so the
-    resulting `d_mean` measures only the variance differential. The "DM on
-    precision" jambe that #10961 documents is exactly this.
-
-    `loss_fn` stays "mse" on purpose: §C forbids `linear` as the conjunction
-    leg, because on raw signed errors `d_mean = bias_a - bias_b` is blind to
-    dispersion -- it measures the very quantity centering removes.
-    """
-    e_a = np.asarray(errors_a, dtype=float)
-    e_b = np.asarray(errors_b, dtype=float)
-    if e_a.shape != e_b.shape:
-        return {"dm_stat": float("nan"), "dm_pvalue": float("nan"), "dm_verdict": "SHAPE_MISMATCH"}
-    if len(e_a) < 10:
-        return {"dm_stat": float("nan"), "dm_pvalue": float("nan"), "dm_verdict": "INSUFFICIENT_DATA"}
-
-    res = dm_verdict(
-        e_a - np.mean(e_a), e_b - np.mean(e_b), horizon=horizon, loss_fn="mse",
-    )
-    return {
-        "dm_stat": float(res["dm_statistic"]),
-        "dm_pvalue": float(res["p_value"]),
-        "dm_verdict": str(res["verdict"]),
-        "mean_loss_diff": float(res["mean_loss_diff"]),
-    }
-
-
 def fit_hmm_regimes(log_rv_train: np.ndarray, seed: int) -> "GaussianHMM":
     """Fit K-state GaussianHMM on log-RV. State 0 = low vol, state 1 = high vol."""
     _require_hmmlearn()

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/scripts/tests/test_btc_m15.py

@@ -1,6 +1,7 @@
 """Tests for btc_m15.py helpers (issue #12734, slice 2/2).
 
-Covers the pure transformations in `scripts/btc_m15.py`:
+Covers the pure transformations in `scripts/bias_metrics.py`
+(extracted from `btc_vol.py`, issue #14363):
   - `_mse_decomposition`: split MSE into bias^2 + variance.
   - `_dm_centered_mse`: DM test on errors centered by their own mean.
   - `analyze_one_combo`: end-to-end on a synthetic combo dict.
@@ -20,9 +21,11 @@
 # Make the parent directory importable.
 sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
 
-from btc_m15 import (  # noqa: E402
+from bias_metrics import (  # noqa: E402
     _dm_centered_mse,
     _mse_decomposition,
+)
+from btc_m15 import (  # noqa: E402
     aggregate_verdicts,
     analyze_one_combo,
 )

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/scripts/tests/test_btc_vol.py

@@ -1,6 +1,7 @@
 """Tests for btc_vol.py helpers (issue #12734).
 
-These cover the pure transformations in `scripts/btc_vol.py`:
+These cover the pure transformations in `scripts/bias_metrics.py`
+(extracted from `btc_vol.py`, issue #14363):
   - `_mse_decomposition`: split MSE into bias^2 + variance.
   - `_dm_centered_mse`: DM test on errors centered by their own mean.
 
@@ -19,11 +20,13 @@
 # Make the parent directory importable.
 sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
 
-from btc_vol import (  # noqa: E402
+from bias_metrics import (  # noqa: E402
     _dm_centered_mse,
-    _dm_uncentered_mse,
     _mse_decomposition,
 )
+from btc_vol import (  # noqa: E402
+    _dm_uncentered_mse,
+)
 
 
 class TestMseDecomposition:

MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/scripts/tests/test_hmm_regime_vol.py

@@ -5,10 +5,11 @@
 `MSE = bias^2 + variance` means a raw-error DM cannot tell a precision win from
 a calibration artefact. This module tests the control that separates them.
 
-The pure helpers `_mse_decomposition` / `_dm_centered_mse` also have coverage in
-`test_btc_vol.py` (they originate there, PR #12742). What is tested HERE is what
-those tests cannot cover: that the control actually DISCRIMINATES, and that the
-M5 return contract carries it.
+The pure helpers `_mse_decomposition` / `_dm_centered_mse` live in
+`bias_metrics.py` (extracted from `btc_vol.py`, issue #14363) and also have
+coverage in `test_btc_vol.py`. What is tested HERE is what those tests cannot
+cover: that the control actually DISCRIMINATES, and that the M5 return
+contract carries it.
 
 Validation shape (Tell c.856-L1): every silence is paired with a mutation that
 changes only the property under test and must make the control speak. A control
@@ -31,13 +32,15 @@
 import hmm_regime_vol  # noqa: E402
 from hmm_regime_vol import (  # noqa: E402
     _aggregate_debiased_state,
-    _dm_centered_mse,
     _is_beaten,
     _is_beats,
-    _mse_decomposition,
     _require_hmmlearn,
     walk_forward_regime_switching,
 )
+from bias_metrics import (  # noqa: E402
+    _dm_centered_mse,
+    _mse_decomposition,
+)
 
 
 # --------------------------------------------------------------------------