From 5ca43dd77353788ec59cec2494100a92f88c8bfc Mon Sep 17 00:00:00 2001 From: jsboige Date: Sun, 4 Oct 2026 02:45:56 +0200 Subject: [PATCH] feat(qc,#18961): ML-EnhancedPairs pair selection by PCA+OPTICS clustering (book ch. 06-09) Port of the reference notebook grouping step: monthly MonthStart('SPY') refit of StandardScaler -> PCA(3) -> OPTICS on 3y daily returns, candidate pairs drawn within each cluster (noise label -1 excluded). Behind the useClusterPairs parameter, default false = fixed-pairs behavior unchanged. Two QC Cloud backtests (project 37318734, 2015-2024, same universe): baseline Sharpe -2.486 / CAGR -0.361% / DD 4.9% / 158 orders vs cluster Sharpe -1.415 / CAGR -0.857% / DD 12.7% / 540 orders. Honest verdict: NO BEATS -- mechanism demonstrated, not a performance gain, consistent with a 7-ETF universe vs the book's ~1000-asset IWB. BOOK_MAPPING ch.06 row 09: PARTIAL -> COVERED. Co-Authored-By: Claude Sonnet 5.5 --- .../QuantConnect/BOOK_MAPPING.md | 2 +- .../projects/ML-EnhancedPairs/README.md | 29 ++++++-- .../projects/ML-EnhancedPairs/main.py | 72 ++++++++++++++++++- 3 files changed, 93 insertions(+), 10 deletions(-) diff --git a/MyIA.AI.Notebooks/QuantConnect/BOOK_MAPPING.md b/MyIA.AI.Notebooks/QuantConnect/BOOK_MAPPING.md index 6cfff9f010..35b8a29636 100644 --- a/MyIA.AI.Notebooks/QuantConnect/BOOK_MAPPING.md +++ b/MyIA.AI.Notebooks/QuantConnect/BOOK_MAPPING.md @@ -103,7 +103,7 @@ Algorithmes LEAN complets sur données de marché. Les exemples 04, 08, 18 et 19 | 08/01 | Stoploss — Benchmark Fixed Percentage Stop Loss | — | GAP | — | variante de référence, sans apprentissage | | 08/02 | Stoploss — ML Placed Stop Loss | [Stoploss-Volatility-ML](projects/Stoploss-Volatility-ML/) | COVERED | Needs-improvement (tranche 17) | stop placé par régression Lasso | | 08/03 | Stoploss — ML Put Option Hedge | — | GAP | — | | -| 09 | ML Trading Pairs Selection | [ML-EnhancedPairs](projects/ML-EnhancedPairs/), [ETF-Pairs](projects/ETF-Pairs/) | PARTIAL | Vivant (ML-EnhancedPairs) | cointégration présente ; ni PCA ni OPTICS pour la sélection des paires | +| 09 | ML Trading Pairs Selection | [ML-EnhancedPairs](projects/ML-EnhancedPairs/), [ETF-Pairs](projects/ETF-Pairs/) | COVERED | Vivant (ML-EnhancedPairs) | PCA(3) + OPTICS mensuels pour grouper l'univers avant cointégration, derrière le paramètre `useClusterPairs` (mode défaut inchangé) — #18961 | | 10 | Stock Selection through Clustering Fundamental Data | [Clustering-Fundamentals-ML](projects/Clustering-Fundamentals-ML/) | COVERED | Needs-improvement | | | 11 | Inverse Volatility Rank and Allocate to Future Contracts | [InverseVolatility-Rank](projects/InverseVolatility-Rank/) | COVERED | Needs-improvement / near-BROKEN | | | 12 | Trading Costs Optimization | [TradingCosts-Optimization](projects/TradingCosts-Optimization/) | COVERED | Démo | | diff --git a/MyIA.AI.Notebooks/QuantConnect/projects/ML-EnhancedPairs/README.md b/MyIA.AI.Notebooks/QuantConnect/projects/ML-EnhancedPairs/README.md index ba70489398..d3f5371e6e 100644 --- a/MyIA.AI.Notebooks/QuantConnect/projects/ML-EnhancedPairs/README.md +++ b/MyIA.AI.Notebooks/QuantConnect/projects/ML-EnhancedPairs/README.md @@ -1,24 +1,39 @@ # ML-EnhancedPairs **Asset class:** US Equities (pairs) -**Cloud project ID:** None (local only) +**Cloud project ID:** 37318734 ## Description ML-enhanced pairs trading. Uses ML to improve pair selection and signal generation beyond traditional cointegration tests. +## Pair-selection modes (book ch. 06-09, #18961) + +- **`useClusterPairs=false` (défaut)** : la liste fixe de 5 paires sector ETFs, comportement historique inchangé. +- **`useClusterPairs=true`** : chaque mois (premier jour de bourse, `MonthStart("SPY")`), l'algorithme standardise 3 ans de rendements quotidiens de l'univers, les réduit à 3 composantes principales (`PCA(n_components=3)`), regroupe les expositions de facteur par `OPTICS`, puis tire les paires candidates **à l'intérieur de chaque cluster** (le bruit OPTICS, label -1, ne rejoint aucune paire). Le pipeline aval (Engle-Granger/ADF, half-life, RandomForest) s'applique ensuite à ces paires comme aux fixes. + +C'est le portage de l'étape 1-2 du notebook de référence `06 Applied Machine Learning/09 ML Trading Pairs Selection/research.ipynb` (HandsOnAITradingBook, commit `e025f21`) : regroupement AVANT cointégration. Le livre applique la méthode à l'univers IWB (~1000 actifs) ; ce projet la démontre sur les 7 sector ETFs du dépôt — assez pour séparer quelques groupes et mesurer l'effet, pas assez pour en faire un usage de production. + ## How to Run **Lean CLI:** `lean backtest "MyIA.AI.Notebooks/QuantConnect/projects/ML-EnhancedPairs"` -**QC Cloud:** Not yet deployed. Copy files to a new QC Cloud project to run. +**QC Cloud:** Deployed as project 37318734. Backtests avec paramètre `useClusterPairs` (`true`/`false`, défaut `false`). ## Backtest Metrics -| Metric | Value | -|--------|-------| -| Method | ML-enhanced pairs trading | -| Rebalance | Daily | +Période commune 2015-01-01 → 2024-12-31, IBKR MARGIN, cash 100 000 $, même univers : + +| Metric | Baseline (paires fixes) | Cluster (PCA + OPTICS) | +| -------- | ------------------------ | ------------------------ | +| Sharpe Ratio | -2.486 | -1.415 | +| Compounding Annual Return | -0.361% | -0.857% | +| Max Drawdown | 4.900% | 12.700% | +| Total Orders | 158 | 540 | + +Backtests réels QC Cloud (`18961-baseline-fixed-pairs` et `18961-cluster-pca-optics`, projet 37318734, mode sélectionné par le paramètre `useClusterPairs`). + +**Lecture honnête (verdict : NO BEATS).** Sur cet univers de 7 ETFs, le mode cluster améliore le Sharpe (-2.486 → -1.415) mais dégrade le reste : rendement annuel plus faible (-0.361 % → -0.857 %), drawdown plus profond (4.9 % → 12.7 %), 3,4x plus d'ordres (158 → 540) et 3,2x plus de frais ($920 -> $2 984, résultat net -3.55 % -> -8.25 %). Le portage démontre la **mécanique** du ch. 06-09 (regroupement avant cointégration ; l'écart de comportement est massif et mesurable : turnover 1.58 % -> 6.01 %), pas un gain — cohérent avec l'échelle : le livre applique la méthode à ~1000 actifs IWB où le regroupement élimine l'essentiel des fausses paires ; sur 7 ETFs voisins, OPTICS fragmente l'univers sans écarter les rapprochements sectoriels trompeurs. ## Files -- main.py - Strategy +- main.py - Strategy (mode cluster derrière `GetParameter("useClusterPairs")`) diff --git a/MyIA.AI.Notebooks/QuantConnect/projects/ML-EnhancedPairs/main.py b/MyIA.AI.Notebooks/QuantConnect/projects/ML-EnhancedPairs/main.py index 731018603f..a7558d93a6 100644 --- a/MyIA.AI.Notebooks/QuantConnect/projects/ML-EnhancedPairs/main.py +++ b/MyIA.AI.Notebooks/QuantConnect/projects/ML-EnhancedPairs/main.py @@ -11,6 +11,13 @@ class MLEnhancedPairsAlgorithm(QCAlgorithm): - Use ML classifier to predict entry/exit timing - Features: Z-score, half-life, volatility regime - Long/short based on prediction and pair dynamics + + Pair-selection modes (book ch. 06-09, #18961): + - useClusterPairs=false (default): the fixed etf_pairs list, unchanged. + - useClusterPairs=true: monthly PCA(3) on standardized 3y daily returns, + OPTICS clustering of the factor exposures, and candidate pairs drawn + from within each cluster -- grouping BEFORE cointegration, as the + reference notebook does on a larger universe. """ def Initialize(self): @@ -41,6 +48,12 @@ def Initialize(self): self.max_half_life = 60 self.max_pairs = 3 + # Cluster mode (book 06-09): pairs discovered by PCA + OPTICS each + # month instead of the fixed list. Default off = current behavior. + self.use_cluster_pairs = self.GetParameter("useClusterPairs", "false").lower() == "true" + self.cluster_lookback_years = 3 + self.cluster_pairs = list(self.etf_pairs) + # Rebalance schedule self.Schedule.On(self.DateRules.EveryDay("SPY"), self.TimeRules.AfterMarketOpen("SPY", 30), @@ -51,10 +64,65 @@ def Initialize(self): self.TimeRules.AfterMarketOpen("SPY", 30), self.TrainModel) + # Refresh the clustering monthly (first trading day of the month: + # month_start WITH a symbol -- a bare month_start skips months whose + # 1st is not a trading day, #18941). + if self.use_cluster_pairs: + self.Schedule.On(self.DateRules.MonthStart("SPY"), + self.TimeRules.AfterMarketOpen("SPY", 30), + self.UpdateClusterPairs) + self.model = None self.scaler = None self.active_pairs = [] + def candidate_pairs(self): + """Pairs to evaluate this cycle: fixed list, or cluster-discovered.""" + return self.cluster_pairs if self.use_cluster_pairs else self.etf_pairs + + def UpdateClusterPairs(self): + """PCA(3) + OPTICS grouping of the universe (book 06-09, step 1-2). + + Standardize 3y of daily returns, reduce to 3 principal components, + cluster the factor exposures with OPTICS, then draw candidate pairs + from within each cluster. Noise points (label -1) join no pair. + """ + from sklearn.preprocessing import StandardScaler + from sklearn.decomposition import PCA + from sklearn.cluster import OPTICS + from itertools import combinations + + window = int(252 * self.cluster_lookback_years) + history = self.History(list(self.symbols.values()), window, Resolution.Daily) + if history.empty: + return + + closes = history['close'].unstack(level=0) + symbol_to_ticker = {str(v): k for k, v in self.symbols.items()} + closes.columns = [symbol_to_ticker.get(str(c), str(c)) for c in closes.columns] + + returns = closes.pct_change().dropna() + if returns.shape[1] < 4 or len(returns) < 60: + return + + standardized = StandardScaler().fit_transform(returns) + pca = PCA(n_components=3, random_state=0) + exposures = pca.fit(standardized).components_.T # assets x components + labels = OPTICS().fit(exposures).labels_ + + tickers = list(returns.columns) + pairs = [] + for cluster_id in sorted(set(labels)): + if cluster_id == -1: + continue # noise: asset joins no pair + members = [tickers[i] for i, lab in enumerate(labels) if lab == cluster_id] + pairs.extend(combinations(members, 2)) + + if pairs: + self.cluster_pairs = pairs + self.Debug(f"[cluster] clusters={len(set(labels)) - (1 if -1 in labels else 0)}" + f" noise={(labels == -1).sum()} candidate_pairs={len(pairs)}") + def TrainModel(self): """Train ML model for entry/exit prediction.""" self.Debug("Training ML model for pairs...") @@ -93,7 +161,7 @@ def PrepareMLFeatures(self, history): features_list = [] labels_list = [] - for ticker1, ticker2 in self.etf_pairs: + for ticker1, ticker2 in self.candidate_pairs(): if ticker1 not in closes.columns or ticker2 not in closes.columns: continue @@ -212,7 +280,7 @@ def Rebalance(self): # Evaluate each pair pair_scores = [] - for ticker1, ticker2 in self.etf_pairs: + for ticker1, ticker2 in self.candidate_pairs(): if ticker1 not in closes.columns or ticker2 not in closes.columns: continue