diff --git a/MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/2.14b-XAI-Shap-Attribution-Causal-Bridge.ipynb b/MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/2.14b-XAI-Shap-Attribution-Causal-Bridge.ipynb new file mode 100644 index 0000000000..d771decdc9 --- /dev/null +++ b/MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/2.14b-XAI-Shap-Attribution-Causal-Bridge.ipynb @@ -0,0 +1,1931 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "9039e2fa", + "metadata": { + "papermill": { + "duration": 0.00444, + "end_time": "2026-09-19T15:03:39.254234", + "exception": false, + "start_time": "2026-09-19T15:03:39.249794", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "# 2.14b — SHAP et do-calculus : la jonction attribution ↔ causalité\n", + "\n", + "**Navigation** : [<< 2.14-Explicabilite-SHAP-LIME-Contrefactuels](2.14-Explicabilite-SHAP-LIME-Contrefactuels.ipynb) | [Index](../README.md) | [Suivant >>](../README.md)\n", + "\n", + "**Grain** : DEEP/notebook-python — lane myia-po-2023:CoursIA-2 — prev: DEEP/notebook-python #16619 (2.14 Explicabilite)\n", + "\n", + "Ce notebook creuse un point que **2.14 effleure et que la litterature XAI confond regulierement** :\n", + "\n", + "**Subtilite centrale** (T9 Chen-Covert-Lundberg-Lee 2022 + R4 Bareinboim-Pearl 2016) : la **Shapley value conditionnelle** sous un background dataset $D$ approche l'**effet causal** $do(X=x)$ quand $D$ respecte la **consistance** avec le DAG. La Shapley **value marginale** (Kernel SHAP classique) approxime l'**observation** $P(Y \\mid X=x)$, qui **n'est pas** l'effet causal. C'est la **jonction attribution↔causalite** que ce notebook rend visible par la mesure.\n", + "\n", + "**Sources canoniques** (Tell c.bibliography-hygiene — archivees hors Git dans GDrive) :\n", + "\n", + "| Ref | Auteur(s) | Annee | Substantif |\n", + "|---|---|---|---|\n", + "| R1 | Lundberg & Lee | 2017 | Kernel SHAP, theoreme d'unicite (3 axiomes : local accuracy, missingness, consistency) — arXiv 1706.06060 |\n", + "| R2 | Lundberg et al (10 auteurs) | 2019 | Tree SHAP exact O(TLD^2) — arXiv 1905.04610 |\n", + "| R3 | Chen, Covert, Lundberg, Lee | 2022 | Conditional vs Marginal Shapley <-> do/see — arXiv 2207.07605 (T9 du cadrage) |\n", + "| R4 | Bareinboim & Pearl | 2016 | Jonction do-calculus ~ conditional Shapley — PNAS 10.1073/pnas.1510507113 |\n", + "| R5 | Bareinboim, Correa, Ibeling, Icard | 2026 | Causal Hierarchy Theorem (CHT) 3 niveaux — Causal AI ch. 2.3 |\n", + "\n", + "**Socle du depot** (jonction XAI <-> causal) :\n", + "\n", + "- [2.14-Explicabilite-SHAP-LIME-Contrefactuels](2.14-Explicabilite-SHAP-LIME-Contrefactuels.ipynb) — la base XAI (SHAP Tree/Kernel, LIME, DiCE contrefactuels, acceptance 7/7)\n", + "- [Do-Calculus-Bridge.ipynb](../../../Probas/DecisionTheory/Causal-Bridges/Do-Calculus-Bridge.ipynb) — do-calculus Bareinboim-Pearl R-90, mediation counterfactuelle (P2 EPIC #16620)\n", + "- [Do-Calculus-Bridge.ipynb](../../../Probas/DecisionTheory/Causal-Bridges/Do-Calculus-Bridge.ipynb) — do-calculus ≅ conditional Shapley (P4 EPIC #16620, 30→43 cellules)\n", + "- [Causal-Bridges](../../../Probas/DecisionTheory/Causal-Bridges/README.md) — versant causal pur (DoWhy, contrefactuels sur DAG)\n", + "\n", + "**Acceptance** (8 critères du cadrage c.655 / #16669) :\n", + "\n", + "1. Notebook Python `coursia-ml-training` Papermill 0 erreur.\n", + "2. Sorties reelles commises (C.2 — Tell c.1175-L1 ★ strict JAMAIS hand-edit).\n", + "3. ≥ 2 visualisations SHAP (summary plot + force plot local).\n", + "4. ≥ 1 visualisation LIME.\n", + "5. ≥ 1 contrefactuel DiCE.\n", + "6. Section 5 « Jonction Shap ↔ do-calculus (T9) » mesure l'ecart KernelShap vs conditional Shapley.\n", + "7. Section 6 « Ponts » renvoie aux 6 notebooks causaux.\n", + "8. Section 7 « Note explicatif ≠ causal » cite R4 §3.3 + R3 §2.4." + ] + }, + { + "cell_type": "markdown", + "id": "67d83cda", + "metadata": { + "papermill": { + "duration": 0.003688, + "end_time": "2026-09-19T15:03:39.261926", + "exception": false, + "start_time": "2026-09-19T15:03:39.258238", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## Garde-fous d'honnêtete (Tell c.G.2 ★★★★ metriques honetes)\n", + "\n", + "1. **Pas de pretention d'exhaustivite** : la litterature XAI sur ce sujet est large (CAT, Grad-CAM, attention-rollout, integrated gradients...). Le notebook se limite a **SHAP** parce que c'est l'attribution la plus formalisee (3 axiomes de R1) et la seule ou la jonction conditionnel/marginal est theoriquement etablie.\n", + "2. **Le modele est volontairement simple** : on reutilise la foret aleatoire entrainee dans 2.14 (German Credit, ~0.75 accuracy). Un modele plus complexe ne changerait pas la **question** (l'ecart Kernel vs Tree = ecart marginal vs conditionnel), mais il rendrait les sorties moins lisibles.\n", + "3. **Le DAG est connu** : on simule un DAG ou `age` -> `credit_amount` -> `default` ET `age` -> `default`. Kernel SHAP (marginal) ignore ce DAG ; Tree SHAP (conditionnel) le respecte via l'ordre des features dans l'arbre. L'ecart est mesurable, pas hypothetique.\n", + "4. **Aucune fabrication de chiffres** : chaque valeur numerique de ce notebook est lue depuis les sorties reelles des cellules. Le notebook ne cache pas les cas ou l'ecart marginal/conditionnel est faible — il les montre." + ] + }, + { + "cell_type": "markdown", + "id": "42f183d4", + "metadata": { + "papermill": { + "duration": 0.003694, + "end_time": "2026-09-19T15:03:39.271685", + "exception": false, + "start_time": "2026-09-19T15:03:39.267991", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 1. Setup : le modele et le background dataset\n", + "\n", + "On reutilise la foret aleatoire de 2.14 (chargee via pickle pour eviter de re-entrainer — 13 ko). Le **background dataset** $D$ est crucial : c'est lui qui distingue marginal de conditionnel.\n", + "\n", + "- **Marginal (Kernel SHAP)** : pour chaque coalition $S \\subseteq F \\setminus \\{i\\}$, on tire $D$ uniformement sur $X_S$ et on complete par $X_{\\bar{S}} = E[X_{\\bar{S}}]$ (moyennes marginales).\n", + "- **Conditionnel (Tree SHAP / Kernel SHAP conditionnel)** : pour chaque coalition $S$, on tire $X_S$ observe dans $D$ et on complete par $X_{\\bar{S}}$ tire **conditionnellement** a $X_S$ (preservation des dependances entre features).\n", + "\n", + "Le DAG etant connu (`age` -> `credit_amount` -> `default` + `age` -> `default`), conditionner sur `age` change la distribution de `credit_amount` (les gens plus ages demandent generalement plus). Marginaliser ignore cette dependance." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "6eea5dc9", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-19T15:03:39.280847Z", + "iopub.status.busy": "2026-09-19T15:03:39.280635Z", + "iopub.status.idle": "2026-09-19T15:03:41.155810Z", + "shell.execute_reply": "2026-09-19T15:03:41.155319Z" + }, + "papermill": { + "duration": 1.881119, + "end_time": "2026-09-19T15:03:41.156715", + "exception": false, + "start_time": "2026-09-19T15:03:39.275596", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "!!! Modele 2.14 absent — recreation d'un DAG synthetique (regle F : informer, pas maquiller).\n" + ] + } + ], + "source": [ + "import os\n", + "import pickle\n", + "import warnings\n", + "\n", + "import numpy as np\n", + "import pandas as pd\n", + "\n", + "warnings.filterwarnings(\"ignore\")\n", + "RANDOM_STATE = 42\n", + "np.random.seed(RANDOM_STATE)\n", + "\n", + "# Reuse the trained model from 2.14 (avoid re-training — that's 13 ko of serialized forest)\n", + "NB14_DIR = os.path.dirname(os.path.abspath('2.14b-XAI-Shap-Attribution-Causal-Bridge.ipynb'))\n", + "MODEL_PATH = os.path.join(NB14_DIR, '.cache', 'german_credit_rf_v1.pkl')\n", + "\n", + "if os.path.exists(MODEL_PATH):\n", + " bundle = pickle.load(open(MODEL_PATH, 'rb'))\n", + " rf = bundle['model']\n", + " feature_names = bundle['feature_names']\n", + " X_background = bundle['X_background']\n", + " y_background = bundle['y_background']\n", + " print(f'Modele 2.14 recharge : {rf.n_estimators} arbres, {len(feature_names)} features.')\n", + "else:\n", + " # Fallback minimal : recreer un DAG synthetique (rare en pratique, signale)\n", + " print('!!! Modele 2.14 absent — recreation d\\'un DAG synthetique (regle F : informer, pas maquiller).')\n", + " from sklearn.ensemble import RandomForestClassifier\n", + " n = 1000\n", + " age = np.random.normal(35, 10, n)\n", + " credit_amount = 5000 + 200 * age + np.random.normal(0, 1000, n)\n", + " default_prob = 1 / (1 + np.exp(-(0.05 * age - 0.0001 * credit_amount - 2)))\n", + " default = (np.random.rand(n) < default_prob).astype(int)\n", + " X_background = pd.DataFrame({'age': age, 'credit_amount': credit_amount})\n", + " y_background = default\n", + " rf = RandomForestClassifier(n_estimators=100, random_state=RANDOM_STATE)\n", + " rf.fit(X_background, y_background)\n", + " feature_names = ['age', 'credit_amount']" + ] + }, + { + "cell_type": "markdown", + "id": "1efab912", + "metadata": { + "papermill": { + "duration": 0.003862, + "end_time": "2026-09-19T15:03:41.164507", + "exception": false, + "start_time": "2026-09-19T15:03:41.160645", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "**Lecture.** Si le pickle est present, on reutilise le modele 2.14 (cohérence pedagogique). Sinon, on signale explicitement et on tombe sur un DAG minimal mais honnete : `age` influence `credit_amount` (200€ par an) **et** `default` (les ages plus élevés sont legerement plus risqués, +0.05 par an sur le logit)." + ] + }, + { + "cell_type": "markdown", + "id": "81704243", + "metadata": { + "papermill": { + "duration": 0.003792, + "end_time": "2026-09-19T15:03:41.172292", + "exception": false, + "start_time": "2026-09-19T15:03:41.168500", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 2. Marginal vs conditionnel : la definition formelle\n", + "\n", + "Pour une feature $i$ et un modele $f$, la **Shapley value** s'ecrit :\n", + "\n", + "$$\n", + "\\phi_i(f, x) = \\sum_{S \\subseteq F \\setminus \\{i\\}} \\frac{|S|!(|F|-|S|-1)!}{|F|!} \\left[ v(S \\cup \\{i\\}) - v(S) \\right]\n", + "$$\n", + "\n", + "Le seul choix libre est la definition de $v(S)$ — la « valeur » de la coalition $S$. Deux definitions :\n", + "\n", + "| Definition | $v_{marginale}(S)$ | $v_{conditionnelle}(S)$ |\n", + "|---|---|---|\n", + "| Kernel SHAP (Lundberg-Lee 2017) | $E_{X_{\\bar{S}} \\sim P(X_{\\bar{S}})}[f(x_S, X_{\\bar{S}})]$ | — |\n", + "| Conditional Kernel SHAP (R3 T9) | — | $E_{X_{\\bar{S}} \\sim P(X_{\\bar{S}} \\mid X_S = x_S)}[f(x_S, X_{\\bar{S}})]$ |\n", + "| Tree SHAP (R2) | — | exactement conditionnel (par construction de l'arbre) |\n", + "\n", + "**Consequence directe** : si le DAG contient `age -> credit_amount`, marginaliser sur `credit_amount` quand on sait `age` evalue le modele sur des couples **(age=35, credit_amount=10000)** tires du profil d'un jeune de 20 ans. C'est **hors distribution**. Le conditionnel reste sur des couples realistes." + ] + }, + { + "cell_type": "markdown", + "id": "8a58f1c9", + "metadata": { + "papermill": { + "duration": 0.003638, + "end_time": "2026-09-19T15:03:41.179917", + "exception": false, + "start_time": "2026-09-19T15:03:41.176279", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 3. Mesure : l'ecart entre marginal et conditionnel\n", + "\n", + "On prend un individu test $x^*$ (le plus a risque selon le modele, comme dans 2.14) et on compare les Shapley values sous les deux definitions. Le Kernel SHAP conditionnel s'obtient avec `shap.Explainer` en passant `data=X_background` et en activant le mode conditional via l'API `maskers.Impute`." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "de96657a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-19T15:03:41.188980Z", + "iopub.status.busy": "2026-09-19T15:03:41.188502Z", + "iopub.status.idle": "2026-09-19T15:03:43.070412Z", + "shell.execute_reply": "2026-09-19T15:03:43.069963Z" + }, + "papermill": { + "duration": 1.887522, + "end_time": "2026-09-19T15:03:43.071179", + "exception": false, + "start_time": "2026-09-19T15:03:41.183657", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Individu test idx=773, P(default)=0.940\n", + "Features : {'age': np.float64(44.7255444962673), 'credit_amount': np.float64(14139.716354490702)}\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "50a620a6d232454c83a061a19af1620e", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " 0%| | 0/1 [00:00= 0.45 : shap_values() rend un ndarray (n, n_features, n_classes) -- un flatten\n", + "# brut melange les classes (fix #16680 : age apparaissait deux fois, anti-symetrie phi(c0)=-phi(c1)).\n", + "# Helper : selection explicite de la classe 'default' (1), compatible liste (SHAP < 0.45).\n", + "def _phi_class1(sv):\n", + " if isinstance(sv, list):\n", + " return sv[1] if len(sv) == 2 else sv\n", + " arr = np.asarray(sv)\n", + " return arr[..., 1] if arr.ndim >= 3 else arr\n", + "\n", + "phi_marginal = _phi_class1(shap_values_marginal)\n", + "\n", + "# Kernel SHAP CONDITIONNEL via le mode 'partition' de shap qui est conditionnel par construction\n", + "# Approche alternative : utiliser TreeExplainer qui EST conditionnel par construction (R2)\n", + "tree_explainer = shap.TreeExplainer(rf)\n", + "shap_values_tree = tree_explainer.shap_values(x_test)\n", + "phi_tree = _phi_class1(shap_values_tree)\n", + "\n", + "# Controle d'efficience (additivite Tree SHAP, exacte par construction) :\n", + "# base E[f] + somme(phi) doit redonner P(default|x).\n", + "_ev = tree_explainer.expected_value\n", + "base_value = float(np.asarray(_ev).ravel()[1]) if np.size(_ev) >= 2 else float(_ev)\n", + "somme_phi_tree = float(np.asarray(phi_tree).sum())\n", + "print(f\"Controle d'additivite : base E[f] = {base_value:+.4f} | somme phi(tree) = {somme_phi_tree:+.4f} \"\n", + " f\"| base + somme = {base_value + somme_phi_tree:+.4f} vs P(default|x) = {proba_default[idx_test]:.4f}\")\n", + "\n", + "# Mise en forme\n", + "phi_marginal_flat = np.asarray(phi_marginal).flatten()\n", + "phi_tree_flat = np.asarray(phi_tree).flatten()\n", + "ecart = phi_marginal_flat - phi_tree_flat\n", + "\n", + "print('\\n=== Shapley values pour la classe default ===')\n", + "print(f'{\"Feature\":<20} {\"Marginal (Kernel)\":>18} {\"Conditionnel (Tree)\":>20} {\"Ecart\":>12}')\n", + "for i, name in enumerate(feature_names[:len(phi_marginal_flat)]):\n", + " print(f'{name:<20} {phi_marginal_flat[i]:>+18.4f} {phi_tree_flat[i]:>+20.4f} {ecart[i]:>+12.4f}')" + ] + }, + { + "cell_type": "markdown", + "id": "be633d76", + "metadata": { + "papermill": { + "duration": 0.00363, + "end_time": "2026-09-19T15:03:43.078853", + "exception": false, + "start_time": "2026-09-19T15:03:43.075223", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "**Lecture.** Sur l'individu test (idx=773, P(default)=0.940, age=44.7, credit_amount=14139.7), les deux methodes attribuent maintenant **deux features distinctes** (fix #16680 : l'indexation precedente flatten le tableau (n, features, classes) et montrait la classe 0 et la classe 1 de la seule feature `age` — d'ou l'anti-symetrie exacte phi(c0) = -phi(c1)). Le controle d'additivite valide l'extraction : base E[f] + somme phi(tree) = 0.1863 + 0.7538 = **0.9400 = P(default|x)** a la 4e decimale, comme l'exige Tree SHAP (exact par construction).\n", + "\n", + "Kernel SHAP (marginal) attribue phi(age)=+0.3783 et phi(credit_amount)=+0.3957 ; Tree SHAP (conditionnel) attribue phi(age)=+0.3505 et phi(credit_amount)=+0.4033. **L'ecart marginal-conditionnel depend de la feature** : +0.0279 sur `age` (le marginal amplifie legerement) mais **-0.0076 sur `credit_amount`** (le marginal attenue). Il n'y a PAS de signe systematique — sur ce modele a deux features correlees, la violation causale du marginal (tirages hors distribution) se repartit differemment selon la feature, et les ecarts restent faibles (ordre 0.01-0.03) devant les attributions elles-memes (ordre 0.4)." + ] + }, + { + "cell_type": "markdown", + "id": "741e0c29", + "metadata": { + "papermill": { + "duration": 0.00366, + "end_time": "2026-09-19T15:03:43.086289", + "exception": false, + "start_time": "2026-09-19T15:03:43.082629", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 4. Quand le conditionnel **est** l'effet causal (T9)\n", + "\n", + "**Theoreme R3 (Chen-Covert-Lundberg-Lee 2022, Section 4)** : sous l'hypothese que le **background dataset $D$** est tire du **modele causal** $P^\\text{do}(X)$ — c'est-a-dire que $D$ respecte la **consistance** avec le DAG $G$ — alors la **Shapley value conditionnelle** $\\phi^\\text{cond}_i(f, x^*)$ coïncide avec l'**effet causal** $do(X_i = x^*_i)$ sur la sortie, **modulo** les variables non observees.\n", + "\n", + "**Reciproque (R4 Bareinboim-Pearl 2016)** : dans la **Ladder of Causation** (R5 Bareinboim-Correa-Ibeling-Icard 2026), les trois niveaux sont :\n", + "\n", + "1. **Association** $P(Y \\mid X)$ — niveau 1, ce que le ML classique fait.\n", + "\n", + "2. **Intervention** $P(Y \\mid do(X))$ — niveau 2, l'effet causal. **Conditional Shapley** sous DAG connu y accede.\n", + "\n", + "3. **Contrefactuel** $P(Y_x \\mid X=x', Y=y')$ — niveau 3, « qu'aurait-il fallu changer ? ». Les contrefactuels DiCE (4. ci-dessous) operent a ce niveau, **mais sans garantie causale** : ils cherchent un voisin realiste, pas un chemin causal.\n", + "\n", + "**Implication pratique pour ce notebook** : si on dispose d'un DAG connu et d'un background $D$ consistant, **Tree SHAP est preferable a Kernel SHAP** pour expliquer un modele de credit. C'est un argument normatif, pas juste methodologique." + ] + }, + { + "cell_type": "markdown", + "id": "8a9f0a2d", + "metadata": { + "papermill": { + "duration": 0.003751, + "end_time": "2026-09-19T15:03:43.093682", + "exception": false, + "start_time": "2026-09-19T15:03:43.089931", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 5. Visualisation : le beeswarm Kernel vs Tree\n", + "\n", + "On trace les deux beeswarms sur le background complet. Les features qui dependent d'autres (DAG) montrent des distributions differentes entre les deux methodes." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "b36106a3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-19T15:03:43.102565Z", + "iopub.status.busy": "2026-09-19T15:03:43.102275Z", + "iopub.status.idle": "2026-09-19T15:03:43.886629Z", + "shell.execute_reply": "2026-09-19T15:03:43.886050Z" + }, + "papermill": { + "duration": 0.790227, + "end_time": "2026-09-19T15:03:43.887673", + "exception": false, + "start_time": "2026-09-19T15:03:43.097446", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "ee9fd4813f94456c8b520c892461471d", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " 0%| | 0/50 [00:00 41.48 +0.1235\n", + " credit_amount > 13463.64 -0.0708\n" + ] + } + ], + "source": [ + "import lime\n", + "import lime.lime_tabular\n", + "\n", + "lime_explainer = lime.lime_tabular.LimeTabularExplainer(\n", + " training_data=np.asarray(X_background),\n", + " feature_names=list(feature_names),\n", + " class_names=['non-default', 'default'],\n", + " mode='classification',\n", + " random_state=RANDOM_STATE,\n", + ")\n", + "\n", + "lime_exp = lime_explainer.explain_instance(\n", + " data_row=np.asarray(x_test).flatten(),\n", + " predict_fn=rf.predict_proba,\n", + " num_features=len(feature_names),\n", + ")\n", + "\n", + "print('LIME weights (classe default) :')\n", + "for feat, weight in lime_exp.as_list():\n", + " print(f' {feat:<30} {weight:>+.4f}')" + ] + }, + { + "cell_type": "markdown", + "id": "727dc025", + "metadata": { + "papermill": { + "duration": 0.003992, + "end_time": "2026-09-19T15:03:43.985896", + "exception": false, + "start_time": "2026-09-19T15:03:43.981904", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "**Lecture.** LIME produit une liste de conditions textuelles (`feature > seuil`) avec leur poids lineaire local. La comparaison avec SHAP force/local donne le **diagnostic classique** : les deux methodes **ne s'accordent pas toujours** sur l'importance des features, et la methode la plus stable depend du modele (LIME pour les modeles non-arbre, SHAP pour les modeles-arbre)." + ] + }, + { + "cell_type": "markdown", + "id": "f34371c0", + "metadata": { + "papermill": { + "duration": 0.00399, + "end_time": "2026-09-19T15:03:43.993915", + "exception": false, + "start_time": "2026-09-19T15:03:43.989925", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 7. Contrefactuels DiCE : le niveau 3 sans garantie causale\n", + "\n", + "DiCE (Mothilal-Ribeiro-Singh 2020) cherche des **voisins realistes** de $x^*$ qui changent la prediction. C'est une approche **niveau 3** (Ladder of Causation) **au sens descriptif** : on cherche un monde contrefactuel. Mais DiCE **ne garantit pas** que le voisin est atteignable par une intervention causale valide sur le DAG. C'est exactement la distinction que le depot entretient entre contrefactuels **sur DAG** (DoWhy-2 dans Causal-Bridges) et contrefactuels **sur features independantes** (DiCE dans 2.14 et ici)." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "7f82c084", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-19T15:03:44.003247Z", + "iopub.status.busy": "2026-09-19T15:03:44.002838Z", + "iopub.status.idle": "2026-09-19T15:03:44.240255Z", + "shell.execute_reply": "2026-09-19T15:03:44.239786Z" + }, + "papermill": { + "duration": 0.243175, + "end_time": "2026-09-19T15:03:44.241100", + "exception": false, + "start_time": "2026-09-19T15:03:43.997925", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\r", + " 0%| | 0/1 [00:00 credit_amount`, fixer `age=42` **doit** aussi ajuster `credit_amount`).\n", + "\n", + "C'est la frontiere que **Causal-Bridges/DoWhy-2-Contrefactuel-Individuel.ipynb** explore avec les graphes causaux structures." + ] + }, + { + "cell_type": "markdown", + "id": "fd45a3cb", + "metadata": { + "papermill": { + "duration": 0.004081, + "end_time": "2026-09-19T15:03:44.257650", + "exception": false, + "start_time": "2026-09-19T15:03:44.253569", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 7bis. Counterfactual SHAP : le niveau 3 d'attribution (Bareinboim thm 6.2.6)\n", + "\n", + "Le notebook s'arrete au barreau 2 (marginal vs conditionnel) et mentionne DiCE (section 7) en precisant que ce dernier \"n'est pas une intervention causale\". **Bareinboim thm 6.2.6** (*Causal Artificial Intelligence*, 2026) definit un **troisieme objet** : le **counterfactual SHAP**, qui attribue les responsabilites sur un contrefactuel $Y_{x}(u)$ (couche 3 de l'echelle causale de Pearl), par opposition au contrefactuel DiCE qui cherche un **voisin realiste** mais sans engagement causal.\n", + "\n", + "**Distinction formelle.** Soient $x^*$ l'individu observe, $do(X=x')$ une intervention au sens de Pearl, et $Y_{x'}(u)$ la valeur de $Y$ pour le meme $u$ (meme contexte exogene) sous l'intervention :\n", + "\n", + "- **SHAP (niveau 1-2)** : $\\phi_i = $ contribution de $X_i$ a $f(x^*)$ (Kernel) ou $E[Y \\mid X=x^*]$ (Tree).\n", + "- **Counterfactual SHAP (niveau 3)** : $\\phi_i^{CF} = \\mathbb{E}[Y_{x'}(u) \\mid X=x^*] - \\mathbb{E}[Y_{x}(u) \\mid X=x^*]$, decompose par feature.\n", + "\n", + "La difference pratique : **memes features $X$ peuvent avoir des attributions differentes** parce que le contrefactuel $Y_{x'}$ evalue l'effet sur $u$ sous une intervention, pas seulement la prediction au point $x'$.\n", + "\n", + "**Reference.** Li, Lee, Dennis, Bareinboim (2026) *Counterfactual Debugging the World Model Transfer Gap* (preprint, archive `G:\\Mon Drive\\MyIA\\IA\\Bibliographie IA\\XAI‚6 - Li, Lee, Dennis, Bareinboim - Counterfactual Debugging the World Model Transfer Gap (preprint).pdf`) applique ce cadre pour **identifier les pas de temps causalement responsables** d'une degradation de performance dans un world model, par divide-and-conquer exploitant la parcimonie des erreurs causales. Bareinboim etant co-auteur, le lien avec `Causal Artificial Intelligence` thm 6.2.6 est direct.\n", + "\n", + "**Ce que cette section montre.** Une **approximation pedagogique** : on prend un contrefactuel $x'$ (en modifiant `age` de +14 ans), on evalue le modele sur $x'$, et on compare la **somme des attributions conditionnelles (Tree SHAP)** au **gap contrefactuel** $f(x') - f(x^*)$. Ce n'est pas le counterfactual SHAP au sens formel Bareinboim thm 6.2.6 (qui necessite le contrefactuel $Y_{x'}(u)$ sur le DAG), mais cela revele la **meme structure** : l'attribution sur le contrefactuel n'est pas la prediction, et la decomposition n'est pas invariante par translation de feature.\n", + "\n", + "**Note methodologique.** DiCE (section 7) reste pertinent pour la **generation de voisins realistes** ; Counterfactual SHAP (cette section) pour l'**attribution causale au contrefactuel**. Les deux sont des outils du niveau 3 mais avec des garanties differentes : DiCE optimise une distance dans l'espace des features, Counterfactual SHAP decompose une difference causale. La section 8 (Ponts) renvoie au notebook `Do-Calculus-Bridge.ipynb` pour l'estimation rigoureuse de $Y_{x}(u)$ via do-calculus.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "371f55b8", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-19T15:03:44.266817Z", + "iopub.status.busy": "2026-09-19T15:03:44.266394Z", + "iopub.status.idle": "2026-09-19T15:03:44.269138Z", + "shell.execute_reply": "2026-09-19T15:03:44.268665Z" + }, + "papermill": { + "duration": 0.008267, + "end_time": "2026-09-19T15:03:44.269891", + "exception": false, + "start_time": "2026-09-19T15:03:44.261624", + "status": "completed" + }, + "tags": [] + }, + "outputs": [], + "source": [ + "# Helper _phi_class1 (defini ici c.687 Tell c.G.9 ★★★★ posture humble fondateur)\n", + "# Cellule idx=21 (apres insertion) appelle _phi_class1(tree_sv_cf)[0].\n", + "# Le helper etait dans la branche c.663 (8ae8c8cd4f) mais pas cherry-picke.\n", + "import numpy as np\n", + "\n", + "# _phi_class1 est definie en cellule [7] des le premier appel shap_values (fix #16680)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "069bf27a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-19T15:03:44.279434Z", + "iopub.status.busy": "2026-09-19T15:03:44.279174Z", + "iopub.status.idle": "2026-09-19T15:03:44.299721Z", + "shell.execute_reply": "2026-09-19T15:03:44.299176Z" + }, + "papermill": { + "duration": 0.026474, + "end_time": "2026-09-19T15:03:44.300477", + "exception": false, + "start_time": "2026-09-19T15:03:44.274003", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Individu test (idx_ref=773, ligne 0 de x_test): f(x*) = 0.9400\n", + "Contrefactuel (age=42): f(x') = 0.7200\n", + "Gap contrefactuel: f(x') - f(x*) = -0.2200\n", + "\n", + "=== Counterfactual SHAP (approximation pedagogique) ===\n", + "Baseline E[f(X)] = 0.1863\n", + "Somme phi Tree SHAP sur x' = +0.5337\n", + "Reconstruction (base + sum phi) = 0.7200\n", + "Reel f(x') = 0.7200\n", + "Additivite Tree SHAP: |base + sum phi - f(x')| = 0.0000\n", + "\n", + "=== Attributions par feature (contrefactuel x') ===\n", + " phi(age) = +0.2560 <-- modifie\n", + " phi(credit_amount) = +0.2778\n", + "\n", + "=== Comparaison au gap contrefactuel ===\n", + "Gap f(x') - f(x*) = -0.2200\n", + "Somme des deltas phi(x') - phi(x*) = +0.5337\n", + "Note : le counterfactual SHAP formel decompose Y_x(u), pas seulement f(x').\n", + " Cette approximation pedagogique montre la structure, pas l'objet formel.\n", + "\n", + "=== References ===\n", + "- Bareinboim (2026) Causal Artificial Intelligence, theorem 6.2.6 (counterfactual SHAP)\n", + "- Li, Lee, Dennis, Bareinboim (2026) Counterfactual Debugging the World Model Transfer Gap\n", + " G:\\Mon Drive\\MyIA\\IA\\Bibliographie IA\\XAI\\2026 - Li, Lee, Dennis, Bareinboim - Counterfactual Debugging the World Model Transfer Gap (preprint).pdf\n" + ] + } + ], + "source": [ + "import shap\n", + "import numpy as np\n", + "\n", + "# Individu de test deja evalue dans la section 3 (idx_ref=773, ref. historique)\n", + "# Fix c.687 Tell c.G.9 ★★★★ : x_test est un DataFrame 1-ligne (X_background.iloc[[idx_test]]),\n", + "# donc x_test.iloc[[idx_test]] etait OOB. On prend l'unique ligne x_test.iloc[[0]].\n", + "idx_test_ref = 773 # reference historique conservee pour la trace\n", + "x_orig = x_test.iloc[[0]]\n", + "f_orig = rf.predict_proba(x_orig)[0, 1]\n", + "\n", + "# Contrefactuel pedagogique : augmenter age de 28 -> 42\n", + "x_cf = x_orig.copy()\n", + "x_cf['age'] = 42\n", + "f_cf = rf.predict_proba(x_cf)[0, 1]\n", + "\n", + "print(\"Individu test (idx_ref=\" + str(idx_test_ref) + \", ligne 0 de x_test): f(x*) = {:.4f}\".format(f_orig))\n", + "print(\"Contrefactuel (age=42): f(x') = {:.4f}\".format(f_cf))\n", + "print(\"Gap contrefactuel: f(x') - f(x*) = {:+.4f}\".format(f_cf - f_orig))\n", + "\n", + "# Attribution Tree SHAP sur le contrefactuel (classe 1, via helper _phi_class1 du fix c.663)\n", + "tree_explainer = shap.TreeExplainer(rf)\n", + "tree_sv_cf = tree_explainer.shap_values(x_cf)\n", + "phi_cf = _phi_class1(tree_sv_cf)[0]\n", + "sum_phi_cf = phi_cf.sum()\n", + "\n", + "# Expected value du modele (baseline SHAP)\n", + "expected_value = tree_explainer.expected_value\n", + "if isinstance(expected_value, (list, np.ndarray)):\n", + " base_value = expected_value[1] if len(expected_value) == 2 else expected_value[0]\n", + "else:\n", + " base_value = expected_value\n", + "\n", + "print()\n", + "print(\"=== Counterfactual SHAP (approximation pedagogique) ===\")\n", + "print(\"Baseline E[f(X)] = {:.4f}\".format(base_value))\n", + "print(\"Somme phi Tree SHAP sur x' = {:+.4f}\".format(sum_phi_cf))\n", + "print(\"Reconstruction (base + sum phi) = {:.4f}\".format(base_value + sum_phi_cf))\n", + "print(\"Reel f(x') = {:.4f}\".format(f_cf))\n", + "print(\"Additivite Tree SHAP: |base + sum phi - f(x')| = {:.4f}\".format(abs(base_value + sum_phi_cf - f_cf)))\n", + "\n", + "print()\n", + "print(\"=== Attributions par feature (contrefactuel x') ===\")\n", + "for name, phi in zip(feature_names, phi_cf):\n", + " marker = ' <-- modifie' if name == 'age' else ''\n", + " print(\" phi({}) = {:+.4f}{}\".format(name, phi, marker))\n", + "\n", + "gap = f_cf - f_orig\n", + "print()\n", + "print(\"=== Comparaison au gap contrefactuel ===\")\n", + "print(\"Gap f(x') - f(x*) = {:+.4f}\".format(gap))\n", + "print(\"Somme des deltas phi(x') - phi(x*) = {:+.4f}\".format(sum_phi_cf))\n", + "print(\"Note : le counterfactual SHAP formel decompose Y_x(u), pas seulement f(x').\")\n", + "print(\" Cette approximation pedagogique montre la structure, pas l'objet formel.\")\n", + "\n", + "print()\n", + "print(\"=== References ===\")\n", + "print(\"- Bareinboim (2026) Causal Artificial Intelligence, theorem 6.2.6 (counterfactual SHAP)\")\n", + "print(\"- Li, Lee, Dennis, Bareinboim (2026) Counterfactual Debugging the World Model Transfer Gap\")\n", + "print(\" G:\\\\Mon Drive\\\\MyIA\\\\IA\\\\Bibliographie IA\\\\XAI\\\\2026 - Li, Lee, Dennis, Bareinboim - Counterfactual Debugging the World Model Transfer Gap (preprint).pdf\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "b64fd69a", + "metadata": { + "papermill": { + "duration": 0.003897, + "end_time": "2026-09-19T15:03:44.308660", + "exception": false, + "start_time": "2026-09-19T15:03:44.304763", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 8. Ponts vers la serie causalite du depot\n", + "\n", + "Ce notebook est le **versant explication predictive**. Le depot couvre par ailleurs le **versant causal**, et les deux se repondent :\n", + "\n", + "- [2.14 Explicabilite (XAI)](2.14-Explicabilite-SHAP-LIME-Contrefactuels.ipynb) — la base : SHAP, LIME, DiCE contrefactuels.\n", + "- [Causal-Bridges](../../../Probas/DecisionTheory/Causal-Bridges/README.md) — do-calculus, DoWhy de bout en bout : [DoWhy-2-Contrefactuel-Individuel](../../../Probas/DecisionTheory/Causal-Bridges/DoWhy-2-Contrefactuel-Individuel.ipynb) fait sur DAG ce que **DiCE fait sur features independantes** — la comparaison est directe.\n", + "- [Do-Calculus-Bridge.ipynb](../../../Probas/DecisionTheory/Causal-Bridges/Do-Calculus-Bridge.ipynb) — do-calculus Bareinboim-Pearl R-90, mediation counterfactuelle formelle.\n", + "- [Do-Calculus-Bridge.ipynb](../../../Probas/DecisionTheory/Causal-Bridges/Do-Calculus-Bridge.ipynb) — 4 taches data-fusion, CHT L1→L2, jonction do-calculus ≅ conditional Shapley.\n", + "- [Infer-5-Causal-Inference.ipynb](../../../Probas/Infer/Infer-5-Causal-Inference.ipynb) — mediation NDE+NIE=TE en Infer.NET (enumeration exacte).\n", + "- [PyMC-05-Causal-Inference.ipynb](../../../Probas/PyMC/PyMC-05-Causal-Inference.ipynb) — version PyMC de la mediation NDE+NIE=TE.\n", + "\n", + "**Le cercle complet** : SHAP explique `f(x)`, DoWhy explique `P(Y \\mid do(X))`, et R3+R4 montrent que les deux **coïncident sous DAG connu + background consistant**." + ] + }, + { + "cell_type": "markdown", + "id": "0682f55c", + "metadata": { + "papermill": { + "duration": 0.004424, + "end_time": "2026-09-19T15:03:44.317071", + "exception": false, + "start_time": "2026-09-19T15:03:44.312647", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 9. Note « explicatif n'est pas causal »\n", + "\n", + "Trois confusions courantes que ce notebook refute par la mesure :\n", + "\n", + "1. **« SHAP mesure les causes »** — non. SHAP mesure les **attributions** sous une definition marginale ou conditionnelle. La jonction causale exige un DAG connu (R4) et un background consistant (R3). Sans ces deux pre-conditions, SHAP reste une **explication** au sens de R1 (3 axiomes), pas au sens causal.\n", + "2. **« DiCE contrefactuel = contrefactuel causal »** — non. DiCE cherche un voisin realiste, pas un chemin causal. Le **contrefactuel au sens de Pearl** (niveau 3) opere sur un DAG avec des equations structurelles ; sans DAG, on n'a pas de contrefactuel causal.\n", + "3. **« Kernel SHAP ≈ Tree SHAP »** — non quand le DAG contient des dependances entre features. La section 3 le montre numeriquement : l'ecart marginal/conditionnel sur `credit_amount` quand on conditionne sur `age` est mesurable et signe la violation causale du marginal.\n", + "\n", + "**Reflexe honnete** : presenter SHAP comme une **explication** (R1 axiomes satisfaits), et **separer** les conclusions causales qui exigent un DAG et un background consistant (R3+R4)." + ] + }, + { + "cell_type": "markdown", + "id": "399a8384", + "metadata": { + "papermill": { + "duration": 0.003909, + "end_time": "2026-09-19T15:03:44.325155", + "exception": false, + "start_time": "2026-09-19T15:03:44.321246", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 10. Exercices\n", + "\n", + "Trois exercices pour passer de lecteur a praticien — stubs conformes (C.1, JAMAIS `raise NotImplementedError`), a completer. Le notebook s'execute de bout en bout meme sans les avoir faits." + ] + }, + { + "cell_type": "markdown", + "id": "61c15485", + "metadata": { + "papermill": { + "duration": 0.004103, + "end_time": "2026-09-19T15:03:44.333311", + "exception": false, + "start_time": "2026-09-19T15:03:44.329208", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Exercice 1 — Mesurer l'ecart marginal/conditionnel sur 5 individus aleatoires\n", + "\n", + "Reprendre la cellule de mesure (section 3) sur **5 individus** tires au hasard (indices `[12, 47, 128, 233, 401]` par exemple) et calculer l'ecart moyen sur chaque feature. **Question** : l'ecart sur `credit_amount` est-il toujours du **meme signe** ? Si oui, c'est le signe de la violation causale systematique du marginal." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "3c9c9d6a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-19T15:03:44.343088Z", + "iopub.status.busy": "2026-09-19T15:03:44.342764Z", + "iopub.status.idle": "2026-09-19T15:03:44.346090Z", + "shell.execute_reply": "2026-09-19T15:03:44.345576Z" + }, + "papermill": { + "duration": 0.009309, + "end_time": "2026-09-19T15:03:44.346818", + "exception": false, + "start_time": "2026-09-19T15:03:44.337509", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Exercice 1 — stub : mesurer l'ecart sur 5 individus.\n" + ] + } + ], + "source": [ + "# STUB etudiant — a completer\n", + "# Indice : repeter la mesure de la cellule [9] sur 5 indices et moyenner les ecarts.\n", + "indices_test = [12, 47, 128, 233, 401]\n", + "# result = None # TODO etudiant : dictionnaire {feature: ecart_moyen_signe}\n", + "print(\"Exercice 1 — stub : mesurer l'ecart sur 5 individus.\")" + ] + }, + { + "cell_type": "markdown", + "id": "25f12c34", + "metadata": { + "papermill": { + "duration": 0.004071, + "end_time": "2026-09-19T15:03:44.355102", + "exception": false, + "start_time": "2026-09-19T15:03:44.351031", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Exercice 2 — Construire un background **inconsistant** et observer l'aggravation\n", + "\n", + "Tirer un background de **memes marginales** mais avec des **dependances inversees** (par exemple, `age` et `credit_amount` negativement correles au lieu de positivement). Mesurer l'ecart marginal/conditionnel : il doit **augmenter** par rapport au background consistant du DAG original." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "3f6e331b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-19T15:03:44.366263Z", + "iopub.status.busy": "2026-09-19T15:03:44.365959Z", + "iopub.status.idle": "2026-09-19T15:03:44.368922Z", + "shell.execute_reply": "2026-09-19T15:03:44.368361Z" + }, + "papermill": { + "duration": 0.008775, + "end_time": "2026-09-19T15:03:44.369614", + "exception": false, + "start_time": "2026-09-19T15:03:44.360839", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Exercice 2 — stub : construire un background inconsistant et observer l'aggravation.\n" + ] + } + ], + "source": [ + "# STUB etudiant — a completer\n", + "# Indice : construire X_background_inconsistent ou les correlations age <-> credit_amount sont inversees.\n", + "# Comparer les ecarts avec la cellule [9].\n", + "# result_inconsistent = None # TODO etudiant : ecart avec background inconsistant\n", + "print(\"Exercice 2 — stub : construire un background inconsistant et observer l'aggravation.\")" + ] + }, + { + "cell_type": "markdown", + "id": "f2318638", + "metadata": { + "papermill": { + "duration": 0.004525, + "end_time": "2026-09-19T15:03:44.378558", + "exception": false, + "start_time": "2026-09-19T15:03:44.374033", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Exercice 3 — Verifier qu'un contrefactuel DiCE **n'est pas** une intervention $do(\\cdot)$\n", + "\n", + "Reprendre le contrefactuel de la section 7 et **comparer** avec un contrefactuel causal **sur DAG** (DoWhy-2 dans Causal-Bridges, ou un simple calcul structurel : `do(age=42)` implique une nouvelle valeur de `credit_amount` via le DAG). Montrer que les deux **different**, et conclure sur la portee de DiCE." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "e39d3400", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-19T15:03:44.388212Z", + "iopub.status.busy": "2026-09-19T15:03:44.387999Z", + "iopub.status.idle": "2026-09-19T15:03:44.391677Z", + "shell.execute_reply": "2026-09-19T15:03:44.391000Z" + }, + "papermill": { + "duration": 0.009684, + "end_time": "2026-09-19T15:03:44.392501", + "exception": false, + "start_time": "2026-09-19T15:03:44.382817", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Exercice 3 — stub : DiCE contrefactuel != intervention do(.).\n" + ] + } + ], + "source": [ + "# STUB etudiant — a completer\n", + "# Indice : appliquer do(age=42) sur le DAG (credit_amount = 5000 + 200*age) et comparer avec DiCE.\n", + "# conclusion = None # TODO etudiant : phrase qui conclut sur la portee de DiCE vs intervention causale.\n", + "print(\"Exercice 3 — stub : DiCE contrefactuel != intervention do(.).\")" + ] + }, + { + "cell_type": "markdown", + "id": "dc2488ed", + "metadata": { + "papermill": { + "duration": 0.004152, + "end_time": "2026-09-19T15:03:44.401077", + "exception": false, + "start_time": "2026-09-19T15:03:44.396925", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## Conclusion\n", + "\n", + "Ce notebook a transforme la **jonction XAI <-> causalite** d'un concept (R3, R4) en une **mesure visible** : l'ecart entre Kernel SHAP marginal et Tree SHAP conditionnel est un proxy direct de la violation causale du marginal. Sur le DAG `age -> credit_amount -> default` simule (ou sur le German Credit reel quand le pickle de 2.14 est present), cet ecart est mesurable, signe, et reproductible.\n", + "\n", + "**Trois takeaways pour le praticien** :\n", + "\n", + "1. **Tree SHAP est preferable a Kernel SHAP** quand on dispose d'un modele-arbre ET d'un DAG connu — c'est un argument normatif (R3 T9 + R4).\n", + "2. **DiCE contrefactuel n'est pas une intervention causale** — c'est un voisin realiste, pas un chemin causal. La comparaison avec DoWhy-2 sur DAG le rend visible.\n", + "3. **SHAP explique, ne cause pas** — sans DAG et sans background consistant, SHAP reste une attribution au sens de R1 (3 axiomes), pas au sens causal (R4 Bareinboim-Pearl).\n", + "\n", + "**Refs** : R1 Lundberg-Lee 2017 · R2 Lundberg et al 2019 · R3 Chen-Covert-Lundberg-Lee 2022 · R4 Bareinboim-Pearl 2016 · R5 Bareinboim-Correa-Ibeling-Icard 2026." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (coursia-ml-training)", + "language": "python", + "name": "coursia-ml-training" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.3" + }, + "papermill": { + "default_parameters": {}, + "duration": 7.572528, + "end_time": "2026-09-19T15:03:45.055613", + "environment_variables": {}, + "exception": null, + "input_path": "D:\\Dev\\CoursIA-16680\\MyIA.AI.Notebooks\\ML\\DataScienceWithAgents\\02-ML-Cours\\2.14b-XAI-Shap-Attribution-Causal-Bridge.ipynb", + "output_path": "D:\\Dev\\CoursIA-16680\\MyIA.AI.Notebooks\\ML\\DataScienceWithAgents\\02-ML-Cours\\2.14b-XAI-Shap-Attribution-Causal-Bridge_output.ipynb", + "parameters": {}, + "start_time": "2026-09-19T15:03:37.483085", + "version": "2.7.0" + }, + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "state": { + "065bf5e565b54ccc81ee83337e4faf75": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_88ecd8bd0da54db2b06e9eee8bc27194", + "max": 50.0, + "min": 0.0, + "orientation": "horizontal", + "style": "IPY_MODEL_b4637bcdfcb249098813702926b05284", + "tabbable": null, + "tooltip": null, + "value": 50.0 + } + }, + "2ff22aaa890d4fefb0715081562a452e": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "354a8c89e9d74f39aaf1524d8386fa1f": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "3fc323ed16834f0eb6aa02c1ffb47c57": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_9d7a964725c6484496c46f1a310b311f", + "max": 1.0, + "min": 0.0, + "orientation": "horizontal", + "style": "IPY_MODEL_61c346b89c8c4111b7025200b898c4f9", + "tabbable": null, + "tooltip": null, + "value": 1.0 + } + }, + "4792b1263d23409daf09d098543d364f": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "4ef609e9d5ed410b858e179494bf4938": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HTMLView", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_2ff22aaa890d4fefb0715081562a452e", + "placeholder": "​", + "style": "IPY_MODEL_96d3d0dd954e411daba0ae088c6042bf", + "tabbable": null, + "tooltip": null, + "value": "100%" + } + }, + "50a620a6d232454c83a061a19af1620e": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_8dda40a15a344c18aa97131617679ca9", + "IPY_MODEL_3fc323ed16834f0eb6aa02c1ffb47c57", + "IPY_MODEL_c2558eb9ad2d4584a3bb718a79a51e59" + ], + "layout": "IPY_MODEL_dff588562aa84c649cf871b2e657fa91", + "tabbable": null, + "tooltip": null + } + }, + "61c346b89c8c4111b7025200b898c4f9": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "6b7aca2bd7a242f093cbeaf830a28381": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "background": null, + "description_width": "", + "font_size": null, + "text_color": null + } + }, + "7d12f671dcee4ceb82b97a7e676285fc": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "88ecd8bd0da54db2b06e9eee8bc27194": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "8dda40a15a344c18aa97131617679ca9": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HTMLView", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_4792b1263d23409daf09d098543d364f", + "placeholder": "​", + "style": "IPY_MODEL_e032ec2de9e84df59baf4b4d879eb867", + "tabbable": null, + "tooltip": null, + "value": "100%" + } + }, + "96d3d0dd954e411daba0ae088c6042bf": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "background": null, + "description_width": "", + "font_size": null, + "text_color": null + } + }, + "9d7a964725c6484496c46f1a310b311f": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "b4637bcdfcb249098813702926b05284": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "b8baefe00ab4436f83935fdf6ac7cf2b": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "c2558eb9ad2d4584a3bb718a79a51e59": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HTMLView", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_354a8c89e9d74f39aaf1524d8386fa1f", + "placeholder": "​", + "style": "IPY_MODEL_dbbead9034c8429795bfbdee9e66591a", + "tabbable": null, + "tooltip": null, + "value": " 1/1 [00:00<00:00, 43.04it/s]" + } + }, + "dbbead9034c8429795bfbdee9e66591a": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "background": null, + "description_width": "", + "font_size": null, + "text_color": null + } + }, + "dc42bad91793491081cd0d848f7a67fa": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HTMLView", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_7d12f671dcee4ceb82b97a7e676285fc", + "placeholder": "​", + "style": "IPY_MODEL_6b7aca2bd7a242f093cbeaf830a28381", + "tabbable": null, + "tooltip": null, + "value": " 50/50 [00:00<00:00, 117.08it/s]" + } + }, + "dff588562aa84c649cf871b2e657fa91": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e032ec2de9e84df59baf4b4d879eb867": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "background": null, + "description_width": "", + "font_size": null, + "text_color": null + } + }, + "ee9fd4813f94456c8b520c892461471d": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_4ef609e9d5ed410b858e179494bf4938", + "IPY_MODEL_065bf5e565b54ccc81ee83337e4faf75", + "IPY_MODEL_dc42bad91793491081cd0d848f7a67fa" + ], + "layout": "IPY_MODEL_b8baefe00ab4436f83935fdf6ac7cf2b", + "tabbable": null, + "tooltip": null + } + } + }, + "version_major": 2, + "version_minor": 0 + } + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/shap_kernel_marginal.png b/MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/shap_kernel_marginal.png new file mode 100644 index 0000000000..02c41fdb6d Binary files /dev/null and b/MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/shap_kernel_marginal.png differ diff --git a/MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/shap_tree_conditional.png b/MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/shap_tree_conditional.png new file mode 100644 index 0000000000..34fbeb9349 Binary files /dev/null and b/MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/shap_tree_conditional.png differ