From 87a30ee1429173b8497fec7441d3c8a561d6db86 Mon Sep 17 00:00:00 2001 From: jsboige Date: Mon, 28 Sep 2026 15:56:16 +0200 Subject: [PATCH 1/2] Add(genai,#17883): 03b_Typed_Decisions_System1 -- contrat choice/score/noul, LLM gele lu sur ses logits vs temoin classique, calibration et escalade (tranche 1) Notebook neuf 29 cellules (12 code, 17 md), execute 12/12 cellules, 0 erreur (kernel bonsai, python 3.11.15). Nav 03 -> 03b, ligne README Tier 2. Co-Authored-By: Claude Sonnet 5 --- .../GenAI/Texte/03_Structured_Outputs.ipynb | 2 +- .../Texte/03b_Typed_Decisions_System1.ipynb | 1561 +++++++++++++++++ MyIA.AI.Notebooks/GenAI/Texte/README.md | 1 + 3 files changed, 1563 insertions(+), 1 deletion(-) create mode 100644 MyIA.AI.Notebooks/GenAI/Texte/03b_Typed_Decisions_System1.ipynb diff --git a/MyIA.AI.Notebooks/GenAI/Texte/03_Structured_Outputs.ipynb b/MyIA.AI.Notebooks/GenAI/Texte/03_Structured_Outputs.ipynb index 1dab50fe24..bcdef80b15 100644 --- a/MyIA.AI.Notebooks/GenAI/Texte/03_Structured_Outputs.ipynb +++ b/MyIA.AI.Notebooks/GenAI/Texte/03_Structured_Outputs.ipynb @@ -43,7 +43,7 @@ }, "source": [ "# 3. Structured Outputs : Sorties JSON Garanties\n", - "**Navigation** : [Index](README.md) | [<< Précédent](02_PromptEngineering.ipynb) | [Suivant >>](04_Function_Calling.ipynb)\n", + "**Navigation** : [Index](README.md) | [<< Précédent](02_PromptEngineering.ipynb) | [Suivant >>](03b_Typed_Decisions_System1.ipynb)\n", "\n", "## Structured Outputs : Sorties JSON Garanties avec OpenAI\n", "\n", diff --git a/MyIA.AI.Notebooks/GenAI/Texte/03b_Typed_Decisions_System1.ipynb b/MyIA.AI.Notebooks/GenAI/Texte/03b_Typed_Decisions_System1.ipynb new file mode 100644 index 0000000000..09b8901c07 --- /dev/null +++ b/MyIA.AI.Notebooks/GenAI/Texte/03b_Typed_Decisions_System1.ipynb @@ -0,0 +1,1561 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "id": "d3b193d9", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:14.756586Z", + "iopub.status.busy": "2026-09-28T13:51:14.756586Z", + "iopub.status.idle": "2026-09-28T13:51:14.762365Z", + "shell.execute_reply": "2026-09-28T13:51:14.761855Z" + }, + "papermill": { + "duration": 0.011529, + "end_time": "2026-09-28T13:51:14.763636", + "exception": false, + "start_time": "2026-09-28T13:51:14.752107", + "status": "completed" + }, + "tags": [] + }, + "outputs": [], + "source": [ + "# Parameters\n", + "BATCH_MODE = \"true\"" + ] + }, + { + "cell_type": "markdown", + "id": "7cb7e45c", + "metadata": { + "papermill": { + "duration": 0.002271, + "end_time": "2026-09-28T13:51:14.768321", + "exception": false, + "start_time": "2026-09-28T13:51:14.766050", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "# 3b. Décisions typées « système 1 » : décider sans générer\n", + "**Navigation** : [Index](README.md) | [<< Précédent](03_Structured_Outputs.ipynb) | [Suivant >>](04_Function_Calling.ipynb)\n", + "\n", + "`03_Structured_Outputs` obtenait une décision d'un LLM **en la générant** : on force le modèle à\n", + "produire du JSON conforme à un schéma, puis on parse. Cette voie marche, mais elle paie le prix\n", + "d'un LLM : génération token par token, hallucination possible, coût par question.\n", + "\n", + "Il existe une autre voie, discutée depuis longtemps et remise en avant en 2026 : **un petit modèle\n", + "de décision** qui reçoit un état et des questions *typées*, et rend en une seule passe une\n", + "distribution calibrée sur des options déclarées d'avance — sans générer une ligne de texte. Le\n", + "contrat public le plus répandu définit trois types de questions : `choice` (catégoriel),\n", + "`score` (ordinal) et `noul` (binaire). Une dizaine d'implémentations ouvertes le servent déjà.\n", + "\n", + "Ce notebook pose la question que le dépôt ne posait nulle part : **quand un petit modèle de\n", + "décision suffit-il, et que perd-on hors du domaine d'entraînement ?** Pour y répondre, il compare\n", + "sur les *mêmes questions typées* deux familles extrêmes :\n", + "\n", + "| Famille | Principe | Coût par question |\n", + "|---|---|---|\n", + "| **Témoin classique** | TF-IDF + régression logistique, entraîné sur le domaine | ~0 (CPU, microsecondes) |\n", + "| **LLM gelé lu sur ses logits** | aucun entraînement : on lit la log-probabilité de chaque option sur un modèle déjà servi | une passe de forward |\n", + "\n", + "Les familles à checkpoint externe (laya, Kev, jevlike — voir conclusion) sont des tranches\n", + "suivantes de l'issue #17883 : elles ne sont pas substituées par un jouet ici." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "b24031dd", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:14.775048Z", + "iopub.status.busy": "2026-09-28T13:51:14.774541Z", + "iopub.status.idle": "2026-09-28T13:51:17.557800Z", + "shell.execute_reply": "2026-09-28T13:51:17.556773Z" + }, + "papermill": { + "duration": 2.787325, + "end_time": "2026-09-28T13:51:17.558588", + "exception": false, + "start_time": "2026-09-28T13:51:14.771263", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Périphérique d'exécution : cuda\n" + ] + } + ], + "source": [ + "import math\n", + "\n", + "import numpy as np\n", + "import torch\n", + "import matplotlib.pyplot as plt\n", + "\n", + "SEED = 42\n", + "np.random.seed(SEED)\n", + "torch.manual_seed(SEED)\n", + "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + "print(\"Périphérique d'exécution :\", DEVICE)" + ] + }, + { + "cell_type": "markdown", + "id": "b467468b", + "metadata": { + "papermill": { + "duration": 0.002304, + "end_time": "2026-09-28T13:51:17.563232", + "exception": false, + "start_time": "2026-09-28T13:51:17.560928", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 1. Le contrat : des questions typées, un espace de réponses fermé\n", + "\n", + "Une question typée déclare **à l'avance** l'espace des réponses possibles. Pour le type\n", + "`choice`, on déclare les N options ; le modèle ne rend jamais autre chose qu'une distribution\n", + "sur ces N options. Le type `noul` est le cas binaire (nommé d'après ses deux options),\n", + "et le type `score` déclare une échelle ordinale de niveaux." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "f62311e1", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:17.567631Z", + "iopub.status.busy": "2026-09-28T13:51:17.567631Z", + "iopub.status.idle": "2026-09-28T13:51:17.575807Z", + "shell.execute_reply": "2026-09-28T13:51:17.575171Z" + }, + "papermill": { + "duration": 0.011455, + "end_time": "2026-09-28T13:51:17.576919", + "exception": false, + "start_time": "2026-09-28T13:51:17.565464", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "État : Ma carte bleue a été refusée trois fois ce matin sur le site\n", + "Options : ('compte', 'facturation', 'livraison', 'technique')\n", + "Distribution : {'compte': np.float64(0.193), 'facturation': np.float64(0.203), 'livraison': np.float64(0.41), 'technique': np.float64(0.193)}\n", + "Décision : livraison | confiance : 0.41\n" + ] + } + ], + "source": [ + "CHOICE, NOUL, SCORE = \"choice\", \"noul\", \"score\"\n", + "\n", + "\n", + "class Question:\n", + " \"\"\"Une question typée : un état soumis au modèle + un espace de réponses déclaré.\"\"\"\n", + "\n", + " def __init__(self, kind, state, options):\n", + " assert kind in (CHOICE, NOUL, SCORE)\n", + " assert len(options) >= 2\n", + " self.kind = kind\n", + " self.state = state\n", + " self.options = tuple(options)\n", + "\n", + " def decide(self, scores, temperature=1.0):\n", + " \"\"\"Ramène n'importe quel vecteur de scores à une distribution sur les options.\n", + "\n", + " C'est tout le contrat : quoi que produise un moteur, la réponse vit dans\n", + " l'espace fermé des options déclarées.\n", + " \"\"\"\n", + " s = np.asarray(scores, dtype=float)\n", + " z = (s - s.max()) / temperature\n", + " p = np.exp(z)\n", + " return p / p.sum()\n", + "\n", + "\n", + "q = Question(\n", + " CHOICE,\n", + " \"Ma carte bleue a été refusée trois fois ce matin sur le site\",\n", + " (\"compte\", \"facturation\", \"livraison\", \"technique\"),\n", + ")\n", + "p_demo = q.decide([0.05, 0.10, 0.80, 0.05])\n", + "print(\"État :\", q.state)\n", + "print(\"Options :\", q.options)\n", + "print(\"Distribution :\", dict(zip(q.options, p_demo.round(3))))\n", + "print(\"Décision :\", q.options[int(p_demo.argmax())], \"| confiance :\", round(float(p_demo.max()), 3))" + ] + }, + { + "cell_type": "markdown", + "id": "c530ef65", + "metadata": { + "papermill": { + "duration": 0.002267, + "end_time": "2026-09-28T13:51:17.581562", + "exception": false, + "start_time": "2026-09-28T13:51:17.579295", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Lecture du résultat\n", + "\n", + "La distribution sort **par construction** sur l'espace déclaré : une « erreur de type »\n", + "(une option mal orthographiée, une classe inventée, un JSON non conforme) est **impossible** —\n", + "c'est la promesse exacte que `03_Structured_Outputs` achetait à coups de schéma et de parsing.\n", + "\n", + "Mais l'inverse n'est pas vrai : l'espace fermé ne dit **rien** de l'exactitude. Une mauvaise\n", + "option peut sortir à 0,9998. La question utile n'est donc pas « la sortie est-elle bien formée ? »\n", + "(toujours oui) mais « **la confiance affichée est-elle honnête ?** » — c'est une question de\n", + "**calibration**, et elle occupera toute la seconde moitié de ce notebook." + ] + }, + { + "cell_type": "markdown", + "id": "151baf39", + "metadata": { + "papermill": { + "duration": 0.00211, + "end_time": "2026-09-28T13:51:17.586035", + "exception": false, + "start_time": "2026-09-28T13:51:17.583925", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 2. Le banc : un jeu de tickets étiquetés, découpé en train / calibration / test\n", + "\n", + "Pour comparer les familles sur les mêmes questions, il faut des étiquettes de référence. Ce\n", + "notebook utilise un **banc didactique embarqué** : 48 tickets de support en français, 12 par\n", + "classe, rédigés pour ce dépôt (licence : celle du dépôt). Le découpage est déterministe —\n", + "6 tickets par classe en entraînement, 3 en calibration, 3 en test — de sorte que la cellule est\n", + "reproductible sans graine d'aléatoire.\n", + "\n", + "Ce banc est **volontairement petit** : il sert la mécanique (contrat, familles, métriques), pas\n", + "la conclusion scientifique. L'écart d'exactitude zéro-shot **hors domaine** documenté dans\n", + "l'issue (~26 points sur un banc indépendant de 49 tâches) exige un banc externe — c'est la\n", + "tranche suivante, déclarée comme telle." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "b7823eeb", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:17.589832Z", + "iopub.status.busy": "2026-09-28T13:51:17.589832Z", + "iopub.status.idle": "2026-09-28T13:51:17.597960Z", + "shell.execute_reply": "2026-09-28T13:51:17.597960Z" + }, + "papermill": { + "duration": 0.011278, + "end_time": "2026-09-28T13:51:17.599539", + "exception": false, + "start_time": "2026-09-28T13:51:17.588261", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "train=24 calibration=12 test=12\n", + "exemple test : Le lien d'activation recu par mail est expire -> compte\n" + ] + } + ], + "source": [ + "CLASSES = (\"compte\", \"facturation\", \"livraison\", \"technique\") # ordre alphabetique, stable\n", + "\n", + "TICKETS = {\n", + " \"compte\": [\n", + " \"Je n'arrive pas a me connecter a mon compte depuis ce matin\",\n", + " \"Mon mot de passe est refuse alors que je suis sur de lui\",\n", + " \"Comment reinitialiser l'acces a mon profil ?\",\n", + " \"Mon compte a ete bloque sans explication\",\n", + " \"Je veux changer l'adresse email associee a mon compte\",\n", + " \"La double authentification n'envoie plus de code\",\n", + " \"J'ai oublie mon identifiant de connexion\",\n", + " \"Mon profil affiche les donnees d'une autre personne\",\n", + " \"Je souhaite fermer mon compte definitivement\",\n", + " \"Le lien d'activation recu par mail est expire\",\n", + " \"Deux comptes ont ete crees avec mon adresse, je veux fusionner\",\n", + " \"Ma session se deconnecte toutes les cinq minutes\",\n", + " ],\n", + " \"facturation\": [\n", + " \"J'ai ete preleve deux fois pour le meme abonnement\",\n", + " \"Ma facture de mars comporte une ligne inconnue\",\n", + " \"Comment obtenir un remboursement du dernier paiement ?\",\n", + " \"Le tarif applique ne correspond pas a l'offre souscrite\",\n", + " \"Je veux un duplicata de ma facture de janvier\",\n", + " \"Ma carte a ete debitee alors que j'ai annule\",\n", + " \"Les frais annuels apparaissent alors que l'offre dit mensuel\",\n", + " \"Puis-je changer de moyen de paiement pour les prelevements ?\",\n", + " \"Le total facture ne correspond pas au panier commande\",\n", + " \"Une remise promise n'apparait pas sur la facture\",\n", + " \"Je conteste un montant preleve sur mon relevé\",\n", + " \"Comment passer a l'offre etudiante avec le tarif reduit ?\",\n", + " ],\n", + " \"livraison\": [\n", + " \"Mon colis est marque livre mais je ne l'ai pas recu\",\n", + " \"La livraison prevue hier n'a pas eu lieu\",\n", + " \"Comment suivre mon colis apres expedition ?\",\n", + " \"Le colis est arrive endommage, que faire ?\",\n", + " \"Il manque un article dans le carton recu\",\n", + " \"Je veux changer l'adresse de livraison en cours de route\",\n", + " \"Le transporteur affiche une adresse erronnee\",\n", + " \"Ma commande est bloquee en entrepot depuis une semaine\",\n", + " \"Puis-je faire livrer au point relais du quartier ?\",\n", + " \"Le suivi n'a pas bouge depuis trois jours\",\n", + " \"Le livreur n'a laisse aucun avis de passage\",\n", + " \"Je souhaite reporter la date de creneau de livraison\",\n", + " ],\n", + " \"technique\": [\n", + " \"L'application plante a l'ouverture de la page de paiement\",\n", + " \"Les pages mettent une minute a charger depuis la mise a jour\",\n", + " \"Le bouton de validation ne reagit pas au clic\",\n", + " \"J'ai une erreur 500 sur mon espace client\",\n", + " \"Les notifications push n'arrivent plus sur mon telephone\",\n", + " \"L'export de mes donnees produit un fichier corrompu\",\n", + " \"La recherche ne renvoie aucun resultat\",\n", + " \"Les pieces jointes ne se telechargent plus\",\n", + " \"Le site affiche mal le texte sur ma tablette\",\n", + " \"L'application se ferme seule apres quelques secondes\",\n", + " \"Impossible de televerser une photo de profil\",\n", + " \"Le mode sombre ne s'applique qu'a moitie de l'interface\",\n", + " ],\n", + "}\n", + "\n", + "assert set(TICKETS) == set(CLASSES) and all(len(v) == 12 for v in TICKETS.values())\n", + "\n", + "X_train, X_cal, X_test, y_train, y_cal, y_test = [], [], [], [], [], []\n", + "for c in CLASSES:\n", + " tickets = TICKETS[c]\n", + " X_train += tickets[0:6]\n", + " X_cal += tickets[6:9]\n", + " X_test += tickets[9:12]\n", + " y_train += [c] * 6\n", + " y_cal += [c] * 3\n", + " y_test += [c] * 3\n", + "\n", + "y_train_idx = np.array([CLASSES.index(c) for c in y_train])\n", + "y_cal_idx = np.array([CLASSES.index(c) for c in y_cal])\n", + "y_test_idx = np.array([CLASSES.index(c) for c in y_test])\n", + "\n", + "print(f\"train={len(X_train)} calibration={len(X_cal)} test={len(X_test)}\")\n", + "print(\"exemple test :\", X_test[0], \"->\", y_test[0])" + ] + }, + { + "cell_type": "markdown", + "id": "1e82d9fd", + "metadata": { + "papermill": { + "duration": 0.002336, + "end_time": "2026-09-28T13:51:17.604278", + "exception": false, + "start_time": "2026-09-28T13:51:17.601942", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 3. Témoin classique : TF-IDF + régression logistique\n", + "\n", + "La base contre laquelle tout le reste doit se mesurer : un classifieur fine-tuné sur un jeu\n", + "d'étiquettes fixe. C'est la famille que l'industrie utilisait avant les LLM — et sur son\n", + "domaine d'entraînement, elle reste très difficile à battre au rapport qualité/prix." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "633e56b6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:17.609750Z", + "iopub.status.busy": "2026-09-28T13:51:17.609750Z", + "iopub.status.idle": "2026-09-28T13:51:19.136013Z", + "shell.execute_reply": "2026-09-28T13:51:19.135053Z" + }, + "papermill": { + "duration": 1.530742, + "end_time": "2026-09-28T13:51:19.137248", + "exception": false, + "start_time": "2026-09-28T13:51:17.606506", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Exactitude du temoin classique sur le test : 0.500\n", + " compte predit=livraison p=0.57 | Le lien d'activation recu par mail est expire\n", + " compte predit=compte p=0.40 | Deux comptes ont ete crees avec mon adresse, je ve\n", + " compte predit=facturation p=0.41 | Ma session se deconnecte toutes les cinq minutes\n", + " facturation predit=technique p=0.42 | Une remise promise n'apparait pas sur la facture\n" + ] + } + ], + "source": [ + "from sklearn.feature_extraction.text import TfidfVectorizer\n", + "from sklearn.linear_model import LogisticRegression\n", + "from sklearn.pipeline import make_pipeline\n", + "\n", + "clf = make_pipeline(\n", + " TfidfVectorizer(min_df=1, ngram_range=(1, 2)),\n", + " LogisticRegression(max_iter=2000, C=10.0),\n", + ")\n", + "clf.fit(X_train, y_train)\n", + "assert tuple(clf.classes_) == CLASSES\n", + "\n", + "P_clf_test = clf.predict_proba(X_test) # colonnes alignees sur CLASSES\n", + "P_clf_cal = clf.predict_proba(X_cal)\n", + "\n", + "acc_clf = float((P_clf_test.argmax(1) == y_test_idx).mean())\n", + "print(f\"Exactitude du temoin classique sur le test : {acc_clf:.3f}\")\n", + "for state, true_c, p in zip(X_test[:4], y_test[:4], P_clf_test[:4]):\n", + " pred = CLASSES[int(p.argmax())]\n", + " print(f\" {true_c:12s} predit={pred:12s} p={p.max():.2f} | {state[:50]}\")" + ] + }, + { + "cell_type": "markdown", + "id": "c5789cec", + "metadata": { + "papermill": { + "duration": 0.002641, + "end_time": "2026-09-28T13:51:19.142504", + "exception": false, + "start_time": "2026-09-28T13:51:19.139863", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Lecture du résultat\n", + "\n", + "Exactitude mesurée : **0,500** — la moitié du test seulement. Vingt-quatre tickets\n", + "d'entraînement, c'est peu : le témoin classique est exact **dans la mesure de son domaine\n", + "étiqueté**, et 24 exemples ne suffisent pas à couvrir les formulations du test. C'est sa\n", + "contrainte structurelle : chaque nouvelle classe, chaque nouveau vocabulaire exige des\n", + "étiquettes. Ses probabilités, elles, ne sont pas nécessairement honnêtes — la régression\n", + "logistique optimise la log-vraisemblance, pas « 0,8 signifie 80 % de chances d'avoir raison ».\n", + "C'est ce que la section calibration mesure." + ] + }, + { + "cell_type": "markdown", + "id": "91b8a4f5", + "metadata": { + "papermill": { + "duration": 0.002394, + "end_time": "2026-09-28T13:51:19.149307", + "exception": false, + "start_time": "2026-09-28T13:51:19.146913", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 4. Famille « LLM gelé, probabilités lues sur les logits »\n", + "\n", + "Le principe (celui des serveurs dits *zero-train*, ex. SemIf) : **aucun entraînement**. On\n", + "prend un LLM déjà servi, on lui soumet l'état et les options déclarées, et au lieu de **générer**\n", + "une réponse, on lit pour chaque option sa **log-probabilité totale** comme continuation de la\n", + "question. La distribution sur les options est le softmax de ces scores. Le modèle ne génère\n", + "rien : une seule passe de forward par option." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "61e9ed86", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:19.157643Z", + "iopub.status.busy": "2026-09-28T13:51:19.157132Z", + "iopub.status.idle": "2026-09-28T13:51:24.723907Z", + "shell.execute_reply": "2026-09-28T13:51:24.723907Z" + }, + "papermill": { + "duration": 5.573019, + "end_time": "2026-09-28T13:51:24.725397", + "exception": false, + "start_time": "2026-09-28T13:51:19.152378", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "1916c79ae2c44770be9ca133f4678832", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Loading weights: 0%| | 0/290 [00:00 1 ramollit un modèle trop sûr de lui, T < 1 durcit un modèle trop timide." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "6c1c41fb", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:26.955145Z", + "iopub.status.busy": "2026-09-28T13:51:26.955145Z", + "iopub.status.idle": "2026-09-28T13:51:29.225702Z", + "shell.execute_reply": "2026-09-28T13:51:29.225702Z" + }, + "papermill": { + "duration": 2.275174, + "end_time": "2026-09-28T13:51:29.227340", + "exception": false, + "start_time": "2026-09-28T13:51:26.952166", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Temperature apprise — LLM gele : T = 1.40 | temoin classique : T = 0.35\n", + "famille T exact. NLL Brier ECE\n", + "temoin classique 1.00 0.500 1.287 0.699 0.127\n", + " + temperature 0.35 0.500 1.518 0.776 0.283\n", + "LLM gele zero-shot 1.00 0.417 1.136 0.585 0.148\n", + " + temperature 1.40 0.417 1.132 0.598 0.266\n" + ] + } + ], + "source": [ + "def nll(P, y):\n", + " return float(-np.log(np.clip(P[np.arange(len(y)), y], 1e-12, 1.0)).mean())\n", + "\n", + "\n", + "def brier(P, y):\n", + " Y = np.eye(P.shape[1])[y]\n", + " return float(((P - Y) ** 2).sum(1).mean())\n", + "\n", + "\n", + "def ece(P, y, bins=10):\n", + " conf = P.max(1)\n", + " ok = (P.argmax(1) == y).astype(float)\n", + " total = 0.0\n", + " for b in range(bins):\n", + " lo, hi = b / bins, (b + 1) / bins\n", + " m = ((conf > lo) & (conf <= hi)) if b > 0 else ((conf >= lo) & (conf <= hi))\n", + " if m.any():\n", + " total += m.mean() * abs(ok[m].mean() - conf[m].mean())\n", + " return float(total)\n", + "\n", + "\n", + "def with_temperature(S, T):\n", + " return np.stack([Question(CHOICE, \"\", CLASSES).decide(s, temperature=T) for s in S])\n", + "\n", + "\n", + "def best_temperature(S_cal, y_cal, grid=None):\n", + " \"\"\"Cherche T minimisant la NLL sur le jeu de calibration (jamais le test).\"\"\"\n", + " grid = np.linspace(0.05, 5.0, 100) if grid is None else grid\n", + " losses = [nll(with_temperature(S_cal, T), y_cal) for T in grid]\n", + " return float(grid[int(np.argmin(losses))])\n", + "\n", + "\n", + "# scores bruts du LLM sur calibration, et log-probs du temoin classique\n", + "S_cal_llm = np.stack([option_scores(x, CLASSES) for x in X_cal])\n", + "S_cal_clf = np.log(np.clip(P_clf_cal, 1e-12, 1.0))\n", + "\n", + "T_llm = best_temperature(S_cal_llm, y_cal_idx)\n", + "T_clf = best_temperature(S_cal_clf, y_cal_idx)\n", + "print(f\"Temperature apprise — LLM gele : T = {T_llm:.2f} | temoin classique : T = {T_clf:.2f}\")\n", + "\n", + "P_llm_test_T = with_temperature(S_test_llm, T_llm)\n", + "P_clf_test_T = with_temperature(np.log(np.clip(P_clf_test, 1e-12, 1.0)), T_clf)\n", + "\n", + "print(f\"{'famille':22s} {'T':>4s} {'exact.':>7s} {'NLL':>7s} {'Brier':>7s} {'ECE':>7s}\")\n", + "for nom, P in [\n", + " (\"temoin classique\", P_clf_test),\n", + " (\" + temperature\", P_clf_test_T),\n", + " (\"LLM gele zero-shot\", P_llm_test),\n", + " (\" + temperature\", P_llm_test_T),\n", + "]:\n", + " print(\n", + " f\"{nom:22s} {1.0 if P is P_clf_test or P is P_llm_test else (T_clf if P is P_clf_test_T else T_llm):4.2f} \"\n", + " f\"{(P.argmax(1) == y_test_idx).mean():7.3f} {nll(P, y_test_idx):7.3f} \"\n", + " f\"{brier(P, y_test_idx):7.3f} {ece(P, y_test_idx):7.3f}\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "088922c2", + "metadata": { + "papermill": { + "duration": 0.002454, + "end_time": "2026-09-28T13:51:29.232346", + "exception": false, + "start_time": "2026-09-28T13:51:29.229892", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Lecture du résultat — y compris quand elle contredit l'attente\n", + "\n", + "1. **La température ne change pas l'exactitude** : mesuré, 0,500 et 0,417 sont identiques\n", + " avant et après — la mise à l'échelle réordonne les scores de façon strictement monotone,\n", + " l'argmax (la décision) est intact. Ce qu'elle doit changer, c'est la confiance.\n", + "2. **Et ici, la calibration ne paie pas** : ECE **empire** pour les deux familles\n", + " (0,127 → 0,283 et 0,148 → 0,266), la NLL du témoin aussi (1,287 → 1,518) ; seule la NLL\n", + " du LLM bouge à peine (1,136 → 1,132). La raison n'est pas un mystère : le jeu de\n", + " calibration compte **12 tickets**. Le T appris (0,35 pour le témoin, 1,40 pour le LLM)\n", + " épouse le bruit de 12 points, pas un défaut systémique de confiance. La calibration est\n", + " une **estimation statistique** — elle exige un jeu dimensionné ; sous-dimensionnée, elle\n", + " peut dégrader la confiance au lieu de la réparer.\n", + "3. **Le sens des T appris reste lisible** : T = 1,40 > 1 pour le LLM zéro-shot (trop sûr de\n", + " lui, tendance confirmée), T = 0,35 < 1 pour le témoin (trop timide sur 24 exemples).\n", + " Le diagnostic est bon ; le remède, sous-estimé." + ] + }, + { + "cell_type": "markdown", + "id": "e1fe1a18", + "metadata": { + "papermill": { + "duration": 0.002254, + "end_time": "2026-09-28T13:51:29.236872", + "exception": false, + "start_time": "2026-09-28T13:51:29.234618", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 6. Décision sélective : agir, ou escalader\n", + "\n", + "Un modèle calibré permet la **décision sélective** : agir quand la confiance dépasse un seuil,\n", + "sinon **escalader** vers un humain (ou un LLM génératif — la voie de `03_Structured_Outputs`).\n", + "La courbe couverture-exactitude montre le compromis : quelle fraction des tickets la famille\n", + "peut-elle traiter seule, et à quelle exactitude." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "fb6c1b92", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:29.242735Z", + "iopub.status.busy": "2026-09-28T13:51:29.242429Z", + "iopub.status.idle": "2026-09-28T13:51:29.357918Z", + "shell.execute_reply": "2026-09-28T13:51:29.357918Z" + }, + "papermill": { + "duration": 0.120393, + "end_time": "2026-09-28T13:51:29.359538", + "exception": false, + "start_time": "2026-09-28T13:51:29.239145", + "status": "completed" + }, + "tags": [] + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAGZCAYAAAB1+tLoAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAbwdJREFUeJzt3Qd4FFX7NvAnBBISem/Se0dAelGqoiKCioh0QRAEQaQovaOCCKIICoiCIAjIK7wgUgQEpIUmvYbeIUACCcl8133e/+w3u9lNdpPte/+uayGZnZ05M2cm+8wz55wJ0jRNEyIiIiIiH5PG0wUgIiIiIkoJBrJERERE5JMYyBIRERGRT2IgS0REREQ+iYEsEREREfkkBrJERERE5JMYyBIRERGRT2IgS0REREQ+iYEsEREREfkkBrJEPmjUqFESFBQkN2/e9HRRiBKZP3++Oj7PnTvnlvUVKVJEOnfu7NRlJiQkSIUKFWT8+PGmaTzvHBcXFycFCxaUr7/+2tNFIT/FQJYoGadPn5Z3331XihUrJunTp5fMmTNL3bp15csvv5SYmBhPF49SadGiRTJt2jRPF8MnTZgwQVauXGnXvAhkEOD6ip9//lkuXLggffr0kUCqp5Tavn27CvTv3r1rNj1dunQyYMAAdUHw6NEjl5aBAhMDWaIkrF69WipWrCi//PKLvPzyyzJjxgyZOHGiFCpUSD766CPp16+fp4tIqcRA1vkBUocOHdRFXuHChX02kP3ss8/kzTfflCxZsoivc1cgO3r06ESBLHTp0kVlsXGuETlbWqcvkchPnD17Vn2R4ct448aNki9fPtN7vXv3llOnTqlA150ePnwoGTJkcOs6/ZWr92V0dLSEh4dLIAoODlYvXxURESEHDhyQKVOmeLoofiFr1qzSrFkzdSHTtWtXTxeH/AwzskQ2fPrpp/LgwQP5/vvvzYJYXYkSJcwysk+ePJGxY8dK8eLFJTQ0VLXb+/jjj+Xx48dmn0MbO9yCS66dn97O8K+//pL33ntPcufOLU899ZTZZ5DleOONN1Rzhxw5cqjyWLt999NPP0m1atUkLCxMsmfPrgJ03Da1x6VLl6Rbt26SP39+tV1FixaVXr16SWxsrGmeM2fOyOuvv66WjeCtVq1aiYJ8W+0mN2/erKbjf8Ct3IwZM6pA0FK7du0kb968Eh8fb5r23//+V+rXr6+C0kyZMsmLL74o//77r9nnsF+xTDQTadGihZqvffv28uyzz6pynj9/XpUBL9SDI+UFLAftKffu3SsNGjRQ+wB1D6j/kSNHquMF+w/tBQcNGpTouEiqrSYyxuXLl1dNW/LkyaOauty5c8c0D5afJk0a2bBhg9lne/ToISEhISooA9TZiBEj1LGATCP2Gfbdpk2brK4XzWdwRwLrzZUrlzz//POyZ88e9T72AS4GfvjhB9O+049fy32HfYo6wbGsz4t9Zmx3asna/tc0TcaNG6fOA+zj5557LlFd65AZ/OCDD9T+xn7H/p88ebLaruQge4n9hrq0xp7zbt68edKoUSN13mL95cqVk2+++SbRsrBvXnrpJdm2bZvUqFFD7Ws0Y1qwYIHVberfv7/6DJaJ/dCxY8ck2+wmVU/6+Y3gEscVlonjbO7cuYmWg7tReA/7PVu2bFK9enVThhV1iDtUgL8P+nqMdde0aVO1jbdv37ZZVqKUYEaWyIb//Oc/6gulTp06ds3/zjvvqC+L1157TT788EP5559/VDOEo0ePyooVK1JcDgSxCCIQgOALyQhfpvhSw3p27twp06dPVwGO8UsQbdOGDx+u5kUZb9y4ob6U8CWNzBOyJbZcvnxZfbniCxRBUZkyZdQX37Jly1SgiS/7a9euqX2E3/v27au+2LEfWrZsqeZ79dVXHdretm3bysyZM1WAieBYh+WjTvAlrGf7fvzxR+nUqZM0b95cBSmYB8FCvXr11LbpQal+oYH58N7nn3+uvpARFN+7d08uXrwoX3zxhZoPAW9K3Lp1S1544QV1kfD222+rwABBE/YDvsCx/8qWLSuHDh1S6zpx4oRdt3sRtCKow+1Z7F/cKfjqq6/U9v3999+qDeKwYcPUvsEFB5aPQH3dunUyZ84cdXFVuXJltayoqCj57rvv1AVB9+7d5f79++pCDftl165dUqVKFdN6sSysF9uE4wb7b+vWreo4QxCDfY/pOD6wbYCLOGsQiL///vtq337yySdqGvaPo3AOIJDFxQhe+/btU5k+40UV4Dho2LChOlax/9AUCLe+hw4dKleuXEm2KQnmxYUJ9q019px3OA4R+KH+06ZNq+oH5zKOCdzRMcLdHfzdwD7H8YxAEsc5LjiwDMBFNS468PcEgWfVqlVVALtq1Sp1/ObMmdNqWZOqJ5y7uOhE0IkLSPydwYUhyoFjBRcCgOMIxx7KqAftBw8eVH/j3nrrLWndurU6ntGuGMe2XhYsT4dtwYUI9i0CdyKn0YgokXv37mk4PV555RW75t+/f7+a/5133jGbPnDgQDV948aNpmn4feTIkYmWUbhwYa1Tp06m3+fNm6fmrVevnvbkyROzefF5vNeyZUuz6e+9956afuDAAfX7uXPntODgYG38+PFm8x06dEhLmzZtoumWOnbsqKVJk0bbvXt3ovcSEhLU/x988IFa59atW03v3b9/XytatKhWpEgRLT4+3mx7zp49a7acTZs2qen4X19ugQIFtDZt2pjN98svv6j5tmzZYlpH1qxZte7du5vNd/XqVS1Llixm07Ff8dkhQ4Yk2o4XX3xR7XtL9pYXGjZsqKbNmjXLbN4ff/xR7T/jvgHMh/n//vtvLSn4HOZbuHCh2fS1a9cmmo46DQkJUcfgnTt31D6sXr26FhcXZ5oHx9Hjx4/NloV58+TJo3Xt2tU0Dccrlt+3b1+b9Q4ZMmQwO2aT2nfly5dX+8mSfiwnt4zr16+r7UN9Gcvw8ccfq/mM5Rg7dqwq24kTJ8yWifrH+RAZGakl5amnnkp0/Dly3kF0dHSizzdv3lwrVqyY2TQce8bjWt/W0NBQ7cMPPzRNGzFihJpv+fLliZZr3B/W2Kqnbt26afny5dNu3rxpNv3NN99U55C+Dfg7iPpLymeffWb1fNFdvnxZvT958uQkl0PkKDYtILIC2QhAZssea9asUf+jd64RMrOQmra0yJzZam9omdlB1stYnuXLl6sMEDJIyN7oL2QiS5YsafWWsg6fQ8YQndyQgbOk3w7GupDtQaZTh8wbsj+4tXjkyBGHthfLRSYWy0UWSrdkyRIpUKCAaT3r169XmWJkF43bhn1Vs2ZNq9uGJhGugtuyyJoaLV26VGVhkck2lhG3nCGp/a9/Hk0AcFvW+Hlkt7CPjZ9HBhGdbZBxRYYV8yEzjmygDvsGWXS9fnGbF5lW1C+ym7pff/1V1QOaLFiy1gzAHf7880+VecUxbiyDnjW03G/IXuIWuHG/NWnSRDVL2bJlS7LZdXzWluTOO0AzHh2y/lg/ssRohoPfjdDsAOXVIZNZunRpNa+xTpBZt3aHIyV1gmtqLBPnN3427iccPyijfkzgrg2yvrt375aU0vcnhy4jZ2PTAiIr0PYNcOvVHmhjiTaKaIdnhIARXwJ4P6XQ5swWBKNGuGWIcuht006ePKm+pCzn09m6dQpogoCAHgFSUrBtCBwtIYDT309uGdaaF+D2L26b4tYlAloECbhNrH9pY9tADwpt1aEOAZ1lG2NnQpCtB4k6lBG3go23WI2uX7+u/kdAabw9jiAIASw+j4AC7SyT+rwO7RQXL16smgmgpzoCJEsIbtGJ6dixY2qMT2vHGdoSo0002jx7C/0csjyWsW8tg07sN9z6Tm6/J+V/N09Sdt4Bmn3gQmDHjh2J2nujTo2jIaDpgyVsk7EdNOqkTZs24iw4v3EhOHv2bPVKaj8NHjxYXUjgghV/49CcA+clhiG0l74/PXUhRP6LgSyRjSAIX+SHDx926HOp+SNt7MBkZMzsOLp+ZN0wDe3erGV1U9oeNCVs7Rtr2412e2iDiGHP8IWJ9oUYzgkBrk7vtIM2gLhgsGTMROoZUwQbriivrXpCGdFZaurUqVY/g45IgDaG6AilQztJtE/F5xHELly40OrnLQM1ZPD0AB9tZa11+kPby1atWqmgF8vGcYG2ngiUPMHR/WwP7DdksdGpzppSpUol+Xm08zYGkY5uA/Zl48aNVSYedY96xkUOLsbQhtSyw5mtOy5JBdOppZcB7blxvFlTqVIl00Xp8ePH5ffff5e1a9eqTC6GU0ObZdwFsIe+P2215SVKKQayRDagQwIyFcio1K5dO8l5MUQXvhgQROiZSL0zBbIexvE0kWmxHGsR2Th0QnEU1mfMpKHTCMqhd3JCpghfhpgnuS9va0ESAvrkgnlsG77kLCHjp78PetbMctttZavRHAK95pEVRrMCbBMCXJ3eYQXBGG4ZOzuQcrS81qCMGDEAQU1SFznIkBoDJ1xE6Z9HJgyZr+QuaFDvCFJRZ7jdjowsOucgSNah8x06MKLJibE8lk0IsF50FkOmOKmsrCMXbvbsZ2PHQ8v9rB9HOOaxDcbMomXQifIji5/S4wIBKDrVpfS8w4UXRqXAHQVjtjW5piRJwTY5emGd1L7H+Y2mU7hgsGc/YYQLXEjihb9XOK7QkRQd6DDSQnLHgr4/jX8fiZyBbWSJbEA2B3+80eMXAaklZF0QaAF6UINlb2g9E4choYxfSJZt9BAwpyQDhd79RhiNANDTHPBlg2wPsiaW2R38jraAtiB7icwdvpT1IZcsP69vO25lI+DXYXQFbBO+2PXb23rgadx2bLOt25r4wkQwgFvhyAIhsDVCOz4EbQjYjLfIjQGOPVDHlm0WU1Jea1Bm9JxHr29LyDDro1CgzSuCCf2l7zN8HuvEyAOW0LbVGGTjWEOPcJQP82MkCbQJNrZJ1DN/xmMBPc+NdQe4hY15rGXbjJ/FvrM2AL41tua1tp/14aKMsF/QFAbHuLEM1kYgwH7DNiEYt4QyYN8lBReuCBptDZGW3HlnbT/jGMOQXCmFOsFFkbURUJLL3Frb9ygjlonsqrUA2Xj+WP6dQHYZxyjWq597+pjMto4HDE2HYDe5pACRwxzuHkYUQH777Tctffr0WrZs2bR+/fppc+bM0WbOnKm1b99e9aDu0aNHop7xb7zxhppH/71Vq1ZWe6y3bt1a++abb7SePXuqHv45c+a0OmqBtRED9N7TFStW1F5++WW1vrfffltNe+utt8zmnThxoppep04d7dNPP1XrHDRokFayZEnV0zgpFy9e1PLmzauFh4er0Qm+/fZbbdSoUaoHM3q766MEoNc7ejkPHz5c++KLL7QqVapoQUFBiXpY16pVSy0L5f/yyy+12rVra9WqVUs0CoCuRIkSWqZMmdT7e/fuTfQ+eu1jVIAKFSpo48aNU+X75JNP1Pp79+5tVjfouW0N9gmW379/f23RokXaqlWrHC4veuNb69WNERtatGih9gV6gs+YMUObNm2aqvPs2bNbrVtL7777rlrfCy+8oPbtV199pY7F/Pnza0uXLlXzHDlyRB2nnTt3Nn0OPfZR9tdff900be7cuaZe99hX6MWPkR9QdsuRGzp06GBaL7Yd68Yxi23QYduwX6dMmaL9/PPP2s6dO22OWoCe/dgPGFEA827YsEFNj42N1QoVKqSOf/Ro//zzz7Vy5cqZ9rNxGUOHDlXTsF7sB/S6x36wPHcePnyoVa1aVY3MgVEccMxjufpxcOPGjST3+Z49e9R61q1bl6Lz7tixY+rvA+ZDOSdNmqQVL15cq1y5cqJtwn7HSAyWcEwZR3nAKB3YLxh1ASNy4O/IhAkT1DGKUVOSYquecO5i/ThOcEzhmMDfCxwz+Junw77EMjDKyXfffadGU8CoCtgHul27dpnqZsGCBWo9Dx48ML3/0ksvqRFYiJyNgSxRMhAQ4IsDQ0nhywmBVd26ddUX+qNHj0zzYZij0aNHq6A0Xbp0WsGCBdUXr3EePbgZPHiw+vLFFwiG5Dl16pTN4beSCmQRwLz22muqTPji6dOnjxYTE5No/l9//VV9ieDLDK8yZcqoQO/48ePJbv/58+fVMFy5cuVSX14YPgifNQ7jdPr0aVUOBEUIqGrUqKH9/vvviZaF+Zo0aaKWg+AXQyetX7/eZiCLoBTvIaC1BZ/DPkQgjXUjYEBAh2DEnkAWX7YIQlB2rMsY0NlbXluBrB6oIUDD+1gO6glBGo4VDPNmj9mzZ6vPhIWFqbpGgISLEQxphCG1nnnmGTVk1N27d80+hwAUZV2yZIlpmCYEP9hGlOXpp59W9YT9YxnIYrm40MGxguMe9Y+g1nhBgYCtQYMGqlzGIbCsBbIImhCw6RcmxiANy6xZs6ZaD4LaqVOnWl0Gzh3sNwwZhXU+++yz2uHDhxOdO3rgh/MPxw6Wi/MNF3MIaFEnyalUqZIKlFN63uGCCMvAMYm/HTgG9AuJlASycOvWLbUuDK2GbUKdY7sth8+yZKue4Nq1a+p8xt8r/N3ChWvjxo3VMadDgIvP58iRQx03OMc++uijRMcvLlJQNlxcGrcTxyXKiyCYyNmC8I/jeVwiIiL/hU6EGGYrMjIyyYeGUPLQ/ANPSkRzLEc6rxLZg21kiYiILOARxuioZdkelhyDNrRov42nzzGIJVdgRpaIiIiIfBIzskRERETkkxjIEhEREZFPYiBLRERERD4p4J7shaevXL58WT3RhM98JiIiIvIu6L51//599ZTD5B4tHnCBLIJY/fnmREREROSdLly4IE899VSS8wRcIItMrL5z8HhL8nyGHI9CxHO/k7vqIt/Gug4MrOfAwboOHAluruuoqCiVdNRjtqQEXCCrNydAEMtA1jtOjkePHqm64B9C/8a6Dgys58DBug4cCR6qa3uagPLIIyIiIiKfxECWiIiIiHwSA1kiIiIi8kkB10aWiIi8X3x8vMTFxXm6GJRMu0nUEdpOso2sf0twcl2nS5dOgoODnVI2BrJERORV40devXpV7t696+mikB11hQAH431yXHb/prmgrrNmzSp58+ZN9fIYyBIRkdfQg9jcuXNLeHg4AyQvD26ePHkiadOmZT35Oc2JdY1lRUdHy/Xr19Xv+fLlS9XyGMgSEZHXNCfQg9gcOXJ4ujiUDAaygUNzcl2HhYWp/xHM4nxPTTMDNmohIiKvoLeJRSaWiPxb+P+d56ltC89AloiIvAqze0T+L8hJ57lHA9ktW7bIyy+/LPnz51cbtHLlymQ/s3nzZqlataqEhoZKiRIlZP78+W4pKxERERF5F4+2kX348KFUrlxZunbtKq1bt052/rNnz8qLL74oPXv2lIULF8qGDRvknXfeUQ2FmzdvLt7YpkSLifF0MbwaekFiHyVER4tw+JZUCQoLYyaLiIgCikcD2RdeeEG97DVr1iwpWrSoTJkyRf1etmxZ2bZtm3zxxRc2A9nHjx+rly4qKsoUQOHlSgjOTlZ/xqXr8BccaCf1wqo+LQV//NFrg1l10fJ/Q7iQ/0pNPeuf1V+UMs8995xKEk2bNs0lyz937pwUK1ZM9u3bJ+XLl1fTXFVfuOvav39/uXPnjngDJNDef/99OXTokNPGQXW3UaNGyW+//SYRERF2xV1r1qyRVatWmerYWXWtn+fW4jFH/n741KgFO3bskCZNmphNQwD7wQcf2PzMxIkTZfTo0Ymm37hxQw3s60rMxpI7xeyLkOuRkSoz643wh+nevXvqDxcHT/dfqalndPrA59E7Gi9fge8lBI56ksXTlixZogacd9U+1JeL/zHSBLjqArpNmzbSrFkzrzkeBg0aJEOGDFHH97PPPquaSNrSoEED+fPPP8XbfPDBB9KrVy+79mnHjh1l3LhxsmnTJqldu7ZT6xrrx/l+69YtdbwaYbxavwxkMb5gnjx5zKbhd2RZY2JiTMM5GA0dOlQGDBhg+h3zFixYUHLlyiWZM2d2aXlxoOfas9ul6/B1OIhv3rwpOXPmZHCTQgkxMXK6fgP1M47rNF7a4xt1jT+Aqoysa7+VmnpGcgFfYBjiBy9fge3Fy1vKjOGMXEnfTvyPrKRlEOJMmTJlUi9vgDvAZ86ckTfeeENt+/LlyyU2Nla9d+HCBalZs6asX7/elKUOCQnxmmNCj0lw4YEHEdgL5W/Xrp188803Uq9ePafWNZaNvxEYai99+vRm71n+niTNS6AoK1asSHKekiVLahMmTDCbtnr1avXZ6Ohou9Zz7949NT/+J8+Lj4/Xrly5ov6nlIl/+FA7UrqMeuFnb8W6DgypqeeYmBjtyJEj6n9ISEjQHj6O88gL67ZHp06d1HeK8XX27Fn13qFDh7Tnn39ey5Ahg5Y7d27t7bff1m7cuGH6bMOGDbU+ffpo/fr107JmzarmmT17tvbgwQOtc+fOWsaMGbXixYtra9asMVvn5s2btWeeeUYLCQnR8ubNqw0ePFiLi4szWy6WqStcuLA2fvx4rUuXLmqZBQsW1L799tsktwv1N3nyZLV+rAefGTdunHoP24ft3LdvnxYbG6vW3bVrV61IkSJa+vTptVKlSmnTpk0zW96mTZtUmcPDw7UsWbJoderU0c6dO6fe279/v/bss8+qsmXKlEmrWrWqtnv3bvXevHnz1PxGEydOVPsK82O92P7KlSvb3H545ZVXVF3pHj16pH344Yda/vz5VZlq1KihypiU3r17a6+99prV9/R9EhERoSVn165dWpMmTbQcOXJomTNn1ho0aKDt3bvXbB4s6+uvv1bHD/Zp0aJFtaVLlyZa388//6zVrl1bCw0N1cqXL6+ODR22B/Pg+ME+TZcunZo2cuRIs/2VVN3AX3/9pY4BxE32nhcpOd9TGqt5z6WCHfAos2vXrplNw+/IrFrLxhIRke+KiYuXciPWeWTdR8Y0l/CQ5L8iv/zySzlx4oRUqFBBxowZo6YhG40HOzRq1Eh1SEY/Dtw1HDx4sMrmbdy40fT5H374Qd2u3rVrl2oSgFu+K1askFdffVU+/vhj9dkOHTpIZGSkGnfz0qVL0qJFC+ncubMsWLBAjh07Jt27d1cZLLR9tAXNHsaOHauWuWzZMrWehg0bSunSpa3Oj7uZc+bMUetHJu7KlStqXbay8E899ZQsXbpUZde2b98uPXr0UB2xsb24hdyqVStVzp9//lllMbG9+i3q9u3by9NPP62yfsjw7t+/32bm75dfflHbOXPmTFWuH3/8UaZPn67a7DqiT58+cuTIEVm8eLEaOQn7/Pnnn1dtX0uWLGn1M1u3bpW33npLUgt3HTp16iQzZsxQWVLUDer05MmTZtnn4cOHy6RJk9Qxhu188803VfnQP0j30UcfqbbQ5cqVk6lTp6qRoNAx3vhAETSF+Pzzz9U+ypYtmxr9SZdc3UD16tXVfJjeuHFj8TY+FciifQYaHRshja+32yAiInKnLFmyqFvICDKRbNF99dVXKjibMGGCadrcuXNV0zYEvqVKlVLT0LZ22LBhpuARgQuaWiGwgBEjRqgA7+DBg1KrVi35+uuv1TKwfAQbZcqUkcuXL6sgGfPaas6BQOm9995TP2NeBKho92gtkEWgheAJ60DABcWLF1eBozUIOo19UdApG31aEHQikEWTPrSbfumll9RywBiMIUhHQIZtAVuBJCBo69atm3oB2m+iHaojfV6wvnnz5qn/EcTCwIEDZe3atWq6sc6Mzp8/b5o/NXCBYzR79mx1u/+vv/5S+0j3+uuvqwshwEUI4h0EvzgGjAF5mzZt1M84TrAN33//vbo40uECq2nTplbLklzdAI5tHOfYfm/k0UD2wYMHcurUKdPvuIrAlVj27NmlUKFC6qTG1SeuOgHDbuHEQgVhyC5c1eJEWb16tQe3goiIXCEsXbDKjHpq3alx4MABFShmzJgx0XunT582BbKVKlUyTUc2Epm0ihUrmqbp/UL059IfPXpUJW+MGbO6deuq79OLFy+q705rjOvBZxF068u0hHVgtB9Hsm/IkCJQR3CI7DMye1WqVFHv4TsdGWR0zkZAhc5xCHCRsQX0Y0HAhqwj3kMApwdV1sqGWMAI+wP72l7IaqKtqF4HOmxzUo9GxnY51HbTBtxJxsULMqOoA5QlOjpa7TsjyyQdfkeMZGuetGnTquwp9pERptmSXN3ocNcb2++NPNrjYs+ePeqKFS/9YMbPuKoE3MowViyu8hC04qpE7yH63XffeeUYskRElDoIuHB73xOv1PbMRmCJ27wIPIwv3D5Gb3ad5S10rNc4TS9Haoets7YeW8t0tKkebs8jo4ks6R9//KG2s0uXLqaOUIBMJ7K0derUUU0oEETu3LlTvYemAv/++68aJx4JKtwmx63+lEJW2nKIKONjUFE3uGjYu3evWd0gAEQm2hZkyp0xDBiy3Fgf1oVmGPgZAbRxfzlThgwZknw/qbrR3b59W22/N/JoIIuhK4xjBuov/Wld+N/YlkP/DMY+w5UTrmpxJUFEROQpaFqgD0OlwxMoEZwVKVJEPYXS+EousEgKbvsi6DAGan///bdqW4l2qs6AW/sIZjFmqj2wfgRBaLqAZBS2Ed/PlvAe7rQieEOb4kWLFpneQ/CE8WIRCOMBSQiubG3/P//8YzbNMuhCG2UkwnSom8OHD5uVA9OQDbWsG2PzEGvlR7va1ML+6tu3r2rugREO8KRSjN5jyXK78Lvlbf+dhnnQjhXBueU89kiqblCXaLqhZ9i9DcfAISIiSgUEqwiu8KAABCTIdPbu3VtlsTB00e7du1UwsG7dOpWptAx6HYFgEUM9YVB+dL7CwPYjR45UdzSdNawdbp+jHS2a8aFpH8qOgAltL20FvrjDiu1D+190UsI2G5sNIkhCAI52lghWkZlGwIXb1WjniaQV3kOQh8/aCsb69eunmjAg0MW6sO24YLBsg4q7t3hhH6FjGzrfGYNmdDDDGKkYQgvlQ0cmjDufVFNF3P3FEFyphf2FZhTIAOO4QVmsZcHReQ7bqm8nyoh9ZdmkY8WKFWo7ccwhY4yml/ZKqm6MndzQUcxWcw9PYyBLRESUCritjlvVuCWObKDeiQhBGYJWDOiPdq8YiB6delITcBYoUEB1ekZQgyZ2aC+KW/p6hzFnQTD64YcfqqZ+CGratm1rs03tu+++q7KomAdjqWKAe71jmd5ZCIEWOiUhiMSIBgi68DnsN8yPoBLvoX0mnvhp7UFGgHWgbAiyq1WrpoIvBKpGCORw+x7LxMgMCMLwtDMjBMJ4H9uIDm/ouY8A2lYbY0DAiaD5+PHjkhq4IEDAiaw9RqRAdtba2L/YB2i2gfbNuKDAqAI4xowmTZqkXjgWEGTjCVyONAFIqm50WK/e6cwbBWEMLgkg6KGH3nfopefqByJQ8pC5wB9HnMQcJD/lj0I+XrWa+rn0vr1e/UAE1rX/S0094/YlMkToD+GMTjXkWggfcDsbnYw8+WhstLFduXJloo5QroDRFRBHfPvtty5dD/YnMq0IsK1B9h/nSUREhEtv+SNwR4YbwTuaxDizrpM63x2J1fhtQkRERGSHTz75RAoXLpzqzne+4sqVKyobjKDSW/nUOLJEREREnoKmIXioRKBo0qSJ+t+bb94zkCUiIiKfhaYFST3VzBclFziig6E3B5fuxKYFREREROSTGMgSERERkU9iIEtEREREPomBLBERERH5JAayREREROSTGMgSERERkU9iIEtERBTgMHyVK58Q5U06d+5s84lZrtKgQQNZtGiR+LKQkBD1BLXkxMbGquHB9uzZ45ZyMZAlIiJyYWCEL/Vp06bZfNQoHvkZHBwsly5dSvRUJf2RoJiPvMf8+fPVwxHssWrVKrl27Zq8+eabsnnzZlWfSb0wjzeKjIyUF154wa6Ad+DAgTJ48GC3lIuBLBERkYcVKFBAPQrU6IcfflDTAxUye/5g+vTp0qVLF0mTJo3UqVNHXaDorzfeeEOef/55s2mYxxvrIW/evBIaGmrXZ9q3by/btm2Tf//918WlYyBLREReCk8uSoiO9sjL3U9N6tSpk8ybN89sGn7H9OQg+HnxxRclLCxMihYtqm5hW2aB7969K++8847kypVLMmfOLI0aNZIDBw4kudzvvvtOypYtK+nTp5cyZcrI119/bXNePbNs+Xr22WdN8yCwqV+/vipnwYIFpW/fvvLw4UPT+yjz2LFjpWPHjqqMPXr0UNN//fVXKV++vAqiMM+UKVOS3SfLli2TihUrqnXlyJFDPWrVuC74/PPPJV++fOr93r17S1xcnOm9O3fuqHJky5ZNwsPDVSby5MmT6j1kTBGY3rt3z7Sdtp4sduPGDdm4caO8/PLLpmwlAkL9hfJhu4zTMI81yHCWKlVKladYsWIyfPhwszLrzUO+/fZbtX8xHwJllNPy7sHo0aNNx0LPnj3NLhpQZ3369JEPPvhAcubMKc2bN0/UtADzYx7sPxwfhQsXlokTJ5qWgf1Wt25dWbx4sbgaH1FLREReSYuJkeNVq3lk3aX37ZWg8HC3ra9ly5Yya9YsFezVq1dP/Y9gCgEQgrukIOC6efOmCrDSpUsnAwYMkOvXr5vN8/rrr6ug6b///a9kyZJFBTuNGzeWEydOSPbs2RMtc+HChTJixAj56quv5Omnn5aIiAjp3r27ZMiQwWpwjcAJAbXu6tWrKnhE21A4ffq0yjyOGzdO5s6dqwI8BEJ4GQN4BJdY78iRI9Xve/fuVcEYgrS2bdvK9u3b5b333lPBJ4Iya1COdu3ayaeffiqvvvqq3L9/X7Zu3Wp2cbJp0yYVhOH/U6dOqWUjCMQ2ApaNwBXNAhDsIYhs0aKFHDlyRGVMcZGAch4/flzNnzFjRqtlQT0ioMQFQWplypRJNWnInz+/HDp0SJU1U6ZMMmjQINM82JZffvlF/vOf/0hUVJR069ZN7S/Up27Dhg0q+MTxggsQBOXYn+PHjze7G9CrVy/5+++/bWaZsW+wrkKFCsmFCxfUy6hGjRpqv7saA1kiIiIPQwD69ttvqyAPgSz+x++YnpRjx47Jn3/+Kbt375bq1aubMqklS5Y0C6Z27dqlglv91jACRmTXkLnUM59GCCSR+WzdurX6HZleBHEIgK0Fsmjji2wiPHr0SGX9ateubcpUIluH283I8gHKh2CoYcOG8s0336jACpAp/vDDD03LxWcQcCP7CMhIohyfffZZkoHskydPVNmRKQRkZ42QMUSQjnIj24yMNgI8BId6AIsgTr/Nj0AQwTr2GS4KcDGATKy+zbacP39e8uTJo5oVpNawYcNMPyMzjXaoixcvNgtkse/RREVvkjJjxgy1bahLvazIrOL4QoCNTPeYMWPko48+UhdMejlRP7gQ0FneoUB7WcyDYxX7Qd/PRgi4sf2uxkCWiIi8UlBYmMqMemrd7ta1a1cVOE2YMEGWLl0qO3bsUAFZUpARRIewqlWrmqaVKFFCBWo6NCF48OCByroZxcTEqEypJdyCx3Rk8/QMJaAsCODs2Q5kQdevX28KjFCGgwcPmmUGVdORhAQ5e/asKWOpB+O6o0ePyiuvvGI2DbeskRGNj49XGVpjByQE2uhUheAXwStuizdr1kxee+01s32CAA5BrA7ZWWQ59XVin9asWdP0PvZd6dKl1XuOwD7Wg/TUWrJkiQr+UTeoT9RH5syZzeZBdtTYrhoXE9jHOE70QLZy5coqiDXOg+Uho6oHpNWqJX0nBBcRTZs2VfsEmfaXXnpJ7Wcj3AGIjo4WV2MgS0REXkm1P3Tj7X1PQ+CF7CBuiyOwq1Chguzfvz/Vy0WQgkDNWm94az3vMT/MmTPHLJgDY/BnDZoOrFu3TmWAcdvbuMx3331XtYu1hOBLh6YLjkDga9xHyH6ijAiiEeT+8ccfKiv5ySefyD///KMyy2CZ6caxhoDP2dDGFE1EUgsXNchOo20rgnNcUCAbO8WO9sIpkVw94MIJFyBoqoI7Amj+gaYkyPDrbt++rdrhuhoDWSIiIi+BbCbaNOJ2uz2QEUNmDm1Y9Swa2kkagycEHWiziiwjbkknB8EgbgufOXNGBU/2Qqcs3KZGcFO8eHGz91AGNAlAttgRCOgt22nidzQxQMCKrJ+1ZSIwReYWL7RlRaZxxYoVqv2wPevEPkXgqzctuHXrlspqlitXznR7Hhnh5KB9MfY96sOYEXYUgnJsAwJynbXb9pGRkXL58mVVf7Bz506VFcdxokN2HJli7Dt9HrTxRdMJRyAbjLbFeCHjjcwsgle9zfXhw4fV9nttIIsea4jGcbDi5CAiIgpU6BlumT3F7Wg9OMAYsZbvW2tXiFv5aINp7xilyOAiE4Z2rgh+kWlEG1MEKQjmAO/j9jHaraLdI4JABDurV69WnaEsb+cDMn/IniLzhwDl8ePHaoB7BGTWgkEELeh0hk5RuG2P4E0P+BDYYHqtWrVU5y6MnoCMHwJbZE7RVtUWbMszzzyj2m8iYEJmEvMnNYICAlC0d8Wt7ty5c6vf0bnM3g5XaPuJ5gyoCzRVQGZ5yJAh6pa93swBFwTIMmM9+q164+16HQI5ZGURfOP2e0qhTAhSkYXF/kDdITC3hGYMaMOMNtDo7IU6RLbU2JYX8RuajaDNLTp7oT006sWRdrxTp05VWX5sHz6HpjBYh/G4RUev5DoqOoPDrY/R3gE7QG8kjB0L77//vkyaNMkVZSQiIvJquG2PL3XjC8GgDoGF5fsIRiwhMYTAx5EEETr3IIuKEQIQmOq92fW2mQho16xZo95HD3UEsmhHqndEsgbBJjqNYUQBNHlApyz0mNdvzVtCkIv4AE0LEODoL72zWKVKleSvv/5SoyRgCC5sPzKleubQFmRy0TMeARyaWuAzyPra6uilZwq3bNmiRhnAtiJgwy14ewbz12G7keFG8ImLALTnxT7UmyQgU4thqxBc4/a5sWOUEbLG2OfGtsEpHdWif//+KuDE6ArI0Ood4IyQncY+x7YjkMd+twz60X4YgTGOB5Qfy7Y1fJgtOL6wzbgIQmCNgBj7Rw+GccGBiztkal0tSHNwsLx+/fqpKws0tMZVGhpvYzyz3377Te0I3N7wZrhCwRUmdrBlI2lyP7RJQk9aXDU7o1dnIMKYl/oQRegYk8ZL2xSyrgNDauoZPa5xpw/BkrM6yASiixcvqkww2i4iaHEVhA+4Ba8/fYysQ3Yaib99+/ZZzcI7y6hRo9SoCkm1q8YFAMYUtudRs6mpawTIyFR//PHHKTrfHYnVHG4TgI1HzzncIjBuDCrJWu9HIiIich0MuI/b3MicYugpDMeEW9/6GK7kWbjl/v3336s72K4MZL0Fmi7gWEQG2R0cDmTRzgRX2taG6+AVGRERkXvh6U7IfKFzFm754rY3bmUnNwYtuQ/aJweKkJAQszFvvS6QRXsItOtBm1jQg1e0pUE7EiIiInIfDMekP0aUAteoUaOSbeuKds7+xuFAFgM1o8E0ehuivcSXX36pfkbDYzTkJiIiIiJyB4d7XOBxZGhIjCAWbSAw2DCaGqCHWnJPgiAiIkqOKwamJyL/PM9TNAAsxo7FEz+IiIic2bYOIx1gjFMMaYTf2ffCe3HUgsChObGusSx0CEOfK5zvOM/dGshiTDT0irTs8IWnXmCaPU+6ICIisoQvNQzFg+8YBLPk3RCQIKuGemMg6980F9Q1nkeAxxOndjhGhwNZW8PO4qkfqY2qiYgosOF7BF9uyP4wMeLdENggiYUnmHFsaP+W4OS6RlLUWZl8uwPZ6dOnq/+xUoxQgOfy6vDHBk/RwKPyiIiIUgPfMxg6isNHeX9wgzrCYPYMZP1bghfXtd2B7BdffGHKyM6aNUtF08YraAy+jOlERERERF4VyOIxYvDcc8/JihUrJGvWrK4sFxERERFRktI4+vQQPGINDfGJiIiIiHwmkEX7iEePHrmuNEREREREdnK4xW7v3r1l8uTJqkcpEREREZGnODz81u7du2XDhg3qiV54sleGDBnM3l++fLkzy0dERERE5JxAFp282rRp4+jHiIiIiIg8G8jOmzfPuSUgIiIiIkqBFI1qi/axf/75p3z77bdy//59NQ2PE3zw4EFKFkdERERE5PqM7Pnz5+X5559Xw3DhsbRNmzaVTJkyqQ5g+J0PRSAiIiIir8zI9uvXT6pXry537tyRsLAw0/RXX31VdQIjIiIiIvLKjOzWrVtl+/bt6rG0RnhE7aVLl5xZNiIiIiIi52VkExISJD4+PtH0ixcvqiYGREREREReGcg2a9ZMpk2bZvo9KChIdfIaOXKktGjRwtnlIyIiIiJyTtOCKVOmSPPmzaVcuXLqcbVvvfWWnDx5UnLmzCk///yzo4sjIiIiInJPRvapp56SAwcOyMcffyz9+/eXp59+WiZNmiQRERGSO3duhwswc+ZM1b42ffr0UrNmTdm1a1eS8yMbXLp0adXRrGDBgqoMCKiJiIiIKLA4nJFF0Iig8+233071ypcsWSIDBgxQQ3YhiEWQimzv8ePHrQbFixYtkiFDhsjcuXOlTp06cuLECencubNq3jB16tRUl4eIPE/TNNFiYsRfBIWFqb9RRETkBYEsAkwMtYVAtnHjxpImTYqeqaAg+Ozevbt06dJF/Y6AdvXq1SpQRcBqCaMl1K1bVzVnAGRy27VrJ//884/NdWBsW7x0UVFRpk5reJFnoQ4QuLAuUs6479TPXrov7alrvB/59tvyKGK/+Iuwqk9LwR9/DJhglud04GBdB44EN9e1I+txOJD94YcfVGb0lVdekSxZskjbtm1VUIuxZR0RGxsre/fulaFDh5qmIShu0qSJ7Nixw+pnkIX96aefVPODGjVqyJkzZ2TNmjXSoUMHm+uZOHGijB49OtH0GzdusEmCF8DBeu/ePXWCpOaiKJAZs5c4rpEB9NW6xrb4UxALMfsi5HpkpNfWi7PxnA4crOvAkeDmutafGuuSQBbZWLywkmXLlqkOXrVq1ZJixYqpgHbEiBF2LefmzZtqGK88efKYTcfvx44ds/oZZGLxuXr16qmdiUfl9uzZU7XXtQWBMpovGDOyaFubK1cuyZw5s93bTa47OZCpQn3wD2HKJERHy93/+1ntx/Bw8dW6Nm5L8a1bJI0PB38JMTFyun4Dr68XZ+M5HThY14Ejwc11jSasLgtkdRgzFk0C8Dpy5Ii0b99eZT7tDWRTYvPmzTJhwgT5+uuvVZvaU6dOqSeNjR07VoYPH271M6GhoeplCRXBE8874ORgfaSCYb95+35Mtq4N09NmyODTwV+CD9WLs/GcDhys68AR5Ma6dmQdKQ5kcVt+1apVqpnB2rVrVSb1o48+svvzGK4rODhYrl27ZjYdv+fNm9fqZxCsohnBO++8o36vWLGiPHz4UHr06CGffPIJTyQiIiKiAOJw5Ldu3Trp1KmTClx79eql/v/jjz/k/Pnzahgue+ERt9WqVZMNGzaYpa7xe+3ata1+Jjo6OlGwimAY0NSAiIiIiAJHitrIvvTSS7JgwQL1JK906dKleOVou4qgGB3F0HkLw28hw6qPYtCxY0cpUKCA6rAFL7/8shrpAGPX6k0LkKXFdD2gJSIiIqLA4HAgi1v/aB/rDBjxAL2s0a726tWrUqVKFVMzBYiMjDTLwA4bNky10cD/ly5dUo2OEcSOHz/eKeUhIiIiIj8OZO/cuaNethQqVMih5fXp00e9bHXuMkqbNq2MHDlSvYiIiIgosDkcyOIhBEkN7I0htYiIiIiIvC6QjYiIMPs9Li5OTUPbVd7iJyIiIiKvDWQrV66caBo6a+XPn18+++wzad26tbPKRkRERERkk9MGXi1durTs3r3bWYsjIiIiInJuRhaPeDXC+K1XrlyRUaNGScmSJR1dHBERERGRewLZrFmzJurshWC2YMGCsnjx4pSVgoicIiEmRrwVHniixcRIQnS02aNofaX8RETkB4Hspk2bzH7HOK8Yz7VEiRJqeCwi8pyTdeuJt7vr6QIQEZHfcDjybNiwoWtKQkQpEhQWJmFVq0rMvn3iL7A92C4iIqKkpCiFevr0afU42aNHj6rfy5UrJ/369ZPixYunZHFElApo6lN44U/qtr03Q9MCPMkPd3CMT+yzBkFsUuNVExERpSiQXbdunbRs2VI9TrZu3bpq2t9//y3ly5eX//znP9K0aVPuWSI3Q9AXFB4uXi0hQQWoacLDkw1kiYiIXBLIDhkyRPr37y+TJk1KNH3w4MEMZImIiIjILRxOi6A5Qbdu3RJN79q1qxw5csRZ5SIiIiIicm4gi/Zt+/fvTzQd03Lnzu3o4oiIiIiI3NO0oHv37tKjRw85c+aM1KlTx9RGdvLkyTJgwICUlYKIiIiIyNWB7PDhwyVTpkwyZcoUGTp0qJqWP39+9WSvvn37Oro4IiIiIiL3BLLoHY3OXnjdv39fTUNgS0RERETk1YHs2bNn5cmTJ1KyZEmzAPbkyZOSLl06KVKkiLPLSERERESU+s5enTt3lu3btyea/s8//6j3iIiIiIi8MpCNiIgwPQjBqFatWlZHMyAiIiIi8opAFm1k9baxRvfu3ZP4+HhnlYuIiIiIyLmBbIMGDWTixIlmQSt+xrR69eo5ujgiIiIiIvd09sJ4sQhmS5cuLfXr11fTtm7dKlFRUbJx48aUlYKIiIiIyNUZ2XLlysnBgwfljTfekOvXr6tmBh07dpRjx45JhQoVHF0cEREREZF7MrL6AxAmTJiQsjUSEREREXkiI0tERERE5A0YyBIRERGRT2IgS0REREQ+iYEsEREREfkkBrJEREREFBijFly7dk0GDhwoGzZsUMNvaZpm9j6f7kW+AsduTJz/HK9h6YLVk/fI+yTExIg/CAoL4zFGRL4dyHbu3FkiIyNl+PDhki9fPv5RI58NYl+btUP2nr8j/qJ64WyytGdtnpNe6GRd/3jqYVjVqlJ44U88xojIdwPZbdu2qSd5ValSxTUlInIDZGL9KYiFPefvqO0KD0nR8NDkguwlAr+YffvEX2BbtJgYCQoP93RRiIgUh7/xChYsmKg5AZEv2zOsiYSHBIuvio6Nl+rj/vR0McgCspbIXiLw84emEf6SVSaiAA9kp02bJkOGDJFvv/1WihQp4ppSEbkRglhmMclVwSyzl0REruPwt3fbtm0lOjpaihcvLuHh4ZIuXTqz92/fvu3M8hEREREROS8jS0RERETkc4Fsp06dXFMSIiIiIiIHpKhhIMaKXblypRw9elT9Xr58eWnZsqUEB/tuhxkiIiIi8vNA9tSpU9KiRQu5dOmSlC5dWk2bOHGiGs1g9erVqu0sEREREZHXPaK2b9++Kli9cOGC7Nu3T73wgISiRYuq94iIiIiIvDIj+9dff8nOnTsle/bspmk5cuSQSZMmSd26dZ1dPiIiIiIi52RkQ0ND5f79+4mmP3jwQEJCQhxdHBERERGRewLZl156SXr06CH//POPesIXXsjQ9uzZU3X4IiIiIiLyykB2+vTpqo1s7dq1JX369OqFJgUlSpSQL7/80jWlJCIiIiJKbRvZrFmzym+//SYnT56UY8eOqWlly5ZVgSwRERERkbuk+AHzJUuWVC8iIiIiIq8NZAcMGCBjx46VDBkyqJ+TMnXqVGeVjYiIiIgodYFsRESExMXFmX4mIiIiIvKJQHbTpk1Wf3aGmTNnymeffSZXr16VypUry4wZM6RGjRo2579796588sknsnz5crl9+7YULlxYpk2bpp42RkRERESBw+FRC7p27Wp1HNmHDx+q9xyxZMkS1VRh5MiR6glhCGSbN28u169ftzp/bGysNG3aVM6dOyfLli2T48ePy5w5c6RAgQKObgYRERERBVpnrx9++EE9xStTpkxm02NiYmTBggUyd+5cu5eF9rTdu3eXLl26qN9nzZolq1evVssYMmRIovkxHVnY7du3S7p06dS0IkWKJLmOx48fq5cuKipK/Z+QkKBe5FmoA4xF7O66MK7P148FX9kWT9U1Of8YkyTqkPUcOFjXgSPBzXXtyHrsDmQRAOoPQEBGFuPH6uLj42XNmjWSO3duu1eM7OrevXtl6NChpmlp0qSRJk2ayI4dO6x+ZtWqVWr82t69e6shwHLlyiVvvfWWDB48WIKDg61+ZuLEiTJ69OhE02/cuCGPHj2yu7zkuoP13r176rhC/btLTFy82bEQls768eMLfGVbPFXXlHpaTIzZMRYUFmZzXtZz4GBdB44EN9e1tTv/qQ5kMX5sUFCQepUqVSrR+5huLWC05ebNmyoAzpMnj9l0/K6PT2vpzJkzsnHjRmnfvr0KnE+dOiXvvfee6oiG5gnWIFA2jrSAgLxgwYIqCM6cObPd5SXXnRw4dlAf7vxDGB37xPQz1h0ekuKR6DzOV7bFU3VNqZcQHS13/+9nVX/h4bbnZT0HDNZ14Ehwc10bk6XJsfsbD528EIk3atRIfv31V8mePbvpvZCQENXpKn/+/OJK2JHI+s6ePVtlYKtVqyaXLl1SncVsBbKhoaHqZQkVwRPPO+DkcHd9GNfl68eCL22LJ+qanMDBY4z1HDhY14EjyI117cg67A5kGzZsqP4/e/asFCpUSG1QauTMmVMFo9euXTObjt/z5s1r9TP58uVTbWONzQjwVDGMeICmCgioiYiIiCgwOBxW49Y+RgywtHTpUtURzF4IOpFR3bBhg1nGFb+jHaw1devWVc0JjI2AT5w4oQJcBrFEREREgcXhQBadp5BNtYRb/hMmTHBoWWi7iuGzEAAfPXpUevXqpYbx0kcx6Nixo1lnMLyPUQv69eunAliMcIB1ovMXEREREQUWh3uFREZGStGiRRNNRxtZvOeItm3bqh6wI0aMUM0DqlSpImvXrjV1AMPyjO0k0Elr3bp10r9/f6lUqZIaPxZBLUYtICIi10swjGBgc5iemBjVQczYttbbYOSF1DaRIyIfDGSReT148GCi8VsPHDggOXLkcLgAffr0US9rNm/enGgamh3s3LnT4fUQEVHqnaxbz6759FEOvFVY1apSeOFPDGaJfJzDl8vt2rWTvn37qlEMMHwWXmg3i8zom2++6ZpSEhGRR7OXCPz8Scy+fWbj4xJRgGRkx44dqx4R27hxY0mbNq3pVhLaszraRpaIiLwfspbIXtoT+OH7AE3GvHVsUTSNsDerTER+GMhidIAlS5aogBbNCcLCwqRixYqqjSwREfkn9UCcJB6EYIKB08PC1EMTvDGQJSL/kuJHAOHpXtae8EVERERE5LWB7MWLF2XVqlVqVAE8iMBo6tSpziobEREREZHzAlk8sKBly5ZSrFgxOXbsmFSoUEG1mcXja6v6WWcAIiIiIvJeDjdgwgMKBg4cKIcOHZL06dPLr7/+KhcuXFCPsH399dddU0oiIiIiotQGsngCF0YoAIxaEBMTIxkzZpQxY8bI5MmTHV0cEREREZF7AtkMGTKY2sXmy5dPTp8+bXrv5s2bKSsFEREREZGr28jWqlVLtm3bJmXLlpUWLVrIhx9+qJoZLF++XL1HREREROSVgSxGJXjw4IH6efTo0epnjCtbsmRJjlhARERERN4ZyOJxtBh6q1KlSqZmBrNmzXJV2YiIiIiInNNGNjg4WJo1ayZ37txx5GNERERERJ7v7IVxY8+cOeP8khARERERuTKQHTdunBpH9vfff5crV65IVFSU2YuIiIiIyCs7e2GkAsDTvYKCgkzT8WQv/I52tEREREREXhfIbtq0yTUlISIiIiJydiDbunVrmT9/vmTOnFnOnz8vbdu2ldDQUEfWQ0RERETk/jayaA/78OFD9XOXLl3k3r17zi0FEREREZErMrJlypSRoUOHynPPPafawv7yyy8qO2tNx44dHS0DEREREZFrAlk89GDAgAGyevVq1aFr2LBhZh29dJjGQNa/4UImJs55HfoSEhLU8qJjn0iaNA4PopFi0bH+2SnRm7fLkboOSxds9W9MoJ8vnuYv9aJ7eO++BMU+EV8XnjmjW/9+uhLPGe+sk+jYJ6pe8LNPBrJ16tSRnTt3qp9xspw4cUJy587t6rKRl8EB/NqsHbL3PB+I4a2qj/tT/EH1wtlkac/aPv0F4I/ni7/Ui+7ic8+KPzifr4Q02/CbzwezPGe8U0xcvFQYtV79fHhUU8kYHCzexOGj/uzZs5IrVy7XlIa8/mD2pz8w+h8ZXDH7MpQf2+FP9py/4/NZGX88X/yhXh6lDZF/sxcRf1L4yimJjnogvo7nDLll+K3ChQunaEXkX/YMayLhIcFOud1848YNdXHkiWyCP9z2Qflxxe/tfyztqWs0jfCXrLIrzhdP8ad6wfkysH5vCY2PlW2DG0lYOt/NYkZHPZRbzRqJP+I5Qy4LZIkAf2DCQ9I6JbhBMIll+fptMU9/OTujPlwpkOvaWecLOUlQkDxOGyoZsmT0+Xq5Jf6J5wzZK7C+TYiIiIjIbzCQJSIiIiKfxECWiIiIiHxSihqgLFu2TD0UITIyUmJjY83e27dvn7PKRkRERETkvIzs9OnT1WNq8+TJIxEREVKjRg3JkSOHnDlzRl544QVHF0dERERE5J5A9uuvv5bZs2fLjBkzJCQkRAYNGiTr16+Xvn37yr1791JWCiIiIiIiVweyaE6AJ31BWFiY3L9/X/3coUMH+fnnnx1dHBERERGRewLZvHnzyu3bt9XPhQoVMj26Fk/88sZn8BIRERGRf3K4s1ejRo1k1apV8vTTT6u2sv3791edv/bs2SOtW7d2TSmJiIiIfPQpX74s2svL73Agi/axeEIP9O7dW3X02r59u7Rs2VLeffddV5SRiIiIyCfxUbVeFsji0ZLGx0u++eab6kVEREREoh7HXb1wNtlz/o74i0r5M6jt8slA9uDBg3YvsFKlSqkpDxERETkoOuqh29eZoCVIdNQDeRCaXtIEpf75SjFxCRL65LE8Dg4RXxcUFCRLe9aWmDjvvi1vL9yJv3/nltounwxkq1SpogqfXGcuzBMf7x+VRkRE5CtuNWsktzy07ktOXNZKEfk3exHRtObi6xAThYek6LlTXhnIPvDCIBbs2sMYkYCIiIi8R3jmjHI+XwkpfOWU+JPyt89J+iexIqHpPF0U8gF2BbKFCxd2fUmIiIjIbuiv0mzDb+r2viegacHNGzclZ66cTmlaoMXEyMXnnlU/e+MtbPJO/pHzJiIiCtBgNmPWzB673Rz9+JFkzJLZrBN4ipfnJ7fhyb141BARkdPGlERwgw4u0bFPnBLcBNqYmETkGAayRERkN46JSUTexKHLZYxIsGXLFrl7967rSkRERF45JqY/wfZ445iYROTCjGxwcLA0a9ZMjh49KlmzZnVwVURE5O9jYqJpwY0bNyRXrlxe2bRAhyCWHYqIkoehVxOio1VnvOSGYfWJpgUVKlSQM2fOSNGiRV1TIiIi8tkxMRHIIkjEvN4cyBKRfRDAnqz+jPo5157dIhkzijdx+K/MuHHjZODAgfL777/LlStXJCoqyuxFREREROSVGdkWLVqo/1u2bGl2WwbpZj7Zi4iIiIi8NpDdtGmT0wsxc+ZM+eyzz+Tq1atSuXJlmTFjhtSoUSPZzy1evFjatWsnr7zyiqxciQfbEREREVGgcDiQbdiwoVMLsGTJEhkwYIDMmjVLatasKdOmTZPmzZvL8ePHJXfu3DY/d+7cOdXEoX79+k4tDxF5B18f79PXy0/kSQkxMeIPgsLC2KnQ2wJZDL+VlAYNGji0vKlTp0r37t2lS5cu6ncEtKtXr5a5c+fKkCFDrH4GzRfat28vo0ePlq1btyY5HNjjx4/VS6e340WHBLzIfsb95az9h2WoHpGsC79nT10b3/On8UoD6e8Nz+nA4ey6Ni7nZN164g/Cqj4tBX/80aeD2QQXfPc7sk6nB7LPPvu/5yAbGSvIkTaysbGxsnfvXhk6dKhpGnq5NmnSRHbs2GHzc2PGjFHZ2m7duqlANikTJ05UAa8lDA/z6NEju8tKYjb0DvafM8ZgxMF679499ceQPZz9mz11jfcq5c8gBy8/FH+B7bl/55Y88OEvMkfwnA4czq5rLCe4QgWJP3xY/EXMvgi5HhmpMrO+PGqB8bs/ODpaXO3+/fuuC2Tv3Llj9ntcXJxERETI8OHDZfz48Q4t6+bNmyrwzZMnj9l0/H7s2DGrn9m2bZt8//33sn//frvWgSAZTReMGdmCBQuqMQ4zZ/bM86l9FR45qcP+s2coHnv+EOJCyNvHnCRxW12v6J3brvFKfUWgjVfKczpwuKKucy9ZbBY4+XLTiNP1/3eHWu2f8HDxVQnR0aLf98a2pHXD8Fvp06e3e16HI5EsWbIkmta0aVMJCQlRASMyrK6M0Dt06CBz5syRnDlz2vWZ0NBQ9bKEk45/ZB1j3F/O3H/4Q8j6CAz21nXGYD5xyZfxnA4cLqlrLxunNCUSXPR96RFp3L8tjqwj9Sk1QxYVHbQcgWAUTwu7du2a2XT8njdv3kTznz59WnXyevnllxO1o0ibNq1af/HixVO8DURERETkOxwOZA8ePJioTQsejDBp0iSpUqWKQ8tCFrdatWqyYcMGadWqlSkwxe99+vRJNH+ZMmXk0KFDZtOGDRumMrVffvmlajJARERERIHB4UAWwSpuJVg+b7dWrVpqpAFHoTlCp06dpHr16mrsWAy/9fDhQ9MoBh07dpQCBQqoTltoM4FH5BplzZpV/W85nYiIiIj8m8OB7NmzZxO1Y0DjX0ca5hq1bdtW9YIbMWKEeiACAuW1a9eaOoBFRkb6dtsSIiIiIvKOQLZw4cKJpmEc15QGsoBmBNaaEsDmzZuT/Oz8+fNTvF4iIiIi8l0OpzonT56snsale+ONNyR79uzq9v+BAwecXT4iIiIiIudkZPHkrYULF6qf169fr15oCvDLL7/IRx99JH/88YejiyQiIiLyS77+uN0ELy+/w4Es2rHqowP8/vvvKiPbrFkzKVKkiNSsWdMVZSQiIiLySf7yuF2/aVqQLVs2uXDhgvoZmVg8ThYwioEjj6clIiIi8kd4JG1Y1ariT4IrVPDKR+06nJFt3bq1vPXWW1KyZEm5deuWvPDCC2o6HlNbokQJV5SRiIiIyGdgmNLCC3/yi8ft6mP837h/3ysft+1wIPvFF1+oZgTIyn766aeS8f8eJYeHIrz33nuuKCMRERGRT0HQFxQeLn4hIUGCHjwQb+RwIJsuXToZOHBgoun9+/d3VpmIiIiIiJLFJw0QERERkU9iIEtEREREPomBLBERERH5JAayRERERBQ4gezdu3flu+++k6FDh8rt27fVtH379smlS5ecXT4iIiIiIueMWnDw4EH1EIQsWbLIuXPnpHv37pI9e3ZZvny5REZGyoIFCxxdJBERERGR6zOyAwYMkM6dO8vJkyclffr0puktWrSQLVu2OF4CIiIiIiJ3BLK7d++Wd999N9H0AgUKyNWrV1NSBiIiIiIi1weyoaGhEhUVlWj6iRMnJFeuXI6XgIiIiIjIHYFsy5YtZcyYMRIXF2d6BBvaxg4ePFjatGmTkjIQEREREbk+kJ0yZYo8ePBAcufOLTExMdKwYUMpUaKEZMqUScaPH+94CYiIiIiI3DFqAUYrWL9+vWzbtk2NYICgtmrVqmokAyIiIiIirw1kdfXq1VMvIiIiIiKvDWSnT59u9wL79u2bmvIQERERETkvkP3iiy/Mfr9x44ZER0dL1qxZTU/6Cg8PV+1mGcgSERERkdd09jp79qzphQ5dVapUkaNHj6rH0+KFn9FOduzYsa4vMRERERFRSkYtGD58uMyYMUNKly5tmoafkbUdNmyYs8tHREREROScQPbKlSvy5MmTRNPj4+Pl2rVrji6OiIiIiMg9gWzjxo3VI2r37dtnmrZ3717p1asXh+AiIiIiIu8NZOfOnSt58+aV6tWrq8fV4lWjRg3JkyePfPfdd64pJRERERFRaseRzZUrl6xZs0ZOnDghx44dU9PKlCkjpUqVcnRRRERERETufyACAlcGr0RERETkM4Fs165dk216QERERETkdYHsnTt3zH6Pi4uTw4cPq4ciNGrUyJllIyIiIiJyXiC7YsWKRNMSEhLUqAXFixd3dHFERERERO4ZtcDqQtKkkQEDBiR6lC0RERERkVcHsnD69GmrD0ogIiIiIvKKpgXIvBppmqae9rV69Wrp1KmTM8tGREREROS8QDYiIiJRswKMLTtlypRkRzQgIiIiIvJYILtp0yanrZyIiIiIyG1tZDHEFobashQVFcXht4iIiIjIewPZzZs3S2xsbKLpjx49kq1btzqrXEREREREzmlacPDgQdPPR44ckatXr5p+j4+Pl7Vr10qBAgXsXRwRERERkXsC2SpVqkhQUJB6WWtCEBYWJjNmzEhdaYiIiIiInB3Inj17Vg21VaxYMdm1a5caqUAXEhIiuXPnluDgYHsXR0RERETknkC2cOHCpsfREhERERH5RCC7atUqeeGFFyRdunTq56S0bNnSWWUjIiIiIkpdINuqVSvVuQvNB/CzLWg/i45fREREREReEcgamxOwaQERERER+eQ4sgsWLJDHjx8nmo6xZfEeEREREZFXBrJdunSRe/fuJZp+//599V5KzJw5U4oUKSLp06eXmjVrqlERbJkzZ47Ur19fsmXLpl5NmjRJcn4iIiIi8k8OB7IYggttYS1dvHhRsmTJ4nABlixZIgMGDJCRI0fKvn37pHLlytK8eXO5fv26zSeLtWvXTjZt2iQ7duyQggULSrNmzeTSpUsOr5uIiIiIAmD4raefftr0QITGjRtL2rT//6Po4IVxZp9//nmHCzB16lTp3r27KZs7a9YsWb16tcydO1eGDBmSaP6FCxea/f7dd9/Jr7/+Khs2bJCOHTs6vH4iIiIi8vNAVh+tYP/+/SpjmjFjRrMHIqBpQJs2bRxaOdrV7t27V4YOHWqaliZNGtVcANlWe0RHR0tcXJxkz57d6vtoz2ts0xsVFWXqtMaOa46x7PTnjP2HZSDLz7rwf6zrwMB6Dhys68CR4Oa6dmQ9dgeyuPUPCFjffPNNCQ0NldS6efOmyubmyZPHbDp+P3bsmF3LGDx4sOTPn18Fv9ZMnDhRRo8enWj6jRs35NGjRykseWCKiYs3239h6YKdcrCizTVOEFzEkP9iXQcG1nPgYF0HjgQ31zX6XTk9kNWVK1dOZWXRKcvon3/+UY+orV69urjLpEmTZPHixardLDqKWYNsL9rgGjOyaFeLR+xmzpzZbWX1B9GxT0w/Y/+Fhzh8+Fg9OdBcBcvjH0L/xroODKznwMG6DhwJbq5rWzGdNQ5HIr1795ZBgwYlCmTR2Wry5MkqoLVXzpw5VfB77do1s+n4PW/evEl+9vPPP1eB7J9//imVKlWyOR8yx9ayx6gInniOMe4vZ+4/nBysj8DAug4MrOfAwboOHEFurGtH1uFwaY4cOSJVq1a12hkM7zkCbWurVaumOmoZo378Xrt2bZuf+/TTT2Xs2LGydu1at2aAiYiIiMh7OBzIIrtpmUGFK1eumI1kYC/c9sfYsD/88IMcPXpUevXqJQ8fPjSNYoCRCIydwZD1HT58uBrVAO118ehcvB48eODwuomIiIgogAJZjNmKwNL4UIS7d+/Kxx9/LE2bNnW4AG3btlXNBEaMGCFVqlRR7W+RadU7gEVGRqogWffNN9+o0Q5ee+01yZcvn+mFZRARERFR4HA4hYqAsUGDBlK4cGHVnAAQfCLw/PHHH1NUiD59+qiXNejIZXTu3LkUrYOIiIiIAjyQLVCggBw8eFA9mODAgQMSFhammgHgaVvp0qVzTSmJiIiIiCykaPykDBkySI8ePVLyUSIiIiIip0jxQKAYoQDtV9Fe1ahly5bOKBcRERERkXMD2TNnzsirr74qhw4dUmOK4SkPgJ8BT+oiIiIiIvK6UQv69esnRYsWlevXr0t4eLj8+++/smXLFjWeq2XHLCIiIiIir8nI7tixQzZu3KieyqU/4aFevXoyceJE6du3r0RERLimpEREREREqcnIoulApkyZ1M8IZi9fvqx+xnBcx48fd3RxRERERETuychWqFBBDbuF5gU1a9ZUj4vFo2Znz54txYoVS1kpiIiIiIhcHcgOGzZMPUIWxowZIy+99JLUr19fcuTIIUuWLHF0cURERERE7glkmzdvbvq5RIkScuzYMbl9+7Zky5bNNHIBEREREZHXtZG9ceNGomnZs2dXQSyG5CIiIiIi8spAtmLFirJ69epE0z///HOpUaOGs8pFREREROTcQHbAgAHSpk0b6dWrl8TExMilS5ekcePGqtPXokWLHF0cEREREZF7AtlBgwapsWS3bt0qlSpVUq/Q0FA5ePCgeuIXEREREZFXBrJ6Jy8Mw3Xu3DmJioqStm3bSt68eZ1fOiIiIiIiZwWyf//9t8rCnjx5UmVhv/nmG3n//fdVMHvnzh1HF0dERERE5J5AtlGjRipo3blzp5QtW1beeecd9VjayMhI1RGMiIiIiMgrx5H9448/pGHDhmbTihcvrjK148ePd2bZiIiIiIicl5G1DGJNC0qTRoYPH+7o4oiIiIiIXBvItmjRQu7du2f6fdKkSXL37l3T77du3ZJy5cqlrBRERERERK4KZNetWyePHz82/T5hwgT1aFrdkydP5Pjx446un4iIiIjItYGspmlJ/k5ERERE5PXjyBIRERER+UwgGxQUpF6W04iIiIiIvHr4LTQl6Ny5s3ocLTx69Eh69uwpGTJkUL8b288SEREREXlNINupUyez399+++1E83Ts2NE5pSIiIiIiclYgO2/ePHtnJSIiIiJyOXb2IiIiIiKfxECWiIiIiHwSA1kiIiIi8kkMZImIiIjIJzGQJSIiIiKfxECWiIiIiHwSA1kiIiIi8kkMZImIiIjIJzGQJSIiIiKfxECWiIiIiHwSA1kiIiIi8kkMZImIiIjIJzGQJSIiIiKfxECWiIiIiHwSA1kiIiIi8kkMZImIiIjIJzGQJSIiIiKfxECWiIiIiHwSA1kiIiIi8kkMZImIiIjIJzGQJSIiIiKf5BWB7MyZM6VIkSKSPn16qVmzpuzatSvJ+ZcuXSplypRR81esWFHWrFnjtrISERERkXfweCC7ZMkSGTBggIwcOVL27dsnlStXlubNm8v169etzr99+3Zp166ddOvWTSIiIqRVq1bqdfjwYbeXnYiIiIg8J0jTNM2D61cZ2GeeeUa++uor9XtCQoIULFhQ3n//fRkyZEii+du2bSsPHz6U33//3TStVq1aUqVKFZk1a1ay64uKipIsWbLIvXv3JHPmzOJK2LUxcfHiL6Jj46X6uD/Vz0fGNJfwkLSpXibqGxctuXPnljRpPH5dRS7Eug4MrOfAwboOHAlurmtHYrXURyKpEBsbK3v37pWhQ4eapmEHNWnSRHbs2GH1M5iODK4RMrgrV660Ov/jx4/Vy7hz9ErBy5WiY59IhVHrxR85a/9hGQj4XV0X5Hms68DAeg4crOvAkeDmunZkPR4NZG/evCnx8fGSJ08es+n4/dixY1Y/c/XqVavzY7o1EydOlNGjRyeafuPGDXn06JG4kj9lY40q5c8g9+/ckgdBQU45WHHFhROEV/T+jXUdGFjPgYN1HTgS3FzX9+/f941A1h2Q7TVmcJGRRdOFXLlyuaVpweFRTcXfhKULliAnBLH6yYFloT74h9C/sa4DA+s5cLCuA0eCm+sanfl9IpDNmTOnBAcHy7Vr18ym4/e8efNa/QymOzJ/aGioellCRbijMjIGB7t8Hb4OJ4e76oM8i3UdGFjPgYN1HTiC3FjXjqzDo0deSEiIVKtWTTZs2GAW9eP32rVrW/0Mphvnh/Xr19ucn4iIiIj8k8ebFuC2f6dOnaR69epSo0YNmTZtmhqVoEuXLur9jh07SoECBVRbV+jXr580bNhQpkyZIi+++KIsXrxY9uzZI7Nnz/bwlhARERFRQAWyGE4LHa9GjBihOmxhGK21a9eaOnRFRkaapZjr1KkjixYtkmHDhsnHH38sJUuWVCMWVKhQwYNbQUREREQBN46su7lzHFlKHschDBys68DAeg4crOvAkeDF48jyyCMiIiIin8RAloiIiIh8EgNZIiIiIvJJDGSJiIiIyCcxkCUiIiIin8RAloiIiIh8EgNZIiIiIvJJHn8ggrvpw+ZijDLyjrHp7t+/L+nTp+c4hH6OdR0YWM+Bg3UdOBLcXNd6jGbPow4CLpBFRUDBggU9XRQiIiIiSiJmw4MRkhJwT/bCVcXly5clU6ZMEhQU5OniBDxcdeGi4sKFC3zSmp9jXQcG1nPgYF0Hjig31zVCUwSx+fPnTzYDHHAZWeyQp556ytPFIAs4MfiHMDCwrgMD6zlwsK4DR2Y31nVymVgdG7UQERERkU9iIEtEREREPomBLHlUaGiojBw5Uv1P/o11HRhYz4GDdR04Qr24rgOusxcRERER+QdmZImIiIjIJzGQJSIiIiKfxECWiIiIiHwSA1kiIiIi8kkMZMnlZs6cKUWKFFHPaK5Zs6bs2rXL5rxz5syR+vXrS7Zs2dSrSZMmSc5PvlvXRosXL1ZP2mvVqpXLy0jur+e7d+9K7969JV++fKrXc6lSpWTNmjVuKy+5r66nTZsmpUuXlrCwMPUkqP79+8ujR4/cVl5y3JYtW+Tll19WT9HC3+GVK1cm+5nNmzdL1apV1flcokQJmT9/vngKA1lyqSVLlsiAAQPUsB379u2TypUrS/PmzeX69es2T4527drJpk2bZMeOHeoPYbNmzeTSpUtuLzu5tq51586dk4EDB6oLGPK/eo6NjZWmTZuqel62bJkcP35cXbAWKFDA7WUn19b1okWLZMiQIWr+o0ePyvfff6+W8fHHH7u97GS/hw8fqrrFRYs9zp49Ky+++KI899xzsn//fvnggw/knXfekXXr1olHYPgtIlepUaOG1rt3b9Pv8fHxWv78+bWJEyfa9fknT55omTJl0n744QcXlpI8Vdeo3zp16mjfffed1qlTJ+2VV15xU2nJXfX8zTffaMWKFdNiY2PdWEryRF1j3kaNGplNGzBggFa3bl2Xl5WcA2HhihUrkpxn0KBBWvny5c2mtW3bVmvevLnmCczIkssgE7N3717VPECXJk0a9TuyrfaIjo6WuLg4yZ49uwtLSp6q6zFjxkju3LmlW7dubiopubueV61aJbVr11ZNC/LkySMVKlSQCRMmSHx8vBtLTu6o6zp16qjP6M0Pzpw5o5qQtGjRwm3lJtdD/RuPC0Cm3t7vdWdL65G1UkC4efOm+rLCl5cRfj927Jhdyxg8eLBqt2N50pDv1/W2bdvUrUfcmiL/rWcEMxs3bpT27duroObUqVPy3nvvqQtU3IIm/6nrt956S32uXr16uNsrT548kZ49e7JpgZ+5evWq1eMiKipKYmJiVPtod2JGlrzWpEmTVCegFStWqI4G5D/u378vHTp0UG0lc+bM6enikAslJCSorPvs2bOlWrVq0rZtW/nkk09k1qxZni4aORn6OCDb/vXXX6s2tcuXL5fVq1fL2LFjPV008mPMyJLLIEAJDg6Wa9eumU3H73nz5k3ys59//rkKZP/880+pVKmSi0tK7q7r06dPq84/6ClrDHggbdq0qkNQ8eLF3VBycvU5jZEK0qVLpz6nK1u2rMrq4PZ1SEiIy8tN7qnr4cOHqwtUdPyBihUrqo5EPXr0UBcvaJpAvi9v3rxWj4vMmTO7PRsLPKrIZfAFhQzMhg0bzIIV/I42c7Z8+umn6gp+7dq1Ur16dTeVltxZ12XKlJFDhw6pZgX6q2XLlqZesBitgvzjnK5bt65qTqBfqMCJEydUgMsg1r/qGn0aLINV/QLmf/2IyB/Url3b7LiA9evXJ/m97lIe6WJGAWPx4sVaaGioNn/+fO3IkSNajx49tKxZs2pXr15V73fo0EEbMmSIaf5JkyZpISEh2rJly7QrV66YXvfv3/fgVpAr6toSRy3wz3qOjIxUI4/06dNHO378uPb7779ruXPn1saNG+fBrSBX1PXIkSNVXf/888/amTNntD/++EMrXry49sYbb3hwKyg5+H6NiIhQL4SFU6dOVT+fP39evY86Rl3rULfh4eHaRx99pB09elSbOXOmFhwcrK1du1bzBAay5HIzZszQChUqpAJUDOeyc+dO03sNGzZUAYyucOHC6kSyfOEPJPlXXVtiIOu/9bx9+3atZs2aKijCUFzjx49XQ6+Rf9V1XFycNmrUKBW8pk+fXitYsKD23nvvaXfu3PFQ6ckemzZtsvq9q9ct/kddW36mSpUq6rjAOT1v3jzNU4Lwj2dywUREREREKcc2skRERETkkxjIEhEREZFPYiBLRERERD6JgSwRERER+SQGskRERETkkxjIEhEREZFPYiBLRERERD6JgSwRERER+SQGskTksNjYWClRooRs377dNO3YsWNSq1YtSZ8+vVSpUsXtZercubO0atXK7es9d+6cBAUFyf79+52+7CJFisi0adPsmnf+/PmSNWtW8Sf+uE2+xlnHt6Pn55EjR+Spp56Shw8fpmq95P8YyFJAuXr1qrz//vtSrFgxCQ0NlYIFC8rLL78sGzZsEF+wefNm9aVy9+5dj5Zj1qxZUrRoUalTp45p2siRIyVDhgxy/Phxl+5PW1+sX375pQp8/Clg2717t/To0cNnji1nH59t27aVEydOmH4fNWqURy6SyP3KlSunLoynTp3q6aKQl2MgSwEDAVC1atVk48aN8tlnn8mhQ4dk7dq18txzz0nv3r3F28XFxXnF8vBU66+++kq6detmNv306dNSr149KVy4sOTIkcOp67RHlixZ/C57lytXLgkPDxd/zOjbIywsTHLnzu3y8pB36tKli3zzzTfy5MkTTxeFvJlGFCBeeOEFrUCBAtqDBw8SvXfnzh3Tz+fPn9datmypZciQQcuUKZP2+uuva1evXjW936lTJ+2VV14x+3y/fv20hg0bqp+//fZbLV++fFp8fLzZPFhmly5dTL+vXLlSe/rpp7XQ0FCtaNGi2qhRo7S4uDjT+zg9v/76a+3ll1/WwsPD1XoxzfjCNChcuLD2xRdfmK2vcuXK2siRI20uT38vuXJY2r17t5YmTRotKirKbNnGF5Z99uxZ9fPixYu1Bg0aqOXPmzdPu3nzpvbmm29q+fPn18LCwrQKFSpoixYtMlsH9t3kyZO14sWLayEhIVrBggW1cePGWV2Xvt8t6+XRo0fa+++/r+XKlUutu27dutquXbtM72/atEl9/s8//9SqVaumylK7dm3t2LFjWlL++ecfrUqVKmqZ+Nzy5cvVciIiIkzzHDp0SHv++efVMZQ7d27t7bff1m7cuGF1eXo5LPeftXrFcdqjRw+1TKy/fPny2n/+8x/1HvZtlixZTPNev35dla9Vq1ZqX2CfTpgwQStSpIiWPn16rVKlStrSpUvVvHpdWTu2MA/qCJ/Jnj271rhxY6vnUFLLQB317t1bnSc5cuTQnn32WTV9ypQpatk4Hp966imtV69e2v37903LNG4TfrZcPqbp+6Vbt25azpw51Tn73HPPafv37zctBz9jnRkzZlTvV61aVR3H1iQkJKj9j2MOxx7OZRxHugULFqj9imXlyZNHa9eunXbt2jWHjitHyuOM7du2bZuqA5Qla9asWrNmzbTbt2+r9/773/+qcwP7GfX74osvaqdOnUpUr/rx/eTJE61r166m46hUqVLatGnTzMqLefr3729a5kcffaR17NjR7PxM6njUPX78WB3n2JdEtjCQpYBw69YtLSgoSP3hTAr+uCJIqVevnrZnzx5t586d6stID5bsCWTxBYEvQOMfX6zfOG3Lli1a5syZtfnz52unT5/W/vjjD/UHHUGkDl8eCFjmzp2r5jl37pz266+/qunHjx/Xrly5ot29e9ehQNa4PATs9pTD0tSpU7UyZcqYTUNZEFR9+OGH6mcEI/oXIJaHcp85c0a7fPmydvHiRe2zzz5TX4xY5/Tp07Xg4GAVIOoGDRqkZcuWTZULX6pbt27V5syZo95DMKoHClgX9q21eunbt68KltesWaP9+++/6n0sU59fDzhq1qypbd68Wc1Tv359rU6dOja3HduFwPitt97SDh8+rILIYsWKmX3RI+jAPEOHDtWOHj2q7du3T2vatKkKPqzBlzUCAdQDtkfff5b1imOzVq1aaj+jnrDvsH5sn2XQFxkZqZUuXVptM4IKwIUA6m3t2rXqs5gfQQK2HfNYO7ZQX2nTplV1jvo8ePCgNnPmTLNgU2drGYBzA0EWAhoEdHpQh23buHGjWvaGDRtUmRHM6ozbFB0drY4vbL++nzANmjRpoi7QELydOHFCzYeAWa9rfAYXE6gPvP/LL7+YBYJGCKZQF9ivOEdwXM6ePdv0/vfff6/ewz7csWOHClJxkayz57hypDyp3T4cl6hn7FdMw3E7Y8YM04XVsmXLVL2dPHlSzYv1VKxY0XQhbhnIxsbGaiNGjFBlwTn9008/qQuRJUuWmMqLi1Cca1jukSNHVBCOANt4fiZ1PBphPxr/jhFZYiBLAQFfRvhjjOxZUhAgIKhCIKDDFxE+q2fzkgtkAe8ja6FDlhZBlf7lgKyWZVD9448/quyPDuv84IMPzObRvySNGWRHAlnL5dlTDkvY1kaNGiWabrk+/QvQMltjDbJA+HIGZHrxhaYHrpYsv1h1xnpBxjBdunTawoULTe/jCxh18OmnnybKnOlWr16tpsXExFhdN+oRAYTx/W+++casPGPHjlUZL6MLFy6YAjxrLLOp1up13bp1KhOe3DIQJCKbiEAe2UVARhbBxvbt280+gwADGUVbx9bevXvVNFxE2cPW8YlzA1n/5CCIxP613CYdji8cZ0a4yEHgiW00QjYf9QUIonBRZA9kiZFlxPFiDwR02GY9uLfnuHKkPKndPtQvMq72QoCLsuKuQlLnmxGy7W3atDH9jr8f+nkGuMODjLt+ftpzPOpeffVVrXPnznaXnwIP28hSQPhfHJe8o0ePqg5geBk7HaDtJd6zV/v27eXXX3+Vx48fq98XLlwob775pqRJ879T7sCBAzJmzBjJmDGj6dW9e3e5cuWKREdHm5ZTvXp1cSbL5dlbDqOYmBg1MkFK1xkfHy9jx46VihUrSvbs2dU6161bJ5GRkep97Gfst8aNG0tKob0u2uPWrVvXNC1dunRSo0aNRPVYqVIl08/58uVT/1+/ft3qcvFZzG/c/tq1ayfap5s2bTLbp2XKlDGVK6XQuQ29uEuVKmVzHtRN/fr1pXXr1qrzGzpewalTp1R9Nm3a1KxcCxYsSLJMlStXVvWAunr99ddlzpw5cufOnRSVH+3TLf35559q+QUKFJBMmTJJhw4d5NatWzaPPWuwvx88eKDaZRu37ezZs6ZtGzBggLzzzjvSpEkTmTRpUpLbjO3EfkSHUJwLK1asMGujuXfvXtVBtFChQqrMDRs2VNP149ee48qR8qR2+3DcJHUunTx5Utq1a6e2N3PmzGqkDGvbYzRz5kxVn2jDjbLMnj3bNP+9e/fU34+aNWua5k+bNq3Z3wFHjke0k3bkeKDAk9bTBSByh5IlS6ovdQwRlVoIRi0DY8tOTPiiwzyrV6+WZ555RrZu3SpffPGF6X18MY0ePVoFHJaMQRJGAXBWmawtz95yGOXMmVN1lLOX5TrR0Q5BFoaVQoCE9z/44ANTByB8cbkTAlydHvglJCSkeHnYp6j/yZMnJ3pPD2hSwp79gpE4EMz8/vvv8tFHH6kAUS8T4HjUpxk/Y0twcLCsX79eDbP2xx9/yIwZM+STTz6Rf/75R41a4QjL4wCdL1966SXp1auXjB8/Xl3UbNu2TXUixLFgbyc3bBv2K0ZMsKR3/sNoB2+99Zba/v/+979qhI3FixfLq6++mugzuIjFyBsIsrHt7733njpm//rrL1Wu5s2bqxcuThHIIYDD75Yd2JI6rhwpT2q3L7njBscqOmjiIiV//vyqjBUqVLDZIQ/LHThwoEyZMkVdxCGYx/7BMWEvR47H27dvS/Hixe1eNgUeZmQpIOBLEl82yCRYG5dQHy6obNmycuHCBfUyjmeI95GZBXx5IeNgZDkUFIJABIf4svv555+ldOnSUrVqVdP7+BlflhiL1fKlZ22tCQkJMWU1jSzLFBUVpTI2yUlJOZ5++ml1QWBvltvS33//La+88oq8/fbbKuOHTJBxiCVcdODL19YQXrb2gRG++DAf1mUM7DGclV6PKYHj4+DBg/Lo0SPTtJ07dybap//++6/KbFnuU1sXJihrUtujZ/guXrxotq8soc5+/PFHlS3DaByXL19W07HNCBAQdFmWSb/7YGu/IghDZhsXPBEREWo+ZCltbYe1ZViDzCaCJgREGGYJmWa9vLZY20/Y3xhWD1k/y23DRZcOy+/fv78KyHFuzps3z+Z6cPwhwJs+fboKIHfs2KEu3nDcI2OMrCcy38i028reJ8fe8qR2+3Dc2DqXsC04/4cNG6aytji+k8u445zCsHsI8PG3AOUwZlExeggCb2Ngi4w26ltnz/GoO3z4sFoPkS0MZClgIIjFlyBuL+O2P26p4VYxvqz028PIZiFLiKYB+/btk127dknHjh3V7UP91lijRo1kz5496jYYloHsB/7YWsIykHGYO3eu+tloxIgR6vMIDhD0oBzIdOALJSnInCCwQMbtxo0bpswGyoQABplffOF26tRJZdOSk5JyIEDCejF/SiBQ1bN8WN+7774r165dM7sIGDx4sAwaNMh0qxHB4vfff6/ex3BMCDQwdBo+h1uZlhAwItOHrCTmw8UIbhPjFqXlsGGOQNYL+x/LwjLXrFkjn3/+udk8GMoNWSTcrkXgjPKj6QSGErIV4CHoxT5FwHHz5k2rt1JxDDZo0EDatGmj9h8uVJB9w/YZod5xAYWLBBwXCIKQNUMWDYHODz/8oMqE4xsZVvxu69hCMDJhwgR1vCPoWL58uXoPAY8jx6c1CFpwcYEynDlzRh2/GJ84KdhP2G5cOGI/oQkKzlmcvxhsH0EcMr04tpA5RrnRTKBPnz4qID1//rwKxFAvtrYBY/riWMM5jXL99NNP6njDtqE5AYJpvcyrVq1SzWQc4Wh5Urt9Q4cOVb8j8MRFGIJxDGmF/ZctWzbVZAFNA3C7H0MToplCcucv1otjGhdVw4cPV8s36tevnwr2V65cqdaHdRvHFrbneARs66VLl9Q+ILLJ0410idwJvbDRMQGdaDCKAIbjwrBY6KBh7/BbgF67GHoHHVEwzEyfPn3MOnsBOnah0wNOM/TKtYTeuujJjCFx0JmjRo0aZr2j8bkVK1Yk+tyYMWO0vHnzqlEY9OGN7t27p7Vt21YtBx190PHDWmcva8tLrhzWvPHGG9qQIUPs6uxl2UkEPa3R6QO92DGKwrBhw6wOzYNezagndNoqVKiQWac0dATDdqLzk63ht9CxBsMmYciipIbfMnZMQlkxDWW3BT3Vsa04fjDChd5T37id6DmOTioY6gj7Fb2z0dFO73xlTc+ePVVHp6SG38K+wxBumA9DFmHoqt9//91qxyh0sGndurVWtmxZNTwU1o2OdxgZAPsUIys0b95c++uvv2weW+hxjnn0IczQCQo93pNi7fhEHaGToCWMhoBzBPsI68HQVsY6sdwmdBJCpyLsV+PwW+ggiLpGZz5sG46N9u3bq06bGBUCw73pw2lhHpyvtjr04RxBT3mcC/gbgJEijB23MFQcRuLA/sCIBatWrTKr/+SOK0fL44ztw0gAOMdRZuw77Gu9fOvXr1fHCN7DEFiY1/i3wvI8Rh2g8xXqBcvCaAj4W2DshIdjD/WNfYh5BgwYkOgct+d4xDmPaURJCcI/tsNcIqLEkNlBRw1kUtBJg4jImdBGF9nfRYsWmXXaJLLEQJaIUgS3YNEWE00xiIicCU0d0NQGTY+IksJAloiIiIh8Ejt7EREREZFPYiBLRERERD6JgSwRERER+SQGskRERETkkxjIEhEREZFPYiBLRERERD6JgSwRERER+SQGskRERETkkxjIEhEREZH4ov8H8agM2lHATwQAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "temoin classique escalade 50 % -> exactitude couverte 0.500\n", + "LLM gele escalade 50 % -> exactitude couverte 0.667\n" + ] + } + ], + "source": [ + "def courbe_couverture_exactitude(P, y):\n", + " \"\"\"Exactitude sur la fraction la plus confiante des questions, pour chaque couverture.\"\"\"\n", + " conf = P.max(1)\n", + " ok = (P.argmax(1) == y)\n", + " ordre = np.argsort(-conf)\n", + " couv, acc = [], []\n", + " for k in range(1, len(y) + 1):\n", + " sel = ordre[:k]\n", + " couv.append(k / len(y))\n", + " acc.append(float(ok[sel].mean()))\n", + " return np.array(couv), np.array(acc)\n", + "\n", + "\n", + "fig, ax = plt.subplots(figsize=(7, 4.2))\n", + "for nom, P, coul in [\n", + " (\"temoin classique (T appris)\", P_clf_test_T, \"tab:blue\"),\n", + " (\"LLM gele zero-shot (T appris)\", P_llm_test_T, \"tab:red\"),\n", + "]:\n", + " c, a = courbe_couverture_exactitude(P, y_test_idx)\n", + " ax.step(c, a, where=\"post\", label=nom, color=coul)\n", + "ax.set_xlabel(\"Couverture (fraction de tickets traites sans escalade)\")\n", + "ax.set_ylabel(\"Exactitude sur la fraction couverte\")\n", + "ax.set_title(\"Courbe couverture-exactitude (banc test)\")\n", + "ax.grid(alpha=0.3)\n", + "ax.legend()\n", + "plt.tight_layout()\n", + "plt.show()\n", + "\n", + "for nom, P in [(\"temoin classique\", P_clf_test_T), (\"LLM gele\", P_llm_test_T)]:\n", + " conf = P.max(1)\n", + " seuil = np.quantile(conf, 0.5) # couverture cible 50 %\n", + " couverts = conf >= seuil\n", + " print(\n", + " f\"{nom:18s} escalade 50 % -> exactitude couverte \"\n", + " f\"{(P.argmax(1)[couverts] == y_test_idx[couverts]).mean():.3f}\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "437d1d02", + "metadata": { + "papermill": { + "duration": 0.003146, + "end_time": "2026-09-28T13:51:29.366293", + "exception": false, + "start_time": "2026-09-28T13:51:29.363147", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Lecture du résultat\n", + "\n", + "La courbe répond à la question posée en ouverture : **quand un petit modèle suffit-il ?** —\n", + "quand, à la couverture visée, son exactitude couverte est au niveau acceptable, **et** que sa\n", + "confiance est calibrée (sinon le seuil ne veut rien dire). Mesuré ici, et contre l'intuition :\n", + "à 50 % d'escalade, c'est le LLM gelé qui couvre le mieux (0,667 contre 0,500) — ses erreurs\n", + "sont moins concentrées sur ses réponses les plus confiante. Sur 12 tickets de test, ce constat\n", + "est une **observation à répliquer**, pas une conclusion. Le résidu au-delà du seuil n'est pas\n", + "un échec du modèle : c'est le flux qu'on **route** délibérément vers l'humain ou le LLM\n", + "génératif. Décider, ce n'est pas toujours répondre — c'est aussi savoir quand ne pas répondre." + ] + }, + { + "cell_type": "markdown", + "id": "a9062fd4", + "metadata": { + "papermill": { + "duration": 0.003127, + "end_time": "2026-09-28T13:51:29.373006", + "exception": false, + "start_time": "2026-09-28T13:51:29.369879", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 7. Exercices\n", + "\n", + "Trois exercices, du plus mécanique au plus ouvert. Chaque stub s'exécute tel quel\n", + "(`return None`), le notebook reste exécutable de bout en bout." + ] + }, + { + "cell_type": "markdown", + "id": "0326b6a5", + "metadata": { + "papermill": { + "duration": 0.002654, + "end_time": "2026-09-28T13:51:29.378716", + "exception": false, + "start_time": "2026-09-28T13:51:29.376062", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Exercice 1 — Étendre le contrat au type ordinal `score`\n", + "\n", + "Le type `score` déclare une échelle ordinale (ex. gravité 1-5) au lieu d'options nominales.\n", + "Écrire `question_score` qui renvoie la distribution sur l'échelle." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "89e68064", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:29.385168Z", + "iopub.status.busy": "2026-09-28T13:51:29.383605Z", + "iopub.status.idle": "2026-09-28T13:51:29.387887Z", + "shell.execute_reply": "2026-09-28T13:51:29.387887Z" + }, + "papermill": { + "duration": 0.008442, + "end_time": "2026-09-28T13:51:29.389768", + "exception": false, + "start_time": "2026-09-28T13:51:29.381326", + "status": "completed" + }, + "tags": [] + }, + "outputs": [], + "source": [ + "def question_score(state, echelle=(\"1\", \"2\", \"3\", \"4\", \"5\")):\n", + " \"\"\"Exercice 1 : distribution typée sur une échelle ordinale de gravité.\n", + "\n", + " TODO etudiant : soumettre l'état au LLM gelé (fonction option_scores) pour chaque\n", + " niveau de l'échelle et renvoyer la distribution.\n", + " Indice : un `score` se traite comme un `choice` sur N niveaux — mais l'exactitude\n", + " stricte n'est plus la bonne métrique (voir Exercice 2 pour pourquoi).\n", + " \"\"\"\n", + " print(\"Exercice a completer\")\n", + " return None\n", + "\n", + "\n", + "# question_score(\"Le site est totalement inaccessible depuis deux heures\")" + ] + }, + { + "cell_type": "markdown", + "id": "02ee89d3", + "metadata": { + "papermill": { + "duration": 0.003302, + "end_time": "2026-09-28T13:51:29.396298", + "exception": false, + "start_time": "2026-09-28T13:51:29.392996", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Exercice 2 — Une métrique ordinale : la distance absolue moyenne\n", + "\n", + "Sur une échelle 1-5, prédire 4 quand la vérité est 5 est une *meilleure* décision que prédire\n", + "1 — l'exactitude stricte compte les deux comme fausses. Écrire `mad_ordinal` qui mesure la\n", + "distance absolue moyenne entre niveau prédit et niveau vrai." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "77208f6f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:29.402510Z", + "iopub.status.busy": "2026-09-28T13:51:29.402510Z", + "iopub.status.idle": "2026-09-28T13:51:29.407792Z", + "shell.execute_reply": "2026-09-28T13:51:29.407269Z" + }, + "papermill": { + "duration": 0.009897, + "end_time": "2026-09-28T13:51:29.409328", + "exception": false, + "start_time": "2026-09-28T13:51:29.399431", + "status": "completed" + }, + "tags": [] + }, + "outputs": [], + "source": [ + "def mad_ordinal(predictions_idx, verite_idx):\n", + " \"\"\"Exercice 2 : distance absolue moyenne entre niveaux prédits et niveaux vrais.\n", + "\n", + " TODO etudiant : renvoyer la moyenne de |prediction - verite| sur le banc.\n", + " Indice : les distributions de l'Exercice 1 se réduisent à un niveau par argmax.\n", + " \"\"\"\n", + " print(\"Exercice a completer\")\n", + " return None\n", + "\n", + "\n", + "# mad_ordinal([4, 2, 5], [5, 2, 3]) # attendu : (1 + 0 + 2) / 3 = 1.0" + ] + }, + { + "cell_type": "markdown", + "id": "73243515", + "metadata": { + "papermill": { + "duration": 0.00291, + "end_time": "2026-09-28T13:51:29.415324", + "exception": false, + "start_time": "2026-09-28T13:51:29.412414", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "### Exercice 3 — Politique d'escalade à budget fixé\n", + "\n", + "Le support peut traiter 60 % des tickets sans humain. Écrire `seuil_pour_couverture` qui,\n", + "à partir des confiances calibrées, trouve le seuil réalisant exactement cette couverture —\n", + "puis mesurer l'exactitude couverte des deux familles à ce budget." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "c345727e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-28T13:51:29.422673Z", + "iopub.status.busy": "2026-09-28T13:51:29.422145Z", + "iopub.status.idle": "2026-09-28T13:51:29.427303Z", + "shell.execute_reply": "2026-09-28T13:51:29.426256Z" + }, + "papermill": { + "duration": 0.009952, + "end_time": "2026-09-28T13:51:29.428198", + "exception": false, + "start_time": "2026-09-28T13:51:29.418246", + "status": "completed" + }, + "tags": [] + }, + "outputs": [], + "source": [ + "def seuil_pour_couverture(confiances, couverture=0.6):\n", + " \"\"\"Exercice 3 : seuil de confiance réalisant la couverture visée.\n", + "\n", + " TODO etudiant : renvoyer le plus petit seuil tel qu'au moins `couverture` des tickets\n", + " aient une confiance >= seuil.\n", + " Indice : np.quantile fait presque tout le travail — attention au sens de l'inégalité.\n", + " \"\"\"\n", + " print(\"Exercice a completer\")\n", + " return None\n", + "\n", + "\n", + "# s = seuil_pour_couverture(P_clf_test_T.max(1), 0.6)" + ] + }, + { + "cell_type": "markdown", + "id": "8ddf7de9", + "metadata": { + "papermill": { + "duration": 0.005773, + "end_time": "2026-09-28T13:51:29.438753", + "exception": false, + "start_time": "2026-09-28T13:51:29.432980", + "status": "completed" + }, + "tags": [] + }, + "source": [ + "## 8. Conclusion — et les familles ouvertes, en tranches suivantes\n", + "\n", + "**Ce que cette tranche a établi.** Un contrat à espace fermé rend les erreurs de type\n", + "impossibles par construction, mais reporte toute la difficulté sur la **calibration** — et la\n", + "calibration s'acquiert avec un jeu dédié **et dimensionné** : sur 12 tickets, la température\n", + "apprise dégrade la confiance au lieu de la réparer. Sur ce banc minuscule, l'écart mesuré\n", + "entre les familles (0,500 contre 0,417 en exactitude brute ; 0,500 contre 0,667 en exactitude\n", + "couverte à 50 % d'escalade) **n'est pas concluant** — et le fait que le LLM gelé couvre mieux\n", + "à budget égal, là où on attendait l'inverse, est exactement le genre de surprise qu'un banc\n", + "didactique sert à rencontrer sans conclure. Le LLM gelé achète l'accès immédiat à tout domaine\n", + "nouveau sans entraînement ; le témoin classique achète des microsecondes par question, au prix\n", + "d'étiquettes à chaque extension du domaine.\n", + "\n", + "**Tranches suivantes de l'issue #17883** (chacune = une PR, déclarées ici pour ne pas les\n", + "substituer par un jouet) :\n", + "\n", + "| Famille | Représentant | Ce qu'elle ajoute |\n", + "|---|---|---|\n", + "| Encodeur + tête de décision | laya, Von (ModernBERT ~400 M) | le fine-tuning bat le zéro-shot… sur le banc d'entraînement |\n", + "| Adaptateur LoRA + tête | Kev (Qwen 0.8B/4B/9B) | sert le contrat HTTP tel quel |\n", + "| Entraîneur minimal CPU | jevlike | tête d'attention par option, contrôle « contexte mélangé » |\n", + "| **Banc externe** | banc indépendant multi-tâches | mesure de l'écart zéro-shot **hors domaine** (~26 points rapportés) |\n", + "\n", + "**Références** : issue #17883 (cadre complet) · #17878 (le volet entraînement, dans\n", + "`PostTraining`) · `03_Structured_Outputs` (la voie générative) · `13_Agentic_Orchestration` (où une\n", + "décision typée alimente l'agent)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.15" + }, + "papermill": { + "default_parameters": {}, + "duration": 18.755496, + "end_time": "2026-09-28T13:51:31.677949", + "environment_variables": {}, + "exception": null, + "input_path": "MyIA.AI.Notebooks/GenAI/Texte/03b_Typed_Decisions_System1.ipynb", + "output_path": "03b_Typed_Decisions_System1.ipynb", + "parameters": {}, + "start_time": "2026-09-28T13:51:12.922453", + "version": "2.6.0" + }, + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "state": { + "014b965eada742e892b77865d139eee8": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_8de54b7102e3436ab3c8231694bc6349", + "max": 290.0, + "min": 0.0, + "orientation": "horizontal", + "style": "IPY_MODEL_7bb33a7081744401b5c76af3b0713559", + "tabbable": null, + "tooltip": null, + "value": 290.0 + } + }, + "1916c79ae2c44770be9ca133f4678832": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_1c686532e0664062a41e34e9715c78c3", + "IPY_MODEL_014b965eada742e892b77865d139eee8", + "IPY_MODEL_2de03d35a20c4ef9ae4605cf4f069a03" + ], + "layout": "IPY_MODEL_cf48b202840140588a172bae95a7d23c", + "tabbable": null, + "tooltip": null + } + }, + "1c686532e0664062a41e34e9715c78c3": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HTMLView", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_c0b8e81b69ec4ea686e6aad3f0ffd0ec", + "placeholder": "​", + "style": "IPY_MODEL_9493c93319ef4ce5b7c827eecb0f07cf", + "tabbable": null, + "tooltip": null, + "value": "Loading weights: 100%" + } + }, + "2de03d35a20c4ef9ae4605cf4f069a03": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HTMLView", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_648cc8c5b4794c4abbf9de456175452f", + "placeholder": "​", + "style": "IPY_MODEL_a400a84ac52b40aea50298d97110d28c", + "tabbable": null, + "tooltip": null, + "value": " 290/290 [00:00<00:00, 6070.42it/s]" + } + }, + "648cc8c5b4794c4abbf9de456175452f": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "7bb33a7081744401b5c76af3b0713559": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "8de54b7102e3436ab3c8231694bc6349": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "9493c93319ef4ce5b7c827eecb0f07cf": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "background": null, + "description_width": "", + "font_size": null, + "text_color": null + } + }, + "a400a84ac52b40aea50298d97110d28c": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "background": null, + "description_width": "", + "font_size": null, + "text_color": null + } + }, + "c0b8e81b69ec4ea686e6aad3f0ffd0ec": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "cf48b202840140588a172bae95a7d23c": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + } + }, + "version_major": 2, + "version_minor": 0 + } + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/MyIA.AI.Notebooks/GenAI/Texte/README.md b/MyIA.AI.Notebooks/GenAI/Texte/README.md index af9fae92e3..d8079bab47 100644 --- a/MyIA.AI.Notebooks/GenAI/Texte/README.md +++ b/MyIA.AI.Notebooks/GenAI/Texte/README.md @@ -37,6 +37,7 @@ La maîtrise des LLMs constitue la pierre angulaire de toute expertise en Géné | # | Notebook | Description | Durée | |---|----------|-------------|-------| | 3 | `3_Structured_Outputs.ipynb` | JSON Schema, Pydantic, mode strict, extraction de données | 55 min | +| 3b | `3b_Typed_Decisions_System1.ipynb` | Décisions typées « système 1 » : contrat choice/score/noul, LLM gelé lu sur ses logits contre témoin classique, calibration (NLL, Brier, ECE, température), escalade | 50 min | | 4 | `4_Function_Calling.ipynb` | Tools API, appels parallèles, boucle agentique | 60 min | ### Tier 3 : Augmentation (Intermédiaire) From fd335d1d1e119093a1179fa42a3c733aefafb3dc Mon Sep 17 00:00:00 2001 From: jsboige Date: Mon, 28 Sep 2026 18:36:40 +0200 Subject: [PATCH 2/2] =?UTF-8?q?fix(genai,#17883):=20review=2018243=20--=20?= =?UTF-8?q?nav=2004=20vers=2003b,=20demo=20expr=C3=A8s=20chiffre=20align?= =?UTF-8?q?=C3=A9,=20coquille=20confiantes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 04_Function_Calling: Précédent pointe vers 03b (le Suivant de 03 y allait deja, la chaine remonte maintenant sans sauter 03b) - 03b cellule lecture demo: la mauvaise option (livraison pour carte refusée) est déclarée choisie exprès, confiance réelle 0,41 citée depuis la sortie, 0,9998 requalifié en plafond illustratif - 03b cellule 20: confiante -> confiantes Markdown uniquement, aucune cellule code touchée (pas de re-exec due, C.2) Co-Authored-By: Claude Sonnet 5 --- .../GenAI/Texte/03b_Typed_Decisions_System1.ipynb | 8 +++++--- MyIA.AI.Notebooks/GenAI/Texte/04_Function_Calling.ipynb | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/MyIA.AI.Notebooks/GenAI/Texte/03b_Typed_Decisions_System1.ipynb b/MyIA.AI.Notebooks/GenAI/Texte/03b_Typed_Decisions_System1.ipynb index 09b8901c07..88a799312f 100644 --- a/MyIA.AI.Notebooks/GenAI/Texte/03b_Typed_Decisions_System1.ipynb +++ b/MyIA.AI.Notebooks/GenAI/Texte/03b_Typed_Decisions_System1.ipynb @@ -221,8 +221,10 @@ "(une option mal orthographiée, une classe inventée, un JSON non conforme) est **impossible** —\n", "c'est la promesse exacte que `03_Structured_Outputs` achetait à coups de schéma et de parsing.\n", "\n", - "Mais l'inverse n'est pas vrai : l'espace fermé ne dit **rien** de l'exactitude. Une mauvaise\n", - "option peut sortir à 0,9998. La question utile n'est donc pas « la sortie est-elle bien formée ? »\n", + "Mais l'inverse n'est pas vrai : l'espace fermé ne dit **rien** de l'exactitude. La démo ci-dessus le fait **exprès** : les scores `[0.05, 0.10, 0.80, 0.05]`, choisis à la main,\n", + "font sortir « livraison » pour une carte bancaire refusée — une mauvaise option, affichée avec\n", + "sa confiance réelle de 0,41 (la sortie de la cellule précédente). Le même contrat laisserait\n", + "sortir une mauvaise option à 0,9998 sans que rien dans l'espace déclaré ne s'y oppose. La question utile n'est donc pas « la sortie est-elle bien formée ? »\n", "(toujours oui) mais « **la confiance affichée est-elle honnête ?** » — c'est une question de\n", "**calibration**, et elle occupera toute la seconde moitié de ce notebook." ] @@ -918,7 +920,7 @@ "quand, à la couverture visée, son exactitude couverte est au niveau acceptable, **et** que sa\n", "confiance est calibrée (sinon le seuil ne veut rien dire). Mesuré ici, et contre l'intuition :\n", "à 50 % d'escalade, c'est le LLM gelé qui couvre le mieux (0,667 contre 0,500) — ses erreurs\n", - "sont moins concentrées sur ses réponses les plus confiante. Sur 12 tickets de test, ce constat\n", + "sont moins concentrées sur ses réponses les plus confiantes. Sur 12 tickets de test, ce constat\n", "est une **observation à répliquer**, pas une conclusion. Le résidu au-delà du seuil n'est pas\n", "un échec du modèle : c'est le flux qu'on **route** délibérément vers l'humain ou le LLM\n", "génératif. Décider, ce n'est pas toujours répondre — c'est aussi savoir quand ne pas répondre." diff --git a/MyIA.AI.Notebooks/GenAI/Texte/04_Function_Calling.ipynb b/MyIA.AI.Notebooks/GenAI/Texte/04_Function_Calling.ipynb index 38465cd8f0..49b21cd322 100644 --- a/MyIA.AI.Notebooks/GenAI/Texte/04_Function_Calling.ipynb +++ b/MyIA.AI.Notebooks/GenAI/Texte/04_Function_Calling.ipynb @@ -42,7 +42,7 @@ "tags": [] }, "source": [ - "**Navigation** : [Index](README.md) | [<< Précédent](03_Structured_Outputs.ipynb) | [Suivant >>](05_RAG_Modern.ipynb)\n", + "**Navigation** : [Index](README.md) | [<< Précédent](03b_Typed_Decisions_System1.ipynb) | [Suivant >>](05_RAG_Modern.ipynb)\n", "\n", "# Function Calling : Connecter les LLMs au Monde Réel\n", "\n",