From 6ce0fd96f4217b1e5e4a70e195d2eb9038087bb0 Mon Sep 17 00:00:00 2001 From: jpramil Date: Tue, 26 May 2026 12:59:46 +0000 Subject: [PATCH 1/4] add onboarding replay --- index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.qmd b/index.qmd index edd3f81..115554b 100644 --- a/index.qmd +++ b/index.qmd @@ -81,7 +81,7 @@ Build a **retrieval-augmented generation** (RAG) system that combines a vector d Before diving into either part, follow these steps to set up your environment. The same setup applies to both parts. -If you need more details for setups, do not hesitate to visite [project 1 introduction page](https://aiml4os.github.io/funathon-project1/). +If you need more details for setups, do not hesitate to visite [project 1 introduction page](https://aiml4os.github.io/funathon-project1/) and/or watch the 2026's Funathon [Onboarding replay](https://aiml4os.github.io/funathon-general-website/onboarding.html). ## Fork this repository From 1dc5db1193fc72a2ccbe5a20687c1f1ce3c276f3 Mon Sep 17 00:00:00 2001 From: jpramil Date: Tue, 26 May 2026 13:07:00 +0000 Subject: [PATCH 2/4] remove confusing gitignore tip --- index.qmd | 4 ---- 1 file changed, 4 deletions(-) diff --git a/index.qmd b/index.qmd index 115554b..8188643 100644 --- a/index.qmd +++ b/index.qmd @@ -154,8 +154,4 @@ df["code"].value_counts().head(10).plot(kind="bar") A small "Run Cell" lens appears above each `# %%` marker. Click it to execute that block. You can also use the shortcut Shift + Enter. This is convenient for iterating on small pieces of code or testing specific functions from the project. -::: {.callout-tip} -Files created at the root of the repository for your own exploration (such as `scratch.py` or `notes.md`) can be safely ignored from git by adding them to your local `.gitignore`, or to `.git/info/exclude` if you don't want to commit the ignore rule. -::: - You're now ready to start. Head to [Part 1](1-ttc.qmd) or [Part 2](2-rag-intro.qmd). From 3f8c984129640d03405f215474c9b894650d0c28 Mon Sep 17 00:00:00 2001 From: jpramil Date: Tue, 26 May 2026 14:50:52 +0000 Subject: [PATCH 3/4] def tokenizer --- 1-ttc.qmd | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/1-ttc.qmd b/1-ttc.qmd index 9215746..754d16f 100644 --- a/1-ttc.qmd +++ b/1-ttc.qmd @@ -23,7 +23,7 @@ Two main approaches exist for this task: | Approach | How it works | Best when | |---|---|---| | **Supervised (this notebook)** | Train a neural network on labelled examples | Large labelled dataset available, low-latency inference needed | -| **RAG** (see notebook 2) | Retrieve similar examples, ask a LLM to decide | Few labelled examples, need for explainability via retrieved context | +| **RAG** (see notebook 2) | Retrieve similar examples, ask a LLM to decide | Few labelled examples | In this notebook, we use the supervised approach: the model learns directly from thousands of labelled examples and can predict at inference time without calling an external API. @@ -42,7 +42,7 @@ flowchart LR ## What you will learn - How to prepare and split a labelled text dataset -- How to train a subword tokenizer (WordPiece) on domain-specific text +- How to train a subword tokenizer (WordPiece, see @tip-tokenizer) on domain-specific text - How to configure and train a neural text classifier end-to-end - How to load a pretrained model from MLflow and run inference - How to interpret predictions using the [Captum](https://captum.ai/docs/introduction) package @@ -100,7 +100,7 @@ The notebook loads this file automatically at startup with `load_dotenv()`, maki # Load the data -The dataset consists of **synthetic labelled examples** generated for the NACE 2025 nomenclature. +The dataset consists of **synthetic labelled examples** generated for the NACE rev 2.1 nomenclature. Each row contains a short text description (`label`) and the corresponding NACE code (`code`). ::: {.callout-tip collapse="false" icon=false} @@ -122,7 +122,7 @@ load_dotenv(override=True) ## Question 2 — Load the dataset from s3 -Use Polars to load the parquet file directly from this public URL: +Use [Polars](https://docs.pola.rs/api/python/stable/reference/index.html) to load the parquet file directly from this public URL: ``` https://minio.lab.sspcloud.fr/projet-formation/diffusion/funathon/2026/project2/generation_None_temp08.parquet @@ -287,6 +287,12 @@ value_encoder = ValueEncoder(label_encoder=encoder) # Applying a tokenizer +::: {#tip-tokenizer .callout-tip collapse="true"} +## What is a tokenizer? + +A **tokenizer** converts raw text into a sequence of integers that a neural network can process. It works in three steps: it first **splits** the text into small units called **tokens** (words, subword fragments, or characters); it then **maps** each token to a unique integer using a fixed vocabulary; finally it **pads or truncates** the sequence to a uniform length so all inputs have the same shape. Because neural networks can only work with numbers, every text classification pipeline starts with this conversion step. +::: + ## Why tokenization? Neural networks can only work with numbers because they have no notion of letters or words. As a a consequence, models cannot use directly the string "I sell croissants and pains au chocolat" to predict a NACE code, it must be transformed into something the model can use. A **tokenizer** is the component that bridges this gap: it converts raw text into a sequence of integers that the model can process. From 80f9f8915288b0a67f3b1ba4917fc52723fdc84e Mon Sep 17 00:00:00 2001 From: jpramil Date: Tue, 26 May 2026 15:11:58 +0000 Subject: [PATCH 4/4] remove tokenizer callout --- 1-ttc.qmd | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/1-ttc.qmd b/1-ttc.qmd index d93e86c..459af9c 100644 --- a/1-ttc.qmd +++ b/1-ttc.qmd @@ -42,7 +42,7 @@ flowchart LR ## What you will learn - How to prepare and split a labelled text dataset -- How to train a subword tokenizer (WordPiece, see @tip-tokenizer) on domain-specific text +- How to train a subword tokenizer (WordPiece) on domain-specific text (tokenizers are introduced in @sec-tokenizer) - How to configure and train a neural text classifier end-to-end - How to load a pretrained model from MLflow and run inference - How to interpret predictions using the [Captum](https://captum.ai/docs/introduction) package @@ -286,13 +286,7 @@ value_encoder = ValueEncoder(label_encoder=encoder) ::: -# Applying a tokenizer - -::: {#tip-tokenizer .callout-tip collapse="true"} -## What is a tokenizer? - -A **tokenizer** converts raw text into a sequence of integers that a neural network can process. It works in three steps: it first **splits** the text into small units called **tokens** (words, subword fragments, or characters); it then **maps** each token to a unique integer using a fixed vocabulary; finally it **pads or truncates** the sequence to a uniform length so all inputs have the same shape. Because neural networks can only work with numbers, every text classification pipeline starts with this conversion step. -::: +# Applying a tokenizer {#sec-tokenizer} ## Why tokenization?