Transformers are the foundational architecture behind the modern artificial intelligence revolution. Introduced in 2017, they discarded sequential processing in favor of the Self-Attention Mechanism, allowing models to process language in parallel and understand deep, long-range context. This architectural breakthrough is the engine powering Large Language Models (LLMs) like GPT, BERT, and LLaMA.
This handbook provides a mathematically rigorous, structurally complete breakdown of how Transformers operate from the ground up.
transformers-study-guide/
├── README.md
│ Complete theoretical study guide
└── Transformers.ipynb
PyTorch implementation notebook
- PART I — Foundations
- PART II — Core Transformer Architecture
- PART III — Scaling Transformers
- PART IV — Transformer Architectures
- PART V — Transformers in Modern LLMs
- PART VI — References
Establish a concrete understanding of the goals and scope of this text.
Upon completing this study guide, students will be able to:
- Mathematically derive the Scaled Dot-Product Attention mechanism.
- Trace exact, multidimensional tensor shapes through every stage of a Transformer layer.
- Implement a generative GPT-style Transformer purely in PyTorch.
- Analyze the computational constraints of
$O(N^2)$ scaling and the engineering solutions required for modern LLMs.
Not applicable for this section.
A student utilizing this guide will transition from treating ChatGPT as a "black box" to understanding the exact floating-point matrix multiplications occurring on the NVIDIA GPUs powering it.
Identify the foundational knowledge required to digest the mathematics in this handbook.
- Basic Python: Array manipulation and loops.
- Linear Algebra: Deep understanding of vectors, matrices, dot products, and matrix multiplication.
- Probability: Understanding distributions and exponential functions.
- Neural Networks: Familiarity with feed-forward networks, loss functions, and backpropagation.
Not applicable for this section.
To understand how an LLM predicts the next word, one must first understand how a basic neural network performs linear regression using linear algebra.
Standardize the dimensional variables used throughout all mathematical and code examples.
This text utilizes standard academic notation for Transformer tensor dimensions.
| Symbol | Meaning |
|---|---|
| B | Batch Size (Number of sequences processed simultaneously) |
| N | Sequence Length (Number of tokens in the sequence) |
| Embedding Dimension (Size of the vector representing a token) | |
| d_k | Key/Query Dimension (Size of a single attention head's projection) |
| d_v | Value Dimension (Usually equal to |
| H | Number of Attention Heads |
| V | Vocabulary Size (Total number of unique tokens in the tokenizer) |
A single batch of text entering a Transformer is mathematically represented as a 3D tensor: $$ X \in \mathbb{R}^{B \times N \times d_{model}} $$
If you process 32 sentences (
Visualize the macro-level data flow through a Generative Transformer.
The entire architecture is a strictly ordered sequence of mathematical transformations designed to extract semantic meaning and predict the next token.
graph TD
Input[Input Text] --> Tokenizer[Tokenizer]
Tokenizer --> TokenIDs[Token IDs]
TokenIDs --> Embed[Embedding Layer]
Embed --> Pos[Positional Encoding]
Pos --> Block
subgraph Transformer Blocks
Block --> MHA[Multi-Head Attention]
MHA --> AddNorm1[Residual Add & LayerNorm]
AddNorm1 --> FFN[Feed-Forward Network]
FFN --> AddNorm2[Residual Add & LayerNorm]
end
AddNorm2 --> Final[Final Hidden States]
Final --> Linear[Linear Layer]
Linear --> Softmax[Softmax]
Softmax --> Output[Output Token]
Not applicable for this section.
When a prompt is inputted into a model, the sequence is processed through the tokenization and embedding layers, passing through the stacked Transformer blocks, before the final linear layer outputs the next token.
Understand the mathematical entities—Vectors, Matrices, Dot Products, and Softmax—that act as the building blocks of neural operations.
Neural networks do not process text strings; they process multidimensional floating-point tensors.
- Vector: A 1D array of numbers. Represents a point or direction in space.
- Matrix: A 2D grid of numbers. Matrices act as linear transformations, projecting vectors from one dimension to another.
The Dot Product: A geometric measurement of similarity between two vectors of equal length. $$ A \cdot B = \sum_{i=1}^n a_i b_i $$
Softmax: Converts a vector of arbitrary raw numbers (logits) into a valid probability distribution. $$ \text{Softmax}(x_i) = \frac{\exp(x_i)}{\sum_{j} \exp(x_j)} $$
If two word vectors, King and Monarch, point in the same direction in a high-dimensional space, their dot product is highly positive, signaling to the network that they share semantic similarity.
Trace the historical progression of NLP algorithms to identify the critical sequential flaws that necessitated the invention of the Transformer.
- Bag of Words / One-Hot Encoding: Resulted in sparse, high-dimensional vectors that did not capture semantic similarity or token order.
- Word2Vec: Mapped words to dense vectors. However, vectors were static. "Bank" had one vector regardless of context.
- RNNs (Recurrent Neural Networks): Attempted to build context by processing text sequentially, passing a "hidden state" from left to right.
RNNs calculate the hidden state at time
When an RNN processes long sequences, the influence of early states (e.g.,
Understand how raw strings are converted into processable integer sequences.
Transformers cannot read text. Before entering the network, text is split into subwords using algorithms like Byte-Pair Encoding (BPE) and mapped to integer IDs from a fixed Vocabulary.
Tokenization is a discrete mapping function
The string "Transformers are great" might be split by a BPE tokenizer into 4 subwords: [Trans, formers, are, great] and converted to IDs: [4512, 891, 312, 908].
Understand the transformation of discrete tokens into dense vectors.
- Embedding Matrix: A massive lookup table where every vocabulary token has a dedicated floating-point vector.
- Dense Vector Representations: Unlike sparse One-Hot vectors, dense vectors pack semantic meaning into a continuous space.
- Semantic Similarity: Words with similar meanings exist geometrically closer to each other in this high-dimensional space.
The Embedding Matrix
Token IDs:
[101, 2057, 2293, 102]
↓
Embedding Vectors:
[
[0.12, -0.33, ...],
[1.21, 0.98, ...],
...
]
Because embeddings capture geometric semantics, they allow algebraic manipulation of concepts. The classic example in a well-trained embedding space is:
Vector(King) - Vector(Man) + Vector(Woman)
Understand the conceptual shift from the Encoder-Decoder bottleneck to dynamic Attention.
- The Bottleneck: Early translation models compressed an entire source sentence into a single vector. For long sentences, this was an impossible bottleneck.
- Attention (2014): A breakthrough allowing a Decoder to "look back" at all source hidden states and dynamically weight which word was most important at the current generation step.
Attention creates a context vector
While generating the French word "chat", the model calculates high attention weights for the English source words "The" and "cat", and near-zero weights for irrelevant words in the sentence.
Understand how the Transformer repurposed Attention to operate on a single sequence in parallel.
- Self-Attention: Every word evaluates its mathematical relevance to every other word in the same sequence, simultaneously.
- Contextual Representation: Words dynamically merge with surrounding context, altering their vector space to resolve ambiguity.
- Parallel Processing: The sequence is processed as a unified matrix, eliminating RNN sequential bottlenecks.
graph LR
X[Input Embeddings]
X --> Q[Query]
X --> K[Key]
X --> V[Value]
Q --> Score[QK^T]
K --> Score
Score --> Softmax
Softmax --> Weights
Weights --> Context
V --> Context
Context --> Output
Self-Attention mathematically relates the entire input sequence
In the sentence "The bank of the river", Self-Attention links the word "bank" heavily to "river", mathematically shifting the vector for "bank" away from "finance" and towards "nature".
Understand the three distinct linear projections that facilitate Self-Attention.
To allow tokens to communicate, the model generates three vectors for every token:
- Query (Q): What semantic information am I looking for?
- Key (K): What semantic information do I contain?
- Value (V): What is my actual underlying meaning?
Given an input embedding matrix
The Query-Key-Value paradigm operates analogously to database retrieval. A query vector (Q) is matched against key vectors (K) representing stored tokens. The resulting similarity coefficients weight the corresponding value vectors (V) to compile the output.
Mathematically derive the core formula of the Transformer architecture.
The formula executes the logic: match Queries to Keys to get a score, scale the score so gradients don't vanish, turn the scores into percentages, and extract the Values.
-
Dot Product (
$QK^T$ ): Multiplies the query matrix$Q$ by the transpose of the key matrix$K$ over the sequence and projection dimensions. This yields the raw attention scores of shape$(B, N, N)$ . -
Scaling (
$\sqrt{d_k}$ ): High dimensionality$d_k$ causes dot products to grow large in magnitude, pushing the Softmax function into regions with near-zero gradients. We divide by$\sqrt{d_k}$ to stabilize the variance of the logits. -
Softmax: Applied along the last dimension to normalize the scores into an
$(B, N, N)$ attention weight probability distribution. -
Value Aggregation (
$\times V$ ): Multiplies the attention weights by the value matrix$V$ , yielding the final contextualized outputs of shape$(B, N, d_v)$ .
If the sequence is "The cat", "cat" dots with "The" to score
Understand how splitting the attention mechanism isolates different linguistic features.
A single attention calculation averages out context. To capture complex grammar, the
- Head 1 finds verbs.
- Head 2 finds adjectives.
$$
\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}1, \dots, \text{head}H)W^O
$$
Where each head is:
$$
\text{head}i = \text{Attention}(Q W_i^Q, K W_i^K, V W_i^V)
$$
And the projection matrices are defined as:
$$
W_i^Q \in \mathbb{R}^{d{model} \times d_k}, \quad W_i^K \in \mathbb{R}^{d{model} \times d_k}, \quad W_i^V \in \mathbb{R}^{d{model} \times d_v}, \quad W^O \in \mathbb{R}^{H d_v \times d_{model}}
$$
In execution, input shape
In the sentence "The bark of the tree", Head A links "bark" to "tree" (identifying it as nature), while Head B ensures "The" properly modifies "bark" grammatically.
Understand the bridge mechanism between Encoder and Decoder modules.
In Encoder-Decoder models (like translation), the Decoder needs to look at the translated source text.
- Query: Comes from the Decoder's current state.
- Key & Value: Come from the Encoder's final output.
Command context maps words dynamically. When the Decoder generates the French word "chien", the Cross-Attention Query searches the English Encoder's Keys to find "dog", and extracts the English "dog" Value representation to guide the generation.
Understand how the Transformer overcomes its native inability to recognize sequence order.
Because Attention evaluates all tokens simultaneously via matrix multiplication, the model has absolutely no concept of word order.
- The Fix: Before entering the Attention layer, we add a mathematical "time-stamp" vector to the embedding vector. (Note: Modern LLMs replace static additions with Rotary Position Embeddings or RoPE).
The original paper used sine and cosine waves: $$ PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) $$ $$ PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) $$
Without positional encodings, "Dog bites man" and "Man bites dog" yield identical mathematical sums in the Attention layer. The encodings force "Dog at position 0" to be mathematically distinct from "Dog at position 2".
Understand the auxiliary components that stabilize the core MHA mechanism in deep networks.
- Residual Connections: Highways that add the original input to the output ($x + \text{Layer}(x)$) to prevent Vanishing Gradients.
- LayerNorm: Normalizes the data to a mean of 0 and variance of 1 to stabilize training. (Modern LLMs often prefer RMSNorm).
- Feed-Forward Network (FFN): Processes each token independently. Expands dimensionality by 4x to memorize factual knowledge. (Modern LLMs often prefer SwiGLU).
graph TD
Input --> LN1[LayerNorm]
LN1 --> MHA[Multi-Head Attention]
MHA --> Add1[Residual Add]
Add1 --> LN2[LayerNorm]
LN2 --> FFN[Feed-Forward Network]
FFN --> Add2[Residual Add]
Add2 --> Output
Pre-Norm formulation: $$ x = x + \text{MHA}(\text{LayerNorm}(x)) $$ $$ x = x + \text{FFN}(\text{LayerNorm}(x)) $$
Residual connections provide a direct path for gradients to propagate backward through the network without attenuation, mitigating the vanishing gradient problem in deep architectures.
Track exact dimensional changes through a block to bridge theory and code.
Let
-
Input IDs:
$(32, 512)$ -
Embeddings:
$(32, 512, 768)$ -
Q, K, V Projections:
$(32, 512, 768)$ -
Split into Heads:
$(32, 12, 512, 64)$ -
Attention Scores (
$QK^T$ ):$(32, 12, 512, 512)$ -
Attention Output (
$\times V$ ):$(32, 12, 512, 64)$ -
Concatenation:
$(32, 512, 768)$ -
FFN Expand:
$(32, 512, 3072)$ -
Final Output:
$(32, 512, 768)$ -
Logits:
$(32, 512, 50000)$
Because the tensor shape entering the block
Understand how tensors handle variable-length sequences in batches.
GPUs require perfect rectangular matrices (e.g., [PAD] token (integer 0).
We use a Padding Mask to force the Attention mechanism to ignore these zeros.
Before Softmax, the
If we don't mask the padding tokens, the word "cat" might accidentally pay 10% of its mathematical attention to blank empty space, corrupting the generation.
Mathematically understand the bottlenecks preventing Transformers from processing infinite text.
Self-Attention demands calculating the relationship between every token and every other token.
For sequence length
-
Time Complexity:
$O(N \cdot d_{model}^2 + N^2 \cdot d_{model})$ per block layer.- Linear projection projection calculations (
$Q, K, V, O$ ) scale as$O(N \cdot d_{model}^2)$ . - Self-attention matrix calculations (
$QK^T$ score product and the value aggregation step) scale quadratically as$O(N^2 \cdot d_{model})$ .
- Linear projection projection calculations (
-
Memory Complexity:
$O(N \cdot d_{model} + H \cdot N^2)$ per block layer, where$H$ is the number of heads.- Storing the activations for the sequence scales linearly as
$O(N \cdot d_{model})$ . - Materializing the attention similarity matrices in memory scales quadratically as
$O(H \cdot N^2)$ .
- Storing the activations for the sequence scales linearly as
A 1-Million token context window (Gemini 1.5 Pro) requires evaluating 1 Trillion relationships per attention head. Standard attention computations result in high activation memory footprints, exceeding GPU memory capacity and necessitating IO-aware algorithms like FlashAttention.
Analyze the architectural hacks used to bypass
Instead of full
- Longformer: Uses a sliding window (tokens only look at nearby neighbors) with global hubs.
- BigBird: Combines random attention, window attention, and global attention.
- Mistral / Mixtral: Uses Sliding Window Attention (SWA) and Grouped Query Attention (GQA) to drastically extend context limits without memory explosions.
SWA reduces complexity from
Mistral-7B can process entire books because it doesn't force word 100,000 to mathematically attend directly to word 1; it propagates context efficiently through sliding windows.
Understand a memory-saving trick for language models.
The input Embedding matrix and the output Logits linear layer have the exact same shape
$$ W_{embed} = W_{logits}^T $$ This assumes the output projection layer has no bias terms. By forcing the input representation projection and the output vocabulary prediction parameters to share weights, parameter footprints are reduced and the geometric learning behavior of token alignment is regularized.
For a vocab of
Understand the breakthrough techniques engineered for Trillion-parameter bounds.
- KV Cache: Caches previous Keys and Values in VRAM to prevent recalculating past tokens during generation.
-
FlashAttention: Tiles attention computations into ultra-fast SRAM, completely bypassing the slow
$O(N^2)$ HBM materialization. - MoE (Mixture of Experts): Slices the massive FFN into 8 smaller "Experts". A router sends tokens to only the top 2 experts, saving FLOPs.
-
LoRA (Low-Rank Adaptation): Fine-tuning by freezing the base model and training tiny
$A$ and$B$ decomposed matrices.
-
LoRA Decomposition:
$$
\Delta W \approx B \times A
$$
Where base model weights
$W \in \mathbb{R}^{d \times k}$ are frozen, and weight update parameter changes$\Delta W$ are approximated by low-rank matrices$B \in \mathbb{R}^{d \times r}$ and$A \in \mathbb{R}^{r \times k}$ where$r \ll \min(d, k)$ . Matrix$A$ is typically initialized using a zero-mean Gaussian distribution$\mathcal{N}(0, \sigma^2)$ , and matrix$B$ is initialized to 0, ensuring$\Delta W = 0$ at training onset. -
Rotary Position Embeddings (RoPE):
RoPE applies a rotation matrix to Query (
$Q$ ) and Key ($K$ ) projections along 2D coordinate slices of head dimensions. For a 2D slice$x = (x_1, x_2)^T$ at position$m$ , the transformation is defined as: $$ R_{\Theta, m}^2 \begin{pmatrix} x_1 \ x_2 \end{pmatrix} = \begin{pmatrix} \cos m\theta & -\sin m\theta \ \sin m\theta & \cos m\theta \end{pmatrix} \begin{pmatrix} x_1 \ x_2 \end{pmatrix} $$ This formulation encodes relative position directly into the dot product relation of keys and queries.
LoRA allows researchers to fine-tune a massive 70B parameter model on a single consumer GPU, as they only train 0.1% of the total weights.
Understand the bidirectional foundation of BERT.
The Encoder uses Unmasked Self-Attention. Word 3 looks at Word 1 (past) and Word 5 (future) simultaneously.
graph LR
Encoder --> Bidirectional[Bidirectional Attention]
Decoder --> Causal[Causal Attention]
Bidirectional --> Understanding
Causal --> Generation
The attention matrix
Used in Google Search algorithms. It reads the entire user query forwards and backwards to grasp perfect grammatical intent before returning search results.
Understand the autoregressive foundation of GPT.
The Decoder uses Masked Self-Attention.
During the
Modern conversational models (e.g., GPT-4) utilize Decoder-only architectures. They ingest a prompt, calculate token probabilities, autoregressively sample the next token, and append it back to the context window.
Quickly contrast model purposes.
| Model | Architecture | Purpose |
|---|---|---|
| Transformer | Encoder-Decoder | Translation |
| BERT | Encoder Only | Understanding / Search |
| GPT | Decoder Only | Autoregressive Generation |
Encoder-only models specialize in language understanding, Decoder-only models specialize in autoregressive text generation, and Encoder-Decoder models combine both capabilities for sequence-to-sequence tasks such as translation and summarization.
Not applicable for this section.
You cannot use BERT to write an essay (it cannot generate text autoregressively). You cannot use GPT for optimal sentence classification (it cannot look backwards from the end of the sentence).
Understand the mathematical pipeline of training an LLM.
- Forward Pass: Pass dataset through model to output next-word probabilities.
- Cross Entropy Loss: Mathematically penalizes wrong predictions.
- Backpropagation: The Chain Rule calculates gradients backwards.
- Gradient Descent (AdamW): Optimizes the weights to minimize loss.
Cross Entropy Loss over a target token distribution $p$ and predicted token distribution $q$ is defined as:
$$
H(p,q)
-\sum_{x} p(x)\log q(x)
$$
For a single training prediction where the target token is index
If the correct next word is "Paris",
Understand probabilistic text generation.
-
Temperature (
$T$ ): Scales logits.$T>1$ increases randomness.$T \rightarrow 0$ makes it greedy. -
Top-P (Nucleus): Samples from tokens whose cumulative probability equals
$P$ .
Temperature scaling: $$ p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} $$
Setting
Understand how large models learn without gradient updates.
Massive LLMs exhibit Few-Shot Learning. By providing examples in the prompt, the Attention mechanism dynamically maps the pattern into the activation space without permanently changing the
Not applicable for this section.
Prompting: "English: Cat
Understand the multi-stage pipeline required to build a conversational AI.
- Pretraining: Massive internet scraping. Next-Token prediction.
- SFT: Supervised Fine-Tuning. High-quality Q&A data.
- RLHF / DPO: Alignment to human preferences.
graph LR
Pretraining --> SFT
SFT --> RLHF
RLHF --> DPO
DPO --> Assistant
Not applicable for this section.
Pretraining establishes general language modeling and factual knowledge representations, SFT aligns the output format to instruction/response conventions, and RLHF aligns model behavior with safety and utility guidelines.
Visualize the unbroken data flow of a generation loop.
Autoregressive generation is an infinite loop terminating on an [EOS] token.
graph TD
Prompt --> Tokenization
Tokenization --> Embeddings
Embeddings --> TransformerLayers
TransformerLayers --> Logits
Logits --> Softmax
Softmax --> Sampling
Sampling --> GeneratedToken
GeneratedToken --> Prompt
Not applicable for this section.
To generate a 10-word sentence, the 96-layer matrix operation executes 10 times, appending one word per loop.
Identify the lineage of modern open-source LLMs.
- LLaMA (Meta): SwiGLU activations, RoPE, RMSNorm. A widely adopted family of open-source foundation models.
- Mistral: Sliding Window Attention, MoE architectures.
- T5 (Google): Classic Encoder-Decoder fine-tuned for text-to-text formatting.
Not applicable for this section.
If you want to build a RAG application locally, you download an architecture derived from the LLaMA family from HuggingFace, because it performs strongly across standardized benchmarks.
Understand how the Transformer generalizes beyond text to vision and audio.
Because Transformers process arbitrary
-
Vision Transformer (ViT): Chops an image into
$16 \times 16$ pixel patches. Each patch is flattened into a 1D vector and projected into$d_{model}$ . - CLIP: Aligns text embeddings and image embeddings into the exact same vector space.
For a Vision Transformer (ViT), we process an image mathematically:
$$
\text{Image} \in \mathbb{R}^{H \times W \times C}
$$
It is chopped into flattened patches:
$$
\text{Patches} \in \mathbb{R}^{N_{patches}\times(P^2C)}
$$
It is then linearly projected into the embedding space:
$$
\text{Embeddings} \in \mathbb{R}^{N_{patches}\times d_{model}}
$$
where
$$
N_{patches} = \frac{HW}{P^2}
$$
This transforms a 2D image matrix into the exact
This patching mechanism allows multimodal models to ingest an image, project it to the token embedding space, and process it through the Self-Attention mechanism alongside text tokens.
The foundational academic literature underlying the engineering of modern LLMs.
- Vaswani et al., "Attention Is All You Need", 2017.
- Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", 2018.
- Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2), 2019.
- Brown et al., "Language Models are Few-Shot Learners" (GPT-3), 2020.
- Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models", 2021.
- Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness", 2022.
- Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (T5), 2019.
- Touvron et al., "LLaMA: Open and Efficient Foundation Language Models", 2023.
- Kudo & Richardson, "SentencePiece: A simple and language independent subword tokenizer", 2018.
- Wu et al., "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation", 2016.
- Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (ViT), 2020.
- Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (CLIP), 2021.
- Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO), 2023.
- Press et al., "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (ALiBi), 2021.
- Peng et al., "YaRN: Efficient Context Window Extension of Large Language Models", 2023.
- Zhang & Sennrich, "Root Mean Square Layer Normalization" (RMSNorm), 2019.
- Shazeer, "GLU Variants Improve Transformer" (SwiGLU), 2020.
- Beltagy et al., "Longformer: The Long-Document Transformer", 2020.
- Jiang et al., "Mistral 7B", 2023.