Build GPT from Scratch: nanoGPT — Andrej Karpathy
“Takes you from an n-gram character model to a full multi-head causal self-attention Transformer matching the Attention Is All You Need paper. The definitive guide.”
Executive Summary: Build GPT from Scratch with nanoGPT
A masterclass deconstruction of how to build GPT from scratch from first principles. In this legendary video tutorial from the “Neural Networks: Zero to Hero” curriculum, Andrej Karpathy constructs a complete, decoder-only Transformer in ~300 lines of pure PyTorch. The lecture systematically progresses from character tokenization and bigram baselines to causal multi-head self-attention, feed-forward projections, residual stream connections, and LayerNorm stabilization, culminating in training nanoGPT live on a GPU to generate Shakespearean English.
Key Mental Models Derived from this Lecture:
- Attention as Communication: Nodes in a sequence aggregate information from previous nodes by weighted averages where weights are determined by content alignment (Query-Key dot product).
- Causal Masking: When building GPT from scratch, future tokens must be strictly isolated by masking attention logits with -inf prior to Softmax so tokens only communicate backward in time.
- Residual Stream as Gradient Superhighway: Residual connections (x = x + sublayer(x)) allow backpropagation gradients to propagate unimpeded across deep Transformer stacks without vanishing.
- Pre-LN Stability: Applying LayerNorm before multi-head attention and feedforward blocks (Pre-LN) stabilizes optimization without requiring delicate learning rate warmup schedules.
- Decoupled Compute and Communication: Multi-head attention handles communication between tokens across time, while MLP feedforward blocks handle per-token compute and feature transformation.
- Positional Embeddings: Transformers are permutation invariant by default; adding learned positional embeddings injects temporal sequence order into token representations.
Key Timestamps: Build GPT from Scratch Lecture Breakdown
Tokenization and Tiny Shakespeare Dataset
The Bigram Baseline Model
Self-Attention Mechanism and Q/K/V Matrices
Multi-Head Attention and Causal Truncation Mask
Residual Connections, LayerNorm and FeedForward
Scaling Up nanoGPT and Text Generation
Mathematical Derivations: Transformers and Build GPT from Scratch
Scaled Dot-Product Attention Formula
Scaling by 1/sqrt(d_k) prevents the dot products from growing excessively large in high dimensions, which would otherwise drive softmax into saturation regions with vanishing gradients during backpropagation.
# Scaled Dot-Product Attention with Causal Masking
B, T, C = x.shape
q = self.query(x) # (B, T, head_size)
k = self.key(x) # (B, T, head_size)
v = self.value(x) # (B, T, head_size)
wei = q @ k.transpose(-2, -1) * (1.0 / math.sqrt(k.size(-1)))
wei = wei.masked_fill(self.tril[:T, :T] == 0, float("-inf"))
wei = F.softmax(wei, dim=-1)
out = wei @ v # (B, T, head_size)Self-Attention Computational Complexity
Because every token queries every preceding token in the sequence of length T, computing the attention matrix requires quadratic memory and compute, motivating modern linear and FlashAttention optimizations.
LayerNorm Normalization and Stabilization
Unlike BatchNorm which normalizes across batch samples, LayerNorm normalizes across feature dimensions independently for each token, ensuring consistent activation scales across deep Transformer layers.
# Pre-LN Block Structure in Transformer
def forward(self, x):
x = x + self.sa(self.ln1(x))
x = x + self.ffwd(self.ln2(x))
return xCross-Entropy Loss for Next-Token Prediction in GPT
Autoregressive language models minimize negative log-likelihood of target tokens. Perplexity is simply the exponential of cross-entropy loss: PPL = exp(L).
Community Consensus: Hacker News Debates on Building GPT from Scratch
Why Decoder-Only Won Over Encoder-Decoder (T5 / Original Transformer)
“This single video demystified LLMs more than reading 50 research papers. It is the pinnacle of technical pedagogy for anyone looking to build GPT from scratch.” — @jeremyphoward
Building GPT from Scratch in Python vs Using High-Level Libraries
“If you want to debug LLM hallucinations, latency, or KV-cache memory spikes in production, you must build GPT from scratch at least once.” — @dr_karpathy_fan
Karpathy’s nanoGPT vs Sebastian Raschka’s “Build a Large Language Model (from Scratch)”
“Start with Karpathy’s video over the weekend, then read Raschka’s book to implement BPE tokenizers and instruction tuning from scratch.” — @transformer_dev
Quadratic Attention Bottleneck: O(T^2) Memory Scaling
“Tri Dao’s FlashAttention is an engineering marvel, but Andrej’s manual tril masking is what makes you understand WHY FlashAttention was necessary.” — @cuda_kernel_dev
Technical FAQ: Build GPT from Scratch, nanoGPT and Transformer Training
In Simple Made Easy, what is the fundamental difference between Simple and Easy?
In the Simple Made Easy lecture by Rich Hickey, Simple is an objective design property (simplex: having a single role, unbraided), whereas Easy is subjective and relative to developer familiarity (facilis: near at hand).
What does the Simple Made Easy lecture mean by the verb “to complect”?
In Simple Made Easy, to complect means to braid, intertwine, or fold multiple distinct concerns together (from Latin complecti). Once concepts are complected, you cannot reason about or test one without dragging the other along.
Why does Simple Made Easy advocate for immutable values over mutable state?
Simple Made Easy proves that mutable state braids value and time together. Immutable values represent permanent historical facts that can be freely shared across concurrent threads without locks, cached indefinitely, and trivially reasoned about.
How does the Epochal Time Model in Simple Made Easy solve concurrency?
In Simple Made Easy, identity is an atomic reference pointing to an immutable value snapshot at time t. Updates apply a pure function to produce a new immutable value at t+1 and swap the reference atomically (e.g. CAS), allowing lock-free reads.
Why does Simple Made Easy recommend passing plain data maps instead of OOP classes?
In Simple Made Easy, plain associative maps represent information directly without braiding data representation with domain logic or class inheritance hierarchies, enabling universal composition.
What architectural constructs does Simple Made Easy recommend for decoupled systems?
Simple Made Easy recommends five foundational constructs: persistent immutable data structures, declarative data (like JSON or EDN), asynchronous queues to decouple space and time, pure functions, and runtime schemas.
