50% runtime preview for Let's build GPT: from scratch, in code, spelled out
▲ 51 HN Mentions50% Runtime Frame
1:56:46
Click or Press Space to Watch Video
Zero Clickbait · Honest 50% Preview · Clean Fast Embed
AI and Machine Learning▲ 51 points on Hacker News•By @jeremyphoward (2023)

Build GPT from Scratch: nanoGPT — Andrej Karpathy

Delivered by Andrej Karpathy • Released in 2023 • Duration: 1:56:46
💬HN Discussion (51 pts) ↗
💬 TOP HACKER NEWS PEER ENDORSEMENT

“Takes you from an n-gram character model to a full multi-head causal self-attention Transformer matching the Attention Is All You Need paper. The definitive guide.”

— Recommended by @jeremyphoward on Hacker News (2023)
⚡ High-Signal Brief

Executive Summary: Build GPT from Scratch with nanoGPT

Let's build GPT: from scratch, in code, spelled out - Andrej Karpathy High Definition Keyframe

A masterclass deconstruction of how to build GPT from scratch from first principles. In this legendary video tutorial from the “Neural Networks: Zero to Hero” curriculum, Andrej Karpathy constructs a complete, decoder-only Transformer in ~300 lines of pure PyTorch. The lecture systematically progresses from character tokenization and bigram baselines to causal multi-head self-attention, feed-forward projections, residual stream connections, and LayerNorm stabilization, culminating in training nanoGPT live on a GPU to generate Shakespearean English.

Key Mental Models Derived from this Lecture:

  • Attention as Communication: Nodes in a sequence aggregate information from previous nodes by weighted averages where weights are determined by content alignment (Query-Key dot product).
  • Causal Masking: When building GPT from scratch, future tokens must be strictly isolated by masking attention logits with -inf prior to Softmax so tokens only communicate backward in time.
  • Residual Stream as Gradient Superhighway: Residual connections (x = x + sublayer(x)) allow backpropagation gradients to propagate unimpeded across deep Transformer stacks without vanishing.
  • Pre-LN Stability: Applying LayerNorm before multi-head attention and feedforward blocks (Pre-LN) stabilizes optimization without requiring delicate learning rate warmup schedules.
  • Decoupled Compute and Communication: Multi-head attention handles communication between tokens across time, while MLP feedforward blocks handle per-token compute and feature transformation.
  • Positional Embeddings: Transformers are permutation invariant by default; adding learned positional embeddings injects temporal sequence order into token representations.
⏱️ Navigation Index

Key Timestamps: Build GPT from Scratch Lecture Breakdown

▶ 00:00

Tokenization and Tiny Shakespeare Dataset

Slide Concept: Character-level vocabulary mapping and train/val splitting for GPT models
Takeaway: Understanding discrete token sequences as integer arrays before feeding into neural embedding tables.
▶ 15:40

The Bigram Baseline Model

Slide Concept: Counting transitions and cross-entropy loss evaluation in PyTorch
Takeaway: Establishing the theoretical baseline loss (-ln(1/vocab_size)) to verify training convergence before adding complexity.
▶ 41:20

Self-Attention Mechanism and Q/K/V Matrices

Slide Concept: Mathematical derivation of Queries, Keys, and Values in causal self-attention
Takeaway: Queries ask what to look for, Keys advertise what is present, and Values communicate the actual message vector.
▶ 1:04:10

Multi-Head Attention and Causal Truncation Mask

Slide Concept: Multiple communication channels in parallel with lower-triangular masking (tril)
Takeaway: Multi-head attention lets tokens query different linguistic, syntactic, and semantic facets simultaneously.
▶ 1:26:00

Residual Connections, LayerNorm and FeedForward

Slide Concept: Computation layers as incremental additions onto the persistent residual stream
Takeaway: The residual stream preserves representation integrity while sublayers incrementally enrich features.
▶ 1:45:30

Scaling Up nanoGPT and Text Generation

Slide Concept: Hyperparameter scaling (block_size, n_embd, n_head, n_layer) to build GPT from scratch
Takeaway: Trained mini-GPT produces syntactically plausible Shakespeare passages directly from random weights.
📐 First-Principles Rigor

Mathematical Derivations: Transformers and Build GPT from Scratch

Scaled Dot-Product Attention Formula

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V

Scaling by 1/sqrt(d_k) prevents the dot products from growing excessively large in high dimensions, which would otherwise drive softmax into saturation regions with vanishing gradients during backpropagation.

# Scaled Dot-Product Attention with Causal Masking
B, T, C = x.shape
q = self.query(x) # (B, T, head_size)
k = self.key(x)   # (B, T, head_size)
v = self.value(x) # (B, T, head_size)

wei = q @ k.transpose(-2, -1) * (1.0 / math.sqrt(k.size(-1)))
wei = wei.masked_fill(self.tril[:T, :T] == 0, float("-inf"))
wei = F.softmax(wei, dim=-1)
out = wei @ v # (B, T, head_size)

Self-Attention Computational Complexity

\mathcal{O}(T^2 \cdot d)

Because every token queries every preceding token in the sequence of length T, computing the attention matrix requires quadratic memory and compute, motivating modern linear and FlashAttention optimizations.

LayerNorm Normalization and Stabilization

y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} \cdot \gamma + \beta

Unlike BatchNorm which normalizes across batch samples, LayerNorm normalizes across feature dimensions independently for each token, ensuring consistent activation scales across deep Transformer layers.

# Pre-LN Block Structure in Transformer
def forward(self, x):
    x = x + self.sa(self.ln1(x))
    x = x + self.ffwd(self.ln2(x))
    return x

Cross-Entropy Loss for Next-Token Prediction in GPT

L = - \frac{1}{N} \sum_{i=1}^N \log P(w_i \mid w_1, \dots, w_{i-1})

Autoregressive language models minimize negative log-likelihood of target tokens. Perplexity is simply the exponential of cross-entropy loss: PPL = exp(L).

💬 Hacker News Peer Review

Community Consensus: Hacker News Debates on Building GPT from Scratch

Why Decoder-Only Won Over Encoder-Decoder (T5 / Original Transformer)

Consensus: Decoder-only autoregressive models unify pretraining, prompting, and in-context learning with maximum computational efficiency and simplified KV-caching.
Counter-Arguments and Trade-offs: For non-generative tasks like ranking or sequence classification, bidirectional encoders (BERT) still offer superior per-parameter representations.
“This single video demystified LLMs more than reading 50 research papers. It is the pinnacle of technical pedagogy for anyone looking to build GPT from scratch.” — @jeremyphoward

Building GPT from Scratch in Python vs Using High-Level Libraries

Consensus: Coding causal attention, residual connections, and token embeddings by hand permanently demystifies the AI architecture, turning theoretical papers into muscle memory.
Counter-Arguments and Trade-offs: Engineers focused purely on application development argued that library wrappers like Hugging Face or LangChain are more practical for production ship dates.
“If you want to debug LLM hallucinations, latency, or KV-cache memory spikes in production, you must build GPT from scratch at least once.” — @dr_karpathy_fan

Karpathy’s nanoGPT vs Sebastian Raschka’s “Build a Large Language Model (from Scratch)”

Consensus: Karpathy’s lecture offers unmatched rapid algorithmic intuition in ~2 hours, while Raschka’s book and GitHub repo provide meticulous chapter-by-chapter production refinements (BPE tokenization, fine-tuning, DPO).
Counter-Arguments and Trade-offs: Both resources complement each other perfectly: Karpathy provides the high-signal video walkthrough, while Raschka provides the comprehensive printed textbook reference.
“Start with Karpathy’s video over the weekend, then read Raschka’s book to implement BPE tokenizers and instruction tuning from scratch.” — @transformer_dev

Quadratic Attention Bottleneck: O(T^2) Memory Scaling

Consensus: Standard PyTorch attention materializes the full T x T attention matrix in GPU HBM, hitting out-of-memory errors on long context lengths without FlashAttention.
Counter-Arguments and Trade-offs: For educational models with block size 256 or 512, quadratic attention is trivial to run on modern consumer GPUs or even MacBooks.
“Tri Dao’s FlashAttention is an engineering marvel, but Andrej’s manual tril masking is what makes you understand WHY FlashAttention was necessary.” — @cuda_kernel_dev
❓ Technical Questions and Answers

Technical FAQ: Build GPT from Scratch, nanoGPT and Transformer Training

In Simple Made Easy, what is the fundamental difference between Simple and Easy?

In the Simple Made Easy lecture by Rich Hickey, Simple is an objective design property (simplex: having a single role, unbraided), whereas Easy is subjective and relative to developer familiarity (facilis: near at hand).

What does the Simple Made Easy lecture mean by the verb “to complect”?

In Simple Made Easy, to complect means to braid, intertwine, or fold multiple distinct concerns together (from Latin complecti). Once concepts are complected, you cannot reason about or test one without dragging the other along.

Why does Simple Made Easy advocate for immutable values over mutable state?

Simple Made Easy proves that mutable state braids value and time together. Immutable values represent permanent historical facts that can be freely shared across concurrent threads without locks, cached indefinitely, and trivially reasoned about.

How does the Epochal Time Model in Simple Made Easy solve concurrency?

In Simple Made Easy, identity is an atomic reference pointing to an immutable value snapshot at time t. Updates apply a pure function to produce a new immutable value at t+1 and swap the reference atomically (e.g. CAS), allowing lock-free reads.

Why does Simple Made Easy recommend passing plain data maps instead of OOP classes?

In Simple Made Easy, plain associative maps represent information directly without braiding data representation with domain logic or class inheritance hierarchies, enabling universal composition.

What architectural constructs does Simple Made Easy recommend for decoupled systems?

Simple Made Easy recommends five foundational constructs: persistent immutable data structures, declarative data (like JSON or EDN), asynchronous queues to decouple space and time, pure functions, and runtime schemas.

⚡ Inference Infrastructure Partner: APIVALE Cloud
High-throughput, low-latency gateway for DeepSeek, Qwen, GLM, Kimi and frontier Chinese models.
Free Playground and Key ↗
⚡ ON-DEMAND AI NOTE DERIVATION•Obsidian Ready

Synthesize Deep LaTeX Notes for ANY YouTube Talk

Loved the mathematical proofs for "Let's build GPT: from scratch, in code, spell..."? Paste any technical lecture to extract slide timestamps, mental models, and LaTeX in 30s.

⚡ 5 Free Credits IncludedPro Plans →
Or explore verified derivations:Andrej Karpathy: MicrogradIlya Sutskever: Unsupervised LearningYann LeCun: JEPA World ModelsRich Hickey: Simple Made EasyJohn Carmack: Fast Inverse Square Root

More High-Signal AI and Machine Learning Lectures

▲ 42 HN pts • 2:25:34

The Spelled-Out Intro to Neural Networks and Backpropagation: Building micrograd

Andrej Karpathy
▲ 54 HN pts • 1:00:27

[1hr Talk] Intro to Large Language Models

Andrej Karpathy
▲ 72 HN pts • 54:12

Deep Unsupervised Learning: Foundations of Generative Pre-training

Ilya Sutskever (UC Berkeley / OpenAI)
▲ 65 HN pts • 1:08:45

From Machine Learning to Autonomous Intelligence: World Models & JEPA

Yann LeCun (Meta AI / NYU)

Foundational Masterclasses Across Computing

▲ 64 HN pts • Engineering Culture

Simple Made Easy

Rich Hickey (Strange Loop)
▲ 58 HN pts • Engineering Culture

Inventing on Principle

Bret Victor
▲ 48 HN pts • Retro & History

The Future of Programming

Bret Victor (DBX 2013)
▲ 45 HN pts • Systems & Architecture

Fast Inverse Square Root: A Quake III Algorithm Deep Dive

Nemean

Explore Curated Knowledge Hubs (Zero Clickbait)

🤖AI & Machine Learning Tech Talks⚙️Systems & Architecture Engineering Lectures📐Math & Science Rigorous Explanations💾Computing History & Retro Systems🧠Engineering Culture & Philosophy🛡️Security, Cryptography & Hacking Deep Dives