Cover of Inside nanoGPT, line by line
The library

Foundation Models · Book 02 of 04

Inside nanoGPT, line by line

A source-pinned, tensor-shape-driven walkthrough of Andrej Karpathy's nanoGPT, from LayerNorm and causal attention through training, checkpoints, and generation.

For ML-literate readers who want to turn Transformer theory into working PyTorch code.

Reading time
133 minutes
Contents
19 chapters + field guide
Verified
August 4, 2026
Read online Download EPUB 3.3 MB Markdown source ↘

A guided route through the book

Inside the book.

  1. 01 Open the file
  2. 02 Map the whole model
  3. 03 LayerNorm and causal attention
  4. 04 The MLP and Transformer block
  5. 05 Construct and run GPT
  6. 06 Train, checkpoint, and generate
  7. 07 Debug and modify the baseline

The complete learning path

In this series.

Book 01 · PreviousInside a Transformer Book 03 · NextPractical Model Training