Understand how language models work, then train your own small GPT.
For programmers and students with some math background and little or no machine learning experience. Follow the path from RNNs to transformers through interactive examples, then complete a training project in Python.
Start Learning →“What I cannot create, I do not understand.” — Richard Feynman
What You Will Build
By the end of the Transformer chapter, you’ll train a small GPT from random weights, inspect training and validation loss, save and reload it, and generate text. The training lab includes a small text corpus and runnable PyTorch code. Its first job is learning patterns in that corpus; a useful chat assistant takes more data and training.
The five chapters follow the path from RNNs to attention to transformers. Detailed derivations and implementations sit in expandable sections. After the main project, you can try fine-tuning an open model as an optional lab.
Prerequisites
No ML experience required. We introduce loss, gradients, and parameter updates before the RNN chapters. You can follow the visual explanations before opening the equations. The training lab uses Python 3.12+ and PyTorch, with a small CPU configuration to start.