Attention & transformers, built from first principles.

You can memorize softmax(Q @ K.T) @ V and still feel lost inside an architecture diagram. This course follows the information instead: what enters a model, how each operation changes it, and why those operations belong together.

Our destination is a small encoder–decoder transformer that learns to copy sequences. It is deliberately modest. A controlled task makes it possible to test causality, padding, shapes, and learning without downloading a dataset or hiding the model behind a high-level transformer wrapper.

What you need before starting

Begin with the four-lesson foundations course if tensor axes, matrix products, weighted means, probability, or gradients are unfamiliar. It uses the same values and typed functions we build on here. The goal is to make these operations recognizable before they appear inside an architecture.

Ready to skip ahead? You should be able to explain why a (3, 4) matrix times a (4, 2) matrix produces (3, 2), why [0.2, 0.3, 0.5] can mix three value vectors, and why a gradient update changes a parameter rather than a training target. If any answer feels uncertain, the foundations are the shortest route to a solid understanding.

Each lesson has a prerequisite, a concrete goal, a visual explanation, implementation details, and a question to check your reasoning. Work through them in order on your first pass. The contents panel jumps within a lesson; site links stay in this tab; external references open a new tab.

How to use the code

Download the complete course code bundle , unzip it, and run the commands in its README.txt. You can also open the individual NumPy forward passes , PyTorch model , training experiment , and verification checks .

python -m pip install numpy torch
python numpy_core.py
python torch_core.py
python verify.py
python train_copy.py --steps 800 --seed 7

The displayed functions come directly from those files. NumPy makes the forward calculations visible. PyTorch performs the same tensor operations and provides automatic differentiation for training. Later snippets reuse functions introduced earlier; run the full files when you want all dependencies together.

The code uses B for batch size, S for source length, T for target length, D for model width, h for head count, and V for vocabulary size. Within an attention equation, the uppercase matrix V instead denotes values; the surrounding shape makes the distinction explicit.

The reading behind this course

The architectural starting point is Attention Is All You Need ↗ , especially §§3.1–3.5. We also use Bahdanau et al. ↗ for attention’s earlier encoder–decoder context, Layer Normalization ↗ , BERT ↗ , and the pre-norm analysis ↗ . Each lesson links to the sources relevant to its claims and to official API documentation where implementation conventions matter.

Our explanations, numerical examples, diagrams, and code are original teaching material. The final model uses pre-norm, one block per stack, and no dropout. The 2017 paper uses a different normalization order and training recipe. We will make that distinction explicit when we assemble the blocks.