Give the sequence a sense of order.

In lesson 1, robot received a vector by selecting one row of a table. The lookup does not know whether robot came first or last. Yet “robot follows person” and “person follows robot” describe different relationships.

We need to expose ordering information to the model. Before choosing a formula, let’s identify the information that is missing.

1. Content alone cannot label the positions

Imagine an operation that compares every row with every other row, mixes their values, and repeats the same computation at each position. If we rearrange the input rows, the operation can simply rearrange its outputs in the same way. There is no intrinsic “position 3” inside a shared matrix multiplication.

For unmasked self-attention without positional information, this is permutation equivariance. If P reorders rows, then:

We will verify that property once we implement attention. Equivariant means the outputs follow the reordering. It does not mean the output tensor is unchanged; that stronger property would be invariance. Causal masking introduces a positional structure of its own, so the unrestricted permutation claim does not apply to arbitrary permutations of a causally masked sequence.

Even if the final output preserves row order, a token’s content-only attention calculation does not know an explicit position label. We supply one.

2. Add a position vector to a token vector

For every position t, construct a vector \(p_t\) with the same D features as the token embedding. The model input becomes:

This is elementwise addition, not concatenation. The model width stays D. The result combines two signals in a shared vector space; the model learns how to use that combination. Addition does not provide a general guarantee that either signal can be uniquely recovered in isolation.

If we reorder the sentence and then assign position 0 to its new first token, we have changed the association between token and position. If we instead move a complete (token, position) pair together, we have preserved that association. These are different experiments.

3. Sinusoids at several frequencies

A useful fixed encoding assigns sine and cosine to pairs of features:

Here, t is the integer position and i indexes a pair of coordinates. With D = 8, there are four frequency pairs. Some coordinates vary rapidly across neighboring positions; others change more slowly. A single sine would repeat too quickly to provide the range of patterns available from several frequencies together.

At position zero, every sine coordinate is zero and every cosine coordinate is one. At later positions, each pair moves around a circle at its assigned rate. This also means the positional vector’s squared norm is D/2 in exact arithmetic, because every sine/cosine pair contributes one.

The sinusoidal construction is specified in §3.5 of Attention Is All You Need ↗ . Being able to evaluate the formula at a new position does not guarantee a trained model will perform well at a longer context length.

def positions(length: int, width: int, start: int = 0) -> FloatArray:
    if width <= 0 or width % 2:
        raise ValueError("This example uses a positive, even model width")
    pos = np.arange(start, start + length, dtype=float)[:, None]
    frequency = 10000.0 ** (-np.arange(0, width, 2) / width)
    angles = pos * frequency[None, :]
    pe = np.empty((length, width))
    pe[:, 0::2] = np.sin(angles)
    pe[:, 1::2] = np.cos(angles)
    return pe

def positions(length: int, width: int, start: int = 0, device: torch.device | str | None = None, dtype: torch.dtype = torch.float32) -> Tensor:
    if width <= 0 or width % 2:
        raise ValueError("This example uses a positive, even model width")
    pos = torch.arange(start, start + length, device=device, dtype=dtype)[:, None]
    frequency = 10000.0 ** (-torch.arange(0, width, 2, device=device, dtype=dtype) / width)
    angles = pos * frequency[None, :]
    pe = torch.empty(length, width, device=device, dtype=dtype)
    pe[:, 0::2] = torch.sin(angles)
    pe[:, 1::2] = torch.cos(angles)
    return pe

Typed Python · Shapes are documented alongside the code. Open the full file for imports and dependencies, or step through a concrete example.

The functions intentionally require an even model width, so every sine has a cosine partner. Broadcasting multiplies a (length, 1) column of positions by a (1, D/2) row of frequencies. The result contains all position–frequency combinations at once.

4. A small experiment

Keep numpy_core.py in the working directory, then run:

import numpy as np
from numpy_core import positions

pe = positions(4, 8)
np.testing.assert_allclose(pe[0, 0::2], 0)
np.testing.assert_allclose(pe[0, 1::2], 1)
np.testing.assert_allclose((pe ** 2).sum(axis=-1), 4)

robot = np.arange(8, dtype=float) / 8
at_zero = robot + pe[0]
at_three = robot + pe[3]
assert not np.allclose(at_zero, at_three)
print(pe.shape)  # (4, 8)

The token vector stayed the same. Its input representation changed because its position changed. That is the property we needed; we have not yet performed any interaction between tokens.

The start argument matters during incremental generation. A newly processed token after a prefix of length 10 belongs at position 10, not at position 0. Our final, deliberately simple decoder recomputes the full prefix, so it assigns positions from zero each time. A cached implementation processing only new tokens must supply the correct offset or position IDs.

5. Learned positions and rotary positions

Learned absolute positions use another trainable table indexed by position. They are conceptually like token embeddings, except the row index means “where” rather than “which token.” A finite table has a supported range; extending that range requires a deliberate adaptation.

Rotary positional embeddings, introduced in RoFormer ↗ , act on query and key coordinate pairs through position-dependent rotations. Their dot products can express relative offsets. This differs from simply adding a vector to the input. We are not implementing RoPE in this course: one positional mechanism is enough to build our first complete model, and mixing the two formulas would obscure which mechanism is doing what.

The shared question is: how can a compatibility calculation depend on position as well as content? Different positional methods answer that question differently. They are architectural choices, not extra attention heads.

Carry this forward

Our tensor is now a sequence of token-plus-position vectors, shape (B, T, D). The next operation must allow one position to gather information from others. We will build it as four visible steps: compatibility scores, scaling, normalized weights, and a weighted sum.