Words become vectors.

We want to understand a machine that reads a sequence and produces another sequence. Eventually, ours will copy short sequences of symbols. Before it can read anything, we need to decide how symbols become numbers.

Consider “the small robot moves.” A computer can store that string, but multiplying character encodings does not automatically give useful language features. We will build a representation in stages. Keep the distinction between a symbol’s identity and its learned representation in mind; much later confusion disappears once these are separate.

1. Tokens are the units, not necessarily words

If arrays or learning updates still feel unfamiliar, start with the foundations course . This lesson now applies those operations to text rather than introducing several mathematical ideas at once.

A tokenizer converts raw input into a sequence of units. A teaching tokenizer can split on spaces. That gives the, small, robot, moves. A real text tokenizer may use word pieces, bytes, punctuation, or other units. One visible word can occupy several token positions.

A vocabulary is a finite collection of supported token types and an assignment of integer IDs to them. If robot has ID 5, that does not mean it is halfway between token 4 and token 6. The IDs are addresses, not semantic coordinates. Renumber the vocabulary and permute the corresponding lookup rows consistently, and the represented input stays the same.

Special tokens express conventions the model needs. In our code, PAD marks an unused batch position, BOS starts an output, and EOS ends one. An UNK token is useful for our tiny text demonstration; byte-based or other production tokenizers can have different unknown-token behavior. These conventions belong to the tokenizer and training setup, not to attention itself.

Subword methods help handle an open-ended set of words with a finite inventory. Sennrich et al. ↗ study this motivation in neural translation. We will keep tokenization deliberately simple so it does not obscure the tensor operations.

2. The minimum useful linear algebra

A scalar is one number. A vector is an ordered list of numbers, and a matrix is a rectangular collection of them. A tensor generalizes this to any number of axes. In code, shape is your map of those axes.

If one token has six features, its vector has shape (6,). Four token vectors make a matrix of shape (4, 6). Two such sequences form a tensor (2, 4, 6): batch, position, feature. A batch lets us process independent examples together; it is not extra context that one example may attend to.

Two operations will recur:

The dot product multiplies matching coordinates and sums them into one number. For example, [1, 2] · [3, 4] = 11. Matrix multiplication applies many such dot products. If X is (T, D) and W is (D, H), Y is (T, H). Each input row becomes H learned combinations of its D features.

The inner dimensions must match. The outer dimensions describe the output. The expression X @ W is not elementwise multiplication X * W. And a transpose changes which axis is being matched; it does not magically fix an incorrect data layout.

3. An embedding is a learned lookup table

Let the vocabulary contain M tokens and the model width be D. An embedding table is a matrix \(E\in\mathbb{R}^{M\times D}\). The representation of token ID i is simply row \(E_i\).

Read that notation as “E is a real-valued matrix with M rows and D columns.” The symbol ∈ means “belongs to,” ℝ denotes real numbers, and the superscript M×D describes the shape rather than raising E to a power. A table row is a vector; selecting it does not involve multiplying the integer ID by its contents.

For our four-token sentence, indexing E with four IDs creates a (4, D) matrix. It does not create a (4, M) vector of vocabulary probabilities. Input embeddings and output probabilities play different roles.

You could represent an ID with a one-hot vector containing a single 1, then multiply by E. Indexing the row computes the same result without constructing that large one-hot vector. PyTorch’s Embedding ↗ implements such a lookup as a trainable module.

def embedding_demo() -> tuple[IntArray, FloatArray, FloatArray]:
    vocab = {"<pad>": 0, "<bos>": 1, "<eos>": 2, "the": 3,
             "small": 4, "robot": 5, "moves": 6, "<unk>": 7}
    tokens = "the small robot moves".split()  # Teaching tokenizer only.
    ids = np.array([vocab.get(token, vocab["<unk>"]) for token in tokens])
    table = np.random.default_rng(0).normal(size=(len(vocab), 6))
    x = table[ids]                           # (4, 6), no arithmetic on IDs.
    assert np.array_equal(x[2], table[vocab["robot"]])
    return ids, table, x

def embedding_demo() -> tuple[Tensor, nn.Embedding, Tensor]:
    vocab = {"<pad>": 0, "<bos>": 1, "<eos>": 2, "the": 3,
             "small": 4, "robot": 5, "moves": 6, "<unk>": 7}
    tokens = "the small robot moves".split()
    ids = torch.tensor([vocab.get(token, vocab["<unk>"]) for token in tokens])
    # Match the NumPy weights exactly, rather than assuming RNGs match.
    weights = torch.from_numpy(np.random.default_rng(0).normal(size=(len(vocab), 6)))
    table = nn.Embedding.from_pretrained(weights, freeze=False)
    x = table(ids)
    torch.testing.assert_close(x[2], table.weight[vocab["robot"]])
    return ids, table, x

Typed Python · Shapes are documented alongside the code. Open the full file for imports and dependencies, or step through a concrete example.

Run embedding_demo(). Both versions return IDs and a (4, 6) representation. The PyTorch example copies the same initial values as NumPy so the comparison is meaningful. Setting unrelated random generators to the same seed would not guarantee identical numbers.

These numbers are random initially. Their geometry becomes useful through training: the loss changes the table and the layers that consume it. A random embedding plot is not evidence that a model understands anything.

4. What does “latent space” actually mean?

Before assigning a meaning to a feature axis, distinguish the representation from the original observation. A word spelling is observed. The model’s feature coordinates are internal quantities learned because they help its objective. “Latent” refers to that internal representation, not to a hidden physical place or a promise that every axis has a name.

For example, the word “bank” can occur near “river” or near “loan.” Its initial embedding row is the same in both cases, but contextual operations can produce different representations for those occurrences. We do not hard-code a “river meaning” axis. The task’s gradients train transformations that can preserve distinctions useful for predictions.

Latent means the representation is internal rather than directly supplied as a labeled property of the input. A token embedding, a contextual hidden vector, or an image representation can live in a learned feature space. In this course, “latent space” is a descriptive phrase, not a claim that each hidden vector is a probabilistic latent variable from a generative model.

Do not imagine one coordinate permanently means “robotness” and another means “motion.” Features can be distributed across dimensions. Their interpretation also depends on the learned transformations that use them. A two-dimensional diagram is a teaching projection, not a literal view of all the model’s dimensions.

The word bank starts from the same lookup row whenever its token ID is the same. After context-sensitive layers, its hidden state in “river bank” can differ from its hidden state in “bank account.” The embedding supplies a starting point; the network computes an updated representation conditioned on available context.

Nor must an encoder compress a sentence into one vector. A transformer encoder usually produces a sequence of hidden vectors. Whether we pool that sequence into one vector depends on the task and output head.

5. A representation is not a prediction

Suppose our hidden vector has D = 32 features but the output vocabulary has M = 12 symbols. A learned output matrix maps 32 features to 12 logits, one score per possible output symbol. Softmax can turn those logits into probabilities. That final choice among symbols is different from the attention weights we will compute over input positions.

When you see a tensor called x, ask which of these it is. Variable names alone are not enough. Keep a shape comment beside each boundary in your code.

What we have built, and what is missing

We can now turn an input into a matrix. But each token was looked up independently, and nothing in the lookup says which position it occupied. The next lesson adds an explicit representation of position. Then attention will allow positions to exchange information.

Try changing robot to an unknown token in the demonstration, then inspect the selected row. Next, duplicate robot in two positions: the lookup vectors are identical before any positional signal is added. That observation sets up the next problem.