Latent Space — Notes on machine intelligence

From pixels to masks: build a baseline you can question.

Sun, 06 Sep 2026 00:00:00 +0000

Imagine inspecting a bright rectangular part against a dark background. Before training a segmentation network, we can ask a simpler question: does a brightness rule already locate the part? A baseline gives us something concrete to test, a way to discover data problems, and a reference a more expensive model should improve upon.

The rule will fail in two different ways. A bright speck outside the rectangle will be selected, and a dark pixel inside it will be missed. We will repair this particular example, then construct another where the same repair destroys a real object. That second result is as important as the first.

Images are arrays. The contract matters.

Sun, 06 Sep 2026 00:00:00 +0000

You load a photograph, run a model, and get a poor prediction. Is the model weak, or did your loader supply blue where the model expected red? Both produce a tensor of the expected shape. A successful function call does not prove that the data means what you think it means.

Our first goal is modest and useful: describe an image precisely enough that another function can interpret it correctly. We will use four pixels, follow one through conversion, and build a small batch. These are the same boundaries that exist in a large inference service.

IoU, Dice, and Hausdorff: what each metric misses.

Sun, 06 Sep 2026 00:00:00 +0000

A segmentation model scores 0.9863 Dice. Almost perfect? In the example below, it also predicts a stray foreground pixel more than four pixels from the target. Another prediction has a lower Dice of 0.8333, but its boundary is never more than one pixel away. Which would you prefer?

You cannot answer without knowing what the segmentation is for. Measuring area, separating objects, locating a precise boundary, and avoiding any distant false detection are different goals. A metric gives a particular summary of an error; it does not decide which errors matter.

RGB, BGR, and alpha: what a channel actually means.

Sun, 06 Sep 2026 00:00:00 +0000

A transparent logo looks correct in your browser, but develops black edges when you resize it for a model. A red object turns blue after loading. A preprocessing function returns three averages, yet you expected one value at every pixel. These failures can happen before a neural network sees a single example.

They share a cause: a shape tells us how many values exist, but not what those values represent. This lesson connects channel meaning to operations you can inspect. We will reuse the four pixels from lesson 1 , then introduce a separate four-pixel transparency experiment. Every diagram is generated from the same source used by the downloadable examples.

Attention, from the ground up.

Sat, 05 Sep 2026 00:00:00 +0000

We have a vector for every input position. Each vector starts with token and position information, but it cannot yet incorporate the rest of the sequence. Attention gives a position a way to gather information from other positions, with weights that depend on the current input.

The central operation is a weighted mixture. We will build that mixture without learned projections, calculate the weights by hand, then put the matrix equation and code around it. If you arrived here directly, the course starts with tokens and vectors .

Encoder, decoder, or both?

Sat, 05 Sep 2026 00:00:00 +0000

We can assemble the same ingredients in several ways. The words encoder and decoder are easiest to understand as descriptions of information flow and purpose, rather than as mysterious types of neural computation.

Ask three questions. What information is available when a prediction must be made? Does the output describe an existing input, or does it extend a sequence? If it generates a sequence, is there a separate source representation it should read?

From scores to a distribution.

Sat, 05 Sep 2026 00:00:00 +0000

A dot product can give scores such as [-2, 0, 3]. Those numbers cannot directly serve as our previous lesson’s weighted-average weights: one is negative, and the total is not one. We need a transformation that preserves which scores are larger while producing a distribution over the available choices.

Softmax does this in two stages. First turn scores into positive evidence. Then divide each evidence value by the total evidence.

Give the sequence a sense of order.

Sat, 05 Sep 2026 00:00:00 +0000

In lesson 1, robot received a vector by selecting one row of a table. The lookup does not know whether robot came first or last. Yet “robot follows person” and “person follows robot” describe different relationships.

We need to expose ordering information to the model. Before choosing a formula, let’s identify the information that is missing.

1. Content alone cannot label the positions

Imagine an operation that compares every row with every other row, mixes their values, and repeats the same computation at each position. If we rearrange the input rows, the operation can simply rearrange its outputs in the same way. There is no intrinsic “position 3” inside a shared matrix multiplication.

How numbers mix information.

Sat, 05 Sep 2026 00:00:00 +0000

Suppose three positions offer information. One contains [1, 0], another [0, 2], and the third [2, 1]. These are value vectors: each has two features. We want one receiving position to combine their information without losing the distinction between the two features.

This is the operation at the end of attention. We will start by choosing the weights ourselves. The following lesson will show how scores can become weights, and the attention course will show where those scores come from.

Several heads, one shared representation.

Sat, 05 Sep 2026 00:00:00 +0000

One attention head gives each query one set of weights over the source. A single weighted mixture may have to combine several useful relationships at once. Multi-head attention allows several learned projections to construct different mixtures before those results are combined.

The key word is learned. We do not assign one head to nouns, one to verbs, and one to position. Different behaviors may emerge, but a head is defined by its projections and computation, not by a guaranteed human-readable role.

The block around attention.

Sat, 05 Sep 2026 00:00:00 +0000

An attention layer computes useful mixtures, but a transformer diagram also contains boxes named Add, Norm, and Feed Forward. These operations are not decoration. They determine how information and gradients pass through the stack and how each position transforms its newly gathered context.

We will build the block used in our final model. Its input and output both have shape (B, T, D), which makes it possible to stack blocks without changing the representation width.

Train it. Test what it learned.

Sat, 05 Sep 2026 00:00:00 +0000

We have assembled the pieces. Now the model must learn a task. Given a source such as [4, 7, 9], we want it to generate [4, 7, 9, EOS]. We choose copying because the target is unambiguous, the data can be generated locally, and every stage of an encoder–decoder transformer is still exercised.

Download and unzip the complete code bundle . It contains the NumPy forward calculations, the PyTorch model, this training script, numerical checks, and a recorded example run. No dataset download or GPU is required.

What actually changes when a model learns?

Sat, 05 Sep 2026 00:00:00 +0000

Our first model has one parameter: a weight of 0.5. Given input 2, it multiplies by that weight and predicts 1. The desired target is 3. Learning will change the weight so that this example’s prediction moves closer to its target.

This tiny model has no attention, but the learning distinction is the same. A model computes outputs with its current parameters; a training procedure adjusts those parameters using an objective and data.

What is actually inside a tensor?

Sat, 05 Sep 2026 00:00:00 +0000

Before a model can attend to anything, it needs a numerical representation. “Tensor” sounds specialized, but our first tensor is just two rows of numbers. The important part is knowing what each axis means.

Imagine that each row describes an item, with two measured features. Our input is [[1, 2], [3, 4]]: two items and two features per item. We will transform each feature pair into a new pair. Nothing here requires language, a model, or a GPU.

Who is allowed to see what?

Sat, 05 Sep 2026 00:00:00 +0000

Attention tells us how to mix a set of values. It does not by itself decide which values a query should be permitted to use. That decision depends on the task.

A model classifying a complete sentence can read both ends of it. A model predicting the next output token cannot consult the answer it is being asked to predict. A model translating a source may read all of that source while seeing only the existing target prefix. We will express each case with two questions: where do the vectors come from, and which query–key pairs are visible?

Words become vectors.

Sat, 05 Sep 2026 00:00:00 +0000

We want to understand a machine that reads a sequence and produces another sequence. Eventually, ours will copy short sequences of symbols. Before it can read anything, we need to decide how symbols become numbers.

Consider “the small robot moves.” A computer can store that string, but multiplying character encodings does not automatically give useful language features. We will build a representation in stages. Keep the distinction between a symbol’s identity and its learned representation in mind; much later confusion disappears once these are separate.

DETR: object detection as set prediction.

Wed, 02 Sep 2026 00:00:00 +0000

A detector must answer two questions: what is in the image, and where is it? DETR makes a clean architectural choice: predict a fixed-size set of candidate objects, then match those candidates to the ground truth during training.

This note is a conceptual walkthrough, with a small matching example rather than a complete detector.

From pixels to a set

A CNN backbone produces a feature map. DETR flattens those spatial features into a sequence, adds positional information, and sends them through a transformer encoder. A decoder uses learned object queries to attend to that representation.

D-FINE: a better way to draw the box.

Sat, 29 Aug 2026 00:00:00 +0000

A bounding box often arrives as four numbers, which makes localization look like four ordinary regression tasks. But a partially occluded or blurry edge can be ambiguous. What if the model represented plausible positions as a distribution before producing a coordinate?

The idea behind D-FINE

D-FINE reframes box regression in DETR-based detection through fine-grained distribution refinement (FDR). Decoder layers iteratively refine localization distributions. Its second component, global optimal localization self-distillation (GO-LSD), transfers localization knowledge from stronger predictions within the model during training.

Linear attention: change the order.

Tue, 25 Aug 2026 00:00:00 +0000

Standard attention constructs pairwise interactions between tokens. As the sequence grows, that matrix grows quadratically. Kernel-based linear attention asks whether we can aggregate the keys and values first, then query the aggregate.

The answer involves changing the attention kernel. You cannot simply move parentheses through the softmax operation.

The associative trick

Replace softmax attention with a positive feature-map kernel $\phi(q)^\top\phi(k)$ . For one query, the normalized output is:

The KV cache, without the mystery.

Fri, 21 Aug 2026 00:00:00 +0000

When a causal language model generates another token, the earlier tokens have not changed. With consistent positional handling and inference settings, their keys and values do not need to be recomputed. A KV cache saves them for the next decoding step.

This note describes ordinary causal self-attention and a deliberately small cache. Real inference engines also manage memory allocation, batching, and varying sequence lengths.

Prefill and decode

During prefill, the model processes the prompt and stores each layer’s key and value tensors. During decode, the new token produces a new query, key, and value. Its query attends to the cached prefix plus the new key/value pair.

An image is a sequence of patches.

Mon, 17 Aug 2026 00:00:00 +0000

A transformer operates on token vectors. An image arrives as a grid. The bridge between them can be surprisingly simple: divide the image into non-overlapping patches, flatten each patch, and project it into an embedding space.

Count the tokens first

For an image with height H and width W, and square patches of side P, the number of patch tokens is:

N = \frac{H}{P}\frac{W}{P}.

READ THE EQUATION

Count patch rows and patch columns, then multiply them to get the number of image tokens.

A little less black box.