How numbers mix information.

Suppose three positions offer information. One contains [1, 0], another [0, 2], and the third [2, 1]. These are value vectors: each has two features. We want one receiving position to combine their information without losing the distinction between the two features.

This is the operation at the end of attention. We will start by choosing the weights ourselves. The following lesson will show how scores can become weights, and the attention course will show where those scores come from.

1. A sum is not a concatenation

Adding [1, 0] and [0, 2] gives [1, 2]: corresponding features are combined, and the output still has two features. Concatenating them gives [1, 0, 0, 2]: both vectors are kept side by side, and the feature count doubles.

Attention uses a weighted sum of values within each head. Multi-head attention later concatenates the outputs of several heads. Those verbs describe different changes in shape. Keep them separate from the start.

2. Weights control each source’s contribution

Choose weights [0.2, 0.3, 0.5]. Multiply the first value by 0.2, the second by 0.3, and the third by 0.5. The contributions are [0.2, 0], [0, 0.6], and [1, 0.5]. Adding those rows gives [1.2, 1.1].

For a weighted average, the weights are nonnegative and sum to one. Ordinary means are the special case of equal weights. Our weights satisfy those conditions. A general weighted sum does not have to, so use the more specific phrase only when its conditions hold.

With nonnegative normalized weights, each output coordinate lies between the smallest and largest corresponding input coordinates. Our first coordinates are 1, 0, and 2; their mixture 1.2 lies between 0 and 2. This is a useful sanity check, not a claim that later projected or residual outputs have the same bounds.

3. A weight alone does not describe an effect

The third source has the largest weight, but the second still contributes 0.6 to feature two because its value there is 2. A large weight attached to an all-zero value contributes nothing. Negative value coordinates can subtract from an output even though the weights are positive.

This is why an attention heatmap is only part of the computation. The heatmap shows which weights were assigned; the values determine what those weights carry. Output projections and later network operations can transform the mixture again.

If all values are identical, every normalized choice of weights gives the same output. Changing a weight matters only in relation to the available values and the normalization of the remaining weights.

4. Matching is a separate operation

A dot product multiplies matching vector coordinates and adds the products. For [1, 0] and [1, 1], it is 1×1 + 0×1 = 1. For [1, 0] and [0, 1], it is zero. This turns two feature vectors into one score.

The score can depend on both direction and magnitude. Doubling one vector doubles its dot product with the other. A dot product is therefore not automatically a normalized measure of similarity, and it is not a probability: it can be negative or larger than one.

Later, attention uses a query–key dot product for matching and a weighted value sum for mixing. A query describes the receiving position’s matching features. A key supplies matching features for a source. A value supplies the content to combine. They may be projected from the same underlying representation and still play different computational roles.

5. Implement the mixture before hiding it in a matrix product

def weighted_mix(weights: FloatArray, values: FloatArray) -> FloatArray:
    """weights: (sources,), values: (sources, features); result: (features,)."""
    if np.any(weights < 0) or not np.isclose(weights.sum(), 1.0):
        raise ValueError("Use nonnegative weights summing to one")
    contributions = weights[:, None] * values
    output = contributions.sum(axis=0)
    return output

def weighted_mix(weights: Tensor, values: Tensor) -> Tensor:
    """weights: (sources,), values: (sources, features); result: (features,)."""
    if torch.any(weights < 0) or not torch.isclose(weights.sum(), weights.new_tensor(1.0)):
        raise ValueError("Use nonnegative weights summing to one")
    contributions = weights[:, None] * values
    output = contributions.sum(dim=0)
    return output

Typed Python · Shapes are documented alongside the code. Open the full file for imports and dependencies, or step through a concrete example.

weights[:, None] has shape (3, 1). Multiplying it by (3, 2) values produces (3, 2) contributions. sum(axis=0) in NumPy, or sum(dim=0) in PyTorch, removes the source axis by addition. Summing on the last axis instead would combine features inside each source and produce the wrong kind of result.

Once you recognize the operation, weights @ values is a shorter way to compute the same mixture. The expanded implementation is useful because Visualize can show every scaled contribution before they are combined.

6. Several receivers mean several weight rows

One receiver has one weight per source. Two receivers need two rows of weights, because they may want different mixtures of the same source values. If weights have shape (T, S) and values have shape (S, D), their matrix product has shape (T, D): one D-feature output for each of T receivers.

The weights need not be symmetric. Receiver A may rely heavily on B while B relies on another source. Attention describes directed information flow, not an undirected friendship between tokens.

Carry this forward

We know what attention weights do, but chosen weights do not adapt to an input. To make a changing mixture, we need a way to convert arbitrary scores into nonnegative weights that sum to one. That conversion is the next lesson.