What is actually inside a tensor?

Before a model can attend to anything, it needs a numerical representation. “Tensor” sounds specialized, but our first tensor is just two rows of numbers. The important part is knowing what each axis means.

Imagine that each row describes an item, with two measured features. Our input is [[1, 2], [3, 4]]: two items and two features per item. We will transform each feature pair into a new pair. Nothing here requires language, a model, or a GPU.

1. Values, axes, shapes, and dtypes

A scalar is one number, such as 2.0. A vector is an ordered list of numbers, such as [1.0, 2.0]. A matrix is a rectangular arrangement of rows and columns. A tensor generalizes this idea to any number of axes.

The shape says how many entries there are along each axis. Our matrix has shape (2, 2). The first 2 counts items; the second counts features. These two axes happen to have equal lengths, but they do different jobs. If we add a third item, the shape becomes (3, 2), not (2, 3).

An index selects a location. Python starts counting at zero: x[0, 1] means row zero, column one, whose value is 2. x[0] selects the entire first row. x[:, 1] selects column one from every row; : means “take everything on this axis.”

The dtype says how each value is represented: an integer, a floating-point value, or a Boolean. Shape and dtype are different properties. A (3, 4) Boolean mask and a (3, 4) floating-point score matrix have the same arrangement and different meanings.

In neural-network code, (B, T, D) commonly means batch examples, token positions, and feature coordinates. A batch collects independent examples so they can be processed together. It does not mean that one sentence should attend to a different sentence in the same batch.

2. A projection creates new features

Choose a weight matrix [[1, 0], [0.5, 2]]. Its rows correspond to input features; its columns correspond to output features. For input row [1, 2], the first output feature is 1×1 + 2×0.5 = 2. The second is 1×0 + 2×2 = 4. A bias [0, 1] shifts that result to [2, 5].

We can describe every output cell using one instruction:

The compact code is x @ weight + bias. If x has shape (T, D) and the weight has shape (D, H), the result is (T, H). D is the axis that is combined and summed out. T items remain; each now has H features. A projection can increase, decrease, or preserve the feature count.

3. The operation is not elementwise multiplication

x * weight multiplies corresponding cells when their shapes are compatible. x @ weight takes row–column dot products. Equal-looking inputs do not make those operations interchangeable.

A transpose swaps axes. A matrix with shape (T, D) becomes (D, T) after .T. It changes which values are rows and which are columns; it does not sort the values. Later, attention transposes keys so each query row can be compared with every key row.

Reshape changes how existing values are grouped without choosing a new order of axes. Transpose and reshape therefore solve different problems. With many axes, write down what each axis means before rearranging it. A wrong transpose can preserve the total number of values and still mix the wrong positions.

4. Broadcasting reuses a compatible value

Our bias has shape (2,), but the matrix output has shape (2, 2). Addition reuses the bias across rows. This is broadcasting. Compare shapes from the right: dimensions must match, or one must be 1; missing leading dimensions act like 1. The resulting operation is as if the smaller values were repeated along compatible axes.

weights[:, None] changes a (3,) vector into (3, 1). The inserted singleton axis makes it clear that each row weight should be reused across feature columns. The next lesson uses exactly this operation. See the NumPy broadcasting guide ↗ for the library’s rules.

5. Read a typed function before its body

def project(x: FloatArray, weight: FloatArray, bias: FloatArray) -> FloatArray:
    """x: (tokens, input_features), weight: (input_features, output_features)."""
    if x.ndim != 2 or weight.ndim != 2 or x.shape[1] != weight.shape[0]:
        raise ValueError("The input feature axis must match the weight's row axis")
    products = x @ weight
    output = products + bias  # (output_features,) broadcasts over tokens.
    return output

def project(x: Tensor, weight: Tensor, bias: Tensor) -> Tensor:
    """x: (tokens, input_features), weight: (input_features, output_features)."""
    if x.ndim != 2 or weight.ndim != 2 or x.shape[1] != weight.shape[0]:
        raise ValueError("The input feature axis must match the weight's row axis")
    products = x @ weight
    output = products + bias
    return output

Typed Python · Shapes are documented alongside the code. Open the full file for imports and dependencies, or step through a concrete example.

x: FloatArray names the input and its type. The colon introduces a type annotation. -> FloatArray declares the returned type. Our alias is NDArray[np.float64]: a NumPy array whose values use float64. It does not encode exact lengths. The docstring and validation explain the shape contract.

The PyTorch version uses Tensor; its shape and dtype still need to be understood from the surrounding contract. Type hints are checked by tools and read by people; Python does not turn them into automatic runtime validation. NumPy’s typing reference ↗ explains the distinction.

Use Visualize line by line. The actual example produces [[2, 5], [5, 9]]. Observe the intermediate products before the bias is added. These are two different operations, even though a model diagram may draw them as one box.

Carry this forward

A learned projection is a feature transformation whose weights come from training. We have used chosen weights so we can inspect the arithmetic. In attention, Q, K, and V are produced by learned projections. Before learning those weights, we need to understand what it means to combine information from several positions.