What actually changes when a model learns?

Our first model has one parameter: a weight of 0.5. Given input 2, it multiplies by that weight and predicts 1. The desired target is 3. Learning will change the weight so that this example’s prediction moves closer to its target.

This tiny model has no attention, but the learning distinction is the same. A model computes outputs with its current parameters; a training procedure adjusts those parameters using an objective and data.

1. Separate six things that often get mixed together

The training target is not secretly changed to match the prediction. The model’s parameter is changed to improve the objective. In a language model, a next-token target can be constructed from the text itself; it still plays the role of a target during the update.

2. Make a prediction and measure its error

The forward pass is this input-to-prediction-to-loss calculation. A larger model performs more operations, but it still produces intermediate values using its current parameters.

3. A derivative answers a local question

A derivative asks: if we change one quantity by a very small amount, how quickly does another quantity change? Here we want the loss’s rate of change with respect to the weight.

First, loss changes with error at a rate of twice the error. Second, error changes with the weight at a rate of x, because the prediction is w times x and the target stays fixed. The chain rule multiplies these local rates along the path.

A gradient collects such derivatives for all parameters. Our one-parameter gradient is one number. A weight matrix has a gradient with a matching shape: one local sensitivity for each weight entry.

4. Turn that slope into an update

5. Explicit arithmetic and automatic differentiation

def learn_step(weight: float, x: float, target: float, rate: float = 0.1) -> tuple[float, float, float]:
    """One squared-error gradient step for prediction = weight * x."""
    prediction = weight * x
    error = prediction - target
    loss = error ** 2
    gradient = 2.0 * error * x
    updated = weight - rate * gradient
    return updated, loss, gradient

def learn_step(weight: float, x: float, target: float, rate: float = 0.1) -> tuple[float, float, float]:
    parameter = torch.tensor(weight, dtype=torch.float64, requires_grad=True)
    prediction = parameter * x
    loss = (prediction - target) ** 2
    loss.backward()
    assert parameter.grad is not None
    gradient = parameter.grad.item()
    updated = parameter.item() - rate * gradient
    return updated, loss.item(), gradient

Typed Python · Shapes are documented alongside the code. Open the full file for imports and dependencies, or step through a concrete example.

The NumPy-side example uses scalar arithmetic to make the derivative explicit. It returns the updated weight, loss before the update, and gradient. All three are floats; the return annotation tuple[float, float, float] makes their structure visible. The loss does not retroactively change when we compute a new weight.

PyTorch creates a trainable scalar with requires_grad=True. Operations build a computation graph, and loss.backward() applies derivative rules backward through that graph. The gradient is available in parameter.grad. This is automatic differentiation, not a finite-difference approximation and not an automatic parameter update.

An optimizer performs the update. The simple rule above is gradient descent; the later Transformer example uses Adam, which maintains additional state from gradients to adjust updates. The separation still holds: clear old gradients, compute a loss, run backward, then update parameters. See PyTorch’s autograd explanation ↗ for how the graph is recorded.

6. What this means for a neural network

A neural network composes parameterized operations. A linear projection mixes features; an activation such as ReLU changes how combinations behave by replacing negative values with zero. Without intervening nonlinear operations, a stack of affine feature projections can be collapsed into a single affine projection. Extra boxes alone do not automatically create a more expressive computation.

In attention, embedding rows and projection matrices are parameters. Token representations, Q/K/V arrays, and attention weights are activations recalculated for each input. The loss gradient can flow through the weighted mixture, softmax, matching scores, and projections to change the stored parameters that produced those activations.

This explains why “learned attention” does not mean storing one permanent attention map. Training learns the transformations used to construct a map for a particular input. A new input can produce a different map with the same stored parameters.

7. Learning on examples is not proof of generalization

Training repeatedly uses a dataset to improve its objective. Evaluation asks whether the resulting behavior also works on examples that did not supply those updates. A low training loss can coexist with poor held-out performance.

During inference, we normally keep parameters fixed and compute predictions. We may build temporary state such as a KV cache; that state is not automatically a weight update. Evaluation mode and disabling gradient recording are also separate controls: one selects module behavior, the other avoids building a gradient graph.

You now have the missing foundation

You can read tensor axes, project features, combine values, turn scores into weights, and explain what learning updates. Continue to Attention & Transformers . When a diagram introduces a new box, ask which of these operations it combines and what new problem it solves.