An image is a sequence of patches.
Turning pixels into tokens, and following the shapes into a vision transformer.
On this page Explore the sections +
A transformer operates on token vectors. An image arrives as a grid. The bridge between them can be surprisingly simple: divide the image into non-overlapping patches, flatten each patch, and project it into an embedding space.
Count the tokens first
For an image with height H and width W, and square patches of side P, the number of patch tokens is:
$$ N = \frac{H}{P}\frac{W}{P}. $$Count patch rows and patch columns, then multiply them to get the number of image tokens.
| Term | What it is and does | What it controls |
|---|---|---|
| N | Number of non-overlapping image patches. | The token count before any extra class or special tokens. |
| H, W | Image height and width, in pixels. | Larger images create more patches at fixed patch size. |
| P | Side length of a square patch, in pixels. | Larger patches reduce the number of tokens and increase pixels per patch. |
| H/P, W/P | Counts of patch rows and columns. The product counts all grid cells. | Requires both image dimensions to be divisible by P in this simple formula. |
Check: A 224×224 image with 16×16 patches has 14 rows and 14 columns: 196 tokens. Halving P to 8 gives 784 tokens, four times as many.
This assumes both image dimensions are divisible by P. Each flattened RGB patch has \(P^2 \cdot 3\) elements. A learned matrix maps those elements to an embedding of dimension D.
A 224 × 224 image with 16 × 16 patches yields 196 patch tokens. The original Vision Transformer adds a classification token, producing 197 tokens for its encoder.
Patchify with NumPy
The subtle part is the transpose. A reshape alone does not generally group the pixels into the intended spatial patches. First split height and width into patch-grid and within-patch axes; then bring the grid axes together.
import numpy as np
from numpy.typing import NDArray
def patchify(image: NDArray[np.float64], patch_size: int) -> NDArray[np.float64]:
h, w, c = image.shape
p = patch_size
if p <= 0 or h % p or w % p:
raise ValueError("Patch size must divide height and width")
grid = image.reshape(h // p, p, w // p, p, c)
patches = grid.transpose(0, 2, 1, 3, 4)
return patches.reshape((h // p) * (w // p), p * p * c)
image = np.arange(224 * 224 * 3, dtype=np.float64).reshape(224, 224, 3)
patches = patchify(image, patch_size=16)
assert patches.shape == (196, 768)
np.testing.assert_array_equal(
patches[0], image[:16, :16, :].reshape(-1)
)
np.testing.assert_array_equal(
patches[1], image[:16, 16:32, :].reshape(-1)
)
print(patches.shape) # (196, 768)Next, multiply the patches by a learned matrix of shape (768, D), add positional embeddings, and feed the sequence to a transformer. A convolution with kernel size and stride both equal to P can implement the same learned patch projection with a suitable weight layout.
Why position still matters
Attention without positional information has no built-in knowledge that one patch was above another. Positional embeddings inject spatial ordering into the tokens. When changing image resolution, the number of patches changes, so learned positional embeddings may need interpolation or another adaptation.
Smaller patches, bigger bill
Halving the patch side length quadruples the number of tokens. An explicitly materialized full attention matrix then has sixteen times as many entries. Efficient attention implementations can avoid storing that entire matrix, but the pairwise computation remains a concern.
Further reading
See An Image is Worth 16×16 Words for the original Vision Transformer. Then return to attention from scratch and substitute patch tokens for word tokens.