DETR: object detection as set prediction.

A detector must answer two questions: what is in the image, and where is it? DETR makes a clean architectural choice: predict a fixed-size set of candidate objects, then match those candidates to the ground truth during training.

This note is a conceptual walkthrough, with a small matching example rather than a complete detector.

From pixels to a set

A CNN backbone produces a feature map. DETR flattens those spatial features into a sequence, adds positional information, and sends them through a transformer encoder. A decoder uses learned object queries to attend to that representation.

Each decoder slot predicts a class distribution and a bounding box. Unused slots predict a special no-object class. In the original DETR, the resulting set prediction design removes the need for anchor generation and non-maximum suppression.

FIG. 002 — Image features enter a transformer; object queries decode a fixed-size set of predictions. Empty slots predict no object.

The ordering problem

Suppose an image contains a cat and a chair. Predicting “cat, chair” should be just as valid as predicting “chair, cat.” Ground-truth objects have no meaningful sequence order.

Bipartite matching finds a one-to-one assignment between targets and prediction slots. The assignment minimizes a cost that combines classification and box quality:

$$ \hat{\sigma} = \underset{\sigma}{\arg\min} \sum_{i=1}^{M}\mathcal{C}(y_i,\hat{y}_{\sigma(i)}). $$

READ THE EQUATION

Among valid one-to-one assignments, choose the assignment with the lowest total target–prediction matching cost.

Term	What it is and does	What it controls
$\sigma,\hat\sigma$	A candidate assignment and the selected best assignment. σ maps each target index to a distinct prediction slot.	Which prediction is paired with each target.
$\arg\min_\sigma$	Return the assignment attaining the smallest cost, not just the cost number.	Enforces a global choice across all pairings.
$M,i,\sum_{i=1}^{M}$	Number of target entries, target index, and sum of their matching costs.	Every included target contributes once; padding/no-object treatment depends on the full setup.
$y_i,\hat y_{\sigma(i)}$	Target i and the prediction assigned to it. Hats indicate predictions.	The two objects being compared.
$\mathcal C$	Pairwise matching cost, combining the chosen class/box criteria.	Determines which assignments are preferred; its relative terms affect the assignment.

Check: For pairwise costs [[4,1],[2,5]], assignment [0,1] costs 9 while [1,0] costs 3. Choose [1,0]. Picking a best prediction separately for each target could reuse a slot and violate one-to-one matching.

Here, M is the number of actual objects and there are at least M prediction slots. The loss trains the matched slots on their assigned objects and unmatched slots toward no-object.

A tiny matching experiment

For two targets and three predictions, we can enumerate every assignment. Rows are targets; columns are prediction slots. Lower is better. This toy example demonstrates the assignment only; it does not calculate DETR’s class, L1, or generalized IoU costs.

python
from itertools import permutations
import numpy as np

#              slot 0  slot 1  slot 2
cost = np.array([[0.8,  0.1,    0.5],   # cat
                 [0.2,  0.9,    0.4]])  # chair

assignments = permutations(range(cost.shape[1]), cost.shape[0])
best = min(
    assignments,
    key=lambda slots: sum(cost[i, j] for i, j in enumerate(slots)),
)

assert best == (1, 0)
print(best)  # cat -> slot 1, chair -> slot 0
# Slot 2 is supervised as no-object.

Open complete file

Enumeration grows factorially and is appropriate only for this tiny illustration. A real implementation uses a linear assignment solver such as the Hungarian algorithm.

What to inspect when debugging

Check box conventions first. A normalized center-format box $(c_x,c_y,w,h)$ is not interchangeable with corner coordinates $(x_1,y_1,x_2,y_2)$. An incorrect conversion can quietly poison both matching and regression losses.

Then inspect how empty images are handled, how no-object classification is weighted, and whether image padding is masked. These details affect training even when the transformer dimensions are all correct.

Term	What it is and does	What it controls
\(\sigma,\hat\sigma\)	A candidate assignment and the selected best assignment. σ maps each target index to a distinct prediction slot.	Which prediction is paired with each target.
\(\arg\min_\sigma\)	Return the assignment attaining the smallest cost, not just the cost number.	Enforces a global choice across all pairings.
\(M,i,\sum_{i=1}^{M}\)	Number of target entries, target index, and sum of their matching costs.	Every included target contributes once; padding/no-object treatment depends on the full setup.
\(y_i,\hat y_{\sigma(i)}\)	Target i and the prediction assigned to it. Hats indicate predictions.	The two objects being compared.
\(\mathcal C\)	Pairwise matching cost, combining the chosen class/box criteria.	Determines which assignments are preferred; its relative terms affect the assignment.

From pixels to a set

The ordering problem

A tiny matching experiment

What to inspect when debugging

Further reading