IoU, Dice, and Hausdorff: what each metric misses.

A segmentation model scores 0.9863 Dice. Almost perfect? In the example below, it also predicts a stray foreground pixel more than four pixels from the target. Another prediction has a lower Dice of 0.8333, but its boundary is never more than one pixel away. Which would you prefer?

You cannot answer without knowing what the segmentation is for. Measuring area, separating objects, locating a precise boundary, and avoiding any distant false detection are different goals. A metric gives a particular summary of an error; it does not decide which errors matter.

We will build three metrics from Boolean masks, then test cases chosen to expose their differences. The code, figures, and table use the same deterministic arrays. There is no trained model or benchmark claim hiding behind these numbers.

1. Establish what is being compared

A binary mask is a two-dimensional Boolean array. True means foreground and False background. The prediction and target must describe the same spatial grid. If one is cropped, resized, rotated, or shifted relative to the other, resolve that transform before computing a score.

This article assumes basic array indexing and masks. The two-lesson vision starter builds those concepts through image representation and thresholding. In particular, it distinguishes an independently defined target from a target generated by the algorithm being evaluated.

For now, both masks have shape (12,12). Our target contains a 6×6 square at rows and columns 3 through 8. We treat foreground locations as sets: P is the set of predicted foreground pixels, G the set of target foreground pixels. Set membership concerns locations, not intensity values.

An intersection contains pixels present in both sets. A union contains pixels present in either set, counted once. Vertical bars around a set mean its number of members, not absolute intensity. Thus, the overlap of a correctly aligned square with itself has 36 members in its intersection and 36 in its union.

2. Inspect the failure before reducing it to a number

Use Next to compare the exact match, a one-column shift, a single distant outlier, a disjoint prediction, and two empty-mask cases. The plus and minus marks make errors distinguishable without relying on color alone.

A false positive is in P but not G: a plus-marked extra prediction. A false negative is in G but not P: a minus-marked missed target. Shared foreground pixels are true positives. Shared background pixels are true negatives, but they do not enter the foreground IoU or Dice formulas below.

Leaving true negatives out is deliberate. An enormous correctly predicted background should not drown out a small object failure. It also means these overlap scores do not by themselves evaluate how well an image-level “nothing present” decision is made. Empty images need explicit attention later.

3. IoU asks how much of the combined region agrees

Intersection over Union, or IoU, divides shared foreground area by the area occupied by either mask. It is also called the Jaccard index.

IoU penalizes over- and under-segmentation through the same denominator. But it does not tell you which happened. Keep false-positive and false-negative counts when the distinction affects the application.

It also has no notion of distance. Move a disjoint prediction from just outside the target to the opposite corner: the IoU remains zero. The score tells us there is no overlap, not how far the miss is from becoming useful.

4. Dice reweights overlap; it is not independent evidence

The binary Dice similarity coefficient divides twice the shared area by the sum of the two foreground areas. Shared pixels are counted once in each mask’s area, which is why the numerator is doubled.

For the same binary pair, Dice and IoU are deterministically related:

Reporting both can help readers familiar with different conventions, but the two numbers do not independently establish boundary quality. For one mask pair, a higher Dice always corresponds to a higher IoU under the same definitions. Aggregate rankings can differ when you average transformed per-image scores.

def overlap(pred: Mask, target: Mask) -> tuple[float, float]:
    """Return IoU, Dice; both empty=1, exactly one empty=0."""
    validate_pair(pred, target)
    intersection: int = int(np.count_nonzero(pred & target))
    union: int = int(np.count_nonzero(pred | target))
    total: int = int(np.count_nonzero(pred)) + int(np.count_nonzero(target))
    if union == 0:
        return 1.0, 1.0
    iou: float = intersection / union
    dice: float = 2.0 * intersection / total
    return iou, dice

The helper validate_pair checks Boolean dtype, matching nonempty grids, and two dimensions. “Nonempty grid” means the image has locations; it does not require any foreground. We use count_nonzero to count decisions, not raw byte intensities. No epsilon is added to silently modify a small-object score.

5. A boundary is a convention you must specify

Overlap treats a mask as a filled region. Hausdorff distance compares sets of points. We must decide which points represent the boundary before asking for a distance.

Here, a foreground pixel is a boundary pixel if at least one of its four axial neighbors is background. The neighbors are up, down, left, and right. Outside-image positions count as background. Subtracting a cross-kernel erosion from the mask gives exactly this inner boundary. A 6×6 filled square has 20 such boundary pixels.

def boundary(mask: Mask) -> Mask:
    """Return inner boundary pixels using a center+four-neighbor cross."""
    validate_pair(mask, mask)
    cross: NDArray[np.uint8] = np.array(
        [[0, 1, 0], [1, 1, 1], [0, 1, 0]], dtype=np.uint8,
    )
    interior: Mask = cv2.erode(
        mask.astype(np.uint8), cross,
        borderType=cv2.BORDER_CONSTANT, borderValue=0,
    ) != 0
    edge: Mask = mask & ~interior
    return edge

We include boundaries around holes and around every disconnected component. We represent each boundary pixel by its center, using its integer row and column. We do not extract a subpixel polygon, compare all foreground pixels, or compute a physical surface-area-weighted metric.

These choices can change results, especially on small objects or coarse grids. Eight-neighbor extraction, contour interpolation, 3-D voxel surfaces, and surface-area weighting are valid choices for other protocols, but their scores are not automatically interchangeable with ours. A metric name without its representation convention is incomplete.

6. Find the nearest destination for each source point

Let A and B now denote the boundary-center sets extracted from P and G. These are different objects from the filled foreground sets. For a point a in A, we calculate its Euclidean distance to each point b in B and retain the nearest one.

Euclidean distance is ordinary straight-line distance. With row spacing and column spacing, measured in the same physical unit, its formula is:

def directed_distances(
    source: Mask, destination: Mask, spacing: tuple[float, float] = (1.0, 1.0),
) -> FloatArray:
    """Each source boundary center -> nearest destination center distance.

    spacing=(row, column), in pixels or a declared physical unit. This dense
    reference rejects products above 2 million pairs before allocating them.
    Caller handles empty boundaries; output float64 has shape (source_count,).
    """
    validate_pair(source, destination)
    scale: FloatArray = np.asarray(spacing, dtype=np.float64)
    if scale.shape != (2,) or not np.isfinite(scale).all() or np.any(scale <= 0):
        raise ValueError("Need positive finite row/column spacing")
    a: FloatArray = np.argwhere(boundary(source)).astype(np.float64) * scale
    b: FloatArray = np.argwhere(boundary(destination)).astype(np.float64) * scale
    if len(a) == 0 or len(b) == 0:
        raise ValueError("Directed distances need two nonempty boundaries")
    if len(a) * len(b) > 2_000_000:
        raise ValueError("Use a distance-transform backend for large masks")
    delta: FloatArray = a[:, None, :] - b[None, :, :]
    squared: FloatArray = np.sum(delta * delta, axis=-1)
    nearest: FloatArray = np.sqrt(squared.min(axis=1))
    return nearest

np.argwhere converts selected boundary pixels into rows of coordinates. Multiplying by scale converts row and column coordinates into the declared distance unit. If A contains m points and B contains n, delta has shape (m,n,2): every source, every destination, two coordinate offsets. Squaring and summing removes the coordinate axis; the result has shape (m,n). Taking min(axis=1) selects the nearest destination for each source, leaving m distances.

Visualize uses two source points and two destination points, so you can inspect the whole calculation. Source centers (1,1),(1,2) compare with destination centers (1,0),(1,1). The nearest distances are [0,1].

This implementation allocates arrays proportional to m times n. It deliberately refuses more than two million pairs. It is a transparent reference for small masks, not a large-volume engine. The verification independently checks the nearest distances against SciPy’s exact Euclidean distance transform ↗ , including non-unit spacing. A distance-transform backend avoids constructing every pair on a common raster grid; it still requires the same boundary and empty-mask conventions.

7. Hausdorff selects the worst nearest-boundary mismatch

A directed Hausdorff distance starts from every point in one boundary, finds each nearest point in the other, then takes the largest distance. Symmetric Hausdorff distance checks both directions:

One direction alone can miss a problem. If the prediction contains the entire target boundary plus a distant component, every target point can find an exact prediction match. Starting only from the target would report zero. Starting from the prediction discovers the unmatched component.

For comparison, SciPy’s directed_hausdorff ↗ accepts point arrays and returns a directed distance. It does not infer segmentation boundaries for you. Our test extracts the declared boundaries and takes the larger result from both directions before comparing with our implementation.

8. HD95 reduces outlier sensitivity by choosing what to ignore

The worst point can make ordinary HD sensitive to a single annotation error or isolated prediction. HD95 replaces a maximum over point distances with a 95th percentile. A percentile summarizes the ordered distribution of those distances; its finite-sample value also depends on an interpolation convention.

We use the maximum of the two directional 95th percentiles, with NumPy’s method="linear". Another convention pools both directional distance arrays before computing one percentile. These operations can disagree when boundary sizes or directional errors differ. Report which you use. The NumPy percentile API ↗ documents the interpolation methods.

def hausdorff(
    pred: Mask, target: Mask, spacing: tuple[float, float] = (1.0, 1.0),
) -> tuple[float, float]:
    """Return symmetric HD and max-directional HD95; empty=0 or +inf."""
    validate_pair(pred, target)
    if len(spacing) != 2 or not all(np.isfinite(s) and s > 0 for s in spacing):
        raise ValueError("Need positive finite row/column spacing")
    if not pred.any() and not target.any():
        return 0.0, 0.0
    if not pred.any() or not target.any():
        return float("inf"), float("inf")
    forward: FloatArray = directed_distances(pred, target, spacing)
    backward: FloatArray = directed_distances(target, pred, spacing)
    hd: float = float(max(forward.max(), backward.max()))
    hd95: float = float(max(
        np.percentile(forward, 95, method="linear"),
        np.percentile(backward, 95, method="linear"),
    ))
    return hd, hd95

Our outlier case is especially revealing. The square has 20 boundary centers. The prediction has those same 20 plus one outlier. Its forward distance list therefore has twenty zeros and one value of √18. With 21 sorted values, the linear 95th percentile lands exactly at zero-based index 19, which is still zero. All backward distances are zero too. HD95 is zero despite an actual error.

This is not a numerical bug. We asked the metric to suppress the extreme tail, and it did. HD95 may be useful when rare extreme deviations are not representative of the desired quality, but it cannot certify that no distant error exists. Inspect the tail and connected components when those failures matter.

9. Empty masks need a policy, not an accidental NaN

A nonempty image grid can contain no foreground. With both masks empty, IoU and Dice formulas divide zero by zero. With one boundary empty, there is no nearest destination point for an ordinary finite Hausdorff distance.

Our explicit policy is:

These are evaluation conventions, not a derivation that empty boundaries have ordinary finite point distances. Other systems mark distances unavailable and summarize such cases separately. Whichever policy you choose, preserve counts and reasons. Silently dropping difficult cases changes the population being evaluated.

Do not average infinity and then report a pleasing finite number by replacing it with zero. Report nonempty-pair boundary statistics alongside missed-object and empty-case counts, or define a task-justified finite penalty in advance. JSON also cannot represent infinity as a standard numeric value; our verification report writes the explicit string "infinity".

10. Read the complete experiment

Values are rounded for display; the verification report retains the computed values and library versions. All six cases are available through metric_cases() in the complete source . Download the source bundle and run python verify_cv.py.

The checks go beyond matching this table. They compare HD with SciPy’s independent point-set implementation, compare nearest-boundary distances with an exact distance transform on 24 seeded random mask pairs at two spacings, verify a separate neighbor-by-neighbor boundary extractor, and exercise border, symmetry, dtype, shape, empty-mask, and spacing cases. These checks support the implementation under its stated two-dimensional conventions. They do not validate a trained segmenter or a medical-use decision.

11. Aggregation can change the story again

Suppose one image contains a one-pixel target that you miss completely, and another contains a 99-pixel target that you predict perfectly. Both targets are nonempty. Averaging the two per-image Dice scores gives 0.5: each image gets equal weight. Pooling all pixel counts before computing Dice gives 198/199≈0.9950: the large object dominates.

These are commonly called macro and micro aggregation, but always name the axis: macro over images, classes, objects, patients, or something else? “Mean Dice” alone is ambiguous. The two aggregations answer different questions. Choose based on the evaluation unit and failure cost, then show enough disaggregated results to reveal hidden groups.

Empty images create another trap. If 90 of 100 images have empty targets and are correctly predicted empty, assigning them Dice 1 can dominate an all-image average even if performance on the ten object-containing images is poor. Keep empty-target detection and nonempty-target segmentation visible separately.

For multiclass segmentation, classes are mutually exclusive at a pixel; compute a binary one-versus-rest mask for each class and specify whether background enters the summary. For multilabel segmentation, a pixel may legitimately belong to several labels; evaluate those channels with their own definitions and thresholds. Do not collapse a multilabel output with argmax unless the task really requires one label.

For instance segmentation, separate objects of the same class matter. A semantic union can hide mergers: two objects joined by a bridge may have good region overlap while becoming one component. Instance matching and object-level errors require an additional protocol; this binary article does not implement that protocol.

12. A score is an evaluation choice, not automatically a loss

The functions here consume hard Boolean decisions. A probability threshold discards the underlying confidence, and these NumPy operations do not provide a differentiable training loss. “Soft Dice loss” is a different function that operates on continuous predictions and uses its own reductions and smoothing conventions. Do not substitute it for the stated hard-mask metric without saying so.

Likewise, a high overlap score does not establish probability calibration. A model predicting 0.51 and one predicting 0.99 can produce the same mask at threshold 0.5. If probabilities drive risk-sensitive decisions or threshold changes, inspect probability quality separately.

Boundary distances also do not establish topology, object count, or task utility. The broader Metrics Reloaded recommendations ↗ and metric-related pitfalls study ↗ organize such choices around the problem being evaluated. Their key role here is to motivate a task-first protocol, not a universal winner among metrics.

A boundary experiment needs a tolerance contract

An edge map from Sobel filtering and a filled segmentation mask are different outputs. Comparing their pixels directly can punish an otherwise correctly located thin contour for not containing an object’s interior. Decide whether your target is a region, a boundary, or a collection of instances before selecting the score.

For a boundary task, annotations have localization uncertainty. A one-pixel offset can mean a visible mistake at one resolution and fall within annotation variation at another. If you count a prediction as correct within a distance tolerance, declare that distance, its unit, and how matches are assigned. Allowing many predicted pixels to match the same reference pixel can reward thick or duplicated edges. An assignment rule and a nearest-distance rule are not interchangeable.

Our Hausdorff implementation avoids claiming a detection-style match count: it measures nearest distances between boundary centers under its stated convention. It still does not tell you whether a predicted contour is closed, whether two objects were merged, or how many objects were missed. Add those checks when the application depends on them.

When you inspect a boundary overlay, use a fixed image scale and include the original signal. A colorful contour alone can conceal a systematic offset caused by preprocessing or resizing. Measure in the coordinate system where the prediction will be used, and retain the transformation needed to map model coordinates back there. The metric function cannot repair a coordinate mismatch on its own.

13. Write the evaluation contract before comparing models

For a concrete binary segmentation report, record the foreground definition, coordinate alignment, ignored-region policy, spacing and unit, probability threshold, postprocessing, boundary extraction, empty-mask policy, HD95 convention, and aggregation axis. State which data was used to choose each tunable setting.

Keep a locked test split at the appropriate independent unit. Use development data for thresholds and morphology. Save per-case counts and distances with identifiers so a mean can be traced back to actual predictions. Compare methods on the same cases, and report slices such as small versus large targets when those groups have different failure modes.

Uncertainty estimates should respect that sampling unit too. Resampling highly correlated video frames as if they were independent can give unjustifiably narrow intervals. A later experimental-practice module will build that analysis; for this pilot, we report exact synthetic behavior rather than attaching a population confidence interval to six constructed examples.

Finally, inspect failures with overlays. A surprising score should lead you back to a mask, an annotation, and a convention. If the visualization and the number seem to disagree, first check coordinate alignment, empty handling, and whether you computed distances on the boundary or the filled region.

14. An experiment worth doing next

Add an outlier progressively farther from a correctly predicted square. Keep its area at one pixel. Predict what changes: IoU and Dice stay fixed; HD grows with its nearest-boundary distance; HD95 stays zero for this 20-plus-one boundary configuration. Now make the erroneous component larger. More of the forward distance distribution becomes nonzero, so its 95th percentile may rise.

Then shift a small square and a large square by one pixel. Their worst boundary displacement can be the same while the smaller square loses a larger fraction of its overlap. Neither metric is contradicting the other. They measure absolute distance and size-relative agreement.

You now have a way to read a segmentation score as a question with assumptions: how much region agrees, how far the worst boundary point strays, or how a chosen percentile describes the boundary-distance distribution. A useful evaluation makes those questions explicit and keeps the failures they omit available for inspection.