From pixels to masks: build a baseline you can question.

Imagine inspecting a bright rectangular part against a dark background. Before training a segmentation network, we can ask a simpler question: does a brightness rule already locate the part? A baseline gives us something concrete to test, a way to discover data problems, and a reference a more expensive model should improve upon.

The rule will fail in two different ways. A bright speck outside the rectangle will be selected, and a dark pixel inside it will be missed. We will repair this particular example, then construct another where the same repair destroys a real object. That second result is as important as the first.

If you arrived directly, Images are arrays establishes coordinates, dtype, channel order, and masks. We now use a single grayscale channel; there is no RGB/BGR ambiguity in this experiment.

1. Define the task independently of the algorithm

Binary semantic segmentation assigns every pixel to one of two meanings: foreground or background. Here foreground means “the rectangular part,” not “whatever is bright.” The distinction lets us identify a bright distractor as an error.

Our synthetic target is a 6×6 square on a 12×12 grid. Rows 3 through 8 and columns 3 through 8 belong to the part, giving 36 foreground pixels. The source image uses intensity 180 inside and 30 outside. We then corrupt the observation: set location (1,1) to 220 and location (5,5) to 40.

The target does not change when we corrupt the observation. It was defined from the synthetic scene geometry. If we instead created the target by thresholding the image with the same rule we planned to evaluate, agreement would prove only that we had repeated our own rule.

The complete source function scene() in cv_masks.py contains these exact values. No random noise or external image download is involved. You can reproduce every pixel from the description above.

Matching object area is not enough. Our two masks both contain 36 positive decisions, yet disagree at two locations. The positions of the decisions matter.

2. A threshold converts a measurement into a decision

Choose threshold 100. A pixel is foreground only when its intensity is strictly greater than 100. Equality belongs to background. This matches OpenCV’s binary threshold convention; see Image Thresholding ↗ .

def threshold_mask(gray: ByteImage, threshold: int = 100) -> Mask:
    """Nonempty uint8 (H,W) -> bool (H,W), using STRICT greater-than."""
    if gray.dtype != np.uint8 or gray.ndim != 2 or gray.size == 0:
        raise ValueError("Expected a nonempty uint8 grayscale image")
    if not 0 <= threshold <= 255:
        raise ValueError("Threshold must be in [0,255]")
    mask: Mask = gray > threshold
    _, encoded = cv2.threshold(gray, threshold, 255, cv2.THRESH_BINARY)
    np.testing.assert_array_equal(mask, encoded != 0)
    return mask

gray > threshold compares every pixel and returns a Boolean array with the same spatial shape. OpenCV returns a byte-encoded result with values 0 or 255. We convert that result to Boolean with encoded != 0 before comparing implementations. Comparing a Boolean mask directly with bytes 0 and 255 would confuse the representation with the decision.

Visualize uses the six values [30,100,180; 220,40,101], so every recorded value is visible without truncation. The complete scene diagram uses the full 12×12 grid. These are explicitly different example sizes for different teaching purposes.

3. Count the kinds of mistakes

A true positive is a location selected in both prediction and target. A false positive is selected only by the prediction. A false negative belongs to the target but is missing from the prediction. A true negative is background in both.

Our raw threshold has 35 true positives, one false positive, one false negative, and 107 true negatives. The ordinary pixel accuracy is 142 correct decisions out of 144, or about 98.61%. That sounds strong even though the mask contains both types of error. In a scene dominated by background, predicting background everywhere can yield high accuracy while missing every object.

The purpose of a score is to answer a task question. If a missed defect matters, ask explicitly how often defects are missed. If area matters, measure overlap. If boundary location matters, measure distance. Do not choose a metric because its number looks reassuring.

4. Let nearby decisions inform a cleanup rule

The threshold treats pixels independently. Mathematical morphology uses a neighborhood, called a structuring element or kernel, to modify a mask. Our kernel is a centered 3×3 square: the current pixel and its eight immediate neighbors.

Erosion retains a foreground pixel only if every position covered by the kernel is foreground. Near a boundary, some neighbor is background, so that pixel disappears. A 6×6 filled square away from the image edge becomes a 4×4 square after one such erosion. A one-pixel object disappears entirely.

Dilation marks a pixel as foreground if any covered position is foreground. It expands a 6×6 square to 8×8 when there is sufficient surrounding space. An isolated point expands to a 3×3 block. Dilation does not know whether that point was noise or a real target.

Think of each operation as a small local question: “all?” for erosion and “any?” for dilation. The kernel defines which neighbors answer it. A cross-shaped kernel asks a different question from a full square. We use both later: a square for this cleanup, a cross for the metric article’s boundary extraction.

OpenCV’s morphology tutorial ↗ defines the operations and their compositions. Our scene and its failure tests below are independently constructed examples.

5. Composition encodes an assumption about scale

Closing is dilation followed by erosion with the same kernel. It can fill small background gaps and holes. Opening is erosion followed by dilation. It can remove foreground structures that cannot contain the kernel. They are not inverse operations: erased information does not magically return.

We close first because the rectangle contains a tiny dark hole. Then we open to remove the isolated bright speck. Order matters. Opening first would erode around the hole as well as around the outer boundary, and can reshape the object differently before closing ever sees it.

def clean_mask(mask: Mask) -> tuple[Mask, Mask]:
    """Return closing then opening, 3x3 square, outside-image background=0."""
    if mask.dtype != np.bool_ or mask.ndim != 2 or mask.size == 0:
        raise ValueError("Expected a nonempty bool mask")
    encoded: ByteImage = mask.astype(np.uint8) * np.uint8(255)
    kernel: ByteImage = np.ones((3, 3), dtype=np.uint8)
    closed_u8: ByteImage = cast(ByteImage, cv2.morphologyEx(
        encoded, cv2.MORPH_CLOSE, kernel,
        borderType=cv2.BORDER_CONSTANT, borderValue=0,
    ))
    opened_u8: ByteImage = cast(ByteImage, cv2.morphologyEx(
        closed_u8, cv2.MORPH_OPEN, kernel,
        borderType=cv2.BORDER_CONSTANT, borderValue=0,
    ))
    closed: Mask = closed_u8 != 0
    cleaned: Mask = opened_u8 != 0
    return closed, cleaned

We encode the Boolean input as 0/255 bytes for OpenCV and decode both outputs back to Boolean. iterations is left at one. The kernel is a 3×3 square centered at its default anchor. BORDER_CONSTANT and borderValue=0 explicitly treat locations outside the image as background.

That border rule is consequential. An object touching the crop edge does not have the same surrounding neighborhood as an interior object. Closing with background padding can remove pixels at the image edge during its erosion stage. Do not silently compare border-sensitive implementations with different padding policies.

The numbers and diagrams are generated from the published implementation. The verification checks exact array equality between cleaned prediction and target. We have solved this scene, under its particular object-size and noise assumptions.

6. Construct the counterexample before trusting the cleanup

Replace the rectangle with a legitimate one-pixel object in the center of a 7×7 image. Our opening removes it. The code has not malfunctioned: the rule has correctly applied an assumption that is wrong for the new task.

Similarly, a thin cable, a narrow vessel, a small character stroke, or two neighboring objects separated by a tiny gap can be damaged by a kernel that looked harmless on a solid rectangle. Closing can bridge objects you needed to keep separate. Opening can remove the very structure you wanted to detect.

The test suite includes the one-pixel counterexample. It should keep passing by confirming that the cleanup removes the object. This is a documented limitation test, not a quality target. It prevents a polished success diagram from being the only evidence a reader sees.

A kernel size is a scale assumption. Three pixels represent different physical distances at different image resolutions. If the acquisition geometry changes, “3×3 worked before” is not enough to justify using it again.

7. When brightness alone stops being a useful rule

A global threshold assumes useful separation on a shared intensity scale. Illumination gradients, shadows, texture, sensor changes, and overlapping foreground/background intensities weaken that assumption. There may be no single threshold that separates the target.

Otsu thresholding chooses a global split from the intensity histogram using a within-class variance criterion. It can help when the histogram supports a useful separation; it does not infer your semantic definition of an object. Adaptive thresholding uses a local neighborhood to choose a spatially varying threshold. That helps with some illumination variation while introducing neighborhood-size and offset choices. Neither guarantees that a bright reflection should count as a part.

A histogram counts how often each intensity occurs; it discards where those pixels were. Two images with the same histogram can have completely different object arrangements. This is why a distribution-based threshold still needs spatial and task validation.

Before adding a learned model, investigate whether the information needed to distinguish the classes is even present in the input. A model can learn texture, shape, and context beyond a single intensity, but it still depends on labels, representative examples, and an evaluation protocol.

8. Make an experiment a comparison you can defend

On this synthetic scene, we chose a threshold and kernel to teach operations. That is an illustration, not an estimate of deployment performance. A real experiment needs separate development and held-out evidence.

1. Define the unit of independence. Images from the same patient, manufactured part, video, or acquisition session may be correlated. Split by the unit that prevents information leaking between development and evaluation.
2. Audit the target. Check coordinate alignment, ignored regions, ambiguous boundaries, empty images, and annotation consistency. Specify what foreground means before implementing a rule.
3. Freeze a baseline. Record loader, channel order, scaling, threshold, kernel, operation order, and border convention. Keep raw and postprocessed predictions.
4. Tune using development data. Choose thresholds and morphology settings there. Repeatedly inspecting final test scores turns that test set into development data.
5. Evaluate the locked pipeline. Report per-image results, empty-target counts, and representative failures. Compare the same images with paired results so changes are attributable.
6. Preserve enough artifacts to reproduce it. Save code version, dependencies, configuration, data/split identifiers, prediction masks, and a report explaining excluded cases. A random seed alone does not capture the experiment.

The test suite verifies threshold agreement with OpenCV, the equality-at-100 case, exact cleanup success, and the small-object failure. It also tests the metrics we will use to judge the masks. All examples run on CPU with the downloadable source and checks .

9. A small exercise with a reasoned answer

Move the bright distractor from (1,1) to a location immediately beside the target. Predict whether it will still be removed. Then change the target from a square to a one-pixel-wide line and predict the effect of opening.

The neighboring distractor may become connected to the object and survive or alter its boundary; the exact result depends on the full neighborhood and operation order. Inspect the returned grids rather than deciding by eye from a resized screenshot. A one-pixel-wide line cannot contain the 3×3 square kernel and disappears under erosion, so opening cannot reconstruct it.

10. Should filtering come before the threshold?

A bright background speck can be removed in two different ways: average its intensity with neighboring pixels before thresholding, or remove a small foreground structure after thresholding. These operations use different information. Smoothing sees intensity differences; morphology sees the binary decisions already made. Neither knows whether the speck is a defect, a real small target, or noise.

Consider a single intensity-180 pixel surrounded by intensity-30 background. The normalized binomial kernel from the filtering lesson gives the center weight one quarter and the neighbors a combined weight three quarters. The smoothed center becomes 67.5. At our threshold of 100 it is rejected. That is useful if it is a distractor and a complete miss if it is a valid one-pixel target. Exactly the same array operation supports both stories; only the target definition distinguishes them.

There is also an order effect. Smoothing an image and then thresholding generally differs from thresholding and then smoothing its Boolean mask. In the first case, original intensities determine contributions. In the second, a barely positive pixel and a very bright pixel have already been made identical. A fractional result from a smoothed mask then needs a new decision rule. Calling both pipelines “blur plus threshold” hides these choices.

Keep four outputs in a controlled comparison: the raw threshold baseline, smoothing then threshold, threshold then morphology, and the combined pipeline. Use the same held-out cases. Record the number of removed false positives and newly removed true positives, especially for small structures. An operation earns its place by improving the relevant decisions, not by making a preview look cleaner.

11. Turn a failure into a useful debugging artifact

When an unexpected mask arrives, save the smallest crop that still reproduces the failure, together with enough surrounding context to preserve neighborhood operations. Cropping too tightly changes the border conditions and can accidentally remove the bug. Keep the original coordinate of the crop and its target aligned.

Trace one pixel backward. Was it already wrong immediately after thresholding? Did closing add it? Did opening remove a nearby connection? Compare Boolean arrays before resizing them for display. Nearest-neighbor previews preserve class membership; a smoothly interpolated screenshot can invent intermediate colors that were never labels.

Then write a prediction in words: “this three-pixel gap will be bridged by closing with this kernel.” Check the actual output. If the prediction fails, inspect kernel shape, border policy, or the complete neighborhood. This turns parameter tweaking into a testable explanation and gives the next engineer a precise starting point.

Next: ask what the score leaves out

You can now create a mask, state the assumptions behind it, and reproduce both success and failure. The remaining question is how to summarize those outcomes without hiding what matters. Continue with IoU, Dice, and Hausdorff: what each metric misses : overlap and boundary distance inspect different aspects of the same prediction.