From neighborhoods to edges: what a filter actually measures.

You have a bright object on a dark background. A threshold almost finds it, but a few bright background pixels become false positives. Blur the image and those specks become less prominent. Unfortunately, a narrow part of the real object also disappears. The operation did what you asked; the assumption behind it was incomplete.

This lesson makes that assumption visible. A filter combines neighboring measurements. Its usefulness depends on whether those neighbors should support the same decision. We will first compute one output pixel by hand, move that computation across an image, and then change the weights to measure a transition instead of an average. The final step is deciding what evidence would justify adding this operation to a pipeline.

Return to images as arrays if row, column, shape, or dtype are unfamiliar. The mask baseline supplies the decision we will eventually evaluate. Here we use a single grayscale plane, so a spatial operation cannot accidentally mix color channels.

1. Start with an image you can predict

Our synthetic image has three identical rows. Each row is [0,0,80,80,80], stored as float32. Column 2 is the first bright column. The values are intensity units chosen for the experiment; floating-point storage does not automatically mean unit-range values.

The full file’s step_image() constructs this array. Its repeated rows intentionally remove vertical variation. If a filter reports a vertical change, we should inspect the axes, border handling, or implementation before inventing an explanation about the image.

A neighborhood is the set of pixels used to calculate one result. A kernel is a small array of weights aligned with that neighborhood. The anchor identifies which kernel entry sits over the output location. For our odd-sized 3×3 kernel, the anchor is the center entry. Every result will use the pixel itself, four side neighbors, and four diagonal neighbors.

At row 1, column 1, the neighborhood is three rows of [0,0,80]. Its center is still dark, but its right-hand neighbors are bright. A weighted average will move this output above zero. That is the origin of blur near a boundary: neighboring evidence crosses the boundary.

2. Nine contributions become one number

Use weights with rows [1,2,1], [2,4,2], and [1,2,1], then divide every weight by 16. This is a binomial smoothing kernel. The center receives weight 4/16, a side neighbor 2/16, and a diagonal neighbor 1/16. All weights are nonnegative and their sum is one.

Multiply entries at matching positions. Only the bright right column contributes: 80 times 1/16, 80 times 2/16, and 80 times 1/16. Their contributions are 5, 10, and 5, producing 20. The output is one scalar at the anchor location; the nine contributions are intermediate quantities, not nine new output pixels.

Here is the general expression for this particular 3×3 operation:

def inspect_patch(patch: FloatPlane, kernel: FloatPlane) -> tuple[FloatPlane, float]:
    """Two finite float32 (3,3) arrays -> nine contributions and their sum."""
    check_plane(patch)
    check_plane(kernel)
    if patch.shape != (3, 3) or kernel.shape != (3, 3):
        raise ValueError("This example requires two 3x3 arrays")
    products: FloatPlane = patch * kernel
    response: float = float(products.sum())
    return products, response

ndarray.sum ↗ has no axis argument here, so it combines all nine entries. The result is converted to a Python float for a clear return contract. Both arrays are checked as finite float32 planes before multiplication. The type alias describes dtype; the docstring and checks describe the shape.

The Visualize recording contains all nine patch values, all nine weights, all nine products, and the final scalar. Try predicting the products before advancing. If the center intensity changed from 0 to 16 while the other pixels stayed fixed, its contribution would increase by 4. The response would become 24. That prediction follows directly from the center weight.

3. Slide the operation without overwriting the evidence

To filter the entire image, repeat the same computation at every output coordinate. Always read from the input and write to a separate output. Updating the input in place would let later pixels consume already-filtered values. The result could then depend on whether you traversed left to right or right to left.

def correlate3(image: FloatPlane, kernel: FloatPlane) -> FloatPlane:
    """Reference correlation, centered 3x3 kernel, replicated border, same H,W."""
    check_plane(image)
    check_plane(kernel)
    if kernel.shape != (3, 3):
        raise ValueError("Expected a 3x3 kernel")
    padded: FloatPlane = np.pad(image, 1, mode="edge")
    output: FloatPlane = np.zeros_like(image)
    height, width = image.shape
    for row in range(height):
        for col in range(width):
            patch: FloatPlane = padded[row:row + 3, col:col + 3]
            output[row, col] = np.sum(patch * kernel)
    return output

np.pad(image, 1, mode="edge") ↗ adds one pixel on each side by repeating boundary values. The padded array is two rows and two columns larger than the original. Original location (row,col) now has its center at (row+1,col+1) in the padded array. Consequently, the slice padded[row:row+3,col:col+3] selects exactly its 3×3 neighborhood.

The loop is intentionally a reference implementation, not a recommendation for processing large images in Python. It makes the indexing auditable. The full source also includes smooth_opencv(), which calls cv2.filter2D ↗ with the same weights, floating-point output, and replicated border. The verification script compares the implementations on several shapes, including a one-pixel image.

The first row of the full smoothed step is [0,20,60,80,80]. All three output rows match. The two columns nearest the transition have mixed dark and bright evidence. Farther away, constant regions remain constant.

Correlation versus convolution: this code aligns the stored kernel with the patch without reversing it. That operation is correlation, and filter2D uses that convention. Mathematical convolution reverses the kernel along both spatial axes first. Our smoothing kernel is symmetric, so the two results agree. An asymmetric derivative kernel can change sign when reversed. A symmetric-only test cannot expose that mistake, which is why our numerical tests also use asymmetric kernels.

4. Border policy is part of the algorithm

The top-left pixel has no real neighbor above or to the left. Filling those positions is an assumption, not recovered information. Repeating the nearest edge value assumes a locally constant extension. Zero padding assumes a dark exterior. Reflection continues an edge pattern by mirroring samples; its exact convention determines whether the boundary sample is repeated.

Consider a constant image of intensity 50. At its top-left corner, zero padding leaves only four real pixels in our 3×3 neighborhood. Their binomial weights add to 9/16, so the response is 28.125. Replication gives 50. A dark frame can therefore be introduced by the padding rule even when the source contains no dark boundary.

That does not make zero padding universally wrong. If the exterior really represents zero signal, it may be appropriate. It means that a comparison between two libraries is incomplete until you align kernel, anchor, border, dtype, and scale. Record all five in your experiment configuration.

Do not silently substitute BORDER_DEFAULT for an explicitly chosen policy. The tutorial uses BORDER_REPLICATE throughout. For tiles cut from a larger image, a better option can be to read a surrounding halo from the original image and discard the halo after filtering. Otherwise, artificial tile boundaries can become visible seams.

5. From averaging to measuring change

A bright constant region and a dark constant region can both have zero local change. An edge detector should respond to the transition between them, not simply to brightness. Subtraction gives us that distinction.

For a horizontal Sobel response, use kernel rows [-1,0,1], [-2,0,2], and [-1,0,1]. Each row subtracts a left value from a right value. The middle row has twice the weight of its neighbors. This combines a horizontal difference with vertical smoothing. The vertical kernel is the transpose: it compares lower rows with upper rows.

The term horizontal derivative refers to the direction in which intensity is compared. It responds strongly to a vertical boundary. This is an easy naming trap: the direction of change and the direction along the visible edge are perpendicular.

On our step image, columns 1 and 2 each have horizontal response 320: the right-minus-left difference is 80 in every contributing row, and the row weights add to four. Column 0 and columns 3–4 have response zero. Every vertical response is zero because the rows are identical.

def sobel_components(image: FloatPlane) -> tuple[FloatPlane, FloatPlane, FloatPlane]:
    """Signed column/row differences and L2 magnitude, unscaled 3x3 Sobel."""
    check_plane(image)
    gx: FloatPlane = cast(FloatPlane, cv2.Sobel(
        image, cv2.CV_32F, 1, 0, ksize=3, scale=1,
        borderType=cv2.BORDER_REPLICATE,
    ))
    gy: FloatPlane = cast(FloatPlane, cv2.Sobel(
        image, cv2.CV_32F, 0, 1, ksize=3, scale=1,
        borderType=cv2.BORDER_REPLICATE,
    ))
    magnitude: FloatPlane = np.hypot(gx, gy)
    return gx, gy, magnitude

In cv2.Sobel ↗ , the derivative orders (1,0) request first-order change in the column direction; (0,1) requests it in the row direction. ksize=3 selects the 3×3 operator and scale=1 leaves its gain unnormalized. CV_32F preserves signed floating-point responses. We use image coordinates: columns increase rightward and rows increase downward.

This gain matters when you compare magnitudes. On an interior linear ramp rising by one intensity unit per column, our raw horizontal response is 8. Dividing by 8 would give one unit per pixel for that ramp. It is not a universal conversion from an edge response to physical slope: pixel spacing, kernel, and the smoothness of the signal matter too.

A falling transition, from 80 down to zero, gives −320 instead of +320. An unsigned output cannot preserve that negative result. Clipping it to zero before taking an absolute value destroys information; the later absolute value cannot recover it. Keep a signed representation through the computation and make display conversion a separate step.

6. Magnitude combines directions and discards sign

The horizontal response is called $G_x$ and the vertical response $G_y$. The subscript names a direction; it is not a channel number. At each pixel, these two values form a local change vector. Its Euclidean length is the gradient magnitude:

np.hypot ↗ calculates this two-component length elementwise. It keeps the spatial shape. A magnitude of 320 is neither a probability nor a claim of 320 objects. It depends on intensity units, derivative gain, and local contrast. If you multiply every input intensity by two, this linear derivative doubles, and its magnitude doubles as well.

A direction can also be calculated from the signed components, but it becomes unstable where both are near zero. Avoid giving a visually strong direction arrow to an almost-flat region merely because an angle function returns a number. Strength and direction need to be interpreted together.

7. Smoothing changes both noise and the edge

The profile diagram shows the exact same input before and after smoothing. The raw derivative is [0,320,320,0,0]. After smoothing first, it becomes [80,240,240,80,0] with our replicated border. The response is less concentrated: smaller at the main transition and nonzero farther away. At the left border it is now 80 because the neighboring smoothed value is 20.

This is the tradeoff hidden by the phrase “remove noise.” Local averaging can reduce independent fluctuations, but it also mixes distinct structures. A thin bright line may become a low-contrast band. Two nearby boundaries may overlap. The best amount of smoothing depends on the smallest structure the task must retain, its contrast, and the noise pattern.

Our binomial kernel can be applied as two one-dimensional passes: weights [1,2,1]/4 horizontally and vertically. Multiplying those row and column weights produces the 3×3 matrix above. This property is called separability. It can reduce work for larger filters, provided intermediate precision and boundary conventions agree. The fixed binomial weights are Gaussian-like; we are not claiming they are every Gaussian filter. A Gaussian filter has a scale parameter, sigma, that controls how broadly neighboring evidence is averaged. See the documented GaussianBlur parameters ↗ when selecting that operation.

Averages are also not the only possible neighborhood summary. A median selects the middle ranked value and can resist isolated extreme values, but it is nonlinear and can remove small legitimate features. Do not swap it into a pipeline simply because the corruption looks noisy. First state the kind of variation you expect and the structures you cannot afford to erase.

8. An edge is evidence, not a segmentation

A high gradient can come from an object boundary, a shadow, texture, a reflection, or a compression artifact. Conversely, two touching objects with similar appearance may have a weak visible boundary. A gradient measures image change. A segmentation mask assigns membership in a defined target class. Those questions overlap only under additional assumptions.

Thresholding the magnitude produces selected edge pixels, usually with gaps and finite thickness. It does not label the whole object interior. Filling every enclosed contour can fail when textures create internal loops or the true boundary is broken. Before treating an edge as an object, explain how connectivity, closure, and the task’s definition of foreground turn that evidence into a region.

Canny adds stages to this idea: local gradient comparison thins responses, and two-threshold hysteresis keeps weaker candidates connected to stronger ones while rejecting unsupported candidates. Its thresholds act on gradient strength, not class probability. Read the OpenCV Canny walkthrough ↗ for the full algorithm. Our executable example intentionally exposes smoothing and Sobel separately; it does not pretend that magnitude alone reproduces Canny.

9. Design an experiment that could prove you wrong

Start with a concrete decision: does smoothing improve the threshold mask for the intended images without losing small targets? Hold the downstream threshold rule fixed for an initial controlled comparison. Then, if you tune each pipeline separately, give both the same validation budget and report that protocol explicitly.

Use a small diagnostic set before a large dataset. Include a constant image, rising and falling steps, an isolated bright pixel, a thin line, two nearby objects, and an object touching the border. These fixtures test different assumptions. A smooth result on one rectangle cannot establish robustness to all six cases.

For real images, split by the source of dependence: subject, video, acquisition session, or original image before generating crops. Neighboring frames or crops from the same source can make a validation result look better than the system will be on a new source. Tune kernel choice and thresholds on validation data, then evaluate the fixed choice on a held-out test set.

Write down the input range, grayscale convention, kernel coefficients, border mode, output dtype, intensity threshold, and any morphological cleanup. Change one component when diagnosing a failure. Save an original input, intermediate smoothed plane, signed derivatives, final mask, and target together. Give comparable displays fixed scales; independent autoscaling can make a weak noisy response look as prominent as a strong true boundary.

Evaluate the task output. If the objective is a mask, report mask overlap and inspect boundary errors using the IoU, Dice, and Hausdorff companion . Add a result broken down by object size: a large foreground region can dominate a global score while all tiny objects vanish. If the objective is an edge map, define edge annotations and a spatial tolerance instead of borrowing a region score without explanation.

The downloadable tests check arithmetic and contracts, not production quality. They compare the NumPy implementation with an independent scalar accumulation and OpenCV, verify constant regions and signed ramps, and reject invalid inputs. Such tests establish that the code implements the declared operation. Only a representative evaluation can establish whether that operation helps your application.

10. Make the next idea earn its place

Run verify_cv.py from the updated vision starter bundle . Change the step from 80 to 40 and predict both the smoothed values and the Sobel responses before rerunning. Replace the step with a one-pixel bright line. Ask whether your original threshold would retain it after smoothing, and inspect the precise pixels responsible for the answer.

You now have two kinds of local evidence: an intensity-derived mask and a change-derived edge map. Continue with connected components : grouping selected pixels into components so that you can reason about whole objects, their sizes, and their locations. Keep the distinction clear as you proceed: a filter measures, a threshold decides, and grouping organizes those decisions. None supplies the meaning of the target by itself.