Learning Convolutions by Inventing Computer Vision

Before we can process images, we need to understand what they are to a computer.

Run the helper above to load two test images — one colored, one grayscale.

Tasks:

1. Use show_images to display both the color and grayscale images.
   • For the grayscale image, pass cmap_list=['gray'].
2. What do you notice visually? How do they differ?

An image is just a grid of numbers. Let's prove it.

A PIL Image can be converted to a NumPy array using np.array(img).

Tasks:

1. Convert both images to NumPy arrays. Store them as color_arr and gray_arr.
2. Print color_arr.shape and gray_arr.shape. What are the dimensions?
3. Print color_arr.dtype and gray_arr.dtype. What is the data type of each pixel?
4. What is the range of values? Print color_arr.min() and color_arr.max().
5. Print just the first 5x5 block of the grayscale array. What do the numbers represent?
6. Print color_arr[0, 0]. What does this value represent? What about color_arr[0, 0, 0]?

Write down your answers:

• A color image has shape (H, W, ?). The third dimension is... because...
• A grayscale image has shape (H, W). There is no third dimension because...
• The numbers in the array range from ___ to ___. This makes sense because...
• color_arr[0, 0] gives [R, G, B] — this means the pixel at row 0, column 0 has Red=___, Green=___, Blue=___.

Now that you know the structure, explore the image as a grid.

Tasks:

1. How many rows does the color image have? How many columns?
2. What is the pixel at the very center of the color image? (Compute the center row and column from the shape.)
3. What is the pixel at the bottom-right corner?
4. Print all pixel values in the first row of the grayscale image. How many values are there?
5. Without using any NumPy operations — using only Python loops — compute the average brightness of the grayscale image. (Average brightness = average of all pixel values.)

Important rule for this notebook: When asked to avoid NumPy, that means: no np.mean(), no np.sum(), no array slicing magic. Use Python for loops, range(), and plain arithmetic. You can still access array elements by index like arr[r, c].

Let's look at a small patch of the image zoomed in, so we can literally see the pixels.

Tasks:

1. Extract the top-left 20x20 patch of the grayscale image.
2. Display it using show_images with cmap_list=['gray'].
3. Do you notice anything? Each square in the display is one number from the array.
4. Extract the same 20x20 patch from the color image and display it.
5. Now extract a 20x20 patch from the middle of each image and display them side by side.

A color pixel is stored as three numbers: (R, G, B) — Red, Green, Blue. Each value is between 0 and 255.

• (255, 0, 0) → pure red
• (0, 255, 0) → pure green
• (0, 0, 255) → pure blue
• (0, 0, 0) → black
• (255, 255, 255) → white
• (128, 128, 128) → gray

Tasks:

1. Look at color_arr[0, 0]. What are the R, G, B values of the top-left pixel?

2. Create a tiny 4x4 test array manually (use np.array) where each pixel is a known color:

   • Row 0: all red pixels [255, 0, 0]
   • Row 1: all green pixels [0, 255, 0]
   • Row 2: all blue pixels [0, 0, 255]
   • Row 3: all white pixels [255, 255, 255]

   Display it. Does it look right?

3. Access test[0, 0, 0] — this is the Red channel of the first pixel. Access test[1, 0, 1] — what channel is this? What value do you expect?

Your job is to extract only the Red channel from the color image. The result should be a 2D array (H x W) of numbers — just the red values.

Rules for this exercise:

• No NumPy slicing tricks like color_arr[:, :, 0]
• No np.split, no np.take, no fancy indexing
• Use plain Python for loops and index access arr[r, c, 0]

Tasks:

1. Create an empty 2D array of the right size using np.zeros((height, width), dtype=np.uint8).
2. Loop over every pixel and copy the Red value into your new array.
3. Display the result using show_images with cmap_list=['gray'].
4. What do bright areas in the red channel mean? What do dark areas mean?
5. Extra: Do the same for the Green and Blue channels. Display all three side by side using colormaps ['Reds', 'Greens', 'Blues'].

When we display the Red channel as grayscale, bright = high red value, dark = low red value. But what if we want to display it as an actual red image?

Think: to show the red channel in red, you need a color image where:

• The red channel = the values from red_channel
• The green channel = all zeros
• The blue channel = all zeros

Tasks:

1. Without using NumPy fancy tricks, build a 3D array red_image of shape (H, W, 3) where only the red component is non-zero.
2. Display it. Does it look red-tinted?
3. Do the same for green and blue. Display all three.
4. Now add all three colored arrays together using a loop (not NumPy addition). Display the result. Does it look like the original color image? Why or why not?

Hint for step 4: Be careful about overflow! Adding uint8 values can exceed 255. Think about how to handle this.

A grayscale image has only one number per pixel — its brightness. A color image has three (R, G, B).

The question is: given R, G, B — how do you compute a single brightness value?

There is no single "right" answer. But there are good ones and bad ones. Your job is to invent several formulas and see what happens.

Tasks:

1. Before writing any code: think of 3 ways you could combine R, G, B into one number. Write them as math formulas.

Your 3 ideas (before looking at any answers):

1. Idea 1: ...
2. Idea 2: ...
3. Idea 3: ...

For each formula you invented (and the ones below), write a function rgb_to_gray_METHOD(color_array) that:

• Takes a (H, W, 3) numpy array
• Returns a (H, W) numpy array of type uint8
• Uses only Python loops and arithmetic — no np.mean(), no slicing across channels

Formulas to implement:

Tip on uint8: When you compute a float result, convert it back to uint8 by wrapping in int(...) and clamping: max(0, min(255, value)).

Tasks:

1. Apply all four methods to color_arr.
2. Display all results side by side using show_images with cmap_list=['gray', 'gray', 'gray', 'gray'].
3. Also display the original color image for reference.
4. Do you see any visible differences between the methods?
5. Compute the pixel-wise difference between gray_average and gray_luminosity: For each pixel, compute abs(average[r,c] - luminosity[r,c]). Display this difference image. Where are the biggest differences?
6. Think about why the luminosity formula (0.299, 0.587, 0.114) uses unequal weights. Notice: Green gets the most weight, Blue the least. Why might that be?

Hint for question 6: Think about how human eyes perceive brightness in different colors.

Your reflection (fill in):

• Which method produced the most visually pleasing grayscale image? Why?
• Why does the luminosity formula give more weight to green?
• Where were the biggest differences between methods?

After converting a color image to grayscale, can you recover the original colors?

A. Yes — the grayscale values contain enough information to reconstruct RGB

B. No — grayscale combines three channels (R, G, B) into one number, permanently losing which colors contributed to that brightness

C. Yes, but only if you used the averaging method

D. No, but only because Python deletes the original data

You have a 100x100 image. You want to produce a 50x50 image.

The output has 4x fewer pixels than the input. Each output pixel must somehow be derived from the input pixels.

Before writing any code, answer these questions:

1. Each output pixel at position (r, c) in the 50x50 image corresponds to which input pixel(s) in the 100x100 image?
2. If output pixel (0, 0) covers input pixels (0,0), (0,1), (1,0), (1,1) — what single value should you assign to it? Think of at least 3 different choices.
3. Is there information loss when going from 100x100 to 50x50? Can you ever perfectly reconstruct the original?

Your answers before coding:

1. Output pixel (r, c) corresponds to input pixel(s): ...
2. Three possible values to assign: ...
3. Is there information loss? ...

Rules: Use only Python loops. No cv2.resize, no PIL .resize, no np.mean over blocks.

Method A — Nearest Neighbor (Subsampling) For each output pixel (r, c), simply copy the value from input pixel (2*r, 2*c).

Method B — Average Pooling For each output pixel (r, c), take the average of the 2x2 block: input[2r, 2c], input[2r, 2c+1], input[2r+1, 2c], input[2r+1, 2c+1]

Method C — Max Pooling For each output pixel (r, c), take the maximum of the same 2x2 block.

Method D — Your own! Invent a fourth method. Some ideas: min pooling, median of the 4 values, or weighted average (give corners less weight).

Tasks:

1. Apply all four methods to img_100. Display the results side by side (along with the original).
2. Which method preserves the most visual detail? Which looks smoothest? Which looks sharpest?
3. Compute a difference map: For each pair of methods, compute abs(method_A[r,c] - method_B[r,c]) for every pixel. Where do methods disagree most?
4. Think deeper: In neural networks, max pooling is used instead of average pooling. Based on what you see, why might max pooling be preferred for detecting features (like edges)?

Your observations:

• Nearest neighbor looks...
• Average pooling looks...
• Max pooling looks...
• I think max pooling is preferred in neural networks because...

Now you have a 50x50 image (use small_nearest from Part 4). You want to produce a 100x100 image.

This is the reverse of downsampling, but it's harder: you need to invent information that wasn't there.

Before coding, think:

1. Each output pixel (r, c) in the 100x100 image comes from which input pixel(s) in the 50x50 image?
2. What happens when (r, c) falls exactly between two input pixels — e.g., output pixel (1, 0) lands between input (0,0) and (1,0)?
3. Write down two different strategies.

Your strategies before coding:

1. Strategy A: ...
2. Strategy B: ...

Rules: Use only Python loops. No cv2.resize, no PIL .resize.

Method A — Nearest Neighbor Replication Each output pixel (r, c) copies from input pixel (r//2, c//2). Every input pixel gets "stretched" into a 2x2 block.

Method B — Bilinear Interpolation (1D first, then 2D) This one is harder. Let's build it up:

• First, implement linear interpolation between two values: lerp(a, b, t) = a + t * (b - a) where t is between 0 and 1.
• For upsampling 2x: output pixel (r, c) maps to input position (r/2, c/2).
• If r/2 = 1.5, you interpolate between row 1 and row 2 with weight 0.5.
• Do this for both row and column directions.

Method C — Your own! Ideas: replicate rows/columns, use the average of neighbors, or anything you can think of.

Tasks:

1. Apply all three methods. Display results next to the original 100x100 image.
2. Nearest neighbor will look "blocky" — why? What visual artifact does it create?
3. Bilinear should look smoother — why? What is the trade-off?
4. Compute the pixel-wise error between the upsampled image and the original img_100: error[r, c] = abs(int(upsampled[r,c]) - int(img_100[r,c])) Which method is closest to the original? Does this surprise you?
5. Compute the mean absolute error (you built this in the Loops & Arrays chapter) between each upsampled image and the original img_100.

When upsampling a 50x50 image to 100x100, nearest-neighbor repeats each pixel while bilinear interpolation blends neighboring pixels. Which produces a smoother result, and why?

A. Nearest-neighbor is smoother because it preserves the original pixel values exactly

B. Bilinear interpolation is smoother — it creates gradual transitions between pixels by computing weighted averages, while nearest-neighbor creates visible blocky squares

C. Both produce identical results at 2x scaling

D. Neither is smooth — you need at least 4x scaling for smooth results

So far you've doubled the image (50→100). Now go from 50x50 to 150x150 — a scale factor of 3.

Nearest neighbor is easy: output pixel (r, c) maps to input (r//3, c//3). Bilinear is similar: output (r, c) maps to input coordinates (r/3, c/3).

But here's a deeper challenge: what about going from 50x50 to 75x75? The scale factor is 1.5 — not an integer.

Output pixel (r, c) maps to input position (r / 1.5, c / 1.5) = (r * 2/3, c * 2/3). This is a fractional coordinate — you must interpolate.

The key insight: Your bilinear implementation already handles this! The formula in_r = r / scale works for any scale — integer or fractional.

Tasks:

1. Generalize your upsample_nearest and upsample_bilinear to accept an arbitrary target_h, target_w instead of a fixed scale.
2. Implement resize_nearest(img, target_h, target_w) and resize_bilinear(img, target_h, target_w).
3. Resize img_50 to 150x150 using both methods.
4. Resize img_50 to 75x75 using both methods.
5. Display and compare results.

Now let's push your implementation further.

Tasks:

1. Take a tiny 10x10 patch from img_50 (top-left corner): patch = img_50[:10, :10].
2. Upsample it to 200x200 using both nearest and bilinear.
3. Display all three (original 10x10 patch, nearest 200x200, bilinear 200x200).
4. The nearest result should look very blocky — you'll see the "pixels". The bilinear should be blurry. Why?
5. Design question: Is there a way to upsample that preserves sharp edges? Describe your idea (no code needed — just think and write).

You've learned that a pixel is a single number. Now let's think differently: a pixel together with its neighbors tells you something about local structure.

Consider a grayscale image. Look at a 3x3 patch around any pixel (r, c):

img[r-1, c-1]   img[r-1, c]   img[r-1, c+1]
img[r,   c-1]   img[r,   c]   img[r,   c+1]
img[r+1, c-1]   img[r+1, c]   img[r+1, c+1]

A filter (also called a kernel or weight matrix) is a small grid of numbers, also 3x3:

w[0,0]  w[0,1]  w[0,2]
w[1,0]  w[1,1]  w[1,2]
w[2,0]  w[2,1]  w[2,2]

The convolution output at pixel (r, c) is the sum of element-wise products. Write a mathematical expression for out[r, c] in terms of the image pixels and filter weights. Use summation notation if you can.

Before coding, think:

1. What happens at the edges of the image? (There are no neighbors outside the boundary.)
2. What does the output represent? Is it still an image? What size is it?
3. What would happen if all weights were 1/9? What computation would that be?
4. What would happen if the center weight is 1 and all others are 0?

Your answers before coding:

1. Edge problem: ...
2. Output: ...
3. All weights = 1/9: ...
4. Center weight = 1, rest = 0: ...

Implement conv(image, weights) where:

• image is a 2D numpy array of shape (H, W)
• weights is a 2D numpy array of shape (K, K) where K is odd (3, 5, 7, ...)
• Output is a 2D numpy array

Boundary strategy — "valid" mode: Only compute output where the kernel fits fully inside the image. For a 3x3 kernel on an HxW image, the output will be (H-2) x (W-2). In general, for a KxK kernel the output is (H-K+1) x (W-K+1).

For each output position (r, c), you need to combine the KxK neighborhood of the image starting at (r, c) with the weight grid. Figure out how to compute the output value using what you learned in Exercise 7.1.

Note: The output can have values outside [0, 255] — don't clamp yet. Use float arrays for the output and we'll handle display separately.

Sanity check: Use the identity filter to verify your implementation:

identity_kernel = np.array([
    [0, 0, 0],
    [0, 1, 0],
    [0, 0, 0]
], dtype=np.float64)

result = conv(img_100.astype(np.float64), identity_kernel)
# result[0,0] should equal img_100[1,1]

The output of convolution may have negative values or values > 255. We need a helper to normalize and display it.

Run this cell to make the helper available.

Now the fun part. Apply your conv function with different kernels and discover what each one does to the image.

Rules: You must apply each kernel, display the result, and describe what you see before reading the reveal below.

Kernel A

1/9  1/9  1/9
1/9  1/9  1/9
1/9  1/9  1/9

Kernel B

 0  -1   0
-1   5  -1
 0  -1   0

Kernel C

-1  -2  -1
 0   0   0
 1   2   1

Kernel D

-1   0   1
-2   0   2
-1   0   1

Tasks:

1. Apply each kernel to img_100.
2. For each result, write down what you observe before reading the reveal.

3. For Kernels C and D: can you combine the two output maps? Try:

   combined[r,c] = sqrt(result_C[r,c]**2 + result_D[r,c]**2)
   

   (Implement with a loop.)

Your observations (fill in before reading the reveal):

• Kernel A: I see...
• Kernel B: I see...
• Kernel C: I see...
• Kernel D: I see...
• Combined C + D magnitude: I see...

Now you know what a convolution filter does. Design your own.

Tasks:

1. Invent a 3x3 filter that does something interesting. Apply it and display the result.
2. Experiment with a 5x5 blur kernel. A 5x5 blur has all weights = 1/25. Apply it. Compare to the 3x3 blur. Which is blurrier? Why?

3. Gaussian blur idea: Instead of equal weights, what if the center gets more weight and edges get less? Design a 3x3 kernel where the center = 4, direct neighbors = 2, corners = 1 (then normalize so they sum to 1).

   gaussian_approx = np.array([
       [1, 2, 1],
       [2, 4, 2],
       [1, 2, 1]
   ], dtype=np.float64)
   gaussian_approx = gaussian_approx / gaussian_approx.sum()
   

   Apply it. How does it compare to the flat blur?

4. Try applying the blur filter multiple times in a row (chain: conv(conv(img, blur), blur)). What happens? Apply it 3 times, 5 times.

You may have noticed that each convolution slightly shrinks the image (98x98 from a 100x100 image with a 3x3 kernel).

A common fix is zero-padding: surround the image with a border of zeros before convolving, so the output stays the same size as the input.

• For a 3x3 kernel, add 1 pixel of zeros on all sides.
• For a 5x5 kernel, add 2 pixels.

Formula: pad_size = (K - 1) // 2

Tasks:

1. Implement pad_image(img, pad_size) that returns a new array with pad_size rows/columns of zeros added on all sides. Use loops — no np.pad!
2. Implement conv_same(image, weights) that uses pad_image internally, so the output is the same size as the input.
3. Apply conv_same with Kernel A to img_100. Verify the output shape is (100, 100).
4. Apply conv_same with Kernels C and D. Notice that now edges near the image boundary are included in the output.

So far, conv works on grayscale (2D). But real images are color (3D: H x W x 3).

How should convolution work on a color image?

Approach 1 — Per-channel: Apply the same kernel independently to each of R, G, B. Stack the three results back into a color image.

Tasks:

1. Implement conv_color(image_rgb, weights) that:
   • Takes a (H, W, 3) array and a kernel
   • Applies conv_same to each channel separately
   • Returns a (H, W, 3) float array
2. Apply it to color_arr with Kernel A. Display the result.
3. Apply it with the Sobel horizontal kernel. What do you get?
4. Think: What is a case where you'd want different kernels for different channels?

A 3x3 convolution filter has the values [[-1, -1, -1], [0, 0, 0], [1, 1, 1]]. What does this filter detect?

A. It blurs the image by averaging neighbors

B. It detects horizontal edges — it responds strongly where pixel values change rapidly from top to bottom

C. It detects vertical edges

D. It inverts the colors of the image

You now have all the building blocks. Let's build a real pipeline.

Goal: Given the original color image:

1. Convert to grayscale (using luminosity method)
2. Resize to 50x50
3. Apply a blur filter
4. Apply a Sobel edge detector (both H and V, then combine)
5. Display each step side by side

Use your own implementations of each step — no library functions for the image processing.

Here is a challenge with no given formula — you must invent the filter.

Goal: Design a 3x3 kernel — experiment with different weight patterns and observe what each one does to the image.

Think:

• You've seen kernels where all weights are equal, kernels with a large center weight and negative neighbors, and kernels with positive values on one side and negative on the other.
• What other patterns can you create? What happens if you put opposite signs on diagonal corners? What if only the corners have non-zero weights?
• Try at least 3 different kernels of your own design.

Tasks:

1. Design a 3x3 kernel, sketch it on paper, predict what it will do, then apply it to img_100 and check.
2. Design a second kernel with a very different weight pattern. Apply and compare.
3. Design a third kernel. Can you find one that highlights features the previous kernels missed?
4. For each kernel, write down what you predicted vs. what you observed.

Tasks:

1. Apply Kernel B 10 times in a row. Display the result after 1, 3, 5, 10 applications.
2. Apply Kernel A 20 times. What happens?
3. Apply Kernel C to an image, then apply Kernel A to the result. What does that produce?
4. Think: In a neural network, many convolution layers are stacked. Each layer has learned kernels. Based on what you've seen, what might the first, second, and deeper layers be detecting?

Your thoughts on stacked layers:

• Layer 1 (close to input) likely detects...
• Layer 2 detects...
• Deeper layers likely detect...

Answer these questions in your own words:

1. What is an image? Describe it as a data structure, not visually.
2. What is a convolution? Explain in 3 sentences without using the word "filter".
3. What is the difference between downsampling and upsampling? Which one loses information permanently? Can you recover from either?
4. Why does the blur kernel make the image blurry? Explain using the math — what operation does it actually perform at each pixel?
5. Why does the Sobel kernel detect edges? Think about what happens when you apply it to a flat region vs. a sharp edge.
6. In a Convolutional Neural Network (CNN), the kernels are not hand-designed — they are learned from data using gradient descent. Based on what you've built, what does "learning a kernel" mean? What is being optimized?