Inventing Neural Networks

Here are five people's heights, labeled 1 for adult and 0 for teenager:

heights = np.array([171, 173, 163, 162, 160])
labels  = np.array([1, 1, 0, 0, 0])

A line m*x + c outputs any number — 0.37, 12.5, -3. But the answer we want is yes/no. We need something that squashes a number into a decision.

The bluntest squasher possible: step_function(z) returns 1 if z >= 0, else 0. Write it.

(In 1958 this exact function, wrapped around a weighted sum, was called the perceptron — the first artificial neuron.)

Your task: Implement step_function(z).

Here's the problem with step_function: imagine gradient descent trying to tune m and c inside step_function(m*x + c). Nudge m a little — the output usually doesn't change at all (it's still exactly 0 or exactly 1). The gradient is zero almost everywhere. Gradient descent is blind; there is no downhill to follow.

We need a step with a slope — a function that goes from 0 to 1 but smoothly, so every nudge changes the output a little. What properties must this "smooth step" have?

• Output is always between 0 and 1
• Output is 0.5 when the input is 0 (the tipping point)
• Output approaches 1 for large positive inputs, 0 for large negative inputs
• The function is smooth everywhere — it has a slope at every point

Your challenge: Try the expression 1 / (1 + math.exp(-z)). Does it meet all four properties? Test it at z = 0, z = 100, z = -100, and z = 2. If it works, use it to write sigmoid(z).

After implementing, run the provided plotting code to compare the step and sigmoid side by side.

Our model is now sigmoid(m*x + c) — a line squashed into (0, 1). Read its output as "how confident am I that this is an adult."

Write sigmoid_error(m, c, x, y) — the mean squared error between sigmoid(m*x + c) and the true labels. This is your familiar MSE with a sigmoid inside; numpy broadcasting handles the whole array at once.

Below is the finite-difference gradient helper you've built twice before, provided as-is.

Write train(m, c, x, y, learning_rate, epochs) — the standard loop: compute both gradients of sigmoid_error, step both weights against them, return the final (m, c).

Then run it on the raw heights, starting from m=0.12, c=0.3. The tests expect something strange: the error starts at 0.6 and ends at 0.6. One hundred epochs, no progress.

Before moving on, diagnose it. Print sigmoid(0.12 * 171 + 0.3). That's sigmoid(20.8) — which is 0.99999999.... Every prediction is glued to 1.0 on sigmoid's flat plateau, where the slope is essentially zero. Zero slope means zero gradient means zero learning. This is called saturation, and it is the single most classic failure mode in neural networks.

The plateau problem came from m*x being huge, because heights are ~170. Keep the inputs near zero and sigmoid operates on its steep middle section, where gradients live.

Subtract 165 from the heights (so they become [6, 8, -2, -3, -5]), train again from the same start with learning_rate=0.5, epochs=2000, and this time build the actual classifier: classify(x_centered) returns 1 when sigmoid(m*x + c) >= 0.5.

What you just invented is logistic regression — and it is exactly one neuron: weighted sum, plus bias, through a squashing activation function. The centering trick you used is why every deep-learning tutorial starts by normalizing inputs.

In Exercise 1.4, the sigmoid neuron "refused to learn" when given large input values. What caused this?

A. Large values cause Python to crash with overflow errors

B. The sigmoid function flattens to nearly 0 or 1 for large inputs — the gradient becomes almost zero, so gradient descent makes almost no progress

C. Large values make the learning rate too high

D. The sigmoid can only handle inputs between 0 and 1

Your Part 1 function draws one boundary. But many real problems need more than one boundary — what if the answer depends on combining several simple decisions?

Let's find out what happens when you feed one small function's output into another.

Implement this four-step graph as compute_graph(x, w10, w11, w20, w21, w30, w31):

y1 = step_function(w11 * x + w10)
y2 = sigmoid(y1 * w21 + w20)
y3 = x * y2
y  = w31 * y3 + w30

Work the first test by hand before coding: with all weights 1 and x = 2, y1 = step(3) = 1, y2 = sigmoid(2) ≈ 0.8808, y3 ≈ 1.7616, y ≈ 2.7616.

Run the provided cell: it plots y versus x for the graph above with one set of weights. Then:

1. Change the weights to a second set of your choosing and plot both curves together.
2. Notice the kink where the step function switches — the weights w10, w11 decide where it happens, the later weights decide what happens on each side.
3. Invent your own graph of a similar size — chain 3-5 operations mixing step_function, sigmoid, *, + — and plot it for two weight settings.

What you just built — a recipe of chained steps with tunable weights — is called a computation graph. A neural network is nothing more than a large one. The graph fixes what shapes are possible; the weights choose one shape; gradient descent finds the weights.

A single sigmoid neuron can fit a smooth S-shaped curve. Why do we need multiple neurons?

A. Multiple neurons run faster on modern hardware

B. A single S-curve can only capture one transition (e.g., "low to high"), but real problems often have multiple transitions or complex shapes that require combining several curves

C. Python limits each neuron to 10 weights

D. Multiple neurons are needed only for classification, not regression

A country taxes income like this: nothing on the first 5000; 30% on everything above. So tax = 0.3 * (income - 5000) if income > 5000, else 0. Plotted, it's a hockey stick — flat, then a straight climb, with a sharp kink at 5000.

The cell below generates 200 people and — remembering Part 1's saturation lesson! — scales incomes and taxes down to the 0-1 range before we model anything. Run it.

rng = np.random.default_rng(42)
income = rng.uniform(0, 10000, 200)
tax = np.where(income > 5000, 0.30 * (income - 5000), 0.0)

x = income / 10000     # scaled income, 0..1
y = tax / 1500         # scaled tax, 0..1 (max tax is 1500)

plt.scatter(x, y, s=12, alpha=0.6)
plt.xlabel("income (scaled)"); plt.ylabel("tax (scaled)")
plt.title("The hockey stick no line can fit")
plt.grid(alpha=0.3); plt.show()

Fit the best straight line to (x, y) — you may use np.polyfit(x, y, 1) (you derived what it computes, least squares, in Chapter 14). Compute its RMSE and look at what it predicts for low incomes.

The tests capture two indictments: the RMSE stays above 0.1 no matter what, and for incomes below 3000 the line predicts negative tax — the government paying you. The problem isn't the fitting; the shape of the model is wrong.

Plot the line against the data to see the mismatch.

A human would solve this in two steps: is this person taxable at all? If no, the answer is 0. If yes, it's a simple line. Let's build exactly that.

Step one is a yes/no question — a job for the neuron from Part 1.

Create is_taxable = (tax > 0).astype(int) and train a sigmoid classifier sigmoid(m*x + c) on (x, is_taxable) — your train from Exercise 1.4 works unchanged (inputs are already scaled; use learning_rate=2.0, epochs=3000, starting from m=1, c=0). Report the accuracy of thresholding at 0.5, and print where the decision boundary sits: -c/m (in scaled income; multiply by 10000 to see it in currency — it should land near 5000).

Step two: for the taxable people only, the relationship is a perfect straight line. Fit a line on just the rows where is_taxable == 1 — on the scaled data it should come out at exactly m=2, c=-1 (check: at x=1, i.e. income 10000, scaled tax is 2*1 - 1 = 1, i.e. 1500).

Then chain the two models into predict_tax(xi):

• if the classifier says taxable -> return the line's prediction,
• else -> return 0.0.

Compare its RMSE with the line's from 3.1.

What you invented: a gated model — one network deciding which expert handles the input. Scaled up, this idea is called a mixture of experts, and it's inside several frontier LLMs. It's also the honest answer to "how do I put an if inside a machine-learning model."

The pipeline needed us to know the problem splits into classify-then-regress. Could a single network figure the shape out by itself? Try the smallest one imaginable — two neurons in a chain:

$$\hat{y} = w_1 \cdot \sigma(m x + c) + w_0$$

Neuron 1 squashes, neuron 2 scales and shifts. Four weights.

First write fit_net(err_fn, params, x, y, lr, epochs, h=1e-6) — your gradient-descent loop generalized to a list of parameters: each epoch, finite-difference each parameter (copy the list, nudge one entry), then step all of them. (You wrote the two-parameter and many-weight versions in Chapter 15 — this is the same music.)

Then define net_error(params, x, y) for the two-neuron model and fit it from [1.0, 0.0, 1.0, 0.0] with lr=0.5, epochs=5000.

The tests expect an honest middle result: clearly better than the line (RMSE ≈ 0.05 vs 0.14), clearly worse than the pipeline. Look at the plot to see why — sigmoid rises and then flattens, but real tax keeps climbing. A smooth step times a constant can imitate a kink, not a ramp.

Look at the tax rule one more time: zero below a threshold, a straight ramp above it. There is an activation function that is that shape:

Your challenge: Before looking at any formula, write a function that outputs 0 for negative inputs and the input itself for positive inputs. That's the shape you need. Write relu(z) (use np.maximum so it works on arrays).

Then reuse your fit_net on the model $\hat{y} = w_1 \cdot \text{relu}(m x + c) + w_0$ — same starting point, same settings, only the activation changed.

The tests expect near-perfection: the network can now represent the true rule exactly (one solution: m=2, c=-1, w1=1, w0=0 — but gradient descent may find an equivalent scaling, e.g. m*w1 = 2).

The reveal: ReLU — the rectified linear unit — is the default activation of modern deep learning, powering nearly every large model. Sigmoid saturates at both ends (you met that plateau in Exercise 1.4); ReLU never saturates on the positive side, and stacks of ReLU neurons build arbitrary piecewise-linear shapes — kinks upon kinks, like the one you just fit with a single neuron.

No tests here — experiment and observe:

1. Replace sigmoid with np.tanh in the two-neuron net. Better or worse on the tax data? (tanh is a sigmoid stretched to output -1...1.)
2. Add a second ReLU neuron: $\hat{y} = w_1\,\text{relu}(m_1 x + c_1) + w_2\,\text{relu}(m_2 x + c_2) + w_0$ (seven parameters — fit_net doesn't care). Invent a tax rule with two brackets and see if it can fit that too.
3. Feed x in again at the output: $\hat{y} = w_1\,\text{relu}(mx+c) + w_2 x + w_0$. (Networks with such shortcuts are called residual — the trick that made very deep networks trainable.)

ReLU (max(0, z)) is simpler than sigmoid. What is its main practical advantage for training neural networks?

A. ReLU outputs larger numbers, making predictions more accurate

B. ReLU's gradient is either 0 or 1 — it never becomes a tiny fraction like sigmoid's, so gradient descent doesn't slow down for large inputs

C. ReLU uses less memory than sigmoid

D. ReLU can only be used in the output layer