Differentiation

Feed it two inputs very close together:

Compute $y_1$ and $y_2$.

Now compute:

$\dfrac{y_2 - y_1}{x_2 - x_1} = \;?$

$= \dfrac{2.02 - 2}{1.01 - 1} = \dfrac{0.02}{0.01} = \mathbf{2}$

What does this number mean?

When $x$ increased by $0.01$, $y$ increased by $0.02$ — twice as fast. This ratio measures how fast $y$ changes per unit change in $x$.

Now:

1. Compute f(1)
2. Increase $x$ by a tiny amount — try 0.000001. Compute f(1.000001)
3. Compute $\dfrac{y_2 - y_1}{x_2 - x_1}$

The derivative of $y = 3x$ is $\mathbf{3}$. That makes sense — for every 1 unit increase in $x$, $y$ increases by 3.

You just computed a derivative using Python. Well done!

Using the same technique (nudge $x$ by $0.000001$), find the derivative of $g$ at each of these points:

Notice something? The derivative of $x^2$ at $x$ seems to be $2x$. Unlike $y = 3x$ where the derivative was the same everywhere, here it changes depending on where you are.

At $x = 0$ the function is flat (derivative = 0). At $x = 2$ it’s climbing steeply (derivative = 4).

Your turn: Put this into a reusable Python function. It should work for any function, at any point $x$:

def diff(my_func, x):
    """Return the derivative of my_func at point x."""
    # your code here

Test it:

print(diff(g, 1))    # should be ≈ 2
print(diff(g, -1))   # should be ≈ -2
print(diff(g, 0))    # should be ≈ 0
print(diff(g, 2))    # should be ≈ 4

You now have a function that can compute the derivative of any function at any point. You just invented numerical differentiation.

Let’s try with $y = x^2$.

If our input is $x$, then $y_1 = x^2$.

Now nudge $x$ by a tiny amount $d$. The new input is $x + d$.

What is $y_2$? Expand $(x + d)^2$.

$y_2 = (x + d)^2 = x^2 + 2xd + d^2$

Now compute the derivative:

$\dfrac{y_2 - y_1}{(x + d) - x}$

$= \dfrac{(x^2 + 2xd + d^2) - x^2}{d}$

$= \dfrac{2xd + d^2}{d} = 2x + d$

But $d$ is a tiny nudge — as small as we want. In the limit, $d$ vanishes to zero.

$\dfrac{dy}{dx} = \mathbf{2x}$

That’s exactly the pattern you saw numerically! At $x = 1$ the derivative is $2$, at $x = 2$ it’s $4$, at $x = 0$ it’s $0$.

This is algebraic (or analytical) differentiation — an exact answer, no approximation.

Verify your answers with your diff function from Part 1!

Now verify these algebraic results using your diff function from Part 1:

def c(x):  return 10
def h(x):  return 5*x + 6
def k(x):  return 3*x**2 + 2

print(diff(c, 3))   # should be ≈ 0
print(diff(h, 3))   # should be ≈ 5
print(diff(k, 3))   # should be ≈ 6*3 = 18

Notice the patterns: a constant disappears, $5x$ becomes $5$, and $3x^2$ becomes $6x$. You’re discovering the rules of differentiation.

Can you guess the derivative of $x^3$? What about $x^4$? Try to see the pattern before we verify.

$y_2 = (x + d)^3 = x^3 + 3x^2 d + 3x d^2 + d^3$

$\dfrac{y_2 - y_1}{d} = \dfrac{3x^2 d + 3x d^2 + d^3}{d} = 3x^2 + 3xd + d^2$

Drop terms with $d$ → $\dfrac{dy}{dx} = \mathbf{3x^2}$

Now look at the pattern:

This is the Power Rule. You just invented it.

The derivative of a sum is the sum of the derivatives.

(Shorthand: $f'(x)$ means “the derivative of $f$ at $x$” — same as $\dfrac{df}{dx}$, just shorter to write.)

$\dfrac{d}{dx}\big[f(x) + g(x)\big] = f'(x) + g'(x)$

Question 2: The derivative of $x^2$ is $2x$. The derivative of $3x^2$ is $6x = 3 \cdot 2x$. What happened to the constant multiplier $3$? Can you state a rule for differentiating $c \cdot f(x)$?

A constant multiplier stays put:

$\dfrac{d}{dx}\big[c \cdot f(x)\big] = c \cdot f'(x)$

Verify each answer numerically with your diff function!

You can now differentiate any polynomial — instantly, exactly, by hand. These are the same rules taught in calculus courses. You didn’t memorize them — you derived them.

$y = x_1^2 + 2 x_1 x_2$

How do you find the derivative now? With respect to which input?

The trick: differentiate with respect to one input at a time, treating the other as a constant. This is called a partial derivative.

(Hint: $2 x_1 x_2$ is just “$\text{constant} \cdot x_1$” when $x_2$ is fixed, and “$\text{constant} \cdot x_2$” when $x_1$ is fixed.)

$\dfrac{\partial y}{\partial x_1} = 2x_1 + 2x_2$

(power rule on $x_1^2$ → $2x_1$;   $2x_2$ is a constant times $x_1$ → $2x_2$)

$\dfrac{\partial y}{\partial x_2} = 2x_1$

($x_1^2$ is a constant with respect to $x_2$ → $0$;   $2x_1 \cdot x_2$ → $2x_1$)

The symbol $\partial$ (curly-d) reminds us it’s a partial derivative — we only moved one input.

Compute $\dfrac{\partial y}{\partial x_1}$ and $\dfrac{\partial y}{\partial x_2}$ at these points:

(Remember: to nudge only $x_1$, compute $\dfrac{f(x_1 + h,\; x_2) - f(x_1,\; x_2)}{h}$. Same idea for $x_2$.)

Your algebraic answers and numerical answers should match. Algebra gives the formula; numbers let you verify.

Test it:

print(diff2(f, 1, 1))      # should be ≈ (4, 2)
print(diff2(f, 2, 1.5))    # should be ≈ (7, 4)

Each partial derivative nudges one input while keeping the other fixed — exactly what you did algebraically, but automated.

In machine learning, most functions have hundreds or millions of inputs (weights). Partial derivatives tell the optimizer how to tweak each weight individually. You just invented the building block of gradient descent.

First problem: how do you even call such a function? Writing f(x1, x2, x3, ...) doesn’t scale.

Python has a trick — you can pass a list as arguments using *:

def f(a, b, c):
    return a + b + c

args = [10, 20, 30]
f(*args)            # same as f(10, 20, 30)  → 60

The * unpacks the list into separate arguments. This works for any length.

$3^2 + 4^2 + 5^2 = 9 + 16 + 25 = \mathbf{50}$

Now, you can also modify one element of the list and call the function again:

nudged = inputs.copy()
nudged[0] = inputs[0] + 0.000001    # nudge only x1
print(g(*nudged))                   # ≈ 50.000006

That’s exactly what you need for a partial derivative — nudge one input, keep the rest fixed.

Hints:

• Loop over each index $i$ in arr
• Make a copy, nudge position $i$ by a tiny $h$
• Compute $\dfrac{f(\text{nudged}) - f(\text{original})}{h}$
• Use * to unpack when calling my_func

Test it with a 3-argument function:

def f3(x1, x2, x3):
    return x1**2 + x2**2 + x3**2

print(diffn(f3, [3, 4, 5]))
# ≈ [6, 8, 10]   (i.e. 2·x1, 2·x2, 2·x3)

Verify by hand:

This list of all partial derivatives has a name: the gradient. Your diffn computes the gradient of any function with any number of inputs.

Your turn: Write two Python functions matching the diagram:

• zz(x) — takes $x$, returns $2x + 5$
• yy(z) — takes $z$, returns $z^2$

Now use them together — compute $z$ and $y$ at $x = 1$:

z_val = zz(1)
y_val = yy(z_val)
print(z_val, y_val)

$z = 2(1) + 5 = \mathbf{7}$

$y = 7^2 = \mathbf{49}$

Now, using your diff function from Part 1, compute these two derivatives at $x = 1$:

1. $\dfrac{dz}{dx}$ — how fast does $z$ change when you nudge $x$?
2. $\dfrac{dy}{dz}$ — how fast does $y$ change when you nudge $z$? (use $z = 7$)

$\dfrac{dz}{dx} = \mathbf{2}$   (slope of $2x + 5$)

$\dfrac{dy}{dz} = \mathbf{14}$   ($2z$ at $z = 7$)

Now the big question: can you figure out $\dfrac{dy}{dx}$ — how fast does $y$ change when you nudge $x$ — using only these two numbers?

Think about it before moving on.

$\dfrac{dy}{dx} = \dfrac{dy}{dz} \times \dfrac{dz}{dx} = 14 \times 2 = \mathbf{28}$

This is the Chain Rule: when functions are chained together, you multiply the derivatives along the chain.

$\dfrac{dy}{dx} = \dfrac{dy}{dz} \cdot \dfrac{dz}{dx}$

Now compute its derivative numerically at $x = 1$:

print(diff(yz, 1))    # should be ≈ 28

$\approx \mathbf{28}$ — exactly what the chain rule predicted!

$\dfrac{dy}{dz} \times \dfrac{dz}{dx} = 14 \times 2 = 28 \;\checkmark$

In neural networks, each layer is a function feeding into the next. The chain rule lets you compute how the final output changes when you tweak a weight in any layer. That process is called backpropagation.

Notice: $x_4$ skips block 1 and feeds directly into block 2. This is common in real networks.

Check:

$\dfrac{\partial y}{\partial x_4} = \mathbf{3}$   $\dfrac{\partial y}{\partial z} = \mathbf{3}$

(Both are just constants — $y$ is linear in $z$ and $x_4$.)

Check:

$\dfrac{\partial z}{\partial x_1} = 2x_1 + 2x_2$   $\dfrac{\partial z}{\partial x_2} = 2x_1 - 3x_2^2$   $\dfrac{\partial z}{\partial x_3} = 5$

Your turn: Evaluate at $x_1\!=\!1,\; x_2\!=\!2,\; x_3\!=\!1,\; x_4\!=\!3$.

The gradient is $[18,\; -30,\; 15,\; 3]$.

$\approx [18,\; -30,\; 15,\; 3]$ — matches your algebraic answers. $\checkmark$

With 4 inputs the difference is small. But imagine a neural network with a million weights.

The chain rule lets you compute $\dfrac{\partial y}{\partial z}$ once and reuse it for every weight in block 1 — that’s the efficiency behind backpropagation.

You work backwards from the output: compute each block’s local derivatives, then multiply along the chain. One pass, all gradients.

$\dfrac{2.89 - 2.25}{1.7 - 1.5} = \dfrac{0.64}{0.2} = 3.2$ — exactly the ratio you’ve been computing!

Derivative of $x^2$ is $2x$, so at $x = 1.5$: slope $= 2(1.5) = \mathbf{3}$.

The derivative at a point is the slope of the tangent line at that point.

Your turn: For each tangent line, is the curve rising, falling, or flat? What is the connection between the sign of the slope and the curve’s direction?

This is why gradient descent works: if the derivative is positive, moving left decreases the function. If it’s negative, moving right decreases it. If it’s zero, you’re at the bottom.

The derivative doesn’t just tell you how fast — it tells you which direction to go.

$\dfrac{dy}{dx} = 2x - 10$

Where is $\dfrac{dy}{dx} = 0$?

$2x - 10 = 0 \;\Rightarrow\; x = 5$. That’s where the curve is flat — a candidate for a minimum or maximum. What is $y$ there?

$y = 25 - 50 + 5 = \mathbf{-20}$. The minimum is at $(5, -20)$.

Now check your answers on the next slide.

See the pattern? The slope goes from negative → zero → positive as you cross the minimum.

The slope is increasing as you move left to right: $-6 \to -2 \to 0 \to +2 \to +6$.

Question: is the rate of change of the slope positive or negative at the minimum?

It’s positive — the slope is increasing. The slope keeps growing from negative through zero to positive.

Your turn: $\dfrac{dy}{dx} = 2x - 10$. What is $\dfrac{d^2 y}{dx^2}$?

$\dfrac{d^2 y}{dx^2} = \mathbf{2}$ — a positive constant.

Positive second derivative means the slope is always increasing — the curve is bending upward (like a bowl). That guarantees the flat point ($dy/dx = 0$) is a minimum, not a maximum.

This is why the second derivative matters in optimization: it tells you whether a flat point is a valley you want to reach or a hilltop you want to avoid.

Hint: the second derivative is the derivative of the derivative. You already have a function that computes $\dfrac{dy}{dx}$ — can you write a new function whose output is $\dfrac{dy}{dx}$, and then pass that to diff?

Start with:

def f(x):
    return x**2 - 10*x + 5

def dfdx(x):
    """Return the first derivative of f at x."""
    # YOUR CODE HERE

def d2fdx(x):
    """Return the second derivative of f at x."""
    # YOUR CODE HERE

print(d2fdx(5))    # should be ≈ 2

The second derivative of $x^2 - 10x + 5$ is $\mathbf{2}$ everywhere — matching your algebraic answer. The curve always bends upward at the same rate.

Nobody taught you these — you derived them yourself. The next time you see $\nabla$, backprop, or gradient descent, you’ll know exactly what’s happening under the hood.