Inventing a Loss Function

Imagine you're building a model that predicts house prices (in lakhs) for 5 different houses at a time. You want to compute the overall error of this model — a single number — so that you can try tweaking each individual weight and see how that weight impacts the overall error.

The simplest idea: for each house, compute predicted - actual, then add them all up.

Try it. What number do you get? Does that number accurately reflect how wrong the predictions are?

The problem is clear: positive and negative differences cancel each other out.

Think of at least two different mathematical operations that would prevent this cancellation — operations that make every error contribute positively to the total, regardless of whether the prediction was too high or too low.

Idea 1:

Idea 2:

What you just invented:

You independently arrived at the two most important loss functions in machine learning:

• Absolute differences → leads to Mean Absolute Error (MAE)
• Squared differences → leads to Mean Squared Error (MSE)

The entire field of regression uses one of these (or a combination). Let's formalise them.

Compute MSE and MAE on the original house data. Then imagine house 5's prediction goes wildly wrong — say predicted[4] = 100 instead of 2.0. Compute both metrics again.

1. Which metric changes more dramatically with the outlier?
2. Why does that happen?

You've just seen one drawback of MSE. Let's think more carefully:

1. MSE problem: When a single prediction is very wrong (an outlier), what happens to the MSE? Is this good or bad for training a model?

2. MAE — any issues? Think about the shape of |x| as a function. Is there anything unusual about it at x = 0? (We'll investigate this in the next part.)

MSE drawback:

Possible MAE issue:

Remember why we wanted a loss function in the first place? We said:

"We need to tweak each individual weight and see how that weight impacts the overall error."

That's exactly what differentiation does. The gradient of the loss with respect to a parameter tells you: if I nudge this parameter a tiny bit, how much does the loss change?

Our training algorithm (gradient descent) depends on this. So our loss function must be differentiable — it must have a well-defined gradient everywhere.

Let's check whether our two loss functions qualify.

For the squared loss $L = \text{diff}^2$, compute the gradient $\frac{dL}{d(\text{diff})}$ using the power rule.

Then verify your formula numerically: pick a few values of diff (e.g. -2, 0, 1, 3), compute the gradient using finite differences $\frac{L(\text{diff} + \epsilon) - L(\text{diff} - \epsilon)}{2\epsilon}$ with a small $\epsilon$, and check that it matches your analytical formula.

Now for the absolute loss $L = |\text{diff}|$, compute the gradient $\frac{dL}{d(\text{diff})}$.

Think piecewise:

• When diff > 0, what is |diff|? What's its derivative?
• When diff < 0, what is |diff|? What's its derivative?

Then verify numerically for diff values: -2, -0.5, 0.5, 3.

You found that the gradient of |diff| is +1 when diff > 0, and -1 when diff < 0.

But what is the gradient when diff = 0 exactly? Is it -1 or +1?

Try computing the numerical gradient at diff = 0. Then think about what the graph of y = |x| looks like at x = 0. Is it smooth, or is there something unusual?

Gradient of |diff| at diff = 0:

What does the graph look like at that point?

Why this matters:

Our training algorithm (gradient descent) works by computing the gradient of the loss and using it to update weights. If the gradient doesn't exist at some point, the algorithm breaks down.

• MSE ($\text{diff}^2$): Differentiable everywhere. Gradient = $2 \cdot \text{diff}$ — smooth, well-defined, no problems. But the gradient grows without bound for large errors, making training unstable with outliers.

• MAE ($|\text{diff}|$): Not differentiable at diff = 0. The sharp corner means gradient descent has no direction to follow when the prediction is exactly right.

So MSE has a problem with large errors, and MAE has a problem at zero error. Can we do better?

Here's our situation:

What if we could use the squared loss for small errors (where it's well-behaved) and the absolute loss for large errors (where it stays bounded)?

Here's the simplest idea. Pick some threshold $d$ and define:

$$L(\text{diff}) = \begin{cases} \text{diff}^2 & \text{if } |\text{diff}| \le d \ |\text{diff}| & \text{if } |\text{diff}| > d \end{cases}$$

Try $d = 1$. Now compute:

1. The loss from the squared formula at $|\text{diff}| = d = 1$: what is $d^2$?
2. The loss from the absolute formula at $|\text{diff}| = d = 1$: what is $|d|$?
3. Are they equal?

Now compute the gradients at that same point:

4. Gradient of $\text{diff}^2$ at diff = 1?
5. Gradient of $|\text{diff}|$ at diff = 1?
6. Are they equal?

What's the problem?

Value from squared formula at |diff|=1:

Value from absolute formula at |diff|=1:

Gradient from squared at diff=1:

Gradient from absolute at diff=1:

The problem:

Plot the naive combined loss for $\text{diff}$ ranging from -3 to 3, using $d = 1$.

You should see a clear discontinuity (jump) at diff = 1 and diff = -1 where the two formulas switch.

Also plot the individual squared and absolute losses on the same graph for comparison.

We keep $\text{diff}^2$ for $|\text{diff}| \le d$. For $|\text{diff}| > d$, we use $A \cdot |\text{diff}| + B$.

At the boundary $|\text{diff}| = d$, we need two conditions:

1. Values match: $d^2 = A \cdot d + B$
2. Gradients match: the gradient of $\text{diff}^2$ at $\text{diff} = d$ must equal the gradient of $A \cdot |\text{diff}| + B$ at $\text{diff} = d$.

This gives you two equations and two unknowns ($A$ and $B$). Solve for $A$ and $B$ in terms of $d$.

Then write a Python function smooth_combined_loss(diff, d) using your values of $A$ and $B$, and plot it. Verify there's no jump or kink at $\pm d$.

You just invented the Huber loss!

The standard formulation divides everything by 2 for convenience (so the squared part's gradient is $\text{diff}$ instead of $2 \cdot \text{diff}$):

$$L_{\delta}(\text{diff}) = \begin{cases} \frac{1}{2}\text{diff}^2 & |\text{diff}| \le \delta \ \delta \cdot |\text{diff}| - \frac{1}{2}\delta^2 & |\text{diff}| > \delta \end{cases}$$

This is mathematically identical to yours — just scaled by ½. It was published by the statistician Peter Huber in 1964 and is built into every modern ML framework: torch.nn.HuberLoss, tf.keras.losses.Huber, sklearn's HuberRegressor.

You derived it from scratch by asking three questions:

1. How do we measure error without cancellation? → squared / absolute
2. What are the problems with each? → outlier sensitivity / non-differentiability
3. Can we combine them smoothly? → match value and gradient at the boundary → Huber loss

Let's implement the standard (÷2) Huber loss and see how it compares to MSE and MAE on real data.

Write a function huber_loss(y_true, y_pred, delta) that returns the average Huber loss over all pairs. Use the standard formulation (with the ½ factor):

• If $|y_i - \hat{y}_i| \le \delta$: loss $= \frac{1}{2}(y_i - \hat{y}_i)^2$
• Else: loss $= \delta \cdot |y_i - \hat{y}_i| - \frac{1}{2}\delta^2$

Test it:

huber_loss([5, 2, 7], [4.8, 2.5, 10], delta=1)  # ≈ 0.8817
huber_loss([1, 2, 3], [1, 2, 3], delta=1)        # 0.0

Plot all three loss functions — MSE, MAE, and Huber — for a single prediction as diff varies from -5 to 5. Use delta = 1.

Verify visually:

1. Huber is smooth everywhere (no sharp corner at 0)
2. Huber is quadratic near zero (matches MSE)
3. Huber is linear in the tails (matches MAE)
4. The transition at $\pm \delta$ is seamless

Compute all three losses on the house price data — with and without the outlier.

actual    = [1.0, 1.1, 0.9, 0.5, 3.0]
normal    = [0.9, 2.0, 0.7, 0.3, 2.0]
outlier   = [0.9, 2.0, 0.7, 0.3, 100.0]

Use delta = 1. Which loss function gives the most sensible numbers in both scenarios?

Going further:

• Pseudo-Huber loss: A smooth approximation $L = \delta^2 \left(\sqrt{1 + (\text{diff}/\delta)^2} - 1\right)$ that avoids even the (smooth) transition point.
• Quantile loss: If you care more about under-predictions vs over-predictions, you can weight the positive and negative sides differently.
• Log-cosh loss: $L = \log(\cosh(\text{diff}))$ — another smooth alternative that's twice differentiable everywhere.
• Try different values of $\delta$ in your Huber loss and see how the curve changes. What happens as $\delta \to 0$? As $\delta \to \infty$?