Line Fitting & Curve Fitting

Given a few data points, can you predict what comes next? You'll start with a simple question, discover the formula hiding in the data, and build your way to fitting lines and curves to anything.

Part 1: The Prediction Problem

There's a mystery box. You feed it a number x and it gives back a number y. You tried it twice:

x	y
1	3
2	7

x	y
1	3
2	7
3	?11
2.5	?9
2.1	?7.4
0	?−1
0.5	?1
x₁	?4x₁ − 1

How much will y be when x = 3?

✓ y = 11! When x goes up by 1, y goes up by 4.

How much when x = 2.5?

✓ y = 9! Half a step in x → half the increase.

What about x = 2.1?

✓ y = 7.4! Just 0.1 past x = 2, so 7 + 0.4.

Now try x = 0.

✓ y = −1! Going back from x = 1 subtracts 4.

What about x = 0.5?

✓ y = 1! Halfway between −1 and 3.

For any x₁, can you write y = ?

y = 4x − 1

You just discovered a linear function!

Now write a Python function with name f that takes x as argument and returns y.

def f(x):
    y = # write your code
    return y

✓ That's right! The code looks like this:

def f(x):
    y = 4 * x - 1
    return y

Now, test by calling f(2), f(3), f(0).

Now, try different values of x and check if the value of y is correct as per the table!

Let's look back and embrace what we have done.

We had data (two points) and we came up with a model (a line), and with this model we are able to answer unseen questions.

In machine learning, coming up with the model is called training and using the model to give output is called inference.

Our Line Model

ChatGPT

Training

Inference

Quick Question: can your equation of line give the output for a value that it has not seen before, say x = −100?

Yes, of course.

Similarly, can ChatGPT or LLM answer questions that it has not seen before?

Yes, of course.

The other way to look at this is — it is a linear relationship between input and output: y = a × x + b. Let's solve another problem to get a grasp of it.

Do you know how an electronic thermometer works?

When a metal is heated, its resistance changes and that changes the electric current through it. We can measure the current and estimate the temperature from it.

We have the following data points:

Current (I)	Temperature (T)
1 mA	10 °F
1.2 mA	50 °F

Find the equation: T = a × I + b. What are a and b?

Put the two values in the equation. You get two equations:

10 = a × 1 + b … (1)
50 = a × 1.2 + b … (2)

Now solve them.

(2) − (1): 40 = 0.2a → a = 200
From (1): b = 10 − 200 → b = −190

So, the final equation is:

T = 200 × I − 190

You solved two problems by hand. Both times:

Two data points → two equations → solve for a and b.

Can you write general Python functions for this?

def fit(x1, y1, x2, y2):
    # return a, b

def predict(a, b, x):
    # return y

From y₁ = a × x₁ + b and y₂ = a × x₂ + b, we get:

def fit(x1, y1, x2, y2):
    a = (y2 - y1) / (x2 - x1)
    b = y1 - a * x1
    return a, b

def predict(a, b, x):
    return a * x + b

Verify with both problems:

fit(1, 3, 2, 7) → (4, −1) predict(4, −1, 3) → 11 ✓
fit(1, 10, 1.2, 50) → (200, −190) predict(200, −190, 1.5) → 110 ✓

In the real world, you usually have more than two data points.

x	y
1	2
2	5
3	7

Fitting with the first two points:

fit(1, 2, 2, 5) → a = 3, b = −1 → y = 3x − 1

Does this line work for the third point?

At x = 3: predicted = 3×3 − 1 = 8 but actual = 7 — off by 1!

Try different pairs of points and you get different lines. We need a way to measure which line is best.

For line y = 3x − 1, the residual at each point is the difference between actual and predicted:

x	Actual	Predicted	Residual
1	2	2	0
2	5	5	0
3	7	8	−1

How do we combine these into a single number?

If we just sum them, positive and negative errors would cancel out! What if we squared them first?

(0² + 0² + (−1)²) / 3 = 0.33

This is the Mean Squared Error (MSE).

The best fit line is the one with the lowest MSE. Finding it efficiently is what gradient descent does.

Not all patterns are linear. Consider this data:

x	y
0	0
1	1
2	4
3	9

No straight line fits this well (dashed). But y = x² fits perfectly (green curve)!

This is a polynomial: y = ax² + bx + c

More parameters → more flexibility. But too many and the model memorizes data instead of learning the pattern. This is called overfitting.

There could be more inputs like x₁, x₂:

y = a × x₁ + b × x₂ + c

Quick question: to find a, b and c, how many pairs of (x₁, x₂, y) are needed?

✓ That's right! To solve for a, b and c we need at least three equations. Hence, three instances minimum.

There could be a complex relationship between inputs and outputs:

y = a × x² + b × x + c

Or there could be a much more complex relationship:

y = ax₁² + bx₂² + cx₁ + dx₂ + ex₁x₂ + f

Here we have a, b, c, d, e and f — six parameters.

In case of LLMs, there are 100s of billions of such parameters and the equation is much more complex!