Classification Metrics

How do you know if a classifier is good? You'll start with raw predictions, invent the confusion matrix, derive precision, recall, and F1 — then discover how to choose the best threshold and compare models using ROC curves and AUC.

Here are two arrays — one containing the actual labels and the other the predicted labels from a binary classifier:

actual = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
predicted = [1, 0, 0, 1, 0, 1, 1, 0, 1, 0]

Index	0	1	2	3	4	5	6	7	8	9
Actual	1	0	1	1	0	1	0	0	1	0
Predicted	1	0	0	1	0	1	1	0	1	0

Some predictions match, some don’t. Can we organize these errors into a structured summary?

Each prediction has two properties:

What was the actual label? (0 or 1)
What did the model predict? (0 or 1)

That gives 2 × 2 = 4 possible outcomes. Try arranging them in a table — rows for actual, columns for predicted.

	Predicted
	0 (Negative)	1 (Positive)
Actual 0	TN True Negative	FP False Positive
Actual 1	FN False Negative	TP True Positive

This is called a confusion matrix. Green = correct, Red = errors.
The first word says if the prediction was right (True) or wrong (False). The second is what the model predicted (Positive or Negative).

Go through the arrays and fill in the confusion matrix:

actual = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
predicted = [1, 0, 0, 1, 0, 1, 1, 0, 1, 0]

	Predicted 0	Predicted 1
Actual 0	TN = ?4 True Negative	FP = ?1 False Positive
Actual 1	FN = ?1 False Negative	TP = ?4 True Positive

TP: indices 0, 3, 5, 8 • TN: indices 1, 4, 7, 9 • FP: index 6 • FN: index 2

Write a function that computes the confusion matrix:

def confusion_matrix(actual, predicted):
    """Return (tp, tn, fp, fn)."""
    # your code here

def confusion_matrix(actual, predicted):
    tp, tn, fp, fn = 0, 0, 0, 0
    for a, p in zip(actual, predicted):
        if a == 1 and p == 1: tp += 1
        elif a == 0 and p == 0: tn += 1
        elif a == 0 and p == 1: fp += 1
        elif a == 1 and p == 0: fn += 1
    return tp, tn, fp, fn

confusion_matrix(actual, predicted) → (4, 4, 1, 1)

Now that we have TP=4, TN=4, FP=1, FN=1 — what should we measure?

The model made 5 positive predictions (TP + FP). What fraction of those were actually positive?

4 / 5 = 0.80 → This is Precision = TP / (TP + FP)
“Of everything I called positive, how many really were?”

There were 5 actual positives (TP + FN). How many did the model catch?

4 / 5 = 0.80 → This is Recall = TP / (TP + FN)
“Of all the real positives, how many did I find?”

Can we combine precision and recall into one number?

F1 Score = 2PR / (P + R) — the harmonic mean, not (P+R)/2, because the arithmetic mean lets one bad metric hide behind a good one (e.g. P=1.0, R=0.0 → average 0.5 but F1 = 0)

A few more metrics you’ll need:

Accuracy	(TP+TN) / Total	Overall correctness (misleading if imbalanced)
TPR	TP / (TP+FN)	Same as recall — used in ROC curves
FPR	FP / (FP+TN)	“Of all negatives, how many did I falsely flag?”

Given TP = 4, TN = 4, FP = 1, FN = 1, calculate each metric:

Metric	Formula	Value
Precision	TP / (TP + FP)	?
Recall	TP / (TP + FN)	?
F1 Score	2PR / (P + R)	?
Accuracy	(TP + TN) / Total	?
TPR	TP / (TP + FN)	?
FPR	FP / (FP + TN)	?

Metric	Formula	Calculation	Value
Precision	TP / (TP + FP)	4 / (4 + 1)	0.80
Recall	TP / (TP + FN)	4 / (4 + 1)	0.80
F1 Score	2PR / (P + R)	2 × 0.8 × 0.8 / 1.6	0.80
Accuracy	(TP + TN) / Total	(4 + 4) / 10	0.80
TPR	TP / (TP + FN)	4 / (4 + 1)	0.80
FPR	FP / (FP + TN)	1 / (1 + 4)	0.20

Accuracy looks great at 0.80. But consider this:

A hospital tests 1000 patients for a rare disease. Only 10 actually have it.

A lazy model says: “Nobody is sick.”

What is its accuracy?

Accuracy = 990 / 1000 = 99%!

But what are precision, recall, and F1?

TP = 0, FN = 10 → Recall = 0% — the model missed every sick patient.

Accuracy is misleading when classes are imbalanced.
That’s why we need precision and recall.

Write functions for each metric:

def precision(tp, fp):
    # return ?

def recall(tp, fn):
    # return ?

def f1_score(p, r):
    # return ?

def true_positive_rate(tp, fn):
    # return ?

def false_positive_rate(fp, tn):
    # return ?

def precision(tp, fp):
    return tp / (tp + fp)

def recall(tp, fn):
    return tp / (tp + fn)

def f1_score(p, r):
    return 2 * p * r / (p + r)

def true_positive_rate(tp, fn):
    return tp / (tp + fn)

def false_positive_rate(fp, tn):
    return fp / (fp + tn)

A model doesn’t always output 0 or 1 — it often outputs a probability:

outputs = [0.9, 0.3, 0.4, 0.8, 0.2, 0.7, 0.55, 0.1, 0.85, 0.35]
actual = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]

Write a function that converts outputs to 0s and 1s using a threshold:

def apply_threshold(outputs, threshold):
    # return list of 0s and 1s

def apply_threshold(outputs, threshold):
    return [1 if o >= threshold else 0 for o in outputs]

apply_threshold(outputs, 0.5) → [1, 0, 0, 1, 0, 1, 1, 0, 1, 0]

That’s our original predicted array!

Question is — how do we decide on this threshold?

t = 0.3 → [1,1,1,1,0,1,1,0,1,1] (8 positives)
t = 0.5 → [1,0,0,1,0,1,1,0,1,0] (5 positives)
t = 0.8 → [1,0,0,1,0,0,0,0,1,0] (3 positives)

Lower threshold → more positives.
Higher threshold → fewer positives.

Let’s see what happens to our metrics at each threshold.

For threshold t = 0.5, what is the precision?

outputs = [0.9, 0.3, 0.4, 0.8, 0.2, 0.7, 0.55, 0.1, 0.85, 0.35]
actual = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]

apply_threshold → [1,0,0,1,0,1,1,0,1,0] → TP=4, FP=1 → Precision = 0.80

Now write a function that computes precision for any threshold:

def precision_at_threshold(actual, outputs, t):
    # use apply_threshold, confusion_matrix, precision

def precision_at_threshold(actual, outputs, t):
    pred = apply_threshold(outputs, t)
    tp, tn, fp, fn = confusion_matrix(actual, pred)
    return precision(tp, fp) if (tp + fp) > 0 else 0

Run precision_at_threshold for each threshold:

for t in [0.2, 0.3, 0.4, 0.5, 0.6, 0.8, 0.9]:
    print(f"t={t}: {precision_at_threshold(actual, outputs, t):.2f}")

Threshold	0.2	0.3	0.4	0.5	0.6	0.8	0.9
Precision	0.56	0.63	0.83	0.80	1.00	1.00	1.00

Best precision: t ≥ 0.6 (Precision = 1.00)

Higher threshold → fewer false positives → higher precision.

Now do the same for recall. For t = 0.5, what is the recall?

Same predictions: TP=4, FN=1 → Recall = 0.80

Write a similar function for recall:

def recall_at_threshold(actual, outputs, t):
    # use apply_threshold, confusion_matrix, recall

def recall_at_threshold(actual, outputs, t):
    pred = apply_threshold(outputs, t)
    tp, tn, fp, fn = confusion_matrix(actual, pred)
    return recall(tp, fn) if (tp + fn) > 0 else 0

Run recall_at_threshold for each threshold:

for t in [0.2, 0.3, 0.4, 0.5, 0.6, 0.8, 0.9]:
    print(f"t={t}: {recall_at_threshold(actual, outputs, t):.2f}")

Threshold	0.2	0.3	0.4	0.5	0.6	0.8	0.9
Recall	1.00	1.00	1.00	0.80	0.80	0.60	0.20

Best recall: t ≤ 0.4 (Recall = 1.00)

Lower threshold → fewer misses → higher recall. The opposite of precision!

For threshold t = 0.5, what is the F1 Score?

You already know: Precision = 0.80, Recall = 0.80

F1 = 2 × 0.8 × 0.8 / (0.8 + 0.8) = 0.80

Write a function for F1 at any threshold:

def f1_at_threshold(actual, outputs, t):
    # use precision_at_threshold, recall_at_threshold, f1_score

def f1_at_threshold(actual, outputs, t):
    p = precision_at_threshold(actual, outputs, t)
    r = recall_at_threshold(actual, outputs, t)
    return f1_score(p, r) if (p + r) > 0 else 0

Run f1_at_threshold for each threshold:

for t in [0.2, 0.3, 0.4, 0.5, 0.6, 0.8, 0.9]:
    print(f"t={t}: {f1_at_threshold(actual, outputs, t):.2f}")

Threshold	0.2	0.3	0.4	0.5	0.6	0.8	0.9
F1	0.71	0.77	0.91	0.80	0.89	0.75	0.33

Best F1: t = 0.4 (F1 = 0.91)

Best precision wants a high threshold. Best recall wants a low threshold. F1 finds the sweet spot.

Which metric would you optimize?

Scenario 1: You’re screening patients for cancer.
Missing a sick patient (FN) could be fatal.
→ Optimize ___?

Scenario 2: You’re filtering spam emails.
Blocking a real email (FP) is very annoying.
→ Optimize ___?

1. Recall — catch every positive, even if some healthy patients get extra tests.

2. Precision — only block when sure, even if some spam slips through.

The right metric depends on the cost of each type of error.

Plot Precision, Recall, and F1 vs Threshold to visualize the tradeoff:

def plot_metrics_vs_threshold(actual, outputs, thresholds):
    # Reuse your _at_threshold functions!
    # Plot precision, recall, F1 vs threshold

import matplotlib.pyplot as plt

def plot_metrics_vs_threshold(actual, outputs, thresholds):
    precs = [precision_at_threshold(actual, outputs, t) for t in thresholds]
    recs  = [recall_at_threshold(actual, outputs, t) for t in thresholds]
    f1s   = [f1_at_threshold(actual, outputs, t) for t in thresholds]
    plt.plot(thresholds, precs, label='Precision')
    plt.plot(thresholds, recs, label='Recall')
    plt.plot(thresholds, f1s, label='F1')
    plt.xlabel('Threshold'); plt.ylabel('Score')
    plt.legend(); plt.show()

Now apply this to a real dataset — logistic regression on Iris (virginica vs. rest):

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import numpy as np

iris = load_iris()
mask = iris.target != 0
X = iris.data[mask]
y = (iris.target[mask] == 2).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)

model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

outputs = model.predict_proba(X_test)[:, 1]
actual = y_test

Use your plotting function:

thresholds = [i / 100 for i in range(1, 100)]
plot_metrics_vs_threshold(actual, outputs, thresholds)

You’ve been plotting metrics against threshold. What if you plotted TPR vs FPR instead — one point per threshold?

Think: what would a perfect model look like? What about random guessing?

A perfect model hits TPR=1, FPR=0 — the top-left corner.

Random guessing follows the diagonal (TPR ≈ FPR at every threshold).

This plot is called the ROC curve (Receiver Operating Characteristic).

Write a function to compute and plot the ROC curve:

def plot_roc(actual, outputs, thresholds):
    # For each threshold: compute TPR, FPR
    # Plot TPR vs FPR + diagonal

def plot_roc(actual, outputs, thresholds):
    tprs, fprs = [], []
    for t in thresholds:
        pred = apply_threshold(outputs, t)
        tp, tn, fp, fn = confusion_matrix(actual, pred)
        tprs.append(true_positive_rate(tp, fn) if (tp+fn) > 0 else 0)
        fprs.append(false_positive_rate(fp, tn) if (fp+tn) > 0 else 0)
    plt.plot(fprs, tprs)
    plt.plot([0, 1], [0, 1], '--', color='gray')
    plt.xlabel('FPR'); plt.ylabel('TPR')
    plt.title('ROC Curve'); plt.show()
    return fprs, tprs

What is the area under the line connecting these two points?

(5, 6) and (7, 10)

This is a trapezoid!

Area = width × average height
= (7 − 5) × (6 + 10) / 2
= 2 × 8 = 16

Now three points:

(2, 3), (5, 7), (8, 4)

What is the total area under the curve?

Two trapezoids:

Blue: (5−2) × (3+7)/2 = 3 × 5 = 15
Orange: (8−5) × (7+4)/2 = 3 × 5.5 = 16.5
Total = 15 + 16.5 = 31.5

Write a function to compute the area under any curve given x and y values:

def area_under_curve(xs, ys):
    # sum of trapezoids

Test cases:

assert area_under_curve([5, 7], [6, 10]) == 16
assert area_under_curve([2, 5, 8], [3, 7, 4]) == 31.5

def area_under_curve(xs, ys):
    area = 0
    for i in range(1, len(xs)):
        width = xs[i] - xs[i-1]
        avg_height = (ys[i] + ys[i-1]) / 2
        area += width * avg_height
    return area

The Area Under the ROC Curve (AUC) summarizes model quality in a single number.

A perfect model’s curve covers the entire square. What is its AUC?
A random model follows the diagonal. What area is that?

AUC = 1.0 → entire square (perfect) • AUC = 0.5 → triangle under diagonal (random) • AUC < 0.5 → worse than random!

You already have area_under_curve. Apply it to the ROC points:

def compute_auc(fprs, tprs):
    # Sort by FPR, then use area_under_curve

def compute_auc(fprs, tprs):
    pairs = sorted(zip(fprs, tprs))
    sorted_fprs = [p[0] for p in pairs]
    sorted_tprs = [p[1] for p in pairs]
    return area_under_curve(sorted_fprs, sorted_tprs)

The higher the AUC, the better the model separates the two classes — regardless of which threshold you pick.

Train two models on the same Iris data and compare:

from sklearn.tree import DecisionTreeClassifier

# Model 1: Logistic Regression
model1 = LogisticRegression(max_iter=200)
model1.fit(X_train, y_train)
outputs1 = model1.predict_proba(X_test)[:, 1]

# Model 2: Shallow Decision Tree
model2 = DecisionTreeClassifier(max_depth=1, random_state=42)
model2.fit(X_train, y_train)
outputs2 = model2.predict_proba(X_test)[:, 1]

Plot both ROC curves and compute AUC for each:

thresholds = [i / 100 for i in range(1, 100)]

fprs1, tprs1 = plot_roc(actual, outputs1, thresholds)
print("AUC Model 1:", compute_auc(fprs1, tprs1))

fprs2, tprs2 = plot_roc(actual, outputs2, thresholds)
print("AUC Model 2:", compute_auc(fprs2, tprs2))

The model with the higher AUC is the better classifier!

Expected: Logistic Regression AUC ≈ 0.99 • Decision Stump AUC ≈ 0.85

You just built from scratch what sklearn provides:

from sklearn.metrics import (
    confusion_matrix, precision_score, recall_score,
    f1_score, roc_auc_score
)

y_pred = apply_threshold(outputs1, 0.5)

print("Precision:", precision_score(actual, y_pred))
print("Recall:   ", recall_score(actual, y_pred))
print("F1:       ", f1_score(actual, y_pred))
print("AUC:      ", roc_auc_score(actual, outputs1))

Compare with your implementations — the numbers should match!

Now you know exactly what these functions do under the hood.

Glossary

Confusion Matrix: A 2×2 table summarizing predictions vs actuals: TP, TN, FP, FN.
Precision: Of all positive predictions, the fraction that are correct: TP / (TP + FP).
Recall: Of all actual positives, the fraction correctly detected: TP / (TP + FN).
F1 Score: The harmonic mean of precision and recall: 2PR / (P + R).
ROC Curve: A plot of TPR vs FPR at various thresholds — shows the tradeoff between sensitivity and false alarm rate.
AUC: Area Under the ROC Curve — a single number summarizing model quality (1.0 = perfect, 0.5 = random).

Part 1: Confusion Matrix~10 min

Part 2: Classification Metrics~10 min

Part 3: Choosing the Right Threshold~25 min

Part 4: ROC Curve & AUC~10 min

Glossary