Before any algorithm, let's learn to count carefully. That's all probability is — careful counting.
Here are 14 days of weather observations. For each day we recorded the outlook, temperature, humidity, and wind — and whether we played tennis.
| Day | Outlook | Temp | Humidity | Wind | Play? |
|---|---|---|---|---|---|
| 1 | Sunny | Hot | High | Weak | No |
| 2 | Sunny | Hot | High | Strong | No |
| 3 | Overcast | Hot | High | Weak | Yes |
| 4 | Rainy | Mild | High | Weak | Yes |
| 5 | Rainy | Cool | Normal | Weak | Yes |
| 6 | Rainy | Cool | Normal | Strong | No |
| 7 | Overcast | Cool | Normal | Strong | Yes |
| 8 | Sunny | Mild | High | Weak | No |
| 9 | Sunny | Cool | Normal | Weak | Yes |
| 10 | Rainy | Mild | Normal | Weak | Yes |
| 11 | Sunny | Mild | Normal | Strong | Yes |
| 12 | Overcast | Mild | High | Strong | Yes |
| 13 | Overcast | Hot | Normal | Weak | Yes |
| 14 | Rainy | Mild | High | Strong | No |
Tasks (pen & paper):
Imagine you have no weather information at all. A friend calls and asks: "Should I come to play tennis?"
Your answers:
P(Play = Yes) = 9/14. In probability and machine learning, this is called a "prior" probability. Why is it called "prior"?
A. Because it was computed before computers existed
B. Because it uses the first column of the data
C. It's what you'd predict *before* looking at any features — your default belief based only on past outcomes
D. It always equals 0.5
The prior says "Yes" 64% of the time. But surely the weather matters. Let's see how one feature changes our prediction.
Look at the data table and answer by hand (just count the rows):
Your answers:
It's a Sunny day. You want to decide: is it more likely a Yes day or a No day?
You have two pieces of information:
Tasks:
Your calculation:
Repeat the same calculation from Ex 2.2 for:
You'll need to count how many Overcast and Rainy days appear among Yes days and No days first.
Now let's automate the counting. Write a function that counts how many rows match a set of conditions.
def count_where(data, conditions):
"""
Count rows in data where ALL conditions are met.
conditions: dict like {"Outlook": "Sunny", "Play": "Yes"}
Returns: int
"""
# Your code here
pass
Write a function that computes a conditional probability using count_where.
def probability(data, event, given):
"""
Compute P(event | given).
event: dict of conditions for the event, e.g. {"Outlook": "Sunny"}
given: dict of conditions for the given, e.g. {"Play": "Yes"}
Returns: float
"""
# Your code here
pass
Write a function that predicts the class using a single feature:
def predict_one_feature(data, feature_name, feature_value, label_col, labels):
"""
For each label, compute: score = P(feature_value | label) * P(label)
Return the label with the highest score.
"""
# Your code here
pass
One feature helped. Can two features help more? Let's combine evidence.
It's a Sunny and Hot day. Should we play?
We already know how to use one feature. For two features, try this: multiply the contribution of each feature separately, along with the prior.
Compute by hand:
Your calculation:
Compute Score(Yes) and Score(No) for a Rainy and Cool day.
In Exercise 3.1, you multiplied P(Sunny | Yes) × P(Hot | Yes). This multiplication is only mathematically valid under a specific assumption. What is it?
A. Outlook and Temperature must have the same number of possible values
B. Outlook and Temperature are *independent* given the label — knowing one doesn't tell you about the other (once you already know whether we played)
C. The dataset must have at least 100 rows
D. All feature probabilities must be greater than 0.5
Write a function that predicts using two features:
def predict_two_features(data, f1_name, f1_val, f2_name, f2_val, label_col, labels):
"""
For each label, compute:
score = P(f1_val | label) * P(f2_val | label) * P(label)
Return the label with the highest score.
"""
# Your code here
pass
You've done it for 1 feature and 2 features. Now let's generalize to any number of features.
It's Sunny, Hot, and High humidity. Compute Score(Yes) and Score(No) using all three features.
You've seen the pattern: for each label, multiply the prior by the conditional probability of each feature. Now write the general version:
def predict(data, features, label_col, labels):
"""
features: dict mapping feature_name -> feature_value,
e.g. {"Outlook": "Sunny", "Temperature": "Hot", "Humidity": "High"}
label_col: name of the label column, e.g. "Play"
labels: list of possible labels, e.g. ["Yes", "No"]
For each label, compute:
score = P(label) * P(f1_val|label) * P(f2_val|label) * ... * P(fn_val|label)
Return the label with the highest score.
"""
# Your code here
pass
Run your predict function on every row in the dataset and compare to the actual label.
Your algorithm has a fatal flaw. Let's find it and fix it.
Look at the data: Outlook = Overcast appears on days 3, 7, 12, 13. What is the Play label for all of those days?
Now answer:
The fix is beautifully simple: pretend you've seen every combination at least once.
The idea: Instead of computing P(Overcast | No) = 0/5, add 1 to the numerator (pretend you saw one extra Overcast-No day) and add k to the denominator (where k is the number of possible values for that feature).
For Outlook, k = 3 (Sunny, Overcast, Rainy). So:
Tasks:
probability_smooth(data, event, given, k):def probability_smooth(data, event, given, k):
"""
Smoothed conditional probability.
Adds 1 to the numerator and k to the denominator.
k = number of possible values for the event's feature.
"""
# Your code here
pass
Write predict_smooth — like your predict from Part 4, but using probability_smooth instead of probability.
You'll need to know k (number of possible values) for each feature. Compute this from the data.
def predict_smooth(data, features, label_col, labels):
"""
Like predict(), but uses Laplace smoothing.
For each feature, k = number of distinct values of that feature in the data.
"""
# Your code here
pass
Laplace smoothing adds 1 to every count. What would happen if you added 1000 instead of 1?
A. The classifier would become more accurate because it has more "data"
B. Nothing would change — the relative ordering of scores stays the same
C. All smoothed probabilities would approach 1/k (uniform) — the fake counts would overwhelm the real data, and the classifier would ignore the actual evidence
D. Python would crash with an overflow error
You have all the pieces. Now wrap them into a reusable class with the standard ML interface: fit and predict.
Before writing code, think about what fit(X, y) needs to compute and store so that predict(X) can work later without re-scanning all the training data.
Think about:
Your design:
After fit(X, y), self should contain:
Implement the full class:
class NaiveBayes:
def __init__(self, smoothing=1):
"""smoothing: Laplace smoothing parameter (default 1)."""
pass
def fit(self, X, y):
"""
X : list of dicts (each dict maps feature_name -> feature_value)
y : list of labels
Precompute and store all the counts needed for prediction.
"""
pass
def predict(self, X):
"""
X : list of dicts
Returns: list of predicted labels, one per row.
"""
pass
Test your NaiveBayes class on a spam detection problem. Each email is represented by the presence/absence of certain words:
Tasks:
NaiveBayes on this dataset.new_emails = [
{"free": "yes", "money": "yes", "urgent": "no", "meeting": "no"}, # likely spam?
{"free": "no", "money": "no", "urgent": "no", "meeting": "yes"}, # likely ham?
{"free": "yes", "money": "no", "urgent": "yes", "meeting": "yes"}, # tricky!
]
Right now, predict returns just the winning label. Add a predict_proba method that returns the probability of each class.
To convert scores to probabilities, divide each score by the sum of all scores:
For example, if Score(Yes) = 0.03 and Score(No) = 0.07, then:
def predict_proba(self, X):
"""
X : list of dicts
Returns: list of dicts, each mapping label -> probability.
e.g. [{"Yes": 0.3, "No": 0.7}, ...]
"""
pass
Test: The probabilities for each row should sum to 1.0.
Scikit-learn has a built-in CategoricalNB. Compare your results:
from sklearn.preprocessing import OrdinalEncoder
from sklearn.naive_bayes import CategoricalNB
import numpy as np
# Encode string features as integers
enc = OrdinalEncoder()
X_encoded = enc.fit_transform([[row[f] for f in ["Outlook","Temperature","Humidity","Wind"]]
for row in data])
y_encoded = [1 if row["Play"]=="Yes" else 0 for row in data]
clf = CategoricalNB(alpha=1.0) # alpha=1.0 is Laplace smoothing
clf.fit(X_encoded, y_encoded)
# Compare predictions
sk_preds = clf.predict(X_encoded)
print("sklearn predictions:", ["Yes" if p==1 else "No" for p in sk_preds])
print("Your predictions: ", nb.predict(X))
Do the predictions match? If not, investigate why.
Your classifier works with categorical features (Sunny, Rainy, etc.). But what if a feature is a number, like actual temperature (72°F, 85°F, etc.)?
Try the binning approach on a dataset of your choice.
What you just built:
You started by counting rows in a table. From there, you:
fit and predict.This is the Naive Bayes classifier — one of the oldest and most practical algorithms in machine learning. Despite its "naive" assumption, it excels at text classification (spam detection, sentiment analysis), medical diagnosis, and any problem with many categorical features.
The mathematical name for what you computed is Bayes' theorem: updating a prior belief using evidence. You invented it from scratch.
This is the Naive Bayes classifier — one of the oldest and most practical algorithms in machine learning. Despite its "naive" assumption, it excels at text classification (spam detection, sentiment analysis), medical diagnosis, and any problem with many categorical features. The mathematical name for what you computed is Bayes' theorem: updating a prior belief using evidence. You invented it from scratch.
Source on GitHub · Back to all chapters
© 2026 CloudxLab. All rights reserved.