Learning Naive Bayes by Inventing It

Yes: 9 days, No: 5 days. Fraction of Yes = 9/14 ≈ 0.643. Since Yes is more common, a reasonable default guess is "Yes".

P(Yes) = 9/14 ≈ 0.643, P(No) = 5/14 ≈ 0.357. We'd predict Yes since it's more likely. They sum to 1 because every day is either Yes or No — there are no other possibilities.

Among Yes days: 2 out of 9 are Sunny → 2/9 ≈ 0.222. Among No days: 3 out of 5 are Sunny → 3/5 = 0.600. This is interesting: Sunny is much more common on No days!

Score(Yes) = 2/14 ≈ 0.143. Score(No) = 3/14 ≈ 0.214. No wins — on Sunny days, we predict "No".

Notice what happened: the prior said Yes (64%), but the evidence from the Sunny outlook flipped the prediction to No. The feature overrode the prior because Sunny is strongly associated with No days.

Overcast: Score(Yes) = 4/14 ≈ 0.286, Score(No) = 0. Predict Yes. Every Overcast day was a Yes day!

Rainy: Score(Yes) = (3/9)×(9/14) = 3/14 ≈ 0.214. Score(No) = (2/5)×(5/14) = 2/14 ≈ 0.143. Predict Yes.

def count_where(data, conditions):
    """
    Count rows in data where ALL conditions are met.
    conditions: dict like {"Outlook": "Sunny", "Play": "Yes"}
    Returns: int
    """
    count = 0
    for row in data:
        if all(row[k] == v for k, v in conditions.items()):
            count += 1
    return count

def probability(data, event, given):
    """
    Compute P(event | given).
    event: dict of conditions for the event, e.g. {"Outlook": "Sunny"}
    given: dict of conditions for the given, e.g. {"Play": "Yes"}
    Returns: float
    """
    both = {**given, **event}
    return count_where(data, both) / count_where(data, given)

def predict_one_feature(data, feature_name, feature_value, label_col, labels):
    """
    For each label, compute:  score = P(feature_value | label) * P(label)
    Return the label with the highest score.
    """
    def score(label):
        p_label = count_where(data, {label_col: label}) / len(data)
        p_feature_given_label = probability(data, {feature_name: feature_value}, {label_col: label})
        return p_feature_given_label * p_label

    return max(labels, key=score)

Score(Yes) ≈ 0.032, Score(No) ≈ 0.086. Predict No. This matches our intuition: both Sunny and Hot are associated with No days.

Score(Yes) = (3/9) × (3/9) × (9/14) = 9/126 ≈ 0.071. Score(No) = (2/5) × (1/5) × (5/14) = 2/70 ≈ 0.029. Predict Yes.

def predict_two_features(data, f1_name, f1_val, f2_name, f2_val, label_col, labels):
    """
    For each label, compute:
      score = P(f1_val | label) * P(f2_val | label) * P(label)
    Return the label with the highest score.
    """
    def score(label):
        p_label = count_where(data, {label_col: label}) / len(data)
        p_f1 = probability(data, {f1_name: f1_val}, {label_col: label})
        p_f2 = probability(data, {f2_name: f2_val}, {label_col: label})
        return p_f1 * p_f2 * p_label

    return max(labels, key=score)

Score(Yes) ≈ 0.011, Score(No) ≈ 0.069. Predict No. Adding the humidity feature reinforced the prediction — High humidity is strongly associated with No days.

def predict(data, features, label_col, labels):
    """
    features: dict mapping feature_name -> feature_value
    label_col: name of the label column
    labels: list of possible labels

    For each label, compute:
      score = P(label) * P(f1_val|label) * P(f2_val|label) * ... * P(fn_val|label)
    Return the label with the highest score.
    """
    def score(label):
        s = count_where(data, {label_col: label}) / len(data)
        for fname, fval in features.items():
            s *= probability(data, {fname: fval}, {label_col: label})
        return s

    return max(labels, key=score)

Running the provided code produces:

✓ predicted=No, actual=No       (day 1)
✓ predicted=No, actual=No       (day 2)
✓ predicted=Yes, actual=Yes     (day 3)
✓ predicted=Yes, actual=Yes     (day 4)
✓ predicted=Yes, actual=Yes     (day 5)
✓ predicted=No, actual=No       (day 6)
✓ predicted=Yes, actual=Yes     (day 7)
✓ predicted=No, actual=No       (day 8)
✓ predicted=Yes, actual=Yes     (day 9)
✓ predicted=Yes, actual=Yes     (day 10)
✓ predicted=Yes, actual=Yes     (day 11)
✓ predicted=Yes, actual=Yes     (day 12)
✓ predicted=Yes, actual=Yes     (day 13)
✓ predicted=No, actual=No       (day 14)

Accuracy: 14/14 = 100.0%

The classifier achieves 100% training accuracy on this small dataset. This does not mean it will be perfect on new data — it is trained on the same data it is being tested on. With larger, noisier datasets, the naive independence assumption will cause some errors.

P(Overcast | No) = 0. This makes Score(No) = 0 regardless of all other features. The algorithm becomes blind to all other evidence. Just because we haven't seen Overcast on a No day doesn't mean it's impossible — it just means our dataset is small.

P_smooth(Sunny | No) = (3+1)/(5+3) = 4/8 = 0.5, compared to unsmoothed 3/5 = 0.6. Smoothing pulled the probability slightly toward the uniform value of 1/3.

def probability_smooth(data, event, given, k):
    """
    Smoothed conditional probability.
    Adds 1 to the numerator and k to the denominator.
    k = number of possible values for the event's feature.
    """
    both = {**given, **event}
    return (count_where(data, both) + 1) / (count_where(data, given) + k)

def predict_smooth(data, features, label_col, labels):
    """
    Like predict(), but uses Laplace smoothing.
    For each feature, k = number of distinct values of that feature in the data.
    """
    def score(label):
        s = count_where(data, {label_col: label}) / len(data)
        for fname, fval in features.items():
            k = len(set(row[fname] for row in data))
            s *= probability_smooth(data, {fname: fval}, {label_col: label}, k)
        return s

    return max(labels, key=score)

After fit(X, y), self should contain:

• self.labels — list of unique labels, e.g. ["Yes", "No"]
• self.label_counts — dict mapping each label to its count, e.g. {"Yes": 9, "No": 5}
• self.feature_counts — nested dict: feature_counts[label][feature_name][feature_value] = count, e.g. {"Yes": {"Outlook": {"Sunny": 2, "Overcast": 4, "Rainy": 3}, ...}, ...}
• self.feature_values — dict mapping each feature name to the set of its distinct values, e.g. {"Outlook": {"Sunny", "Overcast", "Rainy"}, ...} (needed to compute k for smoothing)
• self.total — total number of training examples, e.g. 14

With these stored, predict can compute scores using only simple lookups and arithmetic — no need to re-scan the raw data.

from collections import defaultdict

class NaiveBayes:
    def __init__(self, smoothing=1):
        """smoothing: Laplace smoothing parameter (default 1)."""
        self.smoothing = smoothing

    def fit(self, X, y):
        """
        X : list of dicts (each dict maps feature_name -> feature_value)
        y : list of labels
        Precompute and store all the counts needed for prediction.
        """
        self.total = len(y)
        self.labels = list(set(y))
        self.label_counts = defaultdict(int)
        self.feature_counts = defaultdict(lambda: defaultdict(lambda: defaultdict(int)))
        self.feature_values = defaultdict(set)

        for row, label in zip(X, y):
            self.label_counts[label] += 1
            for fname, fval in row.items():
                self.feature_counts[label][fname][fval] += 1
                self.feature_values[fname].add(fval)

    def predict(self, X):
        """
        X : list of dicts
        Returns: list of predicted labels, one per row.
        """
        results = []
        for row in X:
            best_label = None
            best_score = -1
            for label in self.labels:
                score = self.label_counts[label] / self.total
                for fname, fval in row.items():
                    k = len(self.feature_values[fname])
                    count = self.feature_counts[label][fname].get(fval, 0)
                    score *= (count + self.smoothing) / (self.label_counts[label] + self.smoothing * k)
                if score > best_score:
                    best_score = score
                    best_label = label
            results.append(best_label)
        return results

The first email has "free" and "money" — strong spam signals. The second has only "meeting" — clearly ham. The third is tricky: "free" and "urgent" point to spam, while "meeting" points to ham. The classifier weighs all evidence and picks spam because two spam-indicative features outweigh one ham-indicative feature.

nb_spam = NaiveBayes()
nb_spam.fit(X_spam, y_spam)

# Training accuracy
preds_train = nb_spam.predict(X_spam)
accuracy = sum(p == a for p, a in zip(preds_train, y_spam)) / len(y_spam)
print(f"Training accuracy: {accuracy:.1%}")  # 100.0%

# Predict on new emails
new_emails = [
    {"free": "yes", "money": "yes", "urgent": "no", "meeting": "no"},
    {"free": "no",  "money": "no",  "urgent": "no", "meeting": "yes"},
    {"free": "yes", "money": "no",  "urgent": "yes", "meeting": "yes"},
]
preds = nb_spam.predict(new_emails)
print(f"Predictions: {preds}")
# Expected: ["spam", "ham", "spam"]

def predict_proba(self, X):
    """
    X : list of dicts
    Returns: list of dicts, each mapping label -> probability.
    e.g. [{"Yes": 0.3, "No": 0.7}, ...]
    """
    results = []
    for row in X:
        scores = {}
        for label in self.labels:
            score = self.label_counts[label] / self.total
            for fname, fval in row.items():
                k = len(self.feature_values[fname])
                count = self.feature_counts[label][fname].get(fval, 0)
                score *= (count + self.smoothing) / (self.label_counts[label] + self.smoothing * k)
            scores[label] = score
        total = sum(scores.values())
        probs = {label: s / total for label, s in scores.items()}
        results.append(probs)
    return results

# Add to NaiveBayes class:
NaiveBayes.predict_proba = predict_proba

# Test:
probas = nb.predict_proba(X[:3])
for p in probas:
    print(p, "sum =", round(sum(p.values()), 4))
    assert abs(sum(p.values()) - 1.0) < 0.0001, "Probabilities must sum to 1!"
print("All probability sums verified!")

The predictions match exactly. Both implementations use the same algorithm: for each label, multiply the prior by the Laplace-smoothed conditional probability of each feature, then pick the label with the highest score. The only difference is that scikit-learn works with integer-encoded features internally, while our version works with string features directly.

from sklearn.preprocessing import OrdinalEncoder
from sklearn.naive_bayes import CategoricalNB
import numpy as np

# Encode string features as integers
feature_names = ["Outlook", "Temperature", "Humidity", "Wind"]
enc = OrdinalEncoder()
X_encoded = enc.fit_transform([[row[f] for f in feature_names] for row in data])
y_encoded = [1 if row["Play"] == "Yes" else 0 for row in data]

clf = CategoricalNB(alpha=1.0)   # alpha=1.0 is Laplace smoothing
clf.fit(X_encoded, y_encoded)

# Compare predictions
sk_preds = clf.predict(X_encoded)
sk_labels = ["Yes" if p == 1 else "No" for p in sk_preds]
my_labels = nb.predict(X)

print("sklearn predictions:", sk_labels)
print("Your predictions:   ", my_labels)
print("Match:", sk_labels == my_labels)

You cannot count P(Temperature=72.3 | Yes) directly because with continuous values, almost no two rows share the exact same value. The count would be 0 or 1 for most values, making the probability estimate useless. The Gaussian approach (used by scikit-learn's GaussianNB) is more principled: it estimates the mean and standard deviation of each numeric feature per class, then uses the Gaussian probability density function instead of counting. This avoids the arbitrary choice of bin boundaries.

def bin_value(x, boundaries, labels):
    """
    Map a numeric value to a category label.
    boundaries: sorted list of thresholds, e.g. [60, 80]
    labels: category names, one more than boundaries, e.g. ["Cold", "Mild", "Hot"]
    """
    for i, b in enumerate(boundaries):
        if x < b:
            return labels[i]
    return labels[-1]

# Example: a small dataset with numeric temperature
numeric_data = [
    {"outlook": "Sunny", "temp": 85, "play": "No"},
    {"outlook": "Sunny", "temp": 80, "play": "No"},
    {"outlook": "Overcast", "temp": 83, "play": "Yes"},
    {"outlook": "Rainy", "temp": 70, "play": "Yes"},
    {"outlook": "Rainy", "temp": 68, "play": "Yes"},
    {"outlook": "Rainy", "temp": 65, "play": "No"},
    {"outlook": "Overcast", "temp": 64, "play": "Yes"},
    {"outlook": "Sunny", "temp": 72, "play": "No"},
    {"outlook": "Sunny", "temp": 69, "play": "Yes"},
    {"outlook": "Rainy", "temp": 75, "play": "Yes"},
]

# Bin temperature into Cold/Mild/Hot
boundaries = [65, 75]
temp_labels = ["Cold", "Mild", "Hot"]

X_binned = [
    {"outlook": row["outlook"],
     "temp": bin_value(row["temp"], boundaries, temp_labels)}
    for row in numeric_data
]
y_binned = [row["play"] for row in numeric_data]

nb_binned = NaiveBayes()
nb_binned.fit(X_binned, y_binned)
print(nb_binned.predict([{"outlook": "Sunny", "temp": bin_value(90, boundaries, temp_labels)}]))
# Hot + Sunny -> predicts "No"