Inventing a Recommender System

Run the cell below as-is. It loads movies_ratings.csv -- 25 Bollywood movies rated 0-5 by 18 people (a real class!). Rows are movies, columns are people, and an empty cell means that person never rated that movie. Pandas shows these as NaN (Not a Number).

Run the cell below as-is to load the dataset.

Look at the data: the table is full of NaN. Some people rated almost every movie; a few rated nothing at all. Someone with 0 or 2 ratings tells us nothing about their taste -- keeping them would only add noise.

Your task: build a DataFrame called ratings that keeps only the people (columns) who rated at least 3 movies.

Print ratings.mean().sort_values(). Notice something? Bhaskar's average rating is about 1.7 while Venu's is about 4.1. Does Venu love every movie and Bhaskar hate them all? More likely they just use the scale differently -- a 3 from Bhaskar might mean the same as a 5 from Venu.

If we compare raw numbers, the stingy and generous raters will look like they disagree even when their taste is identical. We need to put everyone on the same scale.

The fix -- min-max scaling, per person: You derived a normalization formula in the Loops & Arrays chapter that maps any value to [0, 1] using the minimum and maximum. Apply that same idea here, column by column, so each person's lowest rating becomes 0 and highest becomes 1.

Your task: create scaled from ratings using that formula, applied column-by-column.

The similarity math coming in Part 2 can't handle NaN -- we need a number in every cell. But which number?

Think about the options before reading on:

• Fill with 0? That says "this person would hate this movie" -- a strong claim we have no evidence for.
• Fill with 1? Same problem, opposite direction.
• Fill with that person's average scaled rating? That says "no opinion -- assume they'd feel about it the way they feel about a typical movie." Neutral. This is the one.

Your task: create filled from scaled, replacing each NaN with the mean of that person's column.

Each person is now a column of 25 numbers -- a vector in 25-dimensional space (one dimension per movie).

Two people with similar taste should have vectors pointing in a similar direction. You already know the dot product from the Secrets of Prediction chapter. Think about these questions:

• The dot product of two identical vectors is large. The dot product of perpendicular vectors is 0. So the dot product tells you something about alignment. But it also depends on the vectors' lengths -- how can you remove the length effect?
• What if you divided the dot product by both vectors' lengths? What would that give you for identical vectors? For perpendicular vectors?

This quantity -- the dot product divided by the product of lengths -- is called cosine similarity: 1 means identical taste, 0 means unrelated taste. It measures direction only, which is one more defense against scale differences between raters.

Your task: write similarity(a, b) for two columns of filled.

We want the similarity between every pair of people -- that's $15 \times 15 = 225$ numbers. Calling similarity in a double loop would work, but there is a spectacular shortcut. It starts with one observation:

What happens to the similarity formula when both vectors already have length 1? Simplify the formula and see what's left.

A vector divided by its own length has length 1 (a unit vector).

Your task: build the numpy array uM where every column of M = filled.values is scaled to length 1.

Here's the payoff. Remember how matrix multiplication works: entry $(i, j)$ of $AB$ is the dot product of row $i$ of $A$ with column $j$ of $B$.

So what is $uM^T uM$? Row $i$ of $uM^T$ is column $i$ of $uM$ -- person $i$'s unit vector. So entry $(i, j)$ is the dot product of person $i$'s unit vector with person $j$'s unit vector... which is exactly their cosine similarity. One operation computes the entire similarity table. This is not a toy trick -- it is precisely how real recommender systems (and the attention layers inside LLMs!) compute all-pairs similarity.

Your task: compute S = uM.T @ uM, then wrap it as a labeled DataFrame:

sim = pd.DataFrame(S, index=filled.columns, columns=filled.columns)

Before running the tests, predict: what should the diagonal of S be, and why? Should S equal its own transpose?

Your task: write most_similar(person) returning the name of the person with the highest similarity to person -- excluding the person themselves (everyone's best match would otherwise be themselves, similarity 1.0).

Before computing cosine similarity, you applied min-max scaling to each person's ratings. Why is this normalization step important?

A. It makes the computation faster

B. Without normalization, a generous rater (who gives 4s and 5s to everything) would appear similar to another generous rater even if their actual preferences are completely different

C. Python requires values between 0 and 1 for cosine similarity

D. It removes missing values from the matrix

We only want to recommend movies the person hasn't rated yet -- no point suggesting a movie they already gave 5 stars.

Careful: which table remembers what was actually rated? Not filled -- we erased that information when we filled the NaNs. Go back to ratings.

Your task: write unrated_movies(person) returning a list of movie names the person never rated.

Now assemble the pieces. To score a candidate movie $m$ for a person $p$:

$$\text{score}(m, p) = \sum_{\text{person } q} \text{rating}(m, q) \cdot \text{sim}(p, q)$$

In words: everyone in the class votes with their (filled, scaled) rating of $m$, but people whose taste matches $p$'s get a louder voice. A movie loved by your taste twins scores high; a movie loved only by people you disagree with scores low.

Look closely -- that sum is a dot product: the movie's row of filled dotted with $p$'s column of sim. So all candidate scores at once is:

scores = filled.loc[candidates] @ sim[person]

Your task: write recommend(person, k=3) that scores every unrated movie and returns the k movie names with the highest scores, best first.

A brand-new user signs up and hasn't rated any movies yet. What happens when the recommender tries to find their taste twin?

A. The system recommends the most popular movies by default

B. Cosine similarity can't be computed -- the user's rating vector is all zeros (or NaN), so there's no direction to compare

C. The system assigns random ratings and proceeds normally

D. The user is automatically matched with the most active rater

You just built a working collaborative-filtering recommender on real data. Try recommend(...) on a few more people -- do the suggestions make sense given who their taste twin is?

One honest limitation to notice: what would recommend("YouJustJoined") do for a brand-new person with zero ratings? Nothing useful -- there's no vector to compare. This is the famous cold-start problem, and it's why every streaming service pesters new users to "pick a few titles you like" before showing recommendations.

recommend() let everyone vote, weighted by similarity. Real systems often do something blunter and cheaper: find your top 5 most similar people, pool their top 5 favorite movies, and drop anything you've already seen. No scores, no weighted sums -- just "what do people like me love?"

Your task: write three functions:

• top5_similar_people(person) -- the 5 most similar people (by sim, excluding the person), most similar first.
• top5_favorites(person) -- the person's 5 highest-rated movies. Use ratings (their real opinions), and .dropna() before sorting.
• recommend_by_friends(person) -- pool the friends' favorites in order, remove movies person has already rated, and drop duplicates while keeping first appearance.

Then compare with recommend(person, 3) from Part 3 -- do the two strategies agree?

Everything we built compared people -- the columns of filled. But the same data read the other way compares movies: each row is a movie's vector of 15 ratings, and two movies whose rows point the same way are liked by the same people.

This powers the other famous recommendation sentence: "customers who watched this also watched..." -- no user profile needed at all.

Your task: repeat the Part 2 trick along the other axis:

1. Scale each row of M = filled.values to length 1. Row lengths: np.sqrt((M ** 2).sum(axis=1, keepdims=True)) (keepdims makes the division broadcast row-wise).
2. S_items = uMr @ uMr.T -- note the transpose moved to the other side: entry (i, j) is now row-dot-row, movie i against movie j.
3. Wrap it: msim = pd.DataFrame(S_items, index=filled.index, columns=filled.index).
4. Write most_similar_movie(name) (mirror of most_similar).

What you invented: item-item collaborative filtering -- the variant Amazon famously chose, because with millions of users and thousands of items, item-item similarities are fewer, more stable, and reusable for every visitor.

Our class matrix was 25x15. Netflix has ~300 million users and ~20,000 titles. The user-similarity matrix alone would have $3\times10^8 \times 3\times10^8$ entries -- no single machine can even hold that, let alone compute it.

Two observations rescue us:

1. The matrix is almost entirely zeros. A typical user rates a few dozen titles out of 20,000. Why store the zeros at all? Keep only the non-zero entries as (row, col, value) triples.
2. Matrix multiplication decomposes into independent little jobs. Recall: $(AB)_{ij} = \sum_k A_{ik} B_{kj}$. Each non-zero pair $A_{ik}, B_{kj}$ contributes one product $A_{ik} \cdot B_{kj}$ to one output cell $(i, j)$ -- and every contribution can be computed on a different machine, then summed up per cell. That "compute little contributions anywhere, then group and sum by key" pattern is called MapReduce, and it's how the giants do it.

Your task: write sparse_matmul(A_triples, B_triples) where each argument is a list of (row, col, value) triples of the non-zero entries. Return a dict mapping (i, j) -> value for the non-zero entries of the product.

This is collaborative filtering -- recommending purely from the pattern of who-likes-what, knowing nothing about the movies themselves (no genres, no actors, no plots). The alternative, content-based filtering, compares movie features instead; modern systems blend both.

And that uM.T @ uM trick -- all-pairs similarity as a single matrix product -- will meet you again in the most unexpected place: it's the heart of the attention mechanism inside transformers, where every word in a sentence computes its similarity to every other word. Same math, different vectors.