You are handed a real task at work:
We bought a recommendation engine. It reads a CSV file with three columns —
user_id, product_id, score— and writes its recommendations to another CSV with columnsuser_id, product_id, recommendation_score. It only accepts integers foruser_id.Our data, however, identifies users by email address:
sandeep,1, 1.0 nitin, 2, 3.0 subodh, 2, 1.0 sandeep, 2, 2.0 nitin, 1, 2.0Make it work. And when the engine answers, translate its output back to emails.
You cannot change the engine. So you will wrap it:
our data (emails) final output (emails)
| ^
v |
ENCODER --> engine input (ints) --> RECOMMENDER --> engine output (ints) --> DECODER
The Encoder and Decoder are separate programs that communicate only through files — so the mapping between emails and integers must itself be saved to a file. This pattern (encode → black box → decode) is everywhere: ML pipelines call it label encoding, and you already met its purest form in the Information Theory chapter.
First, create the input data file. Run this provided cell as-is.
The lines are messy: spaces after commas, a newline at the end. Write
parse_line(line) that returns a clean tuple (user, product_id, score)
with the right types: str, int, float.
parse_line("sandeep, 2, 2.0\n") # ("sandeep", 2, 2.0)
parse_line("nitin,1, 3.5") # ("nitin", 1, 3.5)
OUR_DATA = """\
sandeep,1, 1.0
nitin, 2, 3.0
subodh, 2, 1.0
sandeep, 2, 2.0
nitin, 1, 2.0
"""
with open("our_data.csv", "w") as f:
f.write(OUR_DATA)
print(open("our_data.csv").read())
assert parse_line("sandeep, 2, 2.0\n") == ("sandeep", 2, 2.0)
assert parse_line("nitin,1, 3.5") == ("nitin", 1, 3.5)
assert parse_line("a@b.com , 7 , 0.5") == ("a@b.com", 7, 0.5)
Use line.strip().split(",") to break the line into parts. Then call .strip() on each part to remove extra whitespace before converting types.
After splitting, you get three strings. The first stays a str (just strip whitespace), the second becomes int(...), and the third becomes float(...). Return them as a tuple.
def parse_line(line):
parts = line.strip().split(",")
user = parts[0].strip()
product_id = int(parts[1].strip())
score = float(parts[2].strip())
return (user, product_id, score)
Write read_data(filename) that opens the file and returns a list of parsed
tuples, skipping any blank lines.
read_data("our_data.csv")
# [("sandeep", 1, 1.0), ("nitin", 2, 3.0), ("subodh", 2, 1.0),
# ("sandeep", 2, 2.0), ("nitin", 1, 2.0)]
rows = read_data("our_data.csv")
assert rows == [("sandeep", 1, 1.0), ("nitin", 2, 3.0), ("subodh", 2, 1.0),
("sandeep", 2, 2.0), ("nitin", 1, 2.0)]
Open the file, iterate over its lines, skip any line where line.strip() is empty, and call parse_line on each non-blank line. Collect results into a list.
A list comprehension works well: [parse_line(line) for line in open(filename) if line.strip()].
def read_data(filename):
result = []
for line in open(filename):
if line.strip():
result.append(parse_line(line))
return result
Write build_mapping(rows) that gives each distinct user an integer ID:
1 for the first user seen, 2 for the second new user, and so on.
Users repeat in the data — they must keep the same ID.
build_mapping(rows) # {"sandeep": 1, "nitin": 2, "subodh": 3}
mapping = build_mapping(rows)
assert mapping == {"sandeep": 1, "nitin": 2, "subodh": 3}
Iterate over the rows. For each row, check if the user is already in your mapping dictionary. If not, assign the next integer (use len(mapping) + 1 as the next ID).
Start with an empty dict. For each (user, product_id, score) tuple, do: if user not in mapping: mapping[user] = len(mapping) + 1. Return the dict at the end.
def build_mapping(rows):
mapping = {}
for user, product_id, score in rows:
if user not in mapping:
mapping[user] = len(mapping) + 1
return mapping
The mapping file maps email → id. The decoder needs id → email. The Decoder is a separate program — it will not share memory with the Encoder. The mapping must survive as a file.
Write save_mapping(mapping, filename) (one email,id line per user) and
load_mapping(filename) that reads it back — types intact (ids are ints).
The test is the round trip:
save_mapping(mapping, "mapping.csv")
load_mapping("mapping.csv") == mapping # True
save_mapping(mapping, "mapping.csv")
assert load_mapping("mapping.csv") == mapping
For save_mapping: iterate over mapping.items() and write each pair as a line like "sandeep,1\n". For load_mapping: split each line on ",", strip whitespace, and convert the second part to int.
Be careful with types when loading: int("1") works, but int(" 1\n") requires a .strip() first. Build a new dict from the parsed pairs.
def save_mapping(mapping, filename):
with open(filename, "w") as f:
for email, uid in mapping.items():
f.write(f"{email},{uid}\n")
def load_mapping(filename):
mapping = {}
for line in open(filename):
if line.strip():
parts = line.strip().split(",")
email = parts[0].strip()
uid = int(parts[1].strip())
mapping[email] = uid
return mapping
Put it together. Write encode_file(input_file, encoded_file, mapping_file) that reads the raw data, builds and saves the mapping, and writes the encoded CSV with emails replaced by their integer IDs.
Expected encoded.csv for our data:
1,1,1.0
2,2,3.0
3,2,1.0
1,2,2.0
2,1,2.0
encode_file("our_data.csv", "encoded.csv", "mapping.csv")
expected = "1,1,1.0\n2,2,3.0\n3,2,1.0\n1,2,2.0\n2,1,2.0\n"
assert open("encoded.csv").read() == expected
assert load_mapping("mapping.csv") == {"sandeep": 1, "nitin": 2, "subodh": 3}
This is mostly gluing together the functions you already have: read_data, build_mapping, save_mapping. The new part is writing the encoded file — for each row, look up the user's integer ID from the mapping and write f"{id},{product_id},{score}\n".
Open the output file for writing. For each (user, product_id, score) tuple in the rows, write: f.write(f"{mapping[user]},{product_id},{score}\n").
def encode_file(input_file, encoded_file, mapping_file):
rows = read_data(input_file)
mapping = build_mapping(rows)
save_mapping(mapping, mapping_file)
with open(encoded_file, "w") as f:
for user, product_id, score in rows:
f.write(f"{mapping[user]},{product_id},{score}\n")
Here is the engine you bought. You are not allowed to modify it — run the cell as-is. For each user it recommends the products they have not rated yet, scored by that product's average rating from other users.
Notice it would crash on emails: it calls int() on the first column.
That is exactly why you wrote the Encoder.
Only one line came out: user 3 should look at product 1.
Makes sense — everyone else has already rated everything. But who is user 3?
Time for the Decoder.
Run this provided cell as-is. Do not modify the recommender.
def run_recommender(input_file, output_file):
"""BLACK BOX -- the engine you bought. Do not modify."""
ratings = []
for line in open(input_file):
if line.strip():
u, p, s = line.strip().split(",")
ratings.append((int(u), int(p), float(s)))
product_scores = {}
for u, p, s in ratings:
product_scores.setdefault(p, []).append(s)
avg = {p: sum(v) / len(v) for p, v in product_scores.items()}
seen = {}
for u, p, s in ratings:
seen.setdefault(u, set()).add(p)
with open(output_file, "w") as f:
for u in sorted(seen):
for p in sorted(avg):
if p not in seen[u]:
f.write(f"{u},{p},{avg[p]}\n")
run_recommender("encoded.csv", "recommendations.csv")
print(open("recommendations.csv").read())
The mapping file maps email → id. The decoder needs id → email.
Write invert(mapping). (You built this in the Dictionaries chapter — rebuild
it from memory.) Why is inversion safe here — what property must the mapping have?
invert({"sandeep": 1, "nitin": 2}) # {1: "sandeep", 2: "nitin"}
assert invert({"sandeep": 1, "nitin": 2}) == {1: "sandeep", 2: "nitin"}
Build a new dict where every key-value pair from the original is swapped: {v: k for k, v in mapping.items()}.
Inversion is safe because the mapping is one-to-one (bijective): each email gets a unique integer, so no two emails share the same id. If two emails had the same id, inverting would lose one of them.
Inversion is safe here because the mapping is one-to-one (bijective): each email gets a unique integer via len(mapping) + 1, so no two emails share the same id. If two emails had the same id, inverting would silently lose one of them.
def invert(mapping):
return {v: k for k, v in mapping.items()}
Write decode_file(recs_file, mapping_file, output_file) that loads the mapping from its file, inverts it, reads the engine's recommendations, and writes the final CSV with integer IDs translated back to emails.
decode_file("recommendations.csv", "mapping.csv", "final.csv")
open("final.csv").read() # "subodh,1,1.5\n"
So the mystery user 3 was subodh, and the pipeline recommends product 1 with score 1.5.
assert open("final.csv").read() == "subodh,1,1.5\n"
Load the mapping with load_mapping, invert it with invert. Then read the recommendations file line by line, split on ",", look up the integer user id in the inverted mapping, and write the email-based line to the output file.
The recommendation lines look like "3,1,1.5\n". Split on "," to get the user id (convert to int), product id, and score. Replace the user id with inv_mapping[user_id] and write the line.
def decode_file(recs_file, mapping_file, output_file):
mapping = load_mapping(mapping_file)
inv = invert(mapping)
with open(output_file, "w") as f:
for line in open(recs_file):
if line.strip():
parts = line.strip().split(",")
user_id = int(parts[0])
product_id = parts[1]
score = parts[2]
f.write(f"{inv[user_id]},{product_id},{score}\n")
The recommendation engine takes integer IDs instead of email addresses. Why is this encoding step necessary?
Think about what the recommendation engine does internally — can it compute similarity between two email strings the same way it does between two numbers?
Recommendation engines compute similarity scores, distances, and matrix operations that require numerical inputs. Email strings like "alice@example.com" can't be multiplied or added in a meaningful way. The integer encoding gives each user a unique numerical ID that the algorithm can use as an index. This is a common ML pattern: many algorithms need numerical features, so categorical data (names, emails, categories) must be encoded to numbers first.
A pipeline you have run once works once. A pipeline you have tested works.
The provided cell generates a bigger random dataset. Run your whole pipeline on
it, then write check_pipeline(original_file, final_file) that verifies:
"@").Think about: what could silently break if two different emails ever received the same ID? Which of your functions guarantees that cannot happen?
import random
random.seed(7)
users = [f"user{i}@example.com" for i in range(1, 21)]
with open("big_data.csv", "w") as f:
for _ in range(100):
u = random.choice(users)
p = random.randint(1, 10)
s = round(random.uniform(0.5, 5.0), 1)
f.write(f"{u}, {p}, {s}\n")
# run YOUR pipeline:
encode_file("big_data.csv", "big_encoded.csv", "big_mapping.csv")
run_recommender("big_encoded.csv", "big_recs.csv")
decode_file("big_recs.csv", "big_mapping.csv", "big_final.csv")
print(f"{len(open('big_recs.csv').readlines())} recommendations generated")
print("first 5 lines of the final output:")
for line in open("big_final.csv").readlines()[:5]:
print(" ", line, end="")
assert check_pipeline("big_data.csv", "big_final.csv") == True
Read the original file and collect all unique user emails into a set. Read the final file and collect all user fields (first column). Then assert every final user is in the original set, and every final user contains "@".
For the "@" check: if an integer ID leaked through (the decoder failed for some row), the first column would be something like "3" instead of "user3@example.com". Checking for "@" catches that class of bug.
def check_pipeline(original_file, final_file):
# Collect all unique users from the original data
original_users = set()
for line in open(original_file):
if line.strip():
user = line.strip().split(",")[0].strip()
original_users.add(user)
# Check every user in the final output
for line in open(final_file):
if line.strip():
user = line.strip().split(",")[0].strip()
# 1. Every user must exist in the original data
assert user in original_users, f"Unknown user: {user}"
# 2. No integer IDs leaked (every user contains "@")
assert "@" in user, f"Leaked integer ID: {user}"
return True
The same encode → black box → decode shape appears when you one-hot-encode categories for a model, tokenize text for an LLM, or compress data before a network hop (your Huffman coder!). The black box changes; the wrapper pattern never does.
| Piece | What you built | The general pattern |
|---|---|---|
parse_line, read_data |
messy text → typed tuples | every pipeline starts with parsing |
build_mapping |
email → int, first-seen order | label encoding — sklearn.preprocessing.LabelEncoder is exactly this |
save_mapping, load_mapping |
mapping as a file | separate programs share state only through files |
encode_file |
the Encoder program | adapt your data to their interface |
run_recommender |
untouched black box | you wrapped it instead of changing it |
invert, decode_file |
the Decoder program | translate answers back into your domain |
check_pipeline |
end-to-end verification | trust pipelines you have tested, not pipelines that ran |