The Encoder-Decoder Pipeline — A Real Data Engineering Problem

First, create the input data file. Run this provided cell as-is.

The lines are messy: spaces after commas, a newline at the end. Write parse_line(line) that returns a clean tuple (user, product_id, score) with the right types: str, int, float.

parse_line("sandeep, 2, 2.0\n")   # ("sandeep", 2, 2.0)
parse_line("nitin,1, 3.5")        # ("nitin", 1, 3.5)

Write read_data(filename) that opens the file and returns a list of parsed tuples, skipping any blank lines.

read_data("our_data.csv")
# [("sandeep", 1, 1.0), ("nitin", 2, 3.0), ("subodh", 2, 1.0),
#  ("sandeep", 2, 2.0), ("nitin", 1, 2.0)]

Write build_mapping(rows) that gives each distinct user an integer ID: 1 for the first user seen, 2 for the second new user, and so on. Users repeat in the data — they must keep the same ID.

build_mapping(rows)   # {"sandeep": 1, "nitin": 2, "subodh": 3}

The mapping file maps email → id. The decoder needs id → email. The Decoder is a separate program — it will not share memory with the Encoder. The mapping must survive as a file.

Write save_mapping(mapping, filename) (one email,id line per user) and load_mapping(filename) that reads it back — types intact (ids are ints).

The test is the round trip:

save_mapping(mapping, "mapping.csv")
load_mapping("mapping.csv") == mapping   # True

Put it together. Write encode_file(input_file, encoded_file, mapping_file) that reads the raw data, builds and saves the mapping, and writes the encoded CSV with emails replaced by their integer IDs.

Expected encoded.csv for our data:

1,1,1.0
2,2,3.0
3,2,1.0
1,2,2.0
2,1,2.0

Here is the engine you bought. You are not allowed to modify it — run the cell as-is. For each user it recommends the products they have not rated yet, scored by that product's average rating from other users.

Notice it would crash on emails: it calls int() on the first column. That is exactly why you wrote the Encoder.

Only one line came out: user 3 should look at product 1. Makes sense — everyone else has already rated everything. But who is user 3? Time for the Decoder.

Run this provided cell as-is. Do not modify the recommender.

The mapping file maps email → id. The decoder needs id → email.

Write invert(mapping). (You built this in the Dictionaries chapter — rebuild it from memory.) Why is inversion safe here — what property must the mapping have?

invert({"sandeep": 1, "nitin": 2})   # {1: "sandeep", 2: "nitin"}

Write decode_file(recs_file, mapping_file, output_file) that loads the mapping from its file, inverts it, reads the engine's recommendations, and writes the final CSV with integer IDs translated back to emails.

decode_file("recommendations.csv", "mapping.csv", "final.csv")
open("final.csv").read()   # "subodh,1,1.5\n"

So the mystery user 3 was subodh, and the pipeline recommends product 1 with score 1.5.

The recommendation engine takes integer IDs instead of email addresses. Why is this encoding step necessary?

A. Emails take too much memory — integers are smaller

B. The recommendation algorithm works with numerical operations (similarity scores, matrix math) that require numbers, not strings

C. Python can't compare strings, only numbers

D. Encoding makes the data more secure by hiding email addresses

A pipeline you have run once works once. A pipeline you have tested works.

The provided cell generates a bigger random dataset. Run your whole pipeline on it, then write check_pipeline(original_file, final_file) that verifies:

1. Every user in the final output existed in the original data (no invented users).
2. No integer IDs leaked into the final output (every user field contains "@").

Think about: what could silently break if two different emails ever received the same ID? Which of your functions guarantees that cannot happen?

The same encode → black box → decode shape appears when you one-hot-encode categories for a model, tokenize text for an LLM, or compress data before a network hop (your Huffman coder!). The black box changes; the wrapper pattern never does.