Every concept in this curriculum is something you will build before it is named. You will hit a wall, feel the frustration of a broken approach, and then invent the fix yourself.
The central question: what does an LLM actually know, and what happens when it doesn't?
Prerequisites. Basic Python (functions, lists, dicts, loops). No ML background required.
Companion files — download these alongside the curriculum:
acme_datasets.py — all datasets used across every exercise. Import what you need.provider_setup.md — step-by-step instructions for setting up Anthropic or OpenAI.Install the Anthropic SDK and run the observe code below. Both answers come back instantly and correctly.
Now ask the model something it cannot possibly know — an internal company document it has never seen. Read the answer carefully.
Your task:
ask() called honest_ask(). Before calling the API, check whether the question contains the phrase "internal memo" or "Q3 2024". If it does, return a plain string: "I don't have access to that document." Otherwise, call the API normally.# !pip install anthropic
import anthropic
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from env
def ask(question):
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=256,
messages=[{"role": "user", "content": question}]
)
return response.content[0].text
print(ask("What is the capital of France?"))
print(ask("Who wrote Hamlet?"))
# This is a made-up internal document. The model has never seen it.
print(ask("According to Acme Corp's Q3 2024 internal memo, what is the new expense policy for travel?"))
def honest_ask(question):
# your code here
pass
print(honest_ask("What is the capital of France?"))
print(honest_ask("According to the Q3 2024 memo, what is the travel policy?"))
Check if "internal memo" or "Q3 2024" appears in the question string using Python's in operator.
def honest_ask(question):
q_lower = question.lower()
if "internal memo" in q_lower or "q3 2024" in q_lower:
return "I don't have access to that document."
return ask(question)
You have just felt the core problem: a language model's knowledge is frozen at training time. It cannot know what happened yesterday, or what is in your company's private database. Your keyword check is a crude patch. The real fix requires giving the model the document at the moment it is asked.
What you invented: the intuition behind retrieval-augmented generation — the idea that knowledge should come from outside the model, not from inside it.
Import the Acme Corp policy documents and try putting all three directly into the prompt (see the observe code). It works — the model answers correctly from the documents.
The problem. Imagine you have 10,000 documents instead of 3, each 500 words. Run the calculation code below. You cannot stuff all 10,000 into the prompt.
Your task. Write a function find_relevant_docs(question, docs) that uses simple keyword matching to pick the most relevant document:
Test it with at least two different questions. Does it always pick the right document?
from acme_datasets import DOCUMENTS, DOC_A, DOC_B, DOC_C
def ask_with_docs(question, docs):
combined = "\n\n---\n\n".join(docs)
prompt = f"""You are a helpful assistant. Use only the documents below to answer the question.
If the answer is not in the documents, say "I don't know."
DOCUMENTS:
{combined}
QUESTION: {question}
"""
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=256,
messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text
all_docs = [DOC_A, DOC_B, DOC_C]
print(ask_with_docs("What is the hotel allowance?", all_docs))
print(ask_with_docs("How many days can I work remotely?", all_docs))
words_per_doc = 500
num_docs = 10_000
total_words = words_per_doc * num_docs
approx_tokens = total_words / 0.75
print(f"Total tokens needed: {approx_tokens:,.0f}")
print(f"That is roughly {approx_tokens / 200_000:.1f}x a 200k context window")
def find_relevant_docs(question, docs):
# your code here
pass
question = "What is the hotel allowance for travel?"
best_doc = find_relevant_docs(question, all_docs)
print(ask_with_docs(question, [best_doc]))
Use set(question.lower().split()) for the query words, then count len(query_words & set(doc.lower().split())) for each document.
def find_relevant_docs(question, docs):
query_words = set(question.lower().split())
best_doc = None
best_score = -1
for doc in docs:
doc_words = set(doc.lower().split())
score = len(query_words & doc_words)
if score > best_score:
best_score = score
best_doc = doc
return best_doc
You have just reinvented keyword-based retrieval. It is fast and requires no ML. But try this question: "What is the per-night accommodation limit?" — the word "hotel" does not appear, and "accommodation" might not match "lodging." You would miss documents that mean the same thing but use different words.
What you invented: the idea of retrieval as a preprocessing step — only sending relevant context to the model. This is the skeleton of RAG. The next module fixes the keyword-matching weakness.
The central question: how do you find a document that means the same thing, even when it uses different words?
Build a tiny keyword search engine using the mini corpus (see the observe code). The first sentence scores highest because it shares the word "hotel." But the second and third sentences mean exactly the same thing and score zero — they share no words with the query.
Your task. Measure meaning using numbers instead of words.
sentence-transformers and run the provided code. Each sentence becomes a vector of 384 numbers.a[i] * b[i] for all i. Write it with a loop (no numpy).semantic_search(query, corpus) function that embeds the query, computes dot products against all corpus embeddings, and returns results sorted by score.Do the accommodation and lodging sentences now rank above the Python sentence?
from acme_datasets import MINI_CORPUS
mini_corpus = MINI_CORPUS
def keyword_search(query, corpus):
query_words = set(query.lower().split())
scores = []
for doc in corpus:
doc_words = set(doc.lower().split())
overlap = len(query_words & doc_words)
scores.append((overlap, doc))
scores.sort(reverse=True)
return scores
results = keyword_search("what is the hotel allowance", mini_corpus)
for score, doc in results:
print(f"Score {score}: {doc[:60]}...")
# !pip install sentence-transformers
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("all-MiniLM-L6-v2")
sentences = [
"The hotel rate is capped at $250 per night.",
"Accommodation costs must not exceed two hundred and fifty dollars.",
"Python is a programming language.",
]
embeddings = model.encode(sentences)
print(f"Each sentence becomes a vector of {embeddings.shape[1]} numbers.")
zip(vec_a, vec_b) gives you pairs of numbers. Sum a_i * b_i for each pair.
For semantic_search: model.encode([query])[0] gives the query vector. model.encode(corpus) gives all corpus vectors at once.
def dot_product(vec_a, vec_b):
return sum(a * b for a, b in zip(vec_a, vec_b))
a = embeddings[0]
b = embeddings[1]
c = embeddings[2]
print(f"Similarity (sentence 1 vs 2): {dot_product(a, b):.3f}")
print(f"Similarity (sentence 1 vs 3): {dot_product(a, c):.3f}")
def semantic_search(query, corpus):
query_vec = model.encode([query])[0]
corpus_vecs = model.encode(corpus)
scores = []
for i, doc_vec in enumerate(corpus_vecs):
score = dot_product(query_vec, doc_vec)
scores.append((score, corpus[i]))
scores.sort(reverse=True)
return scores
You have just used embeddings — dense numerical representations of meaning. The dot product between two embedding vectors measures how semantically similar two pieces of text are. This is the engine inside every modern search system.
What you invented: semantic search using vector embeddings — the retrieval backbone of production RAG systems.
In the real world, you embed documents once (when they are added to the system) and store those embeddings. At query time, you only embed the question — not all the documents again.
Your task. Build a simple in-memory vector store.
Step 1 — Write an index_documents(docs) function that takes the list of structured documents and returns a list of (document, embedding) pairs.
Step 2 — Write a retrieve(query, index, top_k=2) function that embeds the query, computes dot product with every indexed embedding, and returns the top-k most relevant documents.
Step 3 — If you had 1 million documents, computing a dot product with each one would be slow. In one or two sentences, describe what you might do to speed this up.
from sentence_transformers import SentenceTransformer
from acme_datasets import STRUCTURED_DOCUMENTS as documents
model = SentenceTransformer("all-MiniLM-L6-v2")
print(f"Loaded {len(documents)} documents")
def index_documents(docs):
# hint: model.encode([d["text"] for d in docs]) gives all embeddings at once
pass
index = index_documents(documents)
print(f"Indexed {len(index)} documents.")
def retrieve(query, index, top_k=2):
# 1. embed the query
# 2. compute dot product with every indexed embedding
# 3. sort and return top_k documents
pass
results = retrieve("How much can I spend on food while travelling?", index, top_k=2)
for doc, score in results:
print(f"[{score:.3f}] ({doc['source']}) {doc['text']}")
model.encode([d['text'] for d in docs]) returns an array of embeddings. Pair each doc with its embedding: list(zip(docs, embeddings)).
In retrieve: query_vec = model.encode([query])[0], then loop over the index computing np.dot(query_vec, emb) for each (doc, emb) pair.
def index_documents(docs):
texts = [d["text"] for d in docs]
embeddings = model.encode(texts)
return list(zip(docs, embeddings))
def retrieve(query, index, top_k=2):
query_vec = model.encode([query])[0]
scored = []
for doc, emb in index:
score = float(np.dot(query_vec, emb))
scored.append((doc, score))
scored.sort(key=lambda x: -x[1])
return scored[:top_k]
What you invented: a vector index — the data structure at the heart of systems like Pinecone, Weaviate, and pgvector. You embed documents once, store the vectors, and at query time only embed the question.
The central question: how do you combine retrieval with generation to get answers that are grounded in your documents?
You now have two working pieces: a retrieval system and a language model. Try connecting them naively (see the observe code). It works — but there is a subtle problem.
The problem. Ask a question completely outside the documents: "What is the boiling point of water?" Does the model say "I don't know" or does it answer from general knowledge?
Your task. Improve naive_rag into grounded_rag with two changes:
0.35, skip the API call entirely and return the fallback string directly.import anthropic
from sentence_transformers import SentenceTransformer
client = anthropic.Anthropic()
model = SentenceTransformer("all-MiniLM-L6-v2")
# Paste your index_documents and retrieve functions from Exercise 2.2 here
# index = index_documents(documents)
def naive_rag(question):
results = retrieve(question, index, top_k=2)
context = "\n".join([doc["text"] for doc, score in results])
prompt = f"Context:\n{context}\n\nQuestion: {question}\nAnswer:"
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=200,
messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text
print(naive_rag("How much can I spend on a hotel?"))
def grounded_rag(question, confidence_threshold=0.35):
results = retrieve(question, index, top_k=2)
# your confidence gate here
# your improved prompt here
context = "\n".join([doc["text"] for doc, score in results])
prompt = "" # your prompt here
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=200,
messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text
# Test both cases
print(grounded_rag("How much can I spend on a hotel?"))
print(grounded_rag("What is the boiling point of water?"))
The confidence gate: top_score = results[0][1] then if top_score < confidence_threshold: return "I cannot answer this from the provided documents."
For the prompt, put the instruction before the context: "Answer only using the context provided. If the context does not contain the answer, respond with exactly: I cannot answer this from the provided documents."
def grounded_rag(question, confidence_threshold=0.35):
results = retrieve(question, index, top_k=2)
top_score = results[0][1]
if top_score < confidence_threshold:
return "I cannot answer this from the provided documents."
context = "\n".join([doc["text"] for doc, score in results])
prompt = f"""Answer only using the context provided. If the context does not contain the answer, respond with exactly: I cannot answer this from the provided documents.
Context:
{context}
Question: {question}
Answer:"""
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=200,
messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text
You have just built a complete RAG pipeline: retrieve relevant chunks, gate on confidence, generate a grounded answer. The confidence gate is a simple but important production pattern — it prevents the model from hallucinating when retrieval fails.
What you invented: Retrieval-Augmented Generation (RAG) — the dominant architecture for grounding LLMs in private or recent knowledge.
Real documents are long. If you embed an entire 10-page PDF as one vector, you lose precision — the embedding averages out over too many topics. But if you split every sentence into its own chunk, each chunk loses context. There is a Goldilocks size.
Your task. Write a chunk_document(text, chunk_size, overlap) function that splits a document into overlapping chunks of approximately chunk_size words, with overlap words repeated at the start of each new chunk.
Then index the chunks and test retrieval with the provided questions. Finally, experiment: change chunk_size to 20 (very small) and then to 200 (very large). How does retrieval quality change?
from acme_datasets import LONG_TRAVEL_POLICY
print(LONG_TRAVEL_POLICY[:300])
# After implementing chunk_document:
chunks = chunk_document(LONG_TRAVEL_POLICY, chunk_size=60, overlap=15)
print(f"Created {len(chunks)} chunks")
chunked_docs = [{"id": i, "text": chunk, "source": "travel_policy_full.txt"}
for i, chunk in enumerate(chunks)]
chunk_index = index_documents(chunked_docs)
questions = [
"Can I fly business class?",
"Can I use Airbnb?",
"How much per kilometre for my own car?",
"Is alcohol reimbursable?",
]
for q in questions:
results = retrieve(q, chunk_index, top_k=1)
doc, score = results[0]
print(f"\nQ: {q}")
print(f"Score {score:.3f}: {doc['text'][:100]}...")
Use text.split() to get a word list. Slide a window: start = 0, advance by chunk_size - overlap each iteration, take words[start:start+chunk_size].
def chunk_document(text, chunk_size=100, overlap=20):
words = text.split()
chunks = []
start = 0
while start < len(words):
end = start + chunk_size
chunk = " ".join(words[start:end])
chunks.append(chunk)
start += chunk_size - overlap
return chunks
What you invented: document chunking with overlap — a standard preprocessing step in all production RAG systems. The overlap prevents relevant sentences from being split across chunk boundaries.
The central question: what happens when answering a question requires doing something — not just knowing something?
Ask the model things that require real-time actions (see the observe code). The time question fails — the model cannot check a clock. The calculator question might work for easy math but could fail on harder ones. The model is stateless and sandboxed: it cannot call your system clock, run a calculator, search the web, or write a file.
Your task.
get_current_date(), calculate(expression), days_until(target_date_str).TOOL: tool_name / INPUT: the input, and final answers as ANSWER: the final response. Test it — does the model follow the protocol?import anthropic
import datetime
client = anthropic.Anthropic()
def ask(question):
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=200,
messages=[{"role": "user", "content": question}]
)
return response.content[0].text
print(ask("What time is it right now?"))
print(ask("What is 10% of 847.33?"))
print(ask(f"Today is {datetime.date.today()}. How many days until Christmas?"))
import datetime
import math
def get_current_date():
"""Returns today's date as a string."""
return str(datetime.date.today())
def calculate(expression):
"""Safely evaluate a mathematical expression string."""
allowed = set("0123456789+-*/()., ")
if not all(c in allowed for c in expression):
return "Error: invalid expression"
try:
result = eval(expression)
return str(round(result, 4))
except Exception as e:
return f"Error: {e}"
def days_until(target_date_str):
"""Returns the number of days from today until a target date (YYYY-MM-DD)."""
today = datetime.date.today()
target = datetime.date.fromisoformat(target_date_str)
return str((target - today).days)
Your tool prompt should list each tool by name and describe what it does. Tell the model: when you need a tool, respond ONLY with TOOL: name / INPUT: value. When you have a final answer, respond ONLY with ANSWER: your answer.
TOOL_PROMPT = """You are a helpful assistant. You have access to these tools:
- get_current_date(): returns today's date
- calculate(expression): evaluates a math expression, e.g. calculate(2 + 2)
- days_until(YYYY-MM-DD): returns number of days from today until that date
When you need a tool, respond ONLY with:
TOOL: tool_name
INPUT: the input
When you have a final answer, respond ONLY with:
ANSWER: your answer
Do not include any other text.
"""
def ask_with_tools(question):
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=100,
messages=[
{"role": "user", "content": TOOL_PROMPT + "\n\nQuestion: " + question}
]
)
return response.content[0].text
print(ask_with_tools("What is today's date?"))
print(ask_with_tools("What is 15% of 1247?"))
The model produces structured text that describes what tool to call. You have to parse that text and actually run the tool. The model is a reasoning engine, not an execution engine. You are the executor.
What you invented: the concept of tool use (also called function calling) — the mechanism that allows language models to request external actions.
Complete the loop: parse the model's text output, run the right tool, and feed the result back.
Your task. Write a one_step_agent(question) function that:
parse_response (provided).run_tool (provided) and prints the result.The limitation. Try: "How many days from today until New Year's Day next year?" — this requires two tool calls (get_current_date then days_until). Your one-step agent can only do one.
def parse_response(text):
"""
Returns:
("tool", tool_name, tool_input) — if the model wants a tool
("answer", final_answer) — if the model has a final answer
("unknown", raw_text) — if parsing fails
"""
text = text.strip()
lines = text.split("\n")
if lines[0].startswith("TOOL:"):
tool_name = lines[0].replace("TOOL:", "").strip()
tool_input = lines[1].replace("INPUT:", "").strip() if len(lines) > 1 else ""
return ("tool", tool_name, tool_input)
elif lines[0].startswith("ANSWER:"):
answer = lines[0].replace("ANSWER:", "").strip()
return ("answer", answer)
else:
return ("unknown", text)
def run_tool(tool_name, tool_input):
if tool_name == "get_current_date":
return get_current_date()
elif tool_name == "calculate":
return calculate(tool_input)
elif tool_name.startswith("days_until"):
return days_until(tool_input)
else:
return f"Unknown tool: {tool_name}"
def one_step_agent(question):
raw = ask_with_tools(question)
print(f"Model said: {raw}")
result = parse_response(raw)
if result[0] == "tool":
# your code here: run the tool, print the result
pass
elif result[0] == "answer":
# your code here: return the answer
pass
else:
return f"Could not parse: {result[1]}"
print(one_step_agent("What is today's date?"))
print(one_step_agent("Calculate 18% of 350."))
In the tool branch: _, tool_name, tool_input = result then tool_result = run_tool(tool_name, tool_input). Print and return it.
def one_step_agent(question):
raw = ask_with_tools(question)
print(f"Model said: {raw}")
result = parse_response(raw)
if result[0] == "tool":
_, tool_name, tool_input = result
tool_result = run_tool(tool_name, tool_input)
print(f"Tool result: {tool_result}")
return f"Tool was called. Result: {tool_result}"
elif result[0] == "answer":
return result[1]
else:
return f"Could not parse: {result[1]}"
A single tool call is not enough for complex questions. You need a loop — the model calls a tool, gets a result, decides if it needs another tool, calls again, and so on. That loop is an agent.
What you invented: the tool execution loop — the bridge between a model's text output and real-world actions.
The central question: how do you give a model the ability to act repeatedly until a task is complete?
An agent is a loop where:
Your task. Build run_agent(question, max_steps=5) — a multi-turn agent using a conversation history list. At each step, ask the model, parse the response, and either run a tool (appending the result back to the conversation) or return the final answer.
Test with questions requiring different numbers of steps:
"What is today's date?" — 1 step"What is 22% of 4500?" — 1 step"How many days until July 4th from today?" — 2 steps"What is 15% of the number of days until New Year?" — 3 stepsdef run_agent(question, max_steps=5):
messages = [
{"role": "user",
"content": TOOL_PROMPT + "\n\nQuestion: " + question}
]
for step in range(max_steps):
print(f"\n--- Step {step + 1} ---")
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=150,
messages=messages
)
model_text = response.content[0].text.strip()
print(f"Model: {model_text}")
messages.append({"role": "assistant", "content": model_text})
result = parse_response(model_text)
if result[0] == "answer":
print(f"\nFinal answer: {result[1]}")
return result[1]
elif result[0] == "tool":
# your code here:
# 1. run the tool
# 2. append the result to messages as a user message
# 3. the loop continues
pass
else:
print(f"Parse failed: {result[1]}")
break
return "Agent exceeded maximum steps without finding an answer."
After running the tool, append the result as a user message: messages.append({"role": "user", "content": f"Tool result: {tool_result}\n\nContinue. If you have enough information, provide the ANSWER. Otherwise, call another tool."})
def run_agent(question, max_steps=5):
messages = [
{"role": "user",
"content": TOOL_PROMPT + "\n\nQuestion: " + question}
]
for step in range(max_steps):
print(f"\n--- Step {step + 1} ---")
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=150,
messages=messages
)
model_text = response.content[0].text.strip()
print(f"Model: {model_text}")
messages.append({"role": "assistant", "content": model_text})
result = parse_response(model_text)
if result[0] == "answer":
print(f"\nFinal answer: {result[1]}")
return result[1]
elif result[0] == "tool":
_, tool_name, tool_input = result
tool_result = run_tool(tool_name, tool_input)
print(f"Tool result: {tool_result}")
messages.append({
"role": "user",
"content": f"Tool result: {tool_result}\n\nContinue. If you have enough information, provide the ANSWER. Otherwise, call another tool."
})
else:
print(f"Parse failed: {result[1]}")
break
return "Agent exceeded maximum steps without finding an answer."
What you invented: the agentic loop — the core architecture of every AI agent, from simple assistants to autonomous research systems. This is also known as the ReAct pattern (Reasoning + Acting).
Your agent has no memory between separate run_agent calls. Try: first tell it "My monthly budget is $3000.", then ask "How much is 40% of my budget?" — the second call fails.
Your task. Add a simple external memory store with two new tools:
remember(key=value) — store a piece of informationrecall(key) — retrieve a stored piece of informationUpdate your TOOL_PROMPT to include these tools and update run_tool to dispatch to them. Then test the two-call sequence above.
memory_store = {}
def remember(key_value):
if "=" not in key_value:
return "Error: format must be key=value"
key, value = key_value.split("=", 1)
memory_store[key.strip()] = value.strip()
return f"Remembered: {key.strip()} = {value.strip()}"
def recall(key):
key = key.strip()
if key in memory_store:
return memory_store[key]
return f"Nothing stored under '{key}'"
TOOL_PROMPT_V2 = """You are a helpful assistant with access to these tools:
- get_current_date(): returns today's date
- calculate(expression): evaluates a math expression
- days_until(YYYY-MM-DD): returns days from today to target date
- remember(key=value): stores information for later
- recall(key): retrieves stored information
When you need a tool, respond ONLY with:
TOOL: tool_name
INPUT: the input
When you have a final answer, respond ONLY with:
ANSWER: your answer
"""
Add two branches to run_tool: elif tool_name == "remember": return remember(tool_input) and elif tool_name == "recall": return recall(tool_input). Then update run_agent to use TOOL_PROMPT_V2 and the new dispatch function.
def run_tool_v2(tool_name, tool_input):
if tool_name == "get_current_date":
return get_current_date()
elif tool_name == "calculate":
return calculate(tool_input)
elif tool_name.startswith("days_until"):
return days_until(tool_input)
elif tool_name == "remember":
return remember(tool_input)
elif tool_name == "recall":
return recall(tool_input)
else:
return f"Unknown tool: {tool_name}"
# Update run_agent to use TOOL_PROMPT_V2 and run_tool_v2
memory_store.clear()
run_agent("Remember that my monthly budget is 3000.", max_steps=3)
run_agent("How much is 40% of my monthly budget?", max_steps=5)
You have added persistent state to a stateless model. The model itself still forgets between calls — but the system around it remembers. This pattern (external memory + stateless model) is how real production agents are built.
What you invented: external agent memory — the pattern used in systems like LangChain's memory modules, AutoGPT's file-based memory, and enterprise AI assistants.
Your final exercise combines everything: the vector index from Module 2, the grounded generation from Module 3, and the agentic loop from Module 5.
Your task. Add a search_docs(query) tool that wraps your retrieve function. Update the tool prompt to include it. Then build a complete RAG agent that can both retrieve information from documents AND use tools — and decide which it needs at each step.
Test with:
"What is the hotel allowance per night?" — retrieval only"I have a 5-day trip. What is the maximum I can claim for meals?" — retrieval + calculationremember, then ask about total costs — retrieval + memory + calculationBonus: Use QA_EVAL_SET from acme_datasets to measure your agent's accuracy across 15 question-answer pairs.
def search_docs(query):
results = retrieve(query, index, top_k=1)
if not results:
return "No relevant documents found."
doc, score = results[0]
if score < 0.30:
return "No sufficiently relevant documents found."
return f"[Source: {doc['source']}] {doc['text']}"
TOOL_PROMPT_FINAL = """You are a helpful assistant for Acme Corp employees.
You have access to these tools:
- search_docs(query): searches the internal policy documents
- calculate(expression): evaluates a math expression
- get_current_date(): returns today's date
- days_until(YYYY-MM-DD): returns days from today to a date
- remember(key=value): stores a fact for later
- recall(key): retrieves a stored fact
Use search_docs first when the question is about company policy.
Use calculate for any arithmetic.
Chain tools as needed.
When you need a tool:
TOOL: tool_name
INPUT: the input
When you have a final answer:
ANSWER: your answer
"""
from acme_datasets import QA_EVAL_SET
correct = 0
for qa in QA_EVAL_SET:
print(f"\nQ: {qa['question']}")
print(f"Expected: {qa['expected_answer']}")
agent_answer = run_agent(qa['question'], max_steps=5)
print(f"Agent: {agent_answer}")
Add elif tool_name == "search_docs": return search_docs(tool_input) to your dispatch function. Update run_agent to use TOOL_PROMPT_FINAL.
For the multi-step hotel+meals question, the agent should call search_docs(hotel rate), search_docs(meal allowance), then calculate(5 * 250 + 5 * 60) or similar.
def run_tool_final(tool_name, tool_input):
if tool_name == "get_current_date":
return get_current_date()
elif tool_name == "calculate":
return calculate(tool_input)
elif tool_name.startswith("days_until"):
return days_until(tool_input)
elif tool_name == "remember":
return remember(tool_input)
elif tool_name == "recall":
return recall(tool_input)
elif tool_name == "search_docs":
return search_docs(tool_input)
else:
return f"Unknown tool: {tool_name}"
# Update run_agent to use TOOL_PROMPT_FINAL and run_tool_final
# Then test:
run_agent("What is the hotel allowance per night?", max_steps=3)
run_agent("I have a 5-day business trip. What is the maximum I can claim for meals?", max_steps=5)
What you invented: a RAG-powered tool-using agent — an architecture that appears in production systems like Perplexity, enterprise knowledge assistants, and AI copilots. You built it from scratch, from first principles.
Your Inventions vs. Industry Names
| Exercise | What you built | Industry name |
|---|---|---|
| 1.1 | Detected unanswerable questions | Hallucination detection |
| 1.2 | Selected relevant docs before prompting | Context injection / prompt stuffing |
| 2.1 | Measured meaning with dot products | Semantic similarity |
| 2.2 | Stored embeddings once, queried many times | Vector index |
| 3.1 | Retrieved then generated, with a fallback | Retrieval-Augmented Generation (RAG) |
| 3.2 | Split long docs into overlapping windows | Chunking with overlap |
| 4.1 | Made the model describe actions as text | Tool use / function calling |
| 4.2 | Parsed model text and ran the right function | Tool dispatcher |
| 5.1 | Looped model + tools until task complete | Agentic loop / ReAct pattern |
| 5.2 | Added persistent storage outside the model | External agent memory |
| 5.3 | Combined retrieval, tools, memory, and loop | RAG agent |
Once you have completed all exercises, the natural next steps are: