Inventing RAG and Agentic AI

The central question: what does an LLM actually know, and what happens when it doesn't?

Prerequisites. Basic Python (functions, lists, dicts, loops). No ML background required.

Companion files — download these alongside the curriculum:

• acme_datasets.py — all datasets used across every exercise. Import what you need.
• provider_setup.md — step-by-step instructions for setting up Anthropic or OpenAI.

Install the Anthropic SDK and run the observe code below. Both answers come back instantly and correctly.

Now ask the model something it cannot possibly know — an internal company document it has never seen. Read the answer carefully.

Your task:

1. The model answered confidently. Is that answer trustworthy? How would you know?
2. Write a new version of ask() called honest_ask(). Before calling the API, check whether the question contains the phrase "internal memo" or "Q3 2024". If it does, return a plain string: "I don't have access to that document." Otherwise, call the API normally.
3. This keyword check is brittle. In one sentence, describe why it will fail for most real questions.

You have just felt the core problem: a language model's knowledge is frozen at training time. It cannot know what happened yesterday, or what is in your company's private database. Your keyword check is a crude patch. The real fix requires giving the model the document at the moment it is asked.

What you invented: the intuition behind retrieval-augmented generation — the idea that knowledge should come from outside the model, not from inside it.

Import the Acme Corp policy documents and try putting all three directly into the prompt (see the observe code). It works — the model answers correctly from the documents.

The problem. Imagine you have 10,000 documents instead of 3, each 500 words. Run the calculation code below. You cannot stuff all 10,000 into the prompt.

Your task. Write a function find_relevant_docs(question, docs) that uses simple keyword matching to pick the most relevant document:

1. Split the question into individual words (lowercase).
2. For each document, count how many of those words appear in the document text (lowercase).
3. Return only the single document with the highest count.

Test it with at least two different questions. Does it always pick the right document?

You have just reinvented keyword-based retrieval. It is fast and requires no ML. But try this question: "What is the per-night accommodation limit?" — the word "hotel" does not appear, and "accommodation" might not match "lodging." You would miss documents that mean the same thing but use different words.

What you invented: the idea of retrieval as a preprocessing step — only sending relevant context to the model. This is the skeleton of RAG. The next module fixes the keyword-matching weakness.

The central question: how do you find a document that means the same thing, even when it uses different words?

Build a tiny keyword search engine using the mini corpus (see the observe code). The first sentence scores highest because it shares the word "hotel." But the second and third sentences mean exactly the same thing and score zero — they share no words with the query.

Your task. Measure meaning using numbers instead of words.

1. Install sentence-transformers and run the provided code. Each sentence becomes a vector of 384 numbers.
2. Implement dot-product similarity between two vectors: sum of a[i] * b[i] for all i. Write it with a loop (no numpy).
3. Build a semantic_search(query, corpus) function that embeds the query, computes dot products against all corpus embeddings, and returns results sorted by score.

Do the accommodation and lodging sentences now rank above the Python sentence?

You have just used embeddings — dense numerical representations of meaning. The dot product between two embedding vectors measures how semantically similar two pieces of text are. This is the engine inside every modern search system.

What you invented: semantic search using vector embeddings — the retrieval backbone of production RAG systems.

In the real world, you embed documents once (when they are added to the system) and store those embeddings. At query time, you only embed the question — not all the documents again.

Your task. Build a simple in-memory vector store.

Step 1 — Write an index_documents(docs) function that takes the list of structured documents and returns a list of (document, embedding) pairs.

Step 2 — Write a retrieve(query, index, top_k=2) function that embeds the query, computes dot product with every indexed embedding, and returns the top-k most relevant documents.

Step 3 — If you had 1 million documents, computing a dot product with each one would be slow. In one or two sentences, describe what you might do to speed this up.

What you invented: a vector index — the data structure at the heart of systems like Pinecone, Weaviate, and pgvector. You embed documents once, store the vectors, and at query time only embed the question.

The central question: how do you combine retrieval with generation to get answers that are grounded in your documents?

You now have two working pieces: a retrieval system and a language model. Try connecting them naively (see the observe code). It works — but there is a subtle problem.

The problem. Ask a question completely outside the documents: "What is the boiling point of water?" Does the model say "I don't know" or does it answer from general knowledge?

Your task. Improve naive_rag into grounded_rag with two changes:

1. Improve the prompt. Add a clear instruction: "Answer only using the context provided. If the context does not contain the answer, respond with exactly: I cannot answer this from the provided documents."
2. Add a confidence gate: after retrieval, check the top similarity score. If the best-matching document scores below 0.35, skip the API call entirely and return the fallback string directly.

You have just built a complete RAG pipeline: retrieve relevant chunks, gate on confidence, generate a grounded answer. The confidence gate is a simple but important production pattern — it prevents the model from hallucinating when retrieval fails.

What you invented: Retrieval-Augmented Generation (RAG) — the dominant architecture for grounding LLMs in private or recent knowledge.

Real documents are long. If you embed an entire 10-page PDF as one vector, you lose precision — the embedding averages out over too many topics. But if you split every sentence into its own chunk, each chunk loses context. There is a Goldilocks size.

Your task. Write a chunk_document(text, chunk_size, overlap) function that splits a document into overlapping chunks of approximately chunk_size words, with overlap words repeated at the start of each new chunk.

Then index the chunks and test retrieval with the provided questions. Finally, experiment: change chunk_size to 20 (very small) and then to 200 (very large). How does retrieval quality change?

What you invented: document chunking with overlap — a standard preprocessing step in all production RAG systems. The overlap prevents relevant sentences from being split across chunk boundaries.

The central question: what happens when answering a question requires doing something — not just knowing something?

Ask the model things that require real-time actions (see the observe code). The time question fails — the model cannot check a clock. The calculator question might work for easy math but could fail on harder ones. The model is stateless and sandboxed: it cannot call your system clock, run a calculator, search the web, or write a file.

Your task.

1. Build a small toolkit of Python functions: get_current_date(), calculate(expression), days_until(target_date_str).
2. Invent a simple protocol. Write a prompt that instructs the model to output tool calls as TOOL: tool_name / INPUT: the input, and final answers as ANSWER: the final response. Test it — does the model follow the protocol?

The model produces structured text that describes what tool to call. You have to parse that text and actually run the tool. The model is a reasoning engine, not an execution engine. You are the executor.

What you invented: the concept of tool use (also called function calling) — the mechanism that allows language models to request external actions.

Complete the loop: parse the model's text output, run the right tool, and feed the result back.

Your task. Write a one_step_agent(question) function that:

1. Asks the model a question using the tool prompt.
2. Parses the response using parse_response (provided).
3. If it is a tool call, runs the tool using run_tool (provided) and prints the result.
4. If it is an answer, returns it.

The limitation. Try: "How many days from today until New Year's Day next year?" — this requires two tool calls (get_current_date then days_until). Your one-step agent can only do one.

A single tool call is not enough for complex questions. You need a loop — the model calls a tool, gets a result, decides if it needs another tool, calls again, and so on. That loop is an agent.

What you invented: the tool execution loop — the bridge between a model's text output and real-world actions.

The central question: how do you give a model the ability to act repeatedly until a task is complete?

An agent is a loop where:

1. The model receives the question and any previous tool results.
2. It decides: do I have enough information to answer, or do I need a tool?
3. If it needs a tool, you run it, add the result to the conversation, and go to step 1.
4. If it has an answer, return it.

Your task. Build run_agent(question, max_steps=5) — a multi-turn agent using a conversation history list. At each step, ask the model, parse the response, and either run a tool (appending the result back to the conversation) or return the final answer.

Test with questions requiring different numbers of steps:

• "What is today's date?" — 1 step
• "What is 22% of 4500?" — 1 step
• "How many days until July 4th from today?" — 2 steps
• "What is 15% of the number of days until New Year?" — 3 steps

What you invented: the agentic loop — the core architecture of every AI agent, from simple assistants to autonomous research systems. This is also known as the ReAct pattern (Reasoning + Acting).

Your agent has no memory between separate run_agent calls. Try: first tell it "My monthly budget is $3000.", then ask "How much is 40% of my budget?" — the second call fails.

Your task. Add a simple external memory store with two new tools:

• remember(key=value) — store a piece of information
• recall(key) — retrieve a stored piece of information

Update your TOOL_PROMPT to include these tools and update run_tool to dispatch to them. Then test the two-call sequence above.

You have added persistent state to a stateless model. The model itself still forgets between calls — but the system around it remembers. This pattern (external memory + stateless model) is how real production agents are built.

What you invented: external agent memory — the pattern used in systems like LangChain's memory modules, AutoGPT's file-based memory, and enterprise AI assistants.

Your final exercise combines everything: the vector index from Module 2, the grounded generation from Module 3, and the agentic loop from Module 5.

Your task. Add a search_docs(query) tool that wraps your retrieve function. Update the tool prompt to include it. Then build a complete RAG agent that can both retrieve information from documents AND use tools — and decide which it needs at each step.

Test with:

• "What is the hotel allowance per night?" — retrieval only
• "I have a 5-day trip. What is the maximum I can claim for meals?" — retrieval + calculation
• Store trip details with remember, then ask about total costs — retrieval + memory + calculation

Bonus: Use QA_EVAL_SET from acme_datasets to measure your agent's accuracy across 15 question-answer pairs.

What you invented: a RAG-powered tool-using agent — an architecture that appears in production systems like Perplexity, enterprise knowledge assistants, and AI copilots. You built it from scratch, from first principles.

Going Further

Once you have completed all exercises, the natural next steps are:

• Proper vector databases. Replace your in-memory index with ChromaDB or Qdrant — both have free local modes and Python clients.
• The ReAct paper. Read ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022). It formalises the loop you built in Exercise 5.1.
• Anthropic's native tool use. The Anthropic API supports tool/function calling natively — the model returns structured JSON instead of the text protocol you invented. Rewrite your agent using it.
• Evaluation. Build a tiny evaluation set: 10 questions with known correct answers, and measure how often your agent gets them right.
• Multi-agent systems. What if one agent specialised in retrieval and another in calculation, and a coordinator decided which to call? That is a multi-agent system — the current frontier.