Project: A "Chat With Your Notes" Assistant
Put the whole course together: a small assistant that answers questions from your own study notes with citations, an evaluation set, guardrails and cost limits, ready to put online with Flask.
The brief
Build an assistant that answers questions about a folder of your own notes (lecture notes, a handbook, or this course's further reading). It must:
- answer only from the notes and cite the note and section for every claim;
- say "I don't know" when the notes do not cover the question;
- pass an evaluation set of at least 30 questions you wrote, including 5 it should refuse;
- cap tokens per answer and requests per minute, and log every question, retrieved chunk, answer, cost and response time.
Milestones
- Index: load the notes, chunk by heading, embed, save the vectors (a NumPy file is fine).
- Retrieve: a function that returns the top k chunks; measure recall@k on 15 questions where you know the right section. Try hybrid search if it is below 0.8.
- Answer: a prompt template with sources in tags, citation rules and the "I don't know" rule; structured output
{"answer": ..., "citations": [...]}validated with pydantic. - Evaluate: run all 30 questions; score faithfulness by hand or with an LLM judge checked against your own ratings on 10 answers.
- Harden: try 5 prompt-injection attacks hidden in a note; add limits and logging.
- Ship: wrap it in a Flask API and web page, and deploy it, following the Flask & Deployment course.
Show it off
In your portfolio README, show a screenshot, a diagram of the pipeline, your evaluation table (recall@k, faithfulness, refusal accuracy), the cost per 100 questions, and one failure you found and fixed. Evidence of evaluation is what makes an LLM project stand out to employers; see Portfolio Projects.
Quiz
1. Your assistant answers a question the notes do not cover, with a made-up answer. Which part of the brief catches this?
2. Why measure recall@k before writing the answer prompt?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Ragas documentation ↗ metrics for evaluating RAG pipelines
- Pydantic documentation ↗ data validation for structured LLM output