Generative AI & LLM Apps
Topic 12 · Making it reliable · ⏱ about 10 h

Project: A "Chat With Your Notes" Assistant

Put the whole course together: a small assistant that answers questions from your own study notes with citations, an evaluation set, guardrails and cost limits, ready to put online with Flask.

Show:
In this topic
Beginner

The brief

Build an assistant that answers questions about a folder of your own notes (lecture notes, a handbook, or this course's further reading). It must:

  • answer only from the notes and cite the note and section for every claim;
  • say "I don't know" when the notes do not cover the question;
  • pass an evaluation set of at least 30 questions you wrote, including 5 it should refuse;
  • cap tokens per answer and requests per minute, and log every question, retrieved chunk, answer, cost and response time.
Beginner

Milestones

  1. Index: load the notes, chunk by heading, embed, save the vectors (a NumPy file is fine).
  2. Retrieve: a function that returns the top k chunks; measure recall@k on 15 questions where you know the right section. Try hybrid search if it is below 0.8.
  3. Answer: a prompt template with sources in tags, citation rules and the "I don't know" rule; structured output {"answer": ..., "citations": [...]} validated with pydantic.
  4. Evaluate: run all 30 questions; score faithfulness by hand or with an LLM judge checked against your own ratings on 10 answers.
  5. Harden: try 5 prompt-injection attacks hidden in a note; add limits and logging.
  6. Ship: wrap it in a Flask API and web page, and deploy it, following the Flask & Deployment course.
Beginner

Show it off

In your portfolio README, show a screenshot, a diagram of the pipeline, your evaluation table (recall@k, faithfulness, refusal accuracy), the cost per 100 questions, and one failure you found and fixed. Evidence of evaluation is what makes an LLM project stand out to employers; see Portfolio Projects.

Check yourself

Quiz

1. Your assistant answers a question the notes do not cover, with a made-up answer. Which part of the brief catches this?

2. Why measure recall@k before writing the answer prompt?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready