CyberFreezeDev- Home
- Blog
- How-to guides
- How to evaluate a RAG chatbot with a real test set
How to evaluate a RAG chatbot with a real test set
A RAG chatbot is only as trustworthy as the test you run it against. How to build a test set from real tickets, which retrieval and answer metrics to track, and how to stop a change from quietly breaking it.
To evaluate a RAG chatbot, build a fixed test set of 50 to 100 real questions taken from support tickets and staff emails, each with the correct answer and the document and page that holds it. Score retrieval (did the right passage come back?) separately from the answer (is it correct, faithful to the passages and properly cited?). Rerun the whole set before every change and release only if nothing that passed before now fails.
This guide is for operations and IT leads who have a retrieval-augmented generation pilot that "looks good" and need a way to prove it, or to catch the day it stops being true.
Step 1: build the test set from real tickets
Questions you invent at a whiteboard are cleaner than the ones people actually type. Start from the record.
- Pull three to six months of real questions from the help desk, shared inboxes and team chat channels.
- Remove duplicates and personal data. Keep the wording, typos and all.
- Write the reference answer and its source for each: document, version and page or section. A subject expert signs off, not the developer.
- Cover the hard cases on purpose: questions with part numbers or clause references, answers in tables, answers that need two documents, and questions your documents do not answer at all.
- Add permission cases: the same question asked as a user who may see the answer and as one who may not.
A spreadsheet is enough:
| Column | Example |
|---|---|
| Question | "Can I carry over unused leave into next year?" |
| Reference answer | "Up to 5 days, used by 31 March." |
| Source | Leave policy v3, section 4.2 |
| Type | Policy, single document |
| Asked as | Employee, UK entity |
Ragas includes a TestsetGenerator that writes synthetic questions from your documents. Use it to fill gaps in coverage, not to replace the real questions.
Retrieval metrics vs answer metrics
When an answer is wrong, you need to know which half failed. If the right passage never came back, no prompt change will fix it.
| Metric | What it asks | Needs a reference? |
|---|---|---|
| Hit rate at 5 | Is the source passage in the top 5 results? | Yes, the source |
| Context precision | Are relevant passages ranked above irrelevant ones? | Optional in Ragas |
| Context recall | Did retrieval find everything the answer needs? | Yes |
| Answer correctness | Does the answer match the reference? | Yes |
| Faithfulness | Is every claim supported by the retrieved passages? | No |
| Response relevancy | Does the answer address the question? | No |
Read the last one carefully: Ragas states that response relevancy is assessed without evaluating factual accuracy. A fluent, on-topic, wrong answer can score well on it. TruLens groups the same idea into its "RAG triad" of context relevance, groundedness and answer relevance.
Faithfulness: the hallucination check
Ragas computes faithfulness by splitting the answer into individual claims and checking each against the retrieved passages: supported claims divided by total claims, from 0 to 1. Anything below 1 means the model added something the documents do not say, which is the working definition of a hallucination in a RAG system.
Faithfulness alone is not enough. If retrieval returned last year's policy, the answer can be perfectly faithful and still wrong. That is why answer correctness against your reference, and the version in the source column, both matter.
For the unanswerable questions, the only passing answer is "the documents do not cover this", with a route to a person.
Checking citations
A citation passes when the cited passage actually supports the sentence it is attached to, and points to the current version of the document. Automated judges can check this at scale, but they have known biases: a 2023 study of language models used as judges found position, verbosity and self-enhancement biases, while a strong judge (GPT-4) agreed with humans over 80% of the time, about as often as humans agree with each other. So run the automated check on every question, and have a person check a sample of 20 or so citations by hand on every run.
Regression runs before each change
Rerun the full set whenever anything changes: chunking, the embedding model, the search weights, the prompt, the language model version, or a large batch of new documents. Then:
- compare every metric with the last accepted run, not with a target you set months ago;
- list the questions that flipped from pass to fail, and read each one;
- block the release if a question that passed before now fails, unless you can explain why the new answer is better;
- add every thumbs-down answer from real users to the test set once it is fixed.
Ragas (0.4.3, Apache 2.0) and DeepEval (Apache 2.0) both implement the metrics above. DeepEval runs as a Pytest suite, so the set can sit in your CI pipeline. Both use a language model as the judge, so each run has a model cost. Our glossary page on model evaluation covers the wider picture.
Where we fit
Our RAG chatbot builds start with this question set, written with your team before anything is indexed. Every change to chunking, search or prompts is scored against it, and citations are checked by hand on a sample.
If you have a pilot that demos well but nobody can say how often it is right, we can help you build the test set and run the first evaluation. Book a consultation.
