RAG evaluation: test retrieval before judging the final answer

Separate finding evidence from using evidence

A fluent answer can hide a bad retrieval result. The reverse happens too: the system finds the right passage and then misstates it. Keep separate records for finding the evidence and answering from it. Otherwise a single RAG score makes it difficult to tell which part needs fixing.

Begin with a bounded collection you are authorized to use. Identify who owns the documents, which version is current and whether different users may see different records. Retrieval quality cannot compensate for an index that exposes another customer’s data. Access checks belong in the retrieval path before passages are provided to the model.

Write questions with known supporting passages

For each question, record the expected evidence and the conditions under which the system should ask for clarification or decline to answer. Include questions that the collection cannot answer. If every test question has an answer, a system that always invents one can appear more capable than it is.

Use stable document identifiers and paragraph references. When a source changes, you need to distinguish an outdated test fixture from a model regression. Remove private details when preparing a shareable dataset, but preserve the structural difficulty of the task. Replacing every long document with a short obvious sentence can make the test unrealistically easy.

Compare changes on the same dataset

Change one major variable at a time: chunking, embedding model, search method, reranking or the answer prompt. Preserve the same questions and grading definitions. Record retrieved document IDs and the final answer so you can inspect why a score changed. A difference in answer style alone does not establish better retrieval.

Anthropic describes contextual retrieval as adding contextual information to chunks before indexing. It is a technique to evaluate, not a reason to assume that every document collection will improve by the same percentage. This guide does not reuse a vendor’s benchmark result as evidence for your system.

Use both automated and reviewed checks

Deterministic checks can confirm that a required document ID appears or that a response includes the fields your application expects. A reviewed rubric can assess whether the cited passage actually supports the answer. Model-based grading can help with scale, but calibrate it against human judgments and investigate disagreements.

A citation URL is not sufficient on its own. The linked source may be irrelevant, outdated or inaccessible to the user. Grade the relationship between the claim and evidence. Include a case where a retrieved document contains instructions telling the agent to ignore its task; the document should remain data rather than become authority.

Make the next experiment specific

If retrieval misses the needed passage, inspect indexing and search settings before rewriting the answer prompt. If the passage is present but the answer is unsupported, focus on generation and grading. Keep cost and latency alongside outcome quality, since a configuration that retrieves far more content may create an expensive context burden.

Prepare the dataset with the CSV converter, inspect uncertainty using the evaluation calculator, and retain an incident-to-test feedback loop. Your conclusion should describe the collection, questions and settings tested, rather than declare that one architecture is universally best.

Implementation map

Sources and limits

Anthropic contextual retrieval. No reported vendor benchmark gain is asserted for this website or a client. Documentation checked October 14, 2026. Recommendations and worked examples are editorial methods, not claims of a completed client engagement.