Practice lab: Locate the failure before changing the model

Task, 15 minutes: build a retrieval evaluation table with relevant source ids, retrieved ids, recall@3, and reciprocal rank. Calculate the two answerable cases and their mean scores. Explain how to score an unanswerable case separately. Then describe how a trace distinguishes a retrieval miss from an answer that contradicts retrieved evidence, and propose a negative test for a forbidden source.

Case A: relevant sources are a and b. Top-3 results are x, a, z. Case B: the only relevant source is c. Top-3 results are c, x, z. Case C: no permitted source contains the answer, so the expected behavior is to say the evidence is insufficient.

Check your work:

• Case A has recall@3 of 1/2 and reciprocal rank of 1/2. Case B has both equal to 1. Their mean recall and MRR are each 0.75.
• Score Case C's abstention separately. Recall with zero relevant sources has no useful denominator here.
• If a required source was never retrieved, investigate retrieval. If it was retrieved but the answer contradicts it, investigate generation and grounding.
• Add a forbidden source containing an attractive answer. It must not enter the retrieved context or a reused cache response.

Follow-up: extend the toy fixture with actual passages and a concrete question for each case. Then write one expected supported claim and one forbidden unsupported claim for each answer. Keep a held-out evaluation set when tuning so that improving remembered examples is not mistaken for general improvement.