A fluent answer with a working citation can still use the wrong evidence, omit a condition, or reveal another tenant's information.
RAG evaluation checks whether retrieval augmented generation finds appropriate evidence and produces useful answers supported by that evidence. A fluent response is insufficient. The system can retrieve an obsolete policy, overlook an exception, or attach a citation that does not support its claim.
Evaluate retrieval and generation separately before judging the complete experience. This guide proposes a practical engineering workflow, not reported test results. Every synthetic scenario below is explicitly hypothetical. RAG can help ground answers, but it does not eliminate hallucinations or make source material automatically trustworthy.
Define the answer contract and evaluation dataset
Specify what a successful answer must include
Start with the user's decision. Define required facts, acceptable uncertainty, prohibited disclosures, source authority, and freshness requirements. A short answer may be appropriate; a missing eligibility condition may be consequential. Write grading notes that distinguish those outcomes rather than rewarding length or confident wording.
For each evaluation case, store the question, trusted access context, corpus snapshot, reference answer, required claims, relevant evidence identifiers, and expected answer or abstention behavior. Add language, question type, and risk labels so aggregate results can be inspected by meaningful slices.
Separate development cases from the holdout
Build cases from permitted, redacted user questions and domain expert input. Synthetic questions can expand coverage, but reviewers should confirm that they reflect plausible needs. Group paraphrases and questions about the same source together when splitting data to reduce leakage.
Version the evaluation dataset and ground truth. Keep a holdout separate from prompt tuning and judge calibration. Repeatedly optimizing against its failures turns it into development data; retire or refresh it deliberately.
Build a repeatable RAG evaluation workflow
Capture the evidence path
Record document versions, chunk identifiers, rank, retrieval scores, applied filters, reranking results, and the exact retrieved context supplied to generation. Also record the embedding model, chunk size, overlap, index configuration, prompt version, generator, evaluator, and decoding settings.
Keep sensitive content in a restricted evidence store rather than general telemetry. Without the actual context, a reviewer cannot determine whether a failure came from retrieval, context assembly, or generation.
- Freeze the corpus snapshot and evaluation cases.
- Run retrieval with the case's trusted permissions.
- Score ranked evidence independently of the generated answer.
- Generate using the captured context and record citations.
- Check answer claims, required coverage, and abstention behavior.
- Review failures and compare the candidate with the baseline.
Test whether the checks detect defects
Use negative controls. In a hypothetical control, replace a supporting chunk with an unrelated passage while preserving its citation identifier. Claim support checks should fail even if the citation resolves. In another hypothetical control, remove a required exception from an otherwise correct answer. The completeness check should detect the omission.
Measure retrieval quality with explicit denominators
Label relevance at a declared retrieval depth
Use question specific relevance labels for document or chunk identifiers. Define whether relevance means topical similarity, partial support, or sufficient evidence for a required claim. Choose the evaluation unit before comparing chunking strategies; duplicate chunks should not masquerade as independent evidence.
Microsoft Foundry's RAG evaluators distinguish retrieval evaluation from response evaluation. Its document retrieval evaluator uses relevance judgments and exposes ranking metrics and missing label diagnostics. Those diagnostics matter when comparing new retrieval configurations.
For a binary rubric, precision@k divides relevant results in the first k positions by k. If fewer than k results are returned, document whether empty positions count as misses or the denominator is the returned count; name that convention consistently. Recall@k divides retrieved relevant items by all known relevant items in the declared reference pool. Graded labels support ranking measures such as NDCG@k.
Unknown labels are not negative labels
In a hypothetical example, three of five retrieved chunks are labeled relevant and two are unjudged. Precision is not established merely by reporting three divided by five. Report judgment coverage, inspect unjudged results, and state any conservative scoring assumption. A recall denominator built from incomplete labels estimates coverage of known evidence, not every relevant passage in the knowledge base.
Distinguish context precision from context recall
Ragas context precision offers different implementations. The LLM based context precision variants assess whether useful chunks rank ahead of irrelevant ones. In contrast, IDBasedContextPrecision divides matching retrieved IDs by retrieved IDs and ignores ordering. Do not use that fraction alone to diagnose reranking regressions. Pin the implementation and whether usefulness is judged against a reference answer or another input.
Ragas context recall includes a claim based formulation that checks how much of the reference answer the retrieved context supports. Missing reference claims can inflate apparent coverage. Inspect references before treating either metric as a release signal.
Separate faithfulness accuracy relevance and completeness
Ask four different questions
Faithfulness asks whether answer claims follow from the supplied evidence. Ragas faithfulness evaluates support for individual response claims in the retrieved context. A faithful answer can still be wrong when its source is obsolete or incorrect.
Accuracy asks whether the answer agrees with the accepted facts for the question and its effective date. Evaluate important numbers, conditions, and exceptions against reviewed ground truth, allowing equivalent wording. Reviewers should establish which document governs the decision and whether its rules were effective on the date being evaluated.
For answer relevance, inspect whether the response helps resolve the specific task in the question or drifts into adjacent material. Ragas response relevancy measures alignment with the question rather than factual correctness. A response can earn a strong relevance grade while misstating an important eligibility rule, so factual verification remains a separate check.
For completeness, compare the delivered answer with the obligations recorded in the case, including prerequisites, exceptions, and the next action. Use the case's claim checklist to distinguish omitted facts from unsupported additions. Record these RAG evaluation metrics separately so a high average cannot conceal a critical missing condition.
Correct facts can still produce an unusable answer
In a hypothetical policy question, a response states the correct deadline but omits a mandatory approval step. It may be faithful, accurate in what it says, and relevant, yet incomplete. Route the omission to answer coverage review rather than automatically changing the retriever.
Evaluate citations at the claim level
Verify entailment and source identity
A working URL establishes reachability, not citation entailment. For every material claim, identify the cited passage and determine whether that passage supports the claim's scope, numbers, dates, and qualifications. Evidence for one sentence should not silently justify an entire paragraph.
Preserve the document version and passage used during generation so later edits do not obscure the evaluation. Separately check whether the user can open the source and whether the displayed citation maps to the retrieved identifier.
In a hypothetical citation failure, an answer promises a refund for every subscription while the cited passage applies only to annual plans. The link works, but the evidence does not entail the broader claim. Mark the support failure and the missing qualification independently.
Measure support among cited claims and citation coverage among claims requiring evidence. Review uncited material claims rather than allowing fewer citations to improve the support score artificially.
Include missing conflicting and stale evidence
Make abstention an evaluated outcome
Create separate cases for absent evidence, an ambiguous question, retrieval failure, and insufficient permissions. Define whether the system should clarify, explain uncertainty, or report a service failure. A relevant sounding answer is not success when the case requires abstention.
In a hypothetical missing evidence case, the corpus contains no approved cancellation policy. The expected response acknowledges that limitation and offers an appropriate next step without inventing terms.
Define authority and freshness rules
In a hypothetical conflict case, two policy documents disagree and neither has established precedence. Require the answer to surface the conflict or escalate it. In a hypothetical stale document case, an archived policy remains searchable after a replacement takes effect. Evaluate source selection against the effective date, not just semantic similarity.
Also test ingestion lag, deletion, broken provenance, and context truncation as distinct failure categories. A fresh source that never reaches the prompt needs a different repair from a generator ignoring correctly supplied evidence.
Protect tenant boundaries before generation and caching
Apply trusted access controls to retrieval
The OWASP RAG security guidance addresses inherited chunk permissions, tenant isolation, permission scoped caches, and invalidation. Enforce access control lists using authenticated server context before evidence enters model input or a shared cache. User supplied tenant fields and model instructions cannot establish authorization.
Scope cache lookup and writes to current authorized access. Invalidate affected entries when documents or permissions change. Access failures should fail closed.
Inspect more than the final answer
In a hypothetical isolation test, tenants A and B ask identical questions about different private policies. Inspect returned chunks, model input, citation payloads, and cache hits for leakage. Repeat after revoking access and with concurrent requests. A harmless final answer does not excuse unauthorized evidence reaching the model.
Treat retrieved instructions as untrusted content. Test poisoned passages with synthetic markers and verify that they cannot override the answer contract. Keep authorization checks independent of model behavior.
Calibrate judges with human review
Validate the evaluator before trusting its scores
Ragas judge alignment guidance compares automated decisions with expert judgments. Build a separate calibration set containing clear failures, valid paraphrases, subtle omissions, and acceptable abstentions. Have reviewers label independently, resolve disagreements, and refine ambiguous rubrics.
Measure false passes and false failures by risk category, not only overall agreement. Pin judge model, prompt, inputs, and metric version. Recalibrate after evaluator changes; otherwise a score shift may reflect the judge rather than the application.
Blind reviewers to candidate identity where practical. Send uncertain grades, security failures, and consequential claims to authorized humans. Use a sampled human audit even when automated checks pass, and retain concise reasons with evidence references for adjudication.
Compare candidates alongside uncertainty cost and latency
Use paired comparisons on the same cases
Run baseline and candidate against identical questions, permissions, and corpus snapshots. Compare per case differences and inspect wins, regressions, and ties. Change one major component at a time where practical so failures remain attributable.
Estimate uncertainty with paired resampling or an appropriate paired test. Keep related questions grouped, and repeat generation when randomness materially affects conclusions. Assess the interval for the paired difference against zero and a predefined practical improvement threshold. Overlap between separate baseline and candidate intervals does not establish whether the paired change is meaningful. Small slices warrant caution about precision and generalization.
Set release criteria for your product
Measure retrieval, reranking, generation, and total latency, including tail latency. Record tokens, service costs, retries, and cache state; keep evaluation judge expense separate from serving cost. Report quality by question and risk category alongside operational budgets.
There is no universal passing threshold. Establish criteria from user impact, reviewed failures, and baseline performance. Critical access violations require their own release decision rather than being averaged away by successful answers.
Review the visible answer experience with people
Use the AI agent testing guide when the product also takes actions. For failure states and retries around retrieval services, consult the API rate limit testing guide.
Create a BetaRoast feedback round with synthetic documents and permitted accounts. Ask people to interpret an answer, open its citations, notice uncertainty, and explain what they would do next. That human feedback can reveal confusing presentation and misplaced confidence. Engineers and authorized domain reviewers remain responsible for retrieval scoring, security checks, and factual validation.
Frequently asked questions
What is RAG evaluation?
It assesses whether a retrieval augmented generation system finds suitable evidence and produces supported, correct, relevant, and complete answers. It also checks behavior when evidence is unavailable, contradictory, outdated, or outside the user's permissions.
How do you evaluate a RAG system?
Freeze the corpus and test cases, inspect retrieval independently, then grade answers and citations against reviewed expectations. Compare a candidate with its baseline on the same cases, review consequential failures, and assess cost and latency alongside quality.
Which metrics matter for RAG evaluation?
Use retrieval precision and recall at a declared depth, ranking quality, context precision and recall, faithfulness, accuracy, answer relevance, completeness, and citation support. Report label coverage and denominators. Select release criteria according to the application's risks rather than a universal score.
How do you create a RAG evaluation dataset?
Collect permitted representative questions, add reviewed edge cases, and annotate required claims, reference evidence, permissions, and expected abstentions. Version documents and labels together. Separate development and calibration data from a holdout, grouping related questions to reduce leakage.
Evaluate retrieval and generation separately, preserve explicit metric denominators, and review claims against authorized current evidence. Use a versioned holdout, calibrated judges, human review, and paired comparisons to assess quality alongside cost and latency. BetaRoast can support human feedback on the visible answer experience; engineering evaluation remains a separate responsibility.