RAG Evaluation Metrics: Three Layers

The short version: RAG evaluation has three layers—corpus (is the data clean), retrieval (did it find the right blocks), generation (is the answer correct). Each layer has its own metrics; together they form one complete ruler.

1. Why three layers

Watching only generation quality is the most common evaluation mistake. A wrong answer may come from retrieval missing the block, from a broken parse, or from the model ignoring its context. Three causes, three fixes—mix them up and you cannot act. Without layers, every optimization is guesswork: you tweak chunking, answers seem slightly better, but you cannot say which stage helped or what it broke elsewhere.

Layering makes optimization attributable. Corpus metrics locate data problems, retrieval metrics locate recall problems, generation metrics locate model problems. After any change, check which layer moved and you know what to fix next—and whether the fix worked. Evaluation stops being a one-time acceptance test and becomes a running diagnostic.

2. Corpus level: is the data clean

Corpus metrics measure pre-ingest quality: parse completeness, table structure rate, OCR accuracy, desensitization coverage, and permission hit rate. This layer is the foundation. If it is weak, every tuning effort upstream only compensates for data defects. Industry experience puts roughly 80% of RAG quality problems at the door of data handling—corpus metrics are what turn that claim into something you can measure.

A typical case: a financial report parses with low completeness, tables flattened into scattered text, and no amount of retrieval tuning will let the system answer a revenue question from fragments. UGLYPEAR AI scores parses at ingest on three layers—structure, relations, content. A 443-page OFD layout document scored 0.924 overall after the full pipeline, with a perfect structure score. Get the number at ingest, not after users start downvoting.

3. Retrieval level: did it find the right blocks

Retrieval uses the classic IR metrics: HitRate@K (is the answer in the top K), Recall@K (fraction recalled), MRR (reciprocal rank of the first correct result), NDCG@K (relevance weighted by rank position). They reflect vector and hybrid retrieval directly and are the numbers to watch when tuning.

These metrics settle real questions fast. On an A-share research corpus, a retrieval smoke test hit 10/10 on ten questions each for 000001 and 000002; adding time filtering compressed 17,590 candidates to 829, screening out about 95.5% of stale documents. Without retrieval metrics, those conclusions require manually scrolling through results—slow and unreproducible.

4. Generation level: is the answer correct

Generation metrics measure the model output: Faithfulness checks fidelity to the retrieved context and is the main guard against hallucination; AnswerRelevancy checks that the answer stays on topic; ContextPrecision checks whether the context fed to the model was precise. This layer stands on the first two—clean corpus, accurate retrieval—before it means anything.

This is also the quality users actually feel. If the first two layers are healthy but generation is poor, look at prompts or the model. If generation is poor while ContextPrecision is high, the model is not using its context faithfully—chase hallucination first. Attribute to a layer before attributing to a cause; otherwise you keep swapping models and hoping.

5. Make evaluation a loop

Metrics are not one-shot. Every chunking or cleaning rule change should auto-rerun evaluation against the baseline—that is what a regression gate does, see Regression Gates. Evaluation iterations generate heavy repeated embeddings; a disk embedding cache drives that cost to zero so the loop can run often enough to matter, see Disk Embedding Cache.

The evaluation set itself must grow: downvoted answers are attributed, turned into badcases, and flow back as new evaluation cases, while metric trends land on a dashboard. Evaluation becomes a routine that runs through the whole life of the knowledge base.

Evaluation is the core of continuous improvement. For retrieval metrics see Hybrid Retrieval and Cross-Encoder Reranking; for attribution see Regression Gates and Controlled Experiments.

Real-World Scenario: A Reproducible Evaluation Baseline

Metrics only drive optimization when paired with a baseline. The ashare-web approach is worth copying: run retrieval smoke tests on a fixed query set (10 queries each on 000001 and 000002, requiring 10/10 hits), with ten end-to-end questions on 000002 completing in 4.6 seconds as the latency baseline; the youzi system continuously records hit_rate / MRR / precision on its evaluation set. Both systems share one rule: once a baseline exists, every change to chunking, cleaning, or retrieval parameters must argue with numbers.

The cost problem of running evaluations is solved by DiskEmbeddingCache: disk caching with model-consistency checks makes evaluation iterations free of repeated embedding cost, so regression testing becomes something you run on every change rather than occasionally. With metrics in place, the next question is preventing regressions — see RAG Regression Gate; the full case data chain is in Financial Research RAG in Practice.

FAQ

Q1: Why three evaluation layers?

Generation-only evaluation hides upstream issues: a wrong answer may come from retrieval missing the block or from a broken parse. With layers, every metric movement is attributable to a specific stage, so each optimization has a direction instead of being guesswork.

Q2: What retrieval metrics are used?

HitRate@K (is the answer in the top K), Recall@K (fraction of relevant blocks recalled), MRR (reciprocal rank of the first correct result), and NDCG@K (relevance weighted by rank position). Together they reflect vector and hybrid retrieval quality directly.

Q3: What do the three generation metrics check?

Faithfulness checks fidelity to the retrieved context and is the main guard against hallucination; AnswerRelevancy checks that the answer stays on topic; ContextPrecision checks whether the context fed to the model was itself precise. Read them together to locate generation problems.

Q4: What is in corpus-level metrics?

Parse completeness, table structure rate, OCR accuracy, desensitization coverage, and permission hit rate. They measure pre-ingest data quality—the foundation of the knowledge base—so problems should be fixed at this layer instead of being left to retrieval tuning.