RAG Launch Checklist: 30 Items

The short version: check every item before an enterprise RAG launch; one miss can become a production incident. The 30 items below form six groups—treat them as a release gate and tick each before go-live.

1. Data quality (1–5)

Data quality is the first gate because nothing downstream repairs it: a paragraph lost at parse time never comes back. Industry experience puts 80% of RAG failures on data handling—these five items are the net.

• Parse completeness meets bar; no whole-file mojibake or dropped pages
• Scans go through OCR + layout; table structure restored correctly
• Headers, footers, watermarks cleaned; no structural noise
• Legacy formats (OLE2 .doc / CNINFO .bin / HTML) covered by differential parsing
• parse_quality three-layer scores within acceptable thresholds

Scans and legacy formats are the danger zone: one 443-page OFD document parses end to end in about 124 seconds, scoring 0.924 on parse_quality. Baseline a sample batch first, then commit to bulk ingestion.

2. Retrieval and chunking (6–10)

Chunking and retrieval decide whether questions find answers. Fixed-size chunks tear semantics apart; vector-only retrieval misses exact model numbers and IDs. Plug both holes.

• Chunking uses the four type-aware presets, not brute fixed-size
• Parent-child chunks on: child for retrieval, parent for generation
• Four-level chunk enrichment (breadcrumb/keywords/summary/HyDE) added
• Hybrid retrieval BM25 + vector + RRF fusion enabled
• CJK bigram tokenization used for Chinese BM25 branch

Run a retrieval smoke test on real business questions; in one anonymized case, ten-question smoke tests on two tickers both hit 10/10. Get the smoke green before go-live.

3. Security and permission (11–15)

Security items fail silently—late desensitization and post-retrieval filtering still return answers, just with content that should never appear. Confirm the mechanisms, not the surface behavior.

• Desensitization done before chunking and embedding
• 11-class sensitive-target detection enabled and handled
• Permission tagging three-level inheritance, highest secrecy merge
• Permission filter pre-retrieval, not post-retrieval
• Data stays in perimeter; inference fully local ONNX

Permission filtering must run pre-retrieval: post-filtering wastes Top-K slots and leaks that sensitive documents exist. Detection and secrecy merging act before embedding—the vectors themselves are a leak surface.

4. Evaluation loop (16–20)

Evaluation is a continuous gate: rule changes re-run it automatically, so quality cannot quietly regress.

• Corpus metrics (parse/table/OCR/desensitization/permission) have baseline
• Retrieval HitRate@K / MRR / NDCG@K measured
• Generation Faithfulness / AnswerRelevancy meet target
• Regression gate wired; changes auto-rerun evaluation
• Disk embedding cache on; iteration cost controlled

Read layers separately: corpus metrics find data problems, HitRate@K and MRR find retrieval problems, Faithfulness finds hallucination. One anonymized case filtered about 95.5% of irrelevant candidates before Top-K.

5. Deployment form (21–25)

Four groups green, one hurdle left: a mismatched delivery form lets ops drag a good pipeline down.

• Delivery form (CLI / HTTP / FFI) matches scenario
• HTTP SSE progress push and Prometheus monitoring ready
• FFI interfaces catch_unwind-guarded, no crash leak
• File-based storage; backup and migration operable
• Multi-tenant auth, quota, audit log configured

Before acceptance, run the evidence: 61 unit tests, 12/12 HTTP API tests, 28/28 SDK C API tests, and six-format ingestion E2E passing incremental audit. Numbers off—do not sign.

6. Ops and governance (26–30)

Launch is not the end; governance decides whether the knowledge base is an asset in six months.

• SHA-256 idempotency; re-runs do not redo work
• Incremental diff updates only changed chunks
• Version archive keeps 3 versions, rollback possible
• valid_until governance; obsolete content not cited
• Resume state machine in place; large corpus batchable

Verify idempotency and resumability: re-runs skip finished preprocessing in 0.2 seconds so large corpora batch without rework, and valid_until keeps retired policies uncited.

Wrap-up

Treat the 30 items as a release gate and block anything that fails. Items 1–5 (data quality) and 11–15 (security) trip most often and fail silently—review them first. Supporting capabilities in RAG Data Quality, Evaluation Metrics, and Private Deployment.

The checklist governs process; evaluation governs evidence. With both in place, press the release button.

After Launch: The First 30 Days of Operations

The checklist handles launch; a cadence handles sustainability. For the first 30 days, work week by week:

• Week 1 · Freeze the baseline: run the three-layer evaluation and archive the baseline (retrieval smoke targeting 10/10 hits), switch on thumbs-down collection, and wire up R1–R6 attribution.
• Week 2 · Harvest defects: flow badcases into the evaluation set and let the attribution distribution decide what to fix first — for most teams the first hotspot is chunking or parsing, not the model.
• Week 3 · Put changes on rails: every chunking / cleaning / retrieval parameter change must pass the regression gate; tune via controlled sweeps.
• Week 4 · Make governance routine: version archiving and time governance (valid_until) become scheduled work, and incremental import with resumable runs becomes the default update path.

Two signals tell you the cadence is healthy: evaluations run on every change (DiskEmbeddingCache makes them free of repeated embedding cost), and thumbs-down attribution converges on specific stages rather than vague answers. The gate design is covered in RAG Regression Gate, the 30 static pre-launch checks in the body of this article, and the long-run methodology in RAG Evaluation Metrics.

FAQ

Q1: How are the 30 items grouped?

Six groups of five: data quality (1–5), retrieval and chunking (6–10), security and permission (11–15), evaluation loop (16–20), deployment form (21–25), and ops and governance (26–30). Tick every item before go-live, and block any failure.

Q2: Which items should I check first?

Data quality 1–5 and security 11–15 trip most often, and their failures are silent: lost paragraphs, late desensitization, post-retrieval permission filtering. They surface only after launch, so review these groups before anything else on the list.

Q3: When should the checklist be used?

As a release gate, right before go-live: tick each item and block anything below the bar. It also works as a re-entry check after major changes to chunking, cleaning, or permission rules, so regressions get caught before users do.

Q4: Does the checklist replace evaluation?

No. The checklist is a process gate ensuring everything was done; evaluation is quantitative evidence proving it was done well, with metrics like HitRate@K and Faithfulness. You need both—neither alone justifies pressing the release button.