Disk Embedding Cache: Zero Re-Embed Cost

The short version: the hidden cost of eval loops is re-embedding. DiskEmbeddingCache stores computed vectors as a binary cache keyed by content hash, with model-consistency checks, so identical content is never embedded twice—iteration cost approaches zero.

1. Why re-embedding is expensive

RAG evaluation is a loop: change a rule, rerun, read metrics. The trap is in the rerun—chunking rules changed, but the corpus itself did not. If every round re-sends every chunk to the embedding model, most of the compute goes to redundant work. Embedding is one of the most expensive steps in the pipeline; multiplied by chunk count and rerun count, the bill gets out of control fast, and at scale the waste is fatal.

The practical damage shows up in behavior: the gate is supposed to run on every change, but when a full eval takes too long, teams start batching changes together. Evaluation exists to find problems, yet it becomes the bottleneck itself—something many teams only discover after their corpus passes ten thousand chunks. A cost problem quietly becomes a process problem, and the feedback loop stops closing.

2. How the disk cache works

The fix is simple in principle: compute something once and keep the result for reuse. Key design decides the cache’s lifetime—a key at the content level survives any change to chunking rules and any number of reruns, while a key at the chunk level dies with one re-chunk.

DiskEmbeddingCache keys by content hash and stores embedding results as binary files. On the next encounter with identical content it reads from disk and skips the model forward pass. Embeddings are deterministic—same content, same model, same output—so a cache hit is exactly equivalent to recomputing, with no loss of correctness.

This shares the spirit of the knowledge base's SHA-256 idempotency: unchanged files skip reprocessing on import, and an idempotent re-run finishes in 0.2 seconds. The cache extends the same discipline to the vector stage. From document to chunk to vector, everything is computed exactly once. Resumable runs manage batches; the cache manages vectors—together they make large-corpus iteration safe to do freely.

3. Model-consistency check

A cache is not valid forever. Once the embedding model or version changes, old and new vectors live in different semantic spaces, and mixing them silently degrades retrieval—not a theoretical risk but a certainty, since vectors from different models are simply incomparable. Migrating to, say, Qwen3-Embedding-4B means the old cache must be invalidated and recomputed, not reused.

So the cache runs a model-consistency check: it records which model produced each vector, and on mismatch auto-invalidates the old entries and recomputes. This avoids the trap of running on wrong-version vectors while assuming all is fine—a failure whose symptom is metrics drifting down for no visible reason, and which is miserable to debug because parsing, chunking, and retrieval all look healthy—only the vectors have quietly switched semantic spaces. The check turns that error into an explicit recompute at the first moment, instead of letting it surface slowly inside evaluation.

4. Let the gate run often

By driving re-embedding cost to zero, the cache lets the regression gate run on every change and every night. Running nightly also means the metric curve stays continuous, so regressions are visible at a glance. Together with idempotent re-runs (0.2 seconds to skip processed content), it forms the basis of low-cost, high-frequency verification: feedback arrives the same day, and problems are caught before merge—see Knowledge Base Lifecycle.

The cache serves the continuous-improvement loop. Metric definitions in RAG Evaluation Metrics; gate design in Regression Gates; hash idempotency and resumable runs in Knowledge Base Lifecycle.

Engineering Checklist: Four Details of Disk Embedding Cache

• Cache keys carry model-consistency checks: switching embedding models (such as upgrading Qwen3-Embedding-4B) must invalidate stale caches; binary caching plus model-consistency verification prevents old vectors from passing as new-model outputs.
• Zero repeated embedding during evaluation iterations: DiskEmbeddingCache lets regression tests and controlled sweeps re-run on the same corpus without re-calling the embedding model — the cost precondition for a high-frequency evaluation loop.
• Pair with incremental diff: the knowledge base re-embeds only changed chunks (SHA-256 idempotency plus stable chunk IDs), so cache hit rate determines the update window on large corpora.
• Fallback when the cache is incomplete: silently degrade to pure BM25 retrieval so the service stays up while the cache rebuilds.

The amplifier for cache savings is lifecycle governance — without incremental diff, the cache only speeds up full re-runs; with it, update cost scales with the change set alone, see Knowledge Lifecycle Governance. The retrieval chain after a cache hit is covered in Hybrid Retrieval.

FAQ

Q1: Why does re-embedding kill evaluation?

Evaluation loops change rules, rerun, and read metrics. With an unchanged corpus, re-embedding every round wastes compute on redundant work; at scale it is fatal, the gate slows down, and the feedback loop stops closing.

Q2: How does the disk cache avoid repeats?

It keys vectors by content hash and stores them as binary files; identical content reads from disk and skips the model forward pass. Embeddings are deterministic, so a hit equals recomputing—extending the knowledge base's SHA-256 idempotency.

Q3: What happens when the embedding model changes?

The cache records which model produced each vector. On a model or version change it auto-invalidates old entries and recomputes, preventing silent degradation from mixing semantic spaces.

Q4: How does the cache relate to the regression gate?

By driving re-embedding cost to zero, the cache lets the gate run on every change and nightly. Combined with idempotent re-runs, it supports low-cost, high-frequency verification, with metric feedback the same day a change lands.