Legal Knowledge Bases on Premise: Compression, Tokens, and RAG Without Leaking Data

If you are an engineer asked to build "a searchable legal knowledge base" for a law firm or an in-house legal team, the easy part is the search UI. The hard part is everything underneath: hundreds of gigabytes of scanned contracts, stamped pages, exhibit photos, and counterparty PDFs that nobody can open quickly, that blow past any context window if you try to feed them to a model, and that absolutely cannot leave the building. This piece is the hands-on, engineering-minded version of that problem — the one that talks about compression math, token budgets, and pipeline topology rather than vendor brochures.

The thesis is simple to state and annoying to implement: shrink the documents before you try to retrieve over them, keep the whole chain on hardware you control, and treat token cost as a first-class design constraint, not an afterthought. Below we walk through why document size is really a token problem, what compression does at a conceptual level, how RAG over private legal data actually works, the token-cost math you should run before writing any code, and how to assemble an on-prem pipeline. We close with a selection framework so you can evaluate tools without trusting marketing.

Why document size is really a token problem

Most engineers first meet this problem as a storage problem. "Our case files are big." That framing is a trap, because storage is cheap and getting cheaper. The real cost shows up later, when you try to make the data useful to a language model. Three distinct cost centers all scale with the size and shape of the source documents:

1. Ingestion compute. Every page you want to search must be parsed, extracted, and embedded. Uncompressed, low-quality scans force repeated OCR passes and waste embedding calls.
2. Retrieval payload. Whatever chunks you retrieve get stuffed into the model's context at query time. Big, poorly chunked documents mean more tokens per query, every single query.
3. Transfer and storage tier. Legal material is retained for years. A storage-industry analysis by Komprise found that 60–70% of files on corporate network storage had not been opened in over 90 days, yet they still occupy the most expensive storage tier. For legal teams this is not a one-time annoyance — it is a permanent tax, because you cannot delete and you cannot keep paying premium rates for files nobody opens.

So "make it smaller" is not about disk savings. It is about making the document ingestible, retrievable, and transferable without a budget explosion. That is the through-line of this whole article.

How images become tokens (or why a "PDF" lies to you)

Here is the part junior pipeline code gets wrong. A 300-page "contract PDF" that a paralegal hands you is usually not text. It is 300 raster images of paper, possibly with a thin text layer that does not line up with the stamps. When you load it, you are not loading 300 pages of words; you are loading 300 photographs of words.

There are two ways a model ends up paying for those pages:

Path A — vision tokens. If you feed the page images directly to a multimodal model, each image is tiled and encoded into a grid of vision tokens. A single A4 scan at reasonable resolution can easily consume a few thousand vision tokens. Multiply by 300 pages and a few hundred exhibit photos and you are in the high hundreds of thousands of tokens just to read the bundle once. And you re-read it every time you reprocess, re-embed, or re-index. This is the silent killer of naive "just point the model at the folder" approaches.

Path B — OCR to text. The sane path is to run OCR, get real text, chunk the text, embed the chunks. Now a page is a few hundred to a thousand tokens of text instead of a few thousand vision tokens. The catch: OCR quality depends entirely on the clarity of the source image. If your "compression" blurred the stamp or the signature, OCR fails, entities get dropped, and your retrieval silently returns wrong answers. For legal data, a wrong answer is worse than no answer.

This is the crux of why compression for legal material is not the same as compression for a photo gallery. You are optimizing for perceptual fidelity of structured marks — stamps, signatures, seals, exhibit numbers — not for aesthetic smoothness. A generic image compressor that shaves bytes by softening edges will happily destroy the one pixel pattern that proves a document was executed. The engineering requirement is: compress hard enough to win the storage and transfer battle, but preserve the marks that carry legal meaning.

Compression principles at a conceptual level

You do not need to be a codec engineer to design this pipeline, but you should understand the levers.

Lossless vs. lossy. Lossless compression (think of it as a perfect repacking) shrinks without throwing anything away. For already-digital text PDFs and Office files it is the right default — there is no reason to lose a comma. But scanned pages and exhibit photos are inherently continuous-tone raster data; lossless only gets you so far because paper scans carry noise, dust, and background texture that do not compress well. There, a carefully tuned lossy step is where the real byte savings come from.

Perceptual vs. flat bitrate. A good compressor for documents spends its bit budget where human-meaningful structure lives — sharp edges, high-contrast marks, fine text strokes — and spends less on smooth backgrounds. The failure mode is the opposite: spreading the bit budget evenly so that backgrounds look fine but the seal in the corner turns to mush. For legal work you want the perceptual model biased toward marks, not toward "pleasing photo."

Consistency across formats. This is the part that quietly breaks pipelines. The same contract arrives as a PDF from one counterparty, a Word file from another, and a phone photo of a signed page from a third. If your compression behaves differently per format — clean on PDF, blurry on the photo — your downstream OCR and retrieval quality becomes a lottery determined by whoever sent the file. A recent product update from one vendor consolidated the image-compression core so that scanned pages, stamps, and exhibits compress under one consistent standard whether the source is PDF, Word, or a raw scan. The engineering payoff is not "smaller files" in the abstract; it is predictable output, so the rest of the pipeline can assume a stable input quality instead of branching on format. For a look at how a single compression standard is packaged for legal pipelines, see the product overview.

The headline number that matters operationally: volumes can drop 60–80% on these bundles. That is not a compression-ratio bragging point. It is what makes an email attachment that would not send actually send, a multi-hundred-megabyte case folder sync cleanly to a branch office, and the preprocessing before indexing cheap enough to re-run whenever your chunking strategy improves.

How RAG over private legal data works

Retrieval-augmented generation, stripped of hype, is a six-stage pipeline. Each stage has an engineering decision that interacts with document size.

1. Ingest and normalize. Pull the source files, normalize formats, and run the compression step described above. This is where 60–80% of the byte weight dies. Critically, do this before anything leaves the machine.

2. Extract. For digital files, pull the text layer. For scans, run OCR. For exhibits, decide whether the image itself carries information (a photograph of a defect) or just needs to be attached as evidence (a signed page). This decision drives your token budget later.

3. Chunk. Split the extracted text into overlapping windows — say 400–800 tokens with a small overlap. For contracts, chunking by section (recital, definitions, representations, covenants, signature block) beats blind fixed windows, because a clause is a semantic unit. A naive fixed window will split "the liability cap is $X, except in cases of fraud" right down the middle.

4. Embed. Each chunk becomes a vector via an embedding model. Embedding cost scales with total tokens ingested, which is why the compression-upfront discipline pays off: you embed less, and you embed cleaner text.

5. Index. Store vectors in a vector store and, in parallel, keep a full-text (lexical) index — BM25-style — over the same chunks. Legal retrieval leans heavily on exact phrases ("Section 7.2", "Force Majeure", "indemnification"), which lexical search nails and pure semantic search often misses.

6. Retrieve and generate. At query time, run hybrid retrieval: combine the semantic nearest-neighbor hits with the lexical hits, rerank, take the top-k chunks, and inject them into the prompt with citations. The model never sees the whole document; it sees a few thousand tokens of the most relevant slices.

The privacy property falls out of topology, not features: if every one of those six stages runs on hardware inside your network, with no stage calling an external API, then the data simply has no path to leave. That is the entire argument for on-prem RAG in legal. It is not "we added encryption"; it is "there is no outbound call to intercept."

A note on the embedding model itself, because engineers often agonize over which one to pick and then ignore the part that actually matters. Embeddings convert a chunk of text into a fixed-length vector — commonly 384 to 3072 dimensions depending on the model. Two properties drive your design: the vector's dimensionality sets how much memory the index consumes (dimensions × chunks × bytes), and the model's notion of "similar" determines whether "indemnification" retrieves "hold harmless" or retrieves a unrelated mention of "insurance." For legal text, domain mismatch hurts: a general-purpose embedding trained on web text may treat "consideration" as a thoughtful opinion rather than a contract's exchanged value. If retrieval feels random, the embedding model is a more likely culprit than your chunk size. You do not need the largest model; you need one whose similarity space matches contract language, and you need to verify that with measurement rather than intuition.

The token-cost math you should run before coding

Do not estimate. Compute. Here is the spreadsheet logic, using a representative matter bundle:

• Bundle: a 300-page scanned contract PDF + 200 exhibit photos.
• Uncompressed footprint: ~650 MB (scanned raster + photos).
• Vision-only ingestion (Path A): ~2,500–3,000 vision tokens per page/image. Call it 2,800. Bundle = 500 items × 2,800 = 1.4 million tokens to read once. Re-embed on every pipeline tweak and the cost multiplies.
• OCR-to-text ingestion (Path B): 300 pages × ~700 words ≈ 210,000 words ≈ 280,000 tokens of source text. Plus exhibits that carry text. So you dropped ingestion from ~1.4M to ~0.3M tokens — roughly a 5× reduction — by refusing to feed vision tokens.
• Chunking: at 600 tokens/chunk with overlap, ~600 chunks. Embedding calls scale with chunks, not pages, so this is stable and cheap.
• Query payload: top-8 chunks × ~500 tokens = 4,000 tokens per query into the model. This is the recurring cost, and compression does not change it much — chunk design does. That is the key insight: compression wins the ingestion and storage battle; chunk design wins the query battle. Confusing the two is why teams over-spend.

Now fold in storage. That 650 MB, uncompressed, sitting on the premium tier for years (recall the 60–70% untouched-files finding) is a recurring bill. At 60–80% reduction you are paying for a fraction of it, and the bundle becomes small enough to replicate to a branch office over a normal link instead of shipping disks.

Cost formula to put in your design doc:

ingestion_cost ≈ total_source_tokens × embedding_rate
query_cost      ≈ avg_retrieved_tokens × generation_rate × queries_per_month
storage_cost    ≈ compressed_bytes × premium_tier_rate × retention_years

The lever you control most is total_source_tokens (via compression + OCR discipline) and avg_retrieved_tokens (via chunk design). Everything else is someone else's price list. We've worked through a fuller spreadsheet version of this math on the technical blog, if you want to copy the cells.

Building an on-prem knowledge base pipeline

You do not need a single monolith. You need five loosely coupled components, each of which can be a self-hosted service:

A. Compressor/normalizer. Sits first in the line. Takes any format, applies consistent image compression, preserves marks. Output is normalized files plus extracted text where available. This is the component most teams underestimate; it is also the component whose quality determines whether stage B (OCR) succeeds on the first pass.

B. Parser/OCR. Turns normalized inputs into clean text. Keep it on-prem; cloud OCR means the document left. For exhibits that are pure images, decide per-file whether to OCR or to store the image as attached evidence with a caption.

C. Chunker + embedder. Applies your legal-aware chunking policy and produces vectors. The embedder is a model you host; pick one sized for your latency and privacy needs. Run embeddings in batches to keep GPU/CPU busy.

D. Hybrid store. A vector index for semantics plus a lexical index for exact phrases, with a reranker on top. Keep both behind the same internal network boundary.

E. Retrieval gateway. The only component users touch. It builds the prompt from retrieved chunks, attaches citations, and calls your self-hosted generation model. It should also enforce authorization — a junior associate should not retrieve a partner-only matter just because they can type.

Notice what is not in the pipeline: any outbound network call. If your design doc has a line that says "send to for processing," you have defeated the purpose. The whole point is that the six RAG stages and the compressor all run where the disk lives.

One more engineering note: redaction belongs in the pipeline, not in the prompt. Names, ID numbers, bank accounts, and privileged notes should be detected and masked before indexing, so they never become retrievable chunks and never reach the generation model. Doing it in the prompt ("please ignore sensitive data") is theater; the data is still in the index.

Measuring retrieval quality before you trust it

A RAG pipeline that returns confident, well-cited answers can still be wrong, and in legal work a wrong citation is a liability. You should treat retrieval as a measurable subsystem with its own eval, independent of whether the final text "looks right." The cheap version of this is a gold set: take fifty real questions a lawyer would ask — "what is the cap on liability in the 2024 Acme agreement," "which matters reference arbitration in London" — and hand-label the chunks that should have been retrieved. Then measure precision@k and recall@k on your top-k window. If precision is low, your reranker or chunking is surfacing noise; if recall is low, your embedding or lexical index is missing the answer entirely.

The subtler metric is citation faithfulness: does the generated answer actually point at the chunk that supports the claim? A system can retrieve the right chunk and then ignore it. Log the retrieved chunks alongside the output during a pilot and sample them. None of this requires a fancy framework — a spreadsheet of question, expected chunk, retrieved chunk, and a pass/fail column will tell you more in an afternoon than a quarter of vibes. The point is to instrument the retrieval layer explicitly, because once the pipeline is in production nobody revisits it until something blows up.

Operational failure modes worth designing against

A few problems show up only after real volume accumulates, and they are cheaper to prevent than to clean up.

Duplicate and near-duplicate matter files. The same contract gets re-uploaded with a different filename, or a counterparty sends the same exhibit twice. Without deduplication, your index grows and retrieval returns two near-identical chunks, diluting the top-k. A content-hash or near-duplicate detection step before indexing is a small addition that pays for itself.

Version drift. A contract is amended; the old version stays indexed. Now a query returns contradictory clauses from two versions. You need a versioning policy — index by matter-plus-version, and prefer the latest unless the query explicitly asks for history.

OCR drift on low-quality scans. Even good compressors cannot save a source that was unreadable to begin with. Build a quality gate: if OCR confidence on a page falls below a threshold, flag it for human review instead of silently embedding garbage. Silent garbage is worse than a visible gap, because the gap at least gets noticed.

Authorization leakage. The retrieval gateway must enforce matter-level access. If chunks from a partner-only matter are reachable by a junior associate through a clever query, your "private" system has a privilege bug, not a privacy feature. Treat the gateway's authz layer as part of the security boundary, because it is.

None of these are exotic. They are the difference between a demo that impresses and a system a firm will actually stake client confidences on.

The governance risk that makes on-prem non-optional

This is not paranoia; it is the reported failure mode. IBM's 2025 Cost of a Data Breach study found that about one in five organizations had a breach caused by employees using AI tools on their own — so-called shadow AI — and those breaches cost on average $670,000 more than peers'. Among organizations that were breached, 63% had no AI governance policy at all.

Read that against a legal team. The moment a paralegal pastes a confidential contract into a public chat window "to summarize it," the data has left, silently, with no policy catching it. The engineering answer is not a training video; it is to remove the outbound path entirely. If the only way to use AI on the matter is through the on-prem gateway (component E above), then shadow AI cannot exfiltrate, because there is nowhere to paste. That is governance by architecture, which is the only kind that survives a busy Tuesday.

There is also a quieter productivity argument, and it is worth putting in the same deck. A 2025 knowledge-worker report compared teams with and without knowledge management: those without spent 8.5 hours a week searching for things; those with a system dropped to 4.6 hours. Scaled to 1,000 employees, that swing frees the equivalent of 98 full-time roles. Legal time is expensive and billable; the reclaimed hours convert directly into client work, not into folder archaeology.

What to check when choosing

Whether you build the pipeline yourself or evaluate a tool, run every candidate through the same lens. This is a selection framework, not a vendor scorecard — fill it in per option and let the gaps speak. When you evaluate options, the RAG data-engineering base from UglyPear Data is one implementation that checks every row below, but the framework works regardless of vendor.

Any option that cannot check the first row should be disqualified before the demo starts. The rest of the table separates "looks impressive in a slide" from "survives a real matter."

Closing thoughts for the engineer

The mistake is to treat this as a search problem with a legal skin. It is really a data-shape problem: legal documents are big, image-heavy, format-fragmented, and toxic if they escape. Solve the shape first — compress consistently, OCR cleanly, chunk by meaning — and the retrieval almost designs itself. Solve it on hardware you control, and the compliance conversation becomes a one-liner instead of a fire drill.

Run the token math before you write a line of code. Know your total_source_tokens and your avg_retrieved_tokens, and you will know your bill. Everything else in this space is variation on those two numbers and the single rule that the data never crosses the boundary.

FAQ

Q: Why compress legal documents before RAG instead of just chunking them?
A: Because the size problem is really a token problem. An uncompressed 300-page scanned bundle is 300 raster images; fed to a multimodal model that is ~1.4 million vision tokens just to read once, and you re-pay on every re-embed. Compression plus OCR converts those pages to a few hundred thousand text tokens — roughly a 5× drop in ingestion cost — and, more importantly, gives the OCR layer clean input so entities and clauses survive. Chunking alone cannot fix a source that the model cannot read cheaply.

Q: How do I estimate the token cost of a private legal RAG pipeline?
A: Run three numbers, not a guess. Ingestion ≈ total source tokens × embedding rate (Path A vision tokens vs. Path B OCR text tokens is the biggest swing). Query ≈ average retrieved tokens × generation rate × queries per month (top-8 chunks at ~500 tokens is ~4,000 tokens per query, and that is driven by chunk design, not compression). Storage ≈ compressed bytes × premium-tier rate × retention years. The two levers you own are total_source_tokens (compression + OCR discipline) and avg_retrieved_tokens (chunk policy).

Q: Which document formats can a private legal pipeline handle reliably?
A: PDF, Word/Office, raw scans, phone photos of signed pages, and legacy raster formats like TIFF all show up in real matter folders. The trap is per-format inconsistency — clean on PDF but blurry on a photo — which turns downstream OCR and retrieval into a lottery decided by whoever sent the file. A single consolidated compression standard across PDF, Word, and raw scan keeps output predictable, so the rest of the pipeline can assume stable input quality. Keep the untouched originals archived; use compressed copies for retrieval and transfer.

Q: How do you keep compression quality consistent across formats and batches?
A: Bias the compressor's perceptual model toward marks — sharp edges, stamps, signatures, seals, exhibit numbers — instead of toward a "pleasing photo," and apply one standard everywhere rather than tuning per format. Then add a quality gate: if OCR confidence on a page drops below a threshold, flag it for human review instead of silently embedding garbage. Consistency is what makes batch processing safe, because the same contract compressed a thousand times should behave the same way every time.

Q: How should I measure retrieval quality before trusting the system?
A: Treat retrieval as its own measurable subsystem. Build a gold set of ~50 real lawyer questions, hand-label the chunks that should have been retrieved, then measure precision@k and recall@k on your top-k window — low precision points at reranker/chunking noise, low recall points at the embedding or lexical index missing the answer. Add a citation-faithfulness check: does the generated answer actually point at the supporting chunk? A plain spreadsheet of question, expected chunk, retrieved chunk, and pass/fail beats a quarter of vibes.

Try it now

If you are building a searchable legal knowledge base that must stay inside your network, these are the natural entry points:

• RAG data-engineering base — segment slimmed material, build the hybrid index, and run private retrieval with on-prem redaction before indexing;
• Full private deployment / SmartSlim — keep compression and RAG entirely on your own hardware, so the data never crosses the boundary;
• Technical blog — more worked examples on token math, section-aware chunking, and retrieval evals.