Picture pouring every document your company owns — contracts, research reports, transcripts, hundreds of pages of scans — into one "intelligent system." What do you think it does with them? Most people who have lived through this will tell you the honest answer: it either refuses to take them, or it takes them and then answers the wrong question, or the bill quietly climbs.
Komprise keeps finding the same thing in enterprise storage: somewhere between 60 and 70 percent of the data sitting on primary storage has not been touched in over 90 days. McKinsey puts a human cost on the search problem — knowledge workers burn nearly 19 percent of the work week, about an hour and a half a day, just hunting for and assembling information. Asana adds the interruption tax: it takes roughly 23 minutes to recover focus after a disruption, and people switch between around 35 applications every day. Documents that are mishandled are not just eating disk. They are quietly multiplying the cost of every single interruption.
A newcomer facing this "problem" usually reaches for two things: a tool to shrink the files, or a vector database so the pile becomes searchable. People who have built production systems do not start there.
They ask a different first question. Not "which tool," but "why does this data need to change at all." To cut storage? To make attachments sendable by email? To spend fewer tokens when a model answers questions over it? Or to keep a compliant archive that must survive ten years unaltered? The same pile, aimed at different goals, demands completely different treatment. Someone who knows the terrain nails the constraints down before touching anything, instead of treating "compress" or "add a vector index" as the default reflex.
Once the constraints are fixed, the second move is to split the pile by what it actually is. A signed contract PDF with a red stamp, a 4K engineering scan, and a 200-page regulatory report are three different animals. The contract needs structure and the stamp preserved to the letter. The scan is full of redundant pixels. The report carries repeated layout patterns. Run one "default compression" across all three and you usually get the worst of it: the things that should stay sharp go soft, the things that should shrink refuse to. This is exactly where the amateur and the professional diverge — the amateur watches "how much it shrank," the professional watches "what must never move."
Go one level deeper and the professional returns to the information itself. Why is a file large? A scan is large because it is a dense raster packed with redundancy the eye cannot tell apart but the bytes do. A text document is large because of repeated structure and embedded objects. So whether to compress, and how, should not follow a default preset — it should follow what the file is for. Something kept as a ten-year legal artifact goes lossless and reversible. Something previewed on screen or stuffed into an email goes lossy but keeps what matters. Taking the lossless-or-lossy switch back from the default is the line between a professional and a tool's factory behavior.
Get the classification wrong and everything downstream breaks. Mark a contract as lossy and the stamp blurs — that is an incident. Force a scan that should be lossy to stay lossless and the size does not move while the whole pipeline slows. So the professional would rather spend ten minutes up front asking "which bucket does this belong in, and who uses it after," than clean up a disaster later. Classification is not a formality in the workflow. It is the foundation everything else rests on.
A small example. A scanned contract lives or dies by its stamp and signing date — blur those and legally you have altered the document. A medical image, compress away the wrong edge and you may have hidden exactly the finding that matters. The amateur sees "sixty percent smaller" and is pleased. The professional asks "is there anything in that sixty percent that must not move." Before compressing, the professional asks who gets hurt if this thing "changes." The answer decides whether and how to compress at all.
There is another layer newcomers miss: the same document often lives in several systems with inconsistent versions. The professional reconciles duplicates and versions before talking about compression or search, because indexing dirty data guarantees poor retrieval no matter what comes after. And the cost has to be counted as a whole: embeddings computed once, reranking run once, the model reading it once — where do tokens buy the most value, do you pull extra passages and let the model choose or pre-rank and pull fewer? Until that math is clear, "saving" is just a slogan. Especially in retrieval-augmented setups, every passage pulled gets read again by a model, so dirty documents and careless chunking compound on the token bill monthly. Compress first, clean first — often cheaper than buying a bigger model later.
People who have shipped this treat a knowledge base as a running system, not an upload. The documents change, the business changes, what answers correctly today may falter when next month's version arrives. So they leave room for an evaluation set and rollback from day one, treating "build" and "maintain" as one job. This is why so many demos dazzle and then collapse in production: a demo proves the system can return something; production demands it keep answering correctly on real, messy, shifting documents. The person who treats "returns something" as the finish and the person who treats it as the start build completely different systems.
Most people assume compression and retrieval are two separate jobs. The professional does not. They see "tidy up, shrink, chunk, embed, retrieve, answer" as one line, and they trust only the result of the whole line, not any isolated pretty number. And before compression and retrieval there is usually a tidying step — the same material scattered across a dozen folders with chaotic names. The professional decides first where it belongs and on what axis to classify, because even the best retrieval cannot save a document filed in the wrong place. A cleanly compressed document carries less noise into retrieval, spends fewer tokens, and recalls more reliably — compression here is not a storage freebie, it is a prerequisite step for retrieval quality. And when retrieval keeps answering off-topic, the professional does not blame the model first; they walk back and ask whether a chunk chopped a sentence in half, whether a casual question simply cannot be matched by literal search, whether the relevant passage got truncated.
At the retrieval step, the professional's toolkit looks different from what amateurs imagine. They do not drop the raw question in. Real users ask in slang, in abbreviations, in synonyms, often packing three intents into one sentence — the clean "state the refund policy" from evaluation sets almost never shows up in reality. So they do hybrid retrieval: a keyword path for exact matches, a semantic path for meaning, merged, and the miss rate drops at once. After the first pass, a reranker pushes the truly relevant to the top. For hard questions, they rewrite one casual sentence into several retrieval-friendly sub-queries, search each, then assemble. Permissions and sources are filtered first so nothing confidential leaks out.
None of this is for show. It all points at one thing: making "answers correctly" a measured metric instead of luck. This is precisely the set of things nobody asked for but the professional always does — prepare a batch of labeled question-answer pairs with cited ground truth before writing the first line of code, because without that, "correct" has no definition; bucket the miss rate and the off-topic rate by question type so you can see at a glance which category keeps failing; rerun the old evaluation set after every knowledge-base update to catch the silent decay where "right today, wrong next month."
So why trust any of it? Not because a vendor says "we are accurate," and not because a buyer hopes "good enough." What holds up is what a third party can also reproduce: an independently labeled golden question set, where anyone rerunning your system on the same labels sees your recall@k and faithfulness on the table; blind tests where new and old pipelines answer real questions anonymously reviewed by independent evaluators, with the gap not leaning on "we tested it internally"; and a regression suite where the cases known to be correct must stay correct, with red-green status open to inspection. The same goes for compression itself — can the compressed document open exactly as before, is the stamp there, did the layout shift? Open it and compare to the original. After shrinking, how many tokens actually dropped and does retrieval still hold? Run the downstream task and you have the answer. Correctness does not come from anyone's promise. It comes from actions anyone can verify.
In the end, the whole approach above is not a product feature list. It is a habit: ask why before how; define "correct" before talking about "fast" and "cheap"; and prove it is truly correct with actions anyone can reproduce, not with slogans.
We recently took a small step at the seam between compression and retrieval: we adapted the pre-retrieval compression of documents to work steadily across several mainstream large models, so the same slimmed-down document can be read and used reliably by different models. This is exactly the hump described above — shrink it and it must not become unreadable to the model. It is far from the full engineering depth, but the direction is right: first weld "small" and "readable" into one thing, then layer on evaluation, reranking, and observability.
The table below contrasts the default tool behavior with how someone who has shipped this actually operates, at a few decisive points.
| Decisive point | Default tool behavior | How a seasoned operator behaves |
|---|---|---|
| Starting point | Compress directly / add a vector index directly | Nail down why the data must change |
| Document handling | One default compression for all | Classify by contract / scan / report, then choose |
| Lossless or lossy | Follow the preset | Decide by use: lossless for archive, lossy for preview |
| Compression vs retrieval | Treated as two jobs | One line, trust only the whole-line result |
| Query | Raw sentence search | Hybrid retrieval + rerank + rewrite/decompose |
| Quality proof | Vendor's own claim | Labeled golden set + blind A/B + regression suite |
Try it now
To see an approach that welds "private deployment, data never leaving the network, compression, and semantic retrieval" into one production line:
FAQ
Q: My small company does not have many documents. Is default compression enough?
A: It depends on what you want the documents to do. Passing a file internally, default compression is usually fine. But the moment documents feed retrieval, feed a model, or must stand as evidence, the difference classification and strategy make is real.
Q: Can a compressed document still serve as legal evidence?
A: It depends on the method. Archive-class goes lossless and reversible, so content and stamps are unaltered and it qualifies. Preview-class lossy is only for quick viewing. The key is not "was it compressed" but "which path did this type take."
Q: Hybrid retrieval sounds complex. Can a small team afford it?
A: At its core it is "keyword plus semantic" merged, then a reranker — both have open implementations. The cost is not money, it is the willingness to treat "answers correctly" as something to measure continuously.
Q: How do I tell whether a knowledge base actually answers correctly?
A: Do not trust the demo. You need a labeled standard Q&A set, a rerunnable evaluation, plus blind testing and regression. If a third party can rerun your system on the same standard, the conclusion holds.
Q: Do compression and RAG have to be done together?
A: Not by the same vendor, but they must be seen as one line. A cleanly compressed document entering retrieval carries less noise, spends fewer tokens, and recalls more steadily — splitting the two usually costs you on both ends.