There is an order-of-operations decision buried inside every retrieval-augmented pipeline, and most teams make it without thinking. The default is to retrieve first and then hand whatever came back to the model. The alternative is to shape the source before it is ever retrieved or reasoned over. That small reordering changes the economics of the entire system, because a language model bills by the token and the documents research institutions feed it are, almost by definition, huge.
This post is written as a set of design principles rather than a product description. The numbers cited come from published industry research and from public benchmarks of a document-compression engine that reaches embedded visuals inside Office files.
The token ledger is written upstream
A survey commissioned by M-Files and run by Vanson Bourne found that 96 percent of employees struggle to find the latest version of a document, and 83 percent have recreated a file from scratch because they could not locate the original. So the document layer is already failing people before a model is involved, and token billing only makes the same problem more expensive.
Komprise's customer research shows that 60 to 70 percent of data on enterprise network-attached storage has not been accessed in over 90 days, yet it still occupies the most expensive primary tier. Research institutions are the clearest example: a project closes, a paper publishes, and the raw data is touched by no one — but it cannot be deleted, because a reviewer may ask for it back years later, exactly as it was. You pay premium rates to hoard material that is mostly cold.
A retrieval system that only indexes and never compresses inherits both failures. It makes the unfindable merely searchable, and the bloated merely queryable.
Principle 1: Permission rides on the query, not the file
The most common architectural mistake is attaching access control to the document or the folder. That model forces a binary: a role gets the whole file or nothing. Real research documents are not binary — a grant proposal holds a public abstract, role-restricted budget tables, and collaborator contact details that should not surface in a general search. At folder level you either over-share or lock everything down until it is useless.
The alternative is to push access tiers down to the content-segment level and enforce them at query time. When a query runs, the system evaluates, per section, what the asking role is authorized to see, and returns only that — a gate applied before the content is assembled, not a filter bolted on after.
Principle 2: Do not relocate the source of truth
The second failure mode is the migration phase itself. Copying a living organization's data into a new repository is where version drift, broken references, and accidental exposures tend to happen. The safer pattern is to connect directly to the databases the institution already runs, so the original data stays in place and nothing is copied into a competing silo that can fall out of date.
This avoids a multi-month migration that rarely finishes cleanly, and it avoids creating a second copy of sensitive data that needs its own protection, audit, and deletion policy. A capable pipeline demonstrates this by connecting to existing systems — an overview of the approach is on the products page.
Principle 3: The system has to be recoverable
Indexing is never a one-time event. Incremental indexing keeps the index fresh without re-scanning everything, which matters when a lab accumulates years of literature and logs. But incremental systems need an undo. Snapshot rollback lets a mistaken or partial indexing run be reverted to a known-good state.
The safeguard worth treating as mandatory is a brake against accidental mass deletion. One tired operator, one wrong scope, and a corpus can vanish. A confirmation step or a bounded blast radius is the difference between a minor incident and a major one.
Principle 4: Compression is primary, not an afterthought
Research documents are huge: theses, grant packs, and experiment reports that routinely run to hundreds of pages, full of high-resolution screenshots, plots, and scanned references. Feed that raw into a model and the cost scales with every needless pixel.
A recent benchmark is instructive. A 443-page OFD document — a format common in certain academic and government workflows — was compressed end-to-end in 124 seconds. More importantly for running cost, image compression in the pipeline cut LLM token cost by roughly 85.9 percent measured against a GPT-4o baseline. That is not a tuning improvement. It is the line between a system affordable at scale and one that gets quietly switched off when the bill arrives. The engine that produced these figures is described on the products page.
What recursive compression actually means
The detail that matters for research is how compression treats embedded content. Most documents are not flat text. A figure pasted into a report is itself an image; a chart copied from a spreadsheet is an object; a diagram cropped from a PDF is nested inside the file. In a naive pipeline, compressing the body text does little, because the weight lives in those embedded visuals.
The capable approach is recursive: the engine walks into the document, identifies the embedded images and charts, and compresses them in place, then repeats the pass so that nested objects are also reached. In the current generation of the engine this means Office images and charts inside a document are compressed recursively rather than only the surrounding text. For a paper or a proposal, the embedded visuals that were inflating the file finally get trimmed. The principle is simple to state and hard to do well: shrink the embedded weight, not just the wrapper.
This also reframes the token problem: a figure fed to a model as a high-resolution screenshot can consume thousands of tokens every time, yet compressing the embedded image first makes the same content far cheaper to store and reason over.
Principle 5: Sovereignty is the floor, not a premium
For research and other regulated work, residency is non-negotiable. The data cannot leave your perimeter — not to a shared tenant, not across a border someone else controls. This is the argument for private, on-premise, and where required, air-gapped, local-first operation. "We encrypt it in transit" is not a sufficient answer when the underlying data is a student record or an unreleased dataset. "It runs inside our perimeter and the original never moves" is the only answer that tends to survive an audit. The broader platform context is on the project home.
The ordinary risk is the one to design for
Leakage in research rarely comes from a sophisticated attack. IBM's 2025 cost-of-a-breach study found that 26 percent of breaches trace back to human error — the more common path than external intrusion. A collaborator's contact left inside a shared draft, a sensitive dataset attached to the wrong address: these are the accidents a good system removes by default. Sensitive content detected and masked at ingest time, with permissions enforced at the fragment level, takes a whole class of ordinary mistakes off the table.
A selection framework
| Dimension | What to weigh |
|---|---|
| Deployment model | Can it run on premise or air-gapped, with the original data never leaving your perimeter? |
| Data egress | Does anything leave the domain, or does it connect to existing databases without copying data out? |
| Sensitive-content handling | Are restricted fields detected and masked before ingest, so they never reach retrieval? |
| Permission granularity | Is access enforced at the content-segment level at query time, rather than only at folder level? |
| Compression depth | Does compression reach embedded images and charts recursively, or only the body text? |
| Token efficiency | Is there a measurable reduction in model token cost, evidenced by a public benchmark? |
| Cold-data cost | Can stale material stay findable without occupying the most expensive primary storage? |
| Recoverability | Are snapshot rollback and a mass-deletion guard present, so a bad import is reversible? |
A candidate that scores well on compression but fails residency is a non-starter, while one that nails residency but cannot enforce segment-level access will quietly over-share.
Closing thought for builders
The temptation is to optimize the model and ignore the document layer. The data says otherwise: most organizations cannot even find the current version of a file, and breaches trace largely to ordinary human error. The cheap wins are upstream — segment-level access, no-migration connection, recoverable state, sovereign deployment, and compression that reaches the embedded visuals.
FAQ
Q: What file formats are handled?
A: The engine works at the document level across common Office formats, PDFs, and certain structured academic and government formats such as OFD. Because compression reaches into embedded objects, the wrapper format matters less than what is nested inside it — a chart pasted from a spreadsheet is reached regardless of whether the host file is a Word document, a PowerPoint deck, or a PDF.
Q: How are embedded objects inside Office documents treated?
A: They are compressed recursively rather than skipped. The engine walks into the document, identifies embedded images and charts, and compresses them in place, then repeats the pass so nested objects are also reached. The embedded visuals inflating the file are trimmed, not just the surrounding body text — which is where most of the weight tends to live in a research document.
Q: Is there a scriptable or API path for batch work?
A: A capable pipeline exposes programmatic access so compression and indexing can be driven in batches and folded into existing workflows rather than run by hand. The design intent is that routine jobs — a semester's worth of theses, a lab's accumulated logs — run unattended on a schedule.
Q: Does compression affect retrieval or answer quality?
A: Compression targets the embedded weight — the high-resolution images and charts that inflate token cost — while the textual content retrieval reasons over is preserved. The document becomes cheaper to store, email, and reason over, and the passages a model is asked about remain intact, so answer quality is generally unaffected even as cost drops materially.
Q: How is cold-data storage cost addressed?
A: By keeping findable material off the most expensive primary storage and indexing it so it can be located by content when needed, rather than hoarding raw bytes on premium network-attached storage. Stale research data stays retrievable for an audit or a reviewer request without occupying the tier meant for hot, frequently accessed files.
Try it now
To see how compression-first retrieval lands in your own lab or department, review the product overview, the platform home, or more engineering notes on the blog. When material is under control, findable, and portable, research stops being slowed by its own documents.