I used to think the problem with public-sector knowledge was a lack of information. The information is everywhere. The problem is that the right person cannot reach the right page at the right moment, and when they finally do, there is no guarantee it is the current version. After a decade of watching agencies buy one repository after another, I have come to believe the failures are not merely technical. They are architectural, and they are moral, in that they erode the public's trust.
Let me start with a number that should unsettle anyone who manages documents for a living. According to the IBM Cost of a Data Breach Report 2025, the global average cost of a single data breach reached 4.44 million US dollars, and the average time to contain one stretched to 241 days. Two hundred and forty-one days is nearly eight months of an organization not knowing what left the building. For a public institution, that is eight months in which a citizen's record or a budget line could sit somewhere it should not.
Why the "one big folder" never worked
The instinct, when a team grows, is to centralize. Put everything in one place, give people access, and let search sort it out. It feels democratic. It is also where most of the damage begins.
A survey by M-Files, conducted with Vanson Bourne, found that 96 percent of employees struggle to find the latest version of a document, and 83 percent have recreated a file from scratch because they could not locate the original. Read them together: the first says our systems hide the truth; the second says people stopped trusting them enough to use them. A knowledge base that nobody trusts is worse than no knowledge base, because it creates the illusion of order while quietly rotting.
And the volume is not slowing. There are more than 3 trillion PDFs in circulation worldwide. One major collaboration platform alone receives over 2 billion file uploads every single day. When I first read that, I assumed it was a typo. It was not. We are not drowning because we lack tools. We are drowning because every tool we add makes the pile deeper and the search harder.
The fallacy of "public by default"
Here is the uncomfortable part of my own story. Early in my career I built a portal for a regional authority. It was open, searchable, and proud of it. Then a contractor pulled a confidential procurement scorecard through a shared link that had been left public. No breach in the dramatic sense — just a file that was one degree too visible. Nothing leaked to the press. But the internal review took months, and the trust between departments never fully returned.
That experience taught me the most expensive lesson of my working life: access should never be a property of the file. It should be a property of the moment — who is asking, what are they allowed to see, and is this the authorized version.
The older model attaches permissions to a document or a folder. A role gets the whole thing or none of it. But real public-sector work does not come in whole documents. A single planning report contains redactable sections, financial tables visible only to the treasury role, and public summaries meant for anyone. When you can only protect at the folder level, you end up either over-sharing or locking everything down so tightly that the file becomes useless.
What actually changes the outcome
A few things, in my experience, separate the deployments that survive from the ones that quietly disappear after launch. The design choices behind these properties are on our products page: https://www.uglypear.com/en/products/
The first is permission granularity pushed down to the content segment, not the file or folder. When a query runs, the system applies tiered, segment-level access at query time and decides per section what the asking role may see, returning only that. Different roles, same query, genuinely different answers. This is the difference between "the document is restricted" and "you can see your part of the document." For a public institution, that distinction is the entire compliance posture.
The second is the refusal to migrate. Most knowledge projects fail in the migration phase, because copying an organization's living data into a new silo is where version drift, broken links, and accidental exposures happen. The better approach connects directly to the databases an organization already runs. The original data stays where it lives. Nothing moves. The knowledge layer reads from the source of truth and never becomes a competing copy that can fall out of date.
The third is the ability to recover. Incremental indexing is good because it keeps the index fresh without re-scanning everything. Snapshot rollback is better because it means a mistaken index update can be undone to a known-good state. And I have come to think the most underrated safeguard is a guard against accidental mass deletion — a simple brake that stops one operator from wiping a corpus in a bad afternoon. None are glamorous, but all keep a system running at 3 a.m. when the on-call person is tired.
I will mention one vendor only once, because the pattern matters more than the name: a product called UglyPear Data has recently been validated against exactly this shape of requirement, with a full pipeline validated end-to-end. What I find worth noting is not the brand but the design choices — restricted access enforced at query time, no-migration database connection, snapshot rollback, and a mass-deletion guard. Those are the properties I now screen for, regardless of who sells them.
Compression is not a luxury feature
There is a second thread to this story that I underestimated for years: size. We have written more about the economics of this on the blog: https://www.uglypear.com/en/blog/
Public-sector documents are enormous. Procurement packs, court filings, engineering drawings, and regulatory submissions routinely run to hundreds of pages, full of high-resolution scans. When you feed that material into a large language model for search or summarization, you pay per token. A 443-page OFD document — a format common in certain government workflows — was recently compressed end-to-end in 124 seconds. More strikingly, image compression in the pipeline cut LLM token cost by roughly 85.9 percent against a GPT-4o baseline. That is not a marginal savings. It is the difference between a system affordable to run at scale and one quietly switched off when the bill arrives.
And here is the part that surprised the skeptics on my team: the compression preserved the elements that actually matter legally. Seals and tables survived. In a public institution, a table of figures is not decoration. A seal is not a logo. They are the difference between a document that carries authority and a document that looks like a summary someone typed. If compression throws those away, it is worse than no compression, because it creates plausible-but-invalid records.
The sovereignty question nobody can avoid
For public-sector work there is a final, non-negotiable dimension. The data cannot leave. Not to a third-party cloud, not to a shared tenant, not across a border someone else controls. This is why the private and on-premise deployment model exists, and why air-gapped, local-first operation is a baseline requirement rather than a premium feature for the paranoid. You can read the full deployment model on the main site: https://www.uglypear.com/en/
I have watched procurement teams reject otherwise excellent tools purely on residency grounds, and they were right to do so. When the data is a citizen's health record or a national budget line, "we encrypt it in transit" is not an answer. "It runs inside our perimeter and the original never moves" is the answer that survives legal review.
A practitioner's checklist
If you are evaluating a knowledge base for a public institution, I would screen for these, in order:
- Permission granularity at the content segment, enforced at query time — not the folder.
- Migration-free connection to your existing databases, with the original data staying in place.
- Incremental indexing with snapshot rollback and a guard against mass deletion.
- On-premise or air-gapped operation where residency is required.
- Compression that reduces token cost without destroying seals, tables, or layout.
None of these require a specific vendor. They require a specific discipline: treat the document as something with many readers and many truths, and build the system to honor that.
What I would tell my younger self
The version of me who built the proud, open portal would not have listened. But if I could wave a copy of the IBM breach report in front of him, and the M-Files finding that 96 percent of people cannot find the current version, I think he would at least pause.
Public-sector knowledge does not fail because we lack intelligence in the system. It fails because we forgot that knowledge is a relationship between a person, a question, and a right to know. Build for that relationship — segment-level access, source-of-truth connection, recoverable state, sovereign deployment — and the pile of 3 trillion PDFs stops feeling like a threat. It starts feeling like a library again.
The tools will keep changing. The discipline is what endures.
Try it now
Ready to see how a migration-free, access-aware knowledge layer fits your agency?
- Explore the platform: https://www.uglypear.com/en/
- See the products: https://www.uglypear.com/en/products/
- Read the blog: https://www.uglypear.com/en/blog/
FAQ
How long does a pilot take?
Most agencies run a scoped pilot in weeks rather than months, because the approach connects directly to the databases you already run — there is no migration to schedule and no parallel silo to stand up. The timeline is driven by the scope of documents you include, not by the effort of moving data. A focused pilot exercises segment-level access, incremental indexing with rollback, and the compression pipeline on a real corpus, so stakeholders see token-cost behavior before a wider rollout.
Which document formats are supported?
The pipeline is built around the formats public-sector work actually produces: office documents, PDFs at scale, and government workflow formats such as OFD, including very large multi-hundred-page files. High-resolution scans embedded in those documents are handled as first-class content rather than discarded. A knowledge base is only as useful as the rarest document it can ingest — if one critical filing format is excluded, the whole system develops a blind spot.
How are scans and images handled?
Scans and images are compressed in a way that reduces LLM token cost without throwing away the elements that carry legal weight. In validation, image compression cut token cost by roughly 85.9 percent against a GPT-4o baseline while preserving seals and tables. A seal is not a decorative logo and a table of figures is not ornamentation — both are what make a public document authoritative, so the pipeline shrinks the material without shrinking its evidentiary value.
Can it run fully offline, in an air-gapped environment?
Yes. Air-gapped, local-first operation is a baseline requirement for public-sector deployments rather than a premium add-on. The data stays inside your perimeter, the original never moves to a third-party cloud or shared tenant, and the knowledge layer reads from your source-of-truth databases within the boundary you control. This is the residency posture that survives legal review when the data is a citizen's record or a national budget line.