I want to tell you about a project that failed without ever admitting it. No incident, no outage, no all-hands email. It failed the way a houseplant dies in an office nobody visits — gradually, and then one day someone looks over and there is nothing left to water.
A while back I was adjacent to a knowledge-base project at a mid-sized company. "Adjacent" is the honest word: I didn't own it, but I was the friend they called when things got strange. So I watched the whole arc, from launch-day enthusiasm to the moment last spring when I asked, casually, how the tool was going, and received a shrug.
This is a postmortem of a project that never officially died. I suspect these are among the more common failures in enterprise software, and also the rarely written about, because there is no ticket to link to. Nothing broke. Things just stopped mattering.
Launch week, when everything was possible
The pitch had been simple, and true in spirit: stop making people dig. The company had tens of thousands of PDFs, Word files, and scans scattered across shared drives, and the plan was to feed them into a retrieval system so anyone could ask a question in plain language and get an answer with the source attached.
The first two weeks were genuinely good. People asked where the travel policy lived. The tool answered. A screenshot made it into the internal newsletter, which in enterprise terms is roughly a parade.
Then the questions drifted away from the tidy documents and into the real ones.
The three quiet leaks
Looking back, the decay happened along three seams, and none of them announced themselves with an error message.
The first seam was the scans. A meaningful slice of the company's institutional memory — old contracts, HR paperwork, that one archive of faxes somebody insisted on keeping — existed only as images of text. The pipeline sliced them into chunks anyway. The chunks went into the vector store anyway. And the answers that came back were fluent summaries of garbage. Nothing crashed. The tool just occasionally said things that were confidently hollow, and users learned, the way users always do, which corners of the tool to avoid.
The second seam was the clones. Every important document at a company lives in five versions: the draft, the reviewed draft, the final, the final-final, and the copy someone emailed to a vendor in March. Ask a question and retrieval, doing its job faithfully, surfaced the passage five times with minor variations — and the one version that actually mattered was buried under its own siblings. A survey commissioned by M-Files found that 96% of employees struggle to find the most current version of a document, and 83% have had to recreate one because they couldn't find the original. I remember thinking that sounded like an exaggeration. It was, if anything, understated. A retrieval system doesn't fix that problem; it amplifies it, because the duplicates all float to the top together.
The third seam was the one that eventually got a meeting scheduled: sensitive fields riding into the index unprotected. Customer account numbers inside contracts, personal details inside HR files. The retrieval layer doesn't know those fields are dangerous; it only knows they match the query. An engineer typed a question about a named account and got back a paragraph containing live customer contact details. Nobody had pasted anything anywhere. The leak was the library itself.
The part where everyone blamed the model
The reaction to that incident followed a script I have since seen everywhere. There was talk of switching models. Someone proposed a fine-tuning sprint. The word "architecture" was used in ways I don't fully endorse.
The uncomfortable truth is that the model was probably the healthiest component in the whole system. What we had built was a smooth, expensive funnel feeding on unclean input — and the input is invisible in every demo. Nobody screenshots the preprocessing stage. There is no vendor slide for "we kept the documents exactly as messy as we found them."
Meanwhile, the human cost kept compounding quietly. Research on attention popularized by Asana's Anatomy of Work Index puts the average recovery time after an interruption at 23 minutes before full focus returns, and finds workers switching between 35 different applications in a day. An internal tool that answers wrongly is not a neutral thing. Every bad answer is another interruption, another context switch, another reason to never open it again. The tool didn't fail on a day. It failed in twenty-three-minute increments, one disappointed question at a time.
And there was a darker failure mode waiting underneath: the people who quietly routed around the internal tool and pasted documents into whatever public AI site had the nicest interface. IBM's 2025 Cost of a Data Breach report found that one in five companies has had a breach connected to employees using AI tools on their own — the pattern people call shadow AI — and that these organizations paid an average of $670,000 more per breach than their peers. Among the breached organizations, 63% still had no formal AI governance policy. I used to read numbers like that as other companies' problems. Now I read them as a description of the week after a knowledge base stops being useful.
What I would tell myself at launch week
If I could send one message back to launch week, it would not be about embeddings or chunk sizes. It would be a much less glamorous sentence: the documents are the project.
Spend the first month on the unglamorous layer. Actually read what the OCR produces from your scans, because the OCR's confidence score is not the same as the text being right. Deduplicate ruthlessly — and by that I mean decide which version of each document is the truth, then make the others unreachable. Find the fields that would ruin someone's week if they surfaced in a chat window, and mask them before they enter the index, not after the incident report.
The second thing I would tell myself: measure the boring stuff. Not "does it answer well" but "what fraction of retrieved chunks are duplicates," "how many of our source files are scans," "which fields in the index would scare our compliance officer." Those numbers predicted the death of the project months before the usage dashboard did. The usage dashboard was, I now think, a lagging indicator wearing a leading indicator's clothes.
The third thing is about incentives. People don't leak data because they are careless; they leak it because the official channel returns worse answers than a free website, and they have work to do. You cannot policy your way out of that. You can only make the internal channel genuinely better, and the path to "genuinely better" runs through document hygiene, not through a bigger model.
What we eventually did
After the shrug — after the project was, functionally, over — a smaller group restarted it with the order of operations flipped. Before anything touched a vector store, documents went through a preparation pipeline: cleaning, deduplication, structure recovery for scans, masking of sensitive fields, compression of the oversized files.
They used a data-engineering pipeline built for exactly this stage — the RAG data engineering base from UglyPear Data, which packages cleaning, masking, deduplication and compression as one on-premise stage; the compression side runs on the same engine family as SmartSlim. What mattered to me was not the branding but the sequencing. The scans got their structure restored before chunking, so chunks were made of sentences instead of smears. The sensitive fields were detected and masked across thirteen categories of information before the index ever saw them. And the number that made the finance-adjacent people sit up was the token math: measured on a GPT-4o basis, image-heavy content cut token costs by 85.9%, and every downstream call gets cheaper when the input is smaller and cleaner.
I will not pretend the second attempt was a triumph of consulting wisdom. It was mostly the first attempt, minus the parts we should never have skipped. But it stuck, and the difference was not the model, which stayed the same. The difference was that the funnel stopped feeding on mud.
The quiet part
Here is what I keep coming back to. Projects like this don't die of a single wound. They die of accumulated small disappointments, each one rationalizable, each one survivable. A garbled answer here, a duplicate passage there, a leaked field nobody reported but everybody whispered about. No single event is worth a meeting. The sum of them is worth the whole project.
So if you are standing at your own launch week right now, with the newsletter screenshot scheduled and the enthusiasm high: the model will get the credit and the blame, but it will deserve almost none of either. Go look at your documents. All the way down. That is where the project actually lives, and that is where it will actually die if you look away.
Nobody will send an email when it happens. You will just hear the shrug.
Try it now
- RAG data engineering base (private deployment, cleaning and masking pipeline): https://www.uglypear.com/en/
- SmartSlim product page (compression engine, desktop, SDK and API): https://www.uglypear.com/en/products/
- More field notes on compression and data engineering: https://www.uglypear.com/en/blog/
FAQ
Q: How do you notice a knowledge base is dying before the usage dashboard shows it?
A: Watch the retrieval-level symptoms: duplicate passages crowding the top results, answers built on obviously scanned pages, and the first internal complaint about a sensitive field surfacing. Those are leading indicators; the usage curve is a lagging one.
Q: Where should sensitive-field masking happen?
A: Before embedding and indexing, and inside your own network. Masking after indexing means the raw fields already passed through your pipeline as intermediate artifacts.
Q: Did replacing the model fix anything in this story?
A: In this case the model never changed between the failed attempt and the working one. The difference was entirely in what the documents looked like before they reached it.
Q: Does compression affect answer quality?
A: Fidelity-preserving compression keeps the text and layout intact and only strips redundant data — file size drops 60% to 80%, and image-heavy content cut token costs by 85.9% measured on a GPT-4o basis. Spot-check a few key pages at full zoom before indexing, as with any pipeline change.