Parsing Legacy Document Formats

The short version: about 23.3% of real enterprise corpora are non-standard formats. Legacy formats cannot go through a generic parser; they need format-specific handling, or you lose entire batches of content.

1. Why legacy formats are hidden rocks

Generic parsers target modern Office (docx / xlsx). Hand them an OLE2 compound document, an ambiguously encoded old text file, or an HTML announcement and they fail. In one A-share research corpus of 841 companies, 252,386 documents, 147GB, about 23.3% were non-standard formats (HTML / OLE2 .doc / CNINFO .bin). Any "skip the old files" strategy throws away nearly a quarter of the corpus.

That quarter is usually the hardest to replace: old .doc and .bin files are early announcements and policy documents whose source systems may already be offline, so there is no second copy to fetch. During a busy earnings season, teams see a parse success rate above ninety percent and approve the batch; the rest land silently as empty documents. Nobody notices until a retrieval smoke test cannot recall a specific company’s metrics, and by then a full round of model tuning has been wasted on a data problem.

2. Differential parsing for OLE2 .doc

A legacy .doc is not flat text; it is a CFB (Compound File Binary) compound document. Content is chopped into sectors scattered across the file and organized by a directory structure. The body does not occupy one contiguous region—the piece table (CLX) structure tells you where each real body fragment lives and in what order to stitch it, while styles, revisions, and deleted residue share the same container.

Parsing must therefore descend into the piece table (CLX), extract only the body fragments, and skip style and revision residue. Get this wrong and you get two outcomes: garbled text, or the sneakier one—the parse reports success while the extracted body is nearly empty. The second case is the hardest to catch, because the pipeline shows green. Only character-count statistics and post-ingest sampling expose it.

3. Encoding detection fallback for CNINFO .bin

Old CNINFO text files rarely declare their encoding, and a single batch can mix UTF-8, UTF-16, and GBK. The parser must probe in order BOM / UTF-16 / UTF-8 / GBK: read the byte-order mark first and settle the case if present; otherwise try each candidate encoding until decoding passes validation.

The order is not arbitrary. A BOM is the most reliable explicit signal; UTF-16 variants need a BOM to tell big-endian from little-endian; UTF-8 is naturally compatible with pure ASCII; GBK is the most common default left behind by legacy Chinese systems. Guess one byte wrong and the whole file becomes mojibake—and mojibake still passes through chunking silently, carrying garbage semantics into the vector store.

4. Content locating for HTML announcements

An HTML announcement page mixes the body with navigation bars, breadcrumbs, copyright notices, ad slots, and scripts. Convert the whole page to text and the noise-to-content ratio approaches parity; after chunking, every chunk carries navigation fragments.

The right approach is to use Readability to locate the main content block, then strip irrelevant tags and keep only the actual announcement. Otherwise noise enters the chunks, competes with the body for relevance at retrieval time, and crowds out Top-K results—the page looks like an all-hit, but the answer is wrong.

Legacy fallback is one slice of parsing. For scan-vs-text routing see Scanned PDF OCR and Layout Analysis; for post-parse cleaning see Document Cleaning. If non-standard formats keep dragging your corpus down, try the UGLYPEAR AI parsing pipeline.

Common Pitfalls and Fixes

These pitfalls share one signature: everything works on the development machine, then fails silently on real corpora — no exceptions raised, just unusable output. In our A-share corpus, the 23.3% non-standard formats concentrate in exactly these three categories. The fastest way to check your own pipeline is sampling: pick random legacy files and compare extracted text against the source for readability. The incremental audit (added + unchanged == total) blocks silent loss before ingestion; see Document Parsing Pipeline.

FAQ

Q1: How much of a real corpus is non-standard format?

In one A-share research corpus of 841 companies, 252,386 documents, 147GB, about 23.3% were non-standard formats (HTML / OLE2 .doc / CNINFO .bin). Skipping them means abandoning nearly a quarter of the corpus, usually the hardest part to replace.

Q2: Why can’t OLE2 .doc text be read directly?

A legacy .doc is a CFB compound document where the body is scattered across sectors. You must descend into the piece table (CLX) structure to stitch real body fragments in order and skip style and revision residue; flat reading yields mojibake or an empty document.

Q3: How is CNINFO .bin encoding handled?

The parser probes in order BOM / UTF-16 / UTF-8 / GBK, trying each candidate until decoding passes validation. Guess one byte wrong and the whole file becomes mojibake, which still passes through chunking and pollutes retrieval.

Q4: How is noise removed from HTML announcements?

Use Readability to locate the main content block, then strip navigation, ads, and script tags, keeping only the announcement body. Otherwise noise enters the chunks and competes with real content for Top-K slots.