How to Design a Document Parsing Pipeline

The short version: a parsing pipeline is not about "reading the file," it is about "turning the file into structured, retrievable, governable data." Design it to route by file type—high-fidelity parsing for text, automatic OCR routing for scans.

1. What formats you support defines what data you can ingest

A usable parsing layer should at least cover PDF / Word (including legacy OLE2 .doc) / Excel / PPT / OFD / XPS / TXT / HTML, all normalized to Markdown or JSON. Format coverage sets the ceiling of your knowledge base: miss one format and a whole class of data never gets in. Non-standard formats are also more common than most teams assume—in one A-share research corpus of 841 companies, 252,386 documents and 147GB, about 23.3% were non-standard (HTML / OLE2 .doc / CNINFO .bin).

Legacy formats cannot be handled by converting and hoping. OLE2 .doc requires parsing the CFB compound-document structure and extracting text through the piece table (CLX); legacy CNINFO .bin text files need BOM / UTF-16 / UTF-8 / GBK encoding detection with fallback; HTML announcements call for Readability body detection followed by tag stripping. Every format you add removes a batch of manual copy-paste work. Text-layer PDF extraction can run under 200ms per file, which is the prerequisite for bulk corpus ingestion.

2. Route text and scanned documents differently

Text-layer PDFs yield text and layout directly; scanned files have no text layer and must be routed to OCR. Make the routing decision automatically before ingestion, not by hand: a mixed corpus typically holds native PDFs alongside scanned reports, and one missed scan means one empty paragraph downstream.

Sharing one path for both means either dropping characters from scans or wastefully re-recognizing text files. The subtler cost is inconsistent quality: similar documents on different routes produce different parse-quality scores, so chunking and retrieval behave unpredictably. Bake routing rules into the pipeline and quality gets a reproducible baseline.

3. Layout analysis preserves structure

DeepDoc layout analysis runs layout.onnx and tsr.onnx as local ONNX inference—no cloud dependency. The first locates regions such as titles, paragraphs, tables, and images; the second recognizes table structure. The stage preserves reading order, coordinates, and table HTML structure (including rowspan / colspan merged-cell restoration), with PDFium for rendering.

This is the step that decides whether downstream chunking understands "this is one table." Lose the layout and a cross-page table becomes two orphaned text blocks, a two-column page gets stitched in the wrong order, and chunks arrive semantically shattered. Coordinates also pay off in governance: during an audit you can trace any passage back to its page and region in the source file.

4. Output must be governable

Parse artifacts should not just be handed downstream; they should carry governance metadata: source, page number, coordinates, and parse_quality scores across three layers (structure / relation / content) plus an overall grade. These fields are the foundation for permission tagging, validity governance, and the evaluation loop—without structured metadata, permission filtering and version archiving have nothing to stand on.

Quality scores also serve triage. Batches below threshold route to human review or a parameter retry; passing batches flow straight into the knowledge base, so a few bad documents never drag down a whole corpus. Recording scores per version also gives every parser upgrade a before-and-after baseline.

Parsing is the first gate of RAG quality. For the full parse-to-chunk chain see RAG Chunking Strategies and Table Structure Extraction; for scans specifically, Scanned PDF OCR and Layout Analysis.

Real-World Scenario: 23.3% Non-Standard Formats Separate Winners

Many teams test their parser on clean PDFs and fall apart on real corpora. Our A-share research corpus spans 841 companies, 252,386 documents and 147 GB (dating back to 2000), and 23.3% of it is non-standard: HTML announcements, OLE2 legacy .doc files, and CNINFO .bin plain-text exports. That means one document in five will produce mojibake or an empty output with a generic parser. The engineering value of a parsing pipeline concentrates exactly on that 23.3%: format-specific extraction, encoding detection with fallback, and readability-based body location decide whether this material becomes a searchable asset or dead weight on disk.

Another overlooked acceptance criterion is incremental auditing: six-format ingestion E2E tests (OFD / PDF / Excel / Word / XPS / PPT) all pass the added + unchanged == total check, so no document is ever silently dropped. For legacy format handling, see Legacy Document Formats; for preserving table structure, see Table Structure Extraction.

FAQ

Q1: Why separate text PDFs from scans?

Text-layer PDFs can be extracted directly, often under 200ms per file; scans have no text layer and must go through OCR. One mixed path either drops characters from scans or wastefully re-recognizes text files, and the two document classes end up with incomparable quality baselines.

Q2: What metadata should parsing output carry?

At minimum: source, page number, coordinates, and parse_quality scores across three layers (structure / relation / content) plus an overall grade. These power permission tagging, validity governance, audit traceability, and the evaluation loop.

Q3: Why preserve coordinates and reading order in layout analysis?

They let downstream chunking reconstruct "this is one table" or "this is one paragraph." Without them tables fragment and paragraphs misalign, retrieval and generation degrade, and audits lose the ability to trace a passage back to its source.

Q4: What share of real corpora are non-standard formats?

In one A-share research corpus of 841 companies, 252,386 documents and 147GB, about 23.3% were non-standard formats (HTML / OLE2 .doc / CNINFO .bin), which is why differential parsing with CFB handling and encoding fallback is mandatory.