Building a Financial Research RAG

The short version: equity research is a proving ground for RAG data engineering—messy documents, legacy formats, relentless updates. ashare-web turns A-share filings, broker research, and investor-relations records into a RAG Q&A system, surviving real corpora through differential parsing, idempotent import, Unstructured-to-SQL, and graceful degradation.

1. Corpus scale and challenge

The scale of real research corpora surprises most teams: 841 companies, 252,386 documents, 147GB, spanning from 2000 to today. Formats make it worse—23.3% are non-standard: HTML announcements, OLE2 legacy .doc files, and CNINFO .bin plain text. Without differential parsing at the entrance, nearly a quarter of the corpus is lost, and nothing downstream can recover it.

The update cadence is harsh too. Filings arrive in seasonal bursts, announcements drop as they are disclosed, research notes and IR records accumulate continuously. The import pipeline must absorb batch peaks, handle documents as they arrive, and never redo a whole batch because one file failed—at this scale, any full re-run is an engineering incident.

2. Idempotency key and increment

The system uses the first 16 hex of the file content SHA-256 as its idempotency key. On import it compares first: unchanged files skip, only changed ones reprocess. Filings re-publish yearly and historical files get re-fed repeatedly; this design means the same data occupies exactly one slot no matter how many times it is imported—no duplicates, no conflicts.

At scale you must also plan for interruption: if a batch import dies halfway, resumable runs keep finished batches from being redone. After an incremental import, serve must restart before new data is searchable—a known, explicit boundary of the current version. And there is a safety net: when the embedding cache is incomplete, the system silently degrades to pure BM25. Retrieval quality dips temporarily; availability never breaks, and hybrid search returns once the cache catches up.

3. Table metrics into a structured store

The most valuable part of a filing is its tables—revenue, net profit, margins. Pure vector search is weak on questions like "what was this company's revenue over the past three years", so Unstructured-to-SQL extracts table metrics into a SQLite metric store. The same question can travel both the semantic path and the SQL path, cross-checking each other.

Structured metrics also make filtering composable. Time filtering is near-mandatory here: in one test, 17,590 candidates filtered down to 829, and 18,805 down to 827—about 95.5% removed. Without structured metadata that filter could not exist, and the full candidate set would crush the generator's context window.

4. Reusable lessons

The value of this system is not any single technique but a set of plain principles made to work: lose nothing at the entrance (differential parsing), never rework repeated imports (idempotency keys), do not rely on vectors alone for metrics (structured store), and keep serving when the cache is incomplete (degradation). The results are measurable: retrieval smoke tests on 000001 and 000002 hit 10/10, and ten questions on 000002 ran end to end in 4.6 seconds. In one anonymized delivery of the same kind, 859 documents imported with 0 failures, producing 36,395 chunks and about 35.19 million characters.

To reuse these lessons on your own corpus, go stage by stage: legacy formats in Legacy Document Formats; chunking and enrichment in Chunking Strategies and Chunk Enrichment; large-corpus lifecycle in Knowledge Base Lifecycle; full metrics in the anonymized customer case.

Case Deep-Dive: The Accounting From 147GB of Corpus to 4.6-Second Answers

Only by lining up the numbers of every stage can you see where the results come from. Corpus layer: 841 companies, 252,386 documents, 147 GB, with 23.3% non-standard formats handled by format-specific parsing. Project layer: 859 documents from two listed issuers with zero failures, 36,395 chunks and about 35.19 million characters, 227 research reports from 25 institutions, 112 investor-relations records, and 457 metrics stored in SQLite. Embedding layer: Qwen3-Embedding-4B with disk caching, zero repeated embedding during evaluation iterations. Retrieval layer: BM25 and dense vectors fused at 1.6 : 1.0 weighted RRF, time filtering compressing candidates from 17,590 to 829 (about 95.5%), smoke tests hitting 10/10 on both 000001 and 000002. End to end: ten questions on 000002 in 4.6 seconds.

There is no single-point magic in this chain: zero failures come from parsing and auditing (added + unchanged == total), hit rates from hybrid retrieval and query rewriting, and second-level response from the interplay of filtering, reranking, and layered generation. Cross-industry migration thinking is in Private Deployment and Data Sovereignty; the full evaluation methodology in RAG Evaluation Metrics.

FAQ

Q1: What is the hardest part of research corpora?

Messy documents, legacy formats, relentless updates. Of 841 companies, 252,386 documents and 147GB, 23.3% are non-standard (HTML, OLE2 .doc, CNINFO .bin) and need differential parsing, or a quarter of the corpus is lost at the entrance.

Q2: How is the idempotency key implemented?

The first 16 hex of the file content SHA-256. Import compares first; unchanged files skip and only changed ones reprocess, so repeated imports never duplicate. A serve restart makes new data searchable.

Q3: How are table metrics made queryable?

Unstructured-to-SQL extracts them into a SQLite metric store, so both vector retrieval and SQL queries work. A question like revenue over three years can travel the semantic and structured paths, each validating the other.

Q4: What if the embedding cache is incomplete?

The system silently degrades to pure BM25 so retrieval stays available, then restores hybrid search once embeddings catch up. It is a deliberate availability-first trade-off.