Chinese Retrieval with CJK Bigrams

The short version: Chinese has no word boundaries, and the tokenization choice sets the ceiling on recall—get it wrong and every later stage pays for it. CJK bigrams split text with a two-character sliding window, keeping single-character recall while adding phrase precision: a safe default for enterprise Chinese RAG.

1. The tokenization trade-off

English tokenizes on spaces; Chinese offers no such convenience, so before building an index you must decide the granularity. That decision caps recall: pick wrong and no amount of tuning downstream recovers it. One detail is easy to miss: the index side and the query side must use the same tokenization—mismatch them, and even a perfect index returns nothing.

Word-based segmentation depends on a dictionary. Any term it misses—a new product model, a new regulation, a new institution name—becomes unrecallable as a whole. In research and legal corpora, where terminology churns constantly, out-of-vocabulary is the norm rather than the exception. Character-based splitting is the opposite: every surface form can match and recall maxes out, but noise grows, semantics shred, ranking quality sinks, and the index balloons.

CJK bigrams split the difference: every two characters become one token, sliding forward. 营业收入 becomes 营业, 业收, and 收入. Every surface combination is covered without collapsing to single characters—the most workable balance between recall and noise in practice. The index bloat is absorbed by ordinary inverted-index compression, not a bill this scheme should carry.

2. Why it matters for RAG

Enterprise documents are full of model numbers, IDs, metric terms, and clause numbers. These live in a crowded semantic space—close in meaning, different on the surface—so vectors miss them, and literal BM25 has to back them up. When filings arrive in seasonal bursts, the misses cluster: new terms in research titles, institution abbreviations in announcements—vectors simply do not know them, while literal matching needs no dictionary.

The value of literal matching shows in a test: retrieval smoke tests require queries for codes like 000001 and 000002 to hit 10/10, not through semantic understanding but through deterministic surface matching. For identifier lookups there is no "semantically similar"—it is either right or wrong. Smoke tests should include such queries on every release to pin down the recall floor.

BM25 effectiveness depends directly on tokenization. Bigrams make model-fragment queries easier to hit literally; colloquial query mapping—aligning a user's casual term to the document's exact term—lifts recall further. Tokenization and rewriting are the two switches of the BM25 branch; leave either off and the branch drags.

3. Complements vectors

Bigrams serve the BM25 branch of hybrid retrieval, alongside dense vectors: vectors handle semantic generalization—finding it even when phrased differently; bigram+BM25 handle literal text and IDs—where not a single character can be wrong. The two result lists fuse by weighted RRF (1.6 : 1.0), holding the recall floor for Chinese retrieval. The weighting is not a guess: the literal branch is steady while the semantic branch fluctuates, so tilting weight toward the literal side keeps overall metrics stable.

Two practical add-ons complete the setup: intent classification and time filtering on the query side to keep requests that should not reach retrieval out of it, and a dedicated test set for identifier and terminology queries on the evaluation side so they are not averaged away by general metrics. Neither is complicated, yet both decide whether bigram gains reach the business.

Fusion is a middle station: ordering quality still needs Cross-Encoder reranking, and whether any of it works is a question for retrieval metrics like HitRate@K and MRR, not for feelings. Bigram is the base of the chain. Fusion upstream in Hybrid Retrieval; reranking after fusion in Cross-Encoder Reranking; how to measure recall in RAG Evaluation Metrics.

Real-World Scenario: Exact-Token Recall in Chinese Research Queries

Chinese RAG recall concentrates on three query types: company short names and ticker codes, metric abbreviations such as revenue or net profit, and reporting-period phrasings. These tokens are either missing from general tokenizers or segmented wrongly — word-based segmentation loses any term the dictionary does not cover. CJK bigram indexes text by character pairs: a phrase like operating revenue becomes three overlapping bigrams, so any query containing those consecutive characters hits regardless of dictionary coverage.

Bigrams solve segmentation; colloquial query rewriting solves comprehension: a vague question like that earlier report about revenue is rewritten into a canonical query carrying metric names and reporting periods. The two together form complete Chinese query preprocessing. This is also part of why the BM25 side carries a 1.6 weighting in hybrid retrieval — precise tokens in Chinese queries discriminate better than semantic generalization, see Hybrid Retrieval. How corpus segmentation interacts with chunking is covered in Chunking Strategy.

FAQ

Q1: Why can’t Chinese tokenize on spaces like English?

Chinese has no word boundaries, so granularity must be chosen before indexing. Word segmentation misses dictionary-absent terms entirely and character splitting is too noisy; bigrams are the workable middle ground.

Q2: What is a CJK bigram?

A two-character sliding window over the text: 营业收入 becomes 营业, 业收, and 收入. Every surface combination is covered without collapsing to single characters, balancing recall and noise.

Q3: How do bigrams relate to vector retrieval?

Bigrams serve the BM25 branch of hybrid retrieval for literal and identifier recall, complementary to semantics-focused dense vectors; the two result lists fuse by weighted RRF.

Q4: Why recommend bigrams for enterprise Chinese RAG?

Enterprise docs are full of model numbers and clause IDs that vectors miss, so BM25 must back them up—and BM25 depends on tokenization. Bigrams balance recall and noise best, making them the safe default.