Knowledge Base Lifecycle: Idempotent Ingest

The short version: lifecycle management decides whether a knowledge base stays usable long-term. The core is four things: content SHA-256 idempotency, stable Chunk IDs, incremental diff that updates only changed chunks, and version archive with resumable runs.

1. Idempotency: re-runs do not redo work

A knowledge base is not a build-once asset: documents get revised, policies expire, and new material arrives every week. Re-running the full parse-clean-chunk-embed pipeline on every change means cost grows linearly with corpus size. Idempotency removes that concern: importing the same batch twice produces exactly the same state as importing it once.

Each file uses a content SHA-256 hash as its idempotency key; unchanged files skip preprocessing entirely. In measurement, an idempotent re-run skips processed content in 0.2 seconds, so full-corpus regression checks cost almost nothing. Stable Chunk IDs are the other half: the same passage keeps the same ID across builds, so hit statistics and citation tracing stay aligned across versions instead of being invalidated by every rebuild.

2. Incremental: touch only changed chunks

Idempotency stops unchanged files from re-running; incremental import goes one step further. When a file does change, a diff against the previous version updates only the chunks that actually changed—untouched chunks stay put, their embeddings stay.

This pays off most in scenarios like earnings season, when hundreds of new documents arrive weekly: you never re-run 100% of the corpus for 1% of the changes. The A-share research corpus holds 841 companies, 252,386 documents, and 147 GB; at that scale, no full-rebuild scheme survives a reporting season. The system uses the first 16 hex characters of the content SHA-256 as its key—short enough to compare fast, specific enough to separate documents.

3. Versioning and validity governance

Getting data in is half the job; getting it out on time is the other half. The version archive keeps 3 versions by default and allows rollback to a historical snapshot: if a chunking rule change breaks something, you return to the last good state, and audits can always check which version was ingested. Validity governance marks obsolete content with valid_until, so expired chunks stop participating in retrieval and the model stops citing dead policies.

Together they solve the classic "old version still being cited" problem. After a policy revision the old text remains in the corpus, and retrieval still returns it—users receive clauses that have already been superseded. That failure mode costs the most in compliance settings, and it is fixed not by the model but by governance rules applied before ingestion.

4. Resumable runs enable scale

A large corpus cannot finish in one pass: services restart, machines lose power, batch jobs get preempted by more urgent work. The ingest state machine records progress per file and continues from the breakpoint after an interruption—no work is redone, and completed files are skipped.

The mechanism has been through systematic engineering validation: 26 scenarios covering version archive, resume, permission tagging, and secrecy filtering all passed, and six-format E2E ingestion passed incremental audit checks satisfying added + unchanged == total—nothing missed, nothing processed twice. Lifecycle management attracts little attention, but it decides whether the knowledge base still works in month two.

Lifecycle follows chunking and enrichment. Splitting in Chunking Strategies; permission and time filtering in Permission Tagging and Financial Research RAG; regression after changes in Regression Gates.

Real-World Scenario: Earnings-Season Updates and Resumable Ingestion

Lifecycle governance proves itself during update spikes. Earnings season adds hundreds of documents weekly; full re-ingestion would be unacceptable in both embedding cost and downtime. The working mechanism is a chain: SHA-256 content hashing for idempotency (duplicate imports skip preprocessing in 0.2 seconds), stable chunk IDs plus incremental diff (only changed chunks update, with silent fallback to pure BM25 retrieval when the embedding cache is incomplete), resumable ingestion via an ingest state machine (large corpora run in batches, interruptions never waste work), version archiving keeping 3 versions by default, and valid_until time governance that retires expired policies from retrieval automatically.

In the ashare-web production system, the idempotency key is the first 16 hex digits of the SHA-256 of file content; after incremental import, restarting serve exposes the new data to retrieval. Six-format ingestion E2E all pass the incremental audit (added + unchanged == total), and all 26 lifecycle scenarios (version archiving, resumable runs, permission tagging, and classification filtering) pass. The quality backstop for changes — the regression gate — is covered in RAG Regression Gate; baseline building in RAG Evaluation Metrics.

FAQ

Q1: Why does a knowledge base need lifecycle management?

It is not built once and done: documents get revised and policies expire. Idempotent re-runs, incremental updates, versioning, validity marks, and resumable ingestion decide whether the base stays usable long-term—most visibly in earnings season with hundreds of new documents arriving weekly.

Q2: What is the idempotency key?

Each file uses a content SHA-256 hash as its idempotency key; unchanged files skip preprocessing entirely. A measured idempotent re-run skips processed content in 0.2 seconds. The A-share research system uses the first 16 hex characters of the hash as its key.

Q3: How many versions are archived?

Three versions by default, with rollback to historical snapshots—if a chunking rule change goes wrong, you return to the last good state. Combined with valid_until validity marks, expired content stops participating in retrieval, so the model stops citing superseded policies.

Q4: What is resumable run for?

Large corpora run in batches; the ingest state machine records progress per file and resumes from the breakpoint after interruption without redoing finished work. Together with version archiving, 26 lifecycle scenarios passed engineering validation, and six-format E2E incremental audits all passed.