Three Delivery Forms for a RAG Pipeline

The short version: a RAG data pipeline should not ship in one form only. The same parse → clean → desensitize → chunk → evaluate capability can be delivered as a CLI, an HTTP service, or an FFI library (C SDK), fitting scripted batch, online service, and embedded integration; picking the wrong form multiplies ops and integration cost later.

1. CLI: scripts and batch

The command-line tool compressor offers RAG commands preprocess / chunk / eval plus the full compression set, built for ops scripts, scheduled jobs, and one-off bulk ingestion. The classic case is backfilling historical archives: point a batch script at hundreds of gigabytes of PDF and Word files, and SHA-256 content idempotency means a re-run after interruption skips finished preprocessing in 0.2 seconds—large corpora resume without rework.

Its limits are just as clear: the CLI is human-triggered and process-based, with no resident service or live progress. When a midnight batch dies halfway, ops reads logs and exit codes, and the problem surfaces the next morning. It fits the batch world, not real-time services.

2. HTTP service: online and integration

uglypear-server (axum-based) provides sync/async compression, chunked upload, batch jobs, RAG preprocessing, knowledge base, evaluation, and an admin console, with SSE progress push and Prometheus monitoring. Chunked upload keeps frontends responsive, progress is visible in real time, and Prometheus metrics plug into existing alerting. Behavior is regression-guarded: 12/12 HTTP API tests and 6/6 SSE event-stream scenarios pass.

Once capability is exposed as an API, frontends, internal platforms, and third-party apps all plug in. Private deployment naturally lands here too: start one service process inside the customer intranet and every internal system calls the API—no per-machine tool installs, and upgrades touch the server only.

3. FFI library: embedded

libcompressor.so with sdk.h exposes a C API (rust_ingest_document / rust_resume_ingest and more), all catch_unwind-guarded—a Rust panic never escapes as a host-process crash, a hard requirement for embedding into commercial software. 28/28 SDK C API tests cover the main call paths, so Java, C#, or Go hosts can integrate over FFI with evidence behind them.

One layer up sit the Python SDK and LangChain connector (zero-dependency embedded client, metadata pass-through). ISVs and data platforms wire parsing and chunking into Dify, RAGFlow, or their own apps in a few lines, instead of rebuilding the data pipeline for every project—the biggest repeated cost integrators face.

4. How to choose

One-off or scheduled ingestion → CLI; exposing retrieval Q&A as a service → HTTP; embedding into your own software → FFI / Python SDK. Two questions decide it: who triggers the work (a person, a scheduler, or a host process), and who consumes the result (files on disk, an API caller, or the app above). When unsure, start with HTTP—it has the widest reach—and sink down to CLI or FFI as needed.

All three forms share the same underlying crates (core / compressor / extractor / rag-*), so behavior is consistent—chunking parameters validated in the CLI produce identical results over HTTP or FFI. No drift between what you tested and what you shipped.

Delivery follows the full data pipeline: get the pipeline right first, then pick the form. Pipeline design in Document Parsing Pipeline and Chunking Strategies; private landing in Private Deployment.

Decision Guide: Comparing and Combining the Four Delivery Forms

The four forms are not mutually exclusive — they are different entry points to the same pipeline: prototype with the SDK, go to production on the HTTP service, embed via FFI, and script operations with the CLI. Enterprise add-ons (multi-tenant API Keys, plan quotas, audit logs) are most complete in the HTTP form. The compliance prerequisite of deployment shape is covered in Private Deployment and Data Sovereignty, and the SDK-first integration path in RAG vs Fine-Tuning.

FAQ

Q1: What three delivery forms exist?

CLI (compressor with preprocess/chunk/eval), HTTP service (uglypear-server with SSE progress push and Prometheus monitoring), and FFI library plus Python SDK (the libcompressor.so C API). All three share the same underlying crates, so behavior stays consistent across forms.

Q2: When to use the CLI?

One-off bulk ingestion, ops scripts, and scheduled jobs. The CLI is human-triggered and process-based; with SHA-256 idempotency and resumable runs it excels at batch backfills with no rework, but it is the wrong tool for real-time external services.

Q3: What does the HTTP service offer?

Sync/async compression, chunked upload, batch jobs, RAG preprocessing, knowledge base, evaluation, and an admin console, with SSE progress push and Prometheus monitoring. 12/12 HTTP API tests and 6/6 SSE scenarios pass, so frontends and apps integrate on stable behavior.

Q4: Which path for embedded integration?

The libcompressor.so C API with sdk.h (rust_ingest_document and more, all catch_unwind-guarded so panics never crash the host), plus the Python SDK and LangChain connector. ISVs embed the pipeline into their own products in a few lines of code.