Compression Task Queue Design: Celery+Redis Async Processing

Bottom line: The standard solution for async compression tasks is a Celery + Redis task queue, with the core architecture being Producer (FastAPI dispatches tasks) → Broker (Redis buffers) → Worker (Celery consumes and calls the Rust compression engine) → Backend (Redis stores results). For batch compression of 10,000 files, sharding into 100-file batches with 12 concurrent Workers completes in approximately 40 minutes. Key design points include task sharding, priority queues, exponential backoff retries, and dead letter queue fallback. Below we start from queue architecture, providing Celery configuration parameter tables and a complete practical solution.

If you're not yet familiar with overall compression service deployment, we recommend first reading Complete Guide to Deploying Compression Services with Docker.

1. Why Compression Tasks Need Async Queues

File compression is a classic CPU-intensive + IO-intensive task. Compressing a 100MB PDF may take 5–15 seconds; if handled synchronously, the HTTP connection hangs for a long time, resulting in very poor single-machine concurrency. Async queues decouple "submission" from "execution": the client submits a task and immediately receives a task_id, while the Worker compresses in the background and returns results via callback or polling upon completion.

Processing Method	Concurrency	Response Latency	Failure Handling	Scale
Synchronous	Poor (blocks HTTP)	5–60s	No retry, direct error	<10 files
Thread pool	Medium (limited by thread count)	1–5s	Manual implementation	10–100 files
Celery async queue	High (horizontal Worker scaling)	<200ms	Auto-retry + dead letter queue	100–100,000 files
Kubernetes + queue	Very high (HPA elastic scaling)	<100ms	Complete fault tolerance	>100,000 files

SmartSlim Network Edition uses a FastAPI + Celery + Redis + MinIO architecture, running stably with 12 concurrent tasks on a single machine. In Kubernetes deployments, HPA can auto-scale between 3–10 replicas. This architecture has supported multiple enterprise customers with daily volumes of tens of thousands of files.

2. Celery Task Queue Architecture in Detail

The Celery task queue consists of four core roles: Producer, Broker, Worker, and Backend. Understanding the responsibilities of each layer is essential for correctly configuring a compression task queue.

Component	Technology	Responsibility	Key Config
Producer	FastAPI	Receive HTTP requests, construct tasks, dispatch to Broker	task.apply_async(queue=...)
Broker	Redis 7.x	Buffer pending task messages, support priority queues	broker_url, visibility_timeout
Worker	Celery 5.x	Consume tasks, call Rust compression engine	concurrency, prefork pool
Backend	Redis	Store task status and return results	result_backend, result_expires
Storage	MinIO	Store original and compressed files	S3-compatible protocol, multipart upload

2.1 Celery Core Configuration Parameters

Celery configuration directly determines queue throughput and stability. The table below shows recommended settings for compression scenarios, validated in SmartSlim production environments.

Parameter	Recommended Value	Description
broker_url	redis://:password@redis:6379/0	Redis as message broker, separate DB to avoid conflicts
result_backend	redis://:password@redis:6379/1	Result storage in separate DB, isolated from Broker
task_serializer	json	JSON serialization, cross-language compatible
result_serializer	json	Results also use JSON
accept_content	['json']	Accept JSON only, security hardening
timezone	Asia/Shanghai	Unified timezone
task_acks_late	True	ACK only after task completion, no task loss on crash
worker_prefetch_multiplier	1	Each Worker prefetches only 1 task, preventing long-task starvation
task_time_limit	600	Hard timeout 600s per task
task_soft_time_limit	540	Soft timeout 540s, triggers SoftTimeLimitExceeded
task_reject_on_worker_lost	True	Reject task on Worker crash, re-queue
result_expires	86400	Results auto-cleaned after 24 hours

2.2 Priority Queue Design

Compression tasks vary in priority: real-time user-submitted requests need fast response, while scheduled archival tasks can run slowly. Redis priority queues enable differentiated scheduling, with higher-priority tasks consumed by Workers first.

Queue Name	Priority	Routing Rule	Typical Task
compression_high	9 (highest)	Real-time user requests	Single-file instant compression
compression_normal	5 (default)	Batch tasks	Batch compress 100–500 files
compression_low	1 (lowest)	Scheduled archival	Nightly full archival compression
dlq_queue	— (dead letter)	Failed retries	Manual investigation or compensation

3. Case Study: Async Compression of 10,000 Files

This is an enterprise data archival scenario: 10,000 historical documents (mixed PDF/Word/images, averaging 8MB per file, totaling approximately 80GB) need unified compression and archival. Requirements: complete within 1 hour, compression ratio no less than 60%.

Solution Design: Split into 100 subtasks of 100 files each, dispatch in batch using Celery group, with 12 Workers consuming concurrently. Each subtask sequentially calls the Rust compression engine to compress 100 files.

Execution Parameters and Throughput:

Metric	Parameter	Measured Value	Description
Total files	10,000	—	Mixed formats, avg 8MB each
Shard size	100 files/batch	100 subtasks	Balances scheduling overhead and retry cost
Worker concurrency	12	prefork mode	Single machine, 12-core CPU
Per-file compression time	—	Avg 3.2s	Rust engine medium preset
Per-batch time	—	~5.3 min	100 files sequential
Total time	—	~44 min	100 batches / 12 concurrent
Compression ratio	—	67.3%	80GB→26.2GB
Failures	—	17	Corrupted files, entered DLQ after retry

Result: 10,000 files compressed in 44 minutes, compression ratio 67.3%, with 17 corrupted files automatically entering the dead letter queue for manual processing. The overall architecture was stable, with peak CPU utilization at 89%, peak memory usage at 4.2GB, and no OOM or task loss.

3.1 Retry and Dead Letter Queue Strategy

Compression task failures fall into two categories: transient errors (IO timeout, out of memory, excessive concurrency) and deterministic errors (corrupted files, unsupported formats). Transient errors have a high probability of success on retry, while retrying deterministic errors is pointless. The table below outlines retry and dead letter decision strategies.

Error Type	Typical Exception	Strategy	Retries	Final Destination
Transient-IO	ConnectionError, TimeoutError	Exponential backoff retry	3	Success or DLQ
Transient-Resource	MemoryError, OOMKilled	Extended backoff + degradation	2	Success or DLQ
Deterministic-File	FileCorrupted, ParseError	No retry, direct DLQ	0	dlq_queue
Deterministic-Format	UnsupportedFormat	No retry, direct DLQ	0	dlq_queue
Deterministic-Permission	PermissionDenied	No retry, alert	0	dlq_queue + alert

Retry configuration uses Celery's autoretry_for and retry_backoff, with initial backoff of 60s, maximum 600s, and random jitter to prevent cascading failures. Tasks in the dead letter queue are periodically scanned by an independent monitoring task, triggering WeCom/DingTalk alerts to notify operations for handling.

Monitoring Metric	Collection Method	Alert Threshold	Action
Queue backlog	Redis LLEN	>500	Trigger Worker scaling
Task failure rate	Celery events	>5%	Investigate logs + pause dispatch
DLQ length	Redis LLEN dlq	>10	WeCom alert
Active Workers	Celery inspect	<10	Auto-restart Worker
Avg task duration	Flower monitoring	>30s	Check large files + degrade
CPU utilization	node_exporter	>95%	Throttle + scale

For the complete compression API usage, see Compression API Guide: REST Interface Design.

4. Queue Configuration Recommendations for Different Scenarios

Different business scenarios have different requirements for throughput, latency, and reliability, requiring differentiated queue configurations. The table below provides recommended settings for common scenarios.

Scenario	Workers	Shard Size	Priority Queue	Retry Strategy
Personal instant compression	2	No sharding	high	Fast retry 3x
Enterprise batch archival	12	100 files/batch	normal/low	Exponential backoff 3x
Government classified processing	4	50 files/batch	high	Strict retry + audit
E-commerce platform images	16	200 files/batch	normal	Fast retry 2x
Nightly scheduled archival	8	500 files/batch	low	Slow backoff 5x
Real-time video transcoding	24	Single file	high	No retry, alert on failure

A general principle: real-time scenarios use high-priority queues + small shards + fast retries; batch scenarios use normal queues + large shards + exponential backoff; classified scenarios use strict auditing + small shards + multi-level retries. For the complete enterprise batch compression solution, see Enterprise Batch Compression: 10K+ File Processing in Practice.

5. FAQ

Q1: How to implement async processing for Celery compression tasks?

Build an async task queue with Celery + Redis: FastAPI receives requests and dispatches tasks to the Redis Broker, Celery Workers consume tasks from the Broker and call the Rust compression engine to perform compression, and results are written to the Backend and MinIO storage. A single task.apply_async triggers async execution, and status is polled via task.id. With 12 concurrent Workers on a single machine, 10,000 files sharded into 100 batches can be completed in 40 minutes.

Q2: How to auto-retry failed compression tasks?

Use Celery's autoretry_for parameter to configure automatic retries: set max_retries=3, retry_backoff=True (exponential backoff, initial 60s), retry_backoff_max=600s, and retry_jitter=True (random jitter to prevent cascading failures). Tasks that fail after 3 retries are automatically routed to the dead letter queue dlq_queue for manual or compensating task handling. We recommend retrying transient errors (IO timeout/out of memory) and sending deterministic errors (corrupted files/unsupported formats) directly to the dead letter queue.

Q3: How to shard 10,000 files for batch compression?

Shard into 100 files per batch, yielding 100 subtasks. Use Celery group or chord for batch dispatch, with 12 Workers consuming in parallel. Each subtask compresses 100 files sequentially. Average compression time per file is 3s, about 5 minutes per batch, and 100 batches in parallel complete in approximately 40 minutes total. Too small a shard size (e.g., 1 per batch) incurs high scheduling overhead; too large (e.g., 1000 per batch) increases retry cost on failure. 100 is the empirically optimal value.

Q4: Which is better for compression task queues: Celery or RQ?

Celery is recommended for compression tasks. Celery supports task sharding (group/chord), priority queues, scheduled tasks, task chains, and dead letter queues — a complete feature set; RQ is lighter but lacks sharding and priority support. Compression scenarios commonly require batch sharding, priority scheduling, and failure retries, all natively supported by Celery. In terms of performance, both are Redis-based with comparable throughput. SmartSlim Network Edition uses a FastAPI + Celery + Redis + MinIO architecture, running stably with 12 concurrent workers on a single machine.

Summary

The standard solution for async compression tasks is a Celery + Redis task queue, with the core being Producer/Broker/Worker/Backend four-layer decoupling. For batch compression of 10,000+ files, sharding into 100-file batches with 12 concurrent Workers completes in approximately 40 minutes, with a compression ratio of 60%–70%. Retry strategies should distinguish transient errors (exponential backoff retry) from deterministic errors (direct to dead letter queue), supported by 6 monitoring metrics to ensure queue stability.

Remember three points: first, task_acks_late=True ensures no task loss on crash; second, worker_prefetch_multiplier=1 prevents long-task starvation; third, the dead letter queue must have monitoring and alerts configured. With the right queue architecture and sharding strategy, both throughput and stability of compression services can reach a new level.