Enterprise Batch Compression: How to Batch Process 10000 Files

BLUF: For batch compression of 10000 enterprise files, the core solution is the "task queue + parallel compression + checkpoint recovery" trio. SmartSlim Enterprise uses Celery task queues for distribution, 12 concurrent parallel processing, and MySQL for checkpoint state recording. In testing, a 500GB document library was compressed to 82GB in 4 hours, an 83.6% compression ratio. This article details the challenges, solution architecture, real-world case study, and deployment options.

If you're not yet familiar with CLI/GUI/API selection for batch compression tools, we recommend first reading Batch Compression Tool Selection: CLI vs GUI vs API.

1. Three Major Challenges of Enterprise Batch Compression

Enterprise batch compression and personal compression are entirely different in scale. For personal use, compressing 100 files takes a few minutes with GUI drag-and-drop; for enterprises, compressing 10000 files faces triple challenges of scale, format diversity, and security. Understanding the challenges is essential for designing the right solution.

The most critical factors are file volume and stability. Processing 10000 files serially, even at 10 seconds per file, would take 28 hours — clearly unacceptable. Parallel compression reduces 28 hours to 2–4 hours, but introduces complexity in task scheduling, resource contention, and exception isolation. Checkpoint recovery is the baseline for stability: if a 500GB task crashes at the 2nd hour, without checkpoint recovery you'd have to start over.

2. Enterprise Batch Compression Solution Architecture

SmartSlim Enterprise's enterprise-grade batch compression solution uses a five-layer architecture: frontend presentation layer → API gateway layer → business logic layer → core algorithm layer → data storage layer. The core capabilities are concentrated in the business logic layer's task queue and the core algorithm layer's parallel compression.

Task sharding is the prerequisite for parallelism. 10000 files are not submitted to the compression engine all at once, but split into 200 batches (50 each) by the Celery task queue, with 12 Worker processes consuming the queue in parallel. Each batch is submitted independently with independent state recording, so a single batch failure doesn't affect other batches.

Checkpoint recovery implementation: before each file is compressed, a "pending" status is written to MySQL; when compression completes, the status is updated to "completed" with the compressed size and hash recorded. On service restart, the system scans the status table, skips "completed" files, and continues from the "pending" queue. This mechanism was proven effective in the 500GB case study — recovery after interruption at the 2nd hour cost only 12 minutes of overhead.

Three deployment options, chosen by enterprise scale and budget.

Most enterprises can meet their needs with Docker Compose single-machine deployment — an 8-core 16GB server with 12 concurrent tasks, processing 10000 files per day. K8s clusters are only needed beyond this scale. Classified or Xinchuang environments require private physical server deployment, with data never leaving the intranet.

3. Case Study: 500GB Document Library Compressed to 82GB

A manufacturing enterprise needed to archive and compress its historical document library. The library contained 500GB of files, approximately 12000 in total, covering PDF (35%), images (25%), Office documents (30%), video (5%), and others (5%). The requirement was to compress and store on an archive server, retaining for 3 years. SmartSlim Enterprise was used, deployed via Docker Compose on an 8-core 32GB server with 12 concurrent tasks.

Execution Parameters:

Time and Size Changes by Phase:

Results: 500GB compressed to 82GB, an 83.6% compression ratio, in 4 hours total. The process was interrupted once at 2 hr 10 min due to server memory fluctuation, with checkpoint recovery taking 12 minutes, for a final total of 4 hr 12 min. All file hash verifications passed, and compression logs completely recorded operators, timestamps, file hashes, and compression parameters, meeting enterprise archiving audit requirements. Video and Office documents had the highest compression ratios (83%/78%) because embedded resources have large compression potential; PDF compression ratio was 39% because some PDFs were already optimized scans.

4. Solution Recommendations by Scale and Scenario

Enterprise batch compression solutions aren't about bigger being better — they should match actual scale. The table below provides recommendations by file volume and scenario.

Selection advice: for under 1000 files per day, Server API suffices (Standard license, 4 concurrent) at the lowest cost; for 1000–10000 files, Enterprise single-machine (Professional license, 12 concurrent) offers the best value; K8s clusters are only needed beyond 10000 files. Classified units must use private deployment regardless of file volume, with data never leaving the intranet.

For compliance requirements in government and classified scenarios, refer to Government OA System Document Compression: OFD/PDF Batch Processing Solution. For more technical details on task queue design, see Compression Task Queue Design Explained.

5. Frequently Asked Questions (FAQ)

Q1: What solution for batch compression of 10000 enterprise files?

We recommend SmartSlim Enterprise, based on a task queue + parallel compression + checkpoint recovery architecture. 10000 files are distributed through Celery task queues, processed with 12 concurrent tasks, using MinIO storage and Redis cache. A single Server edition (12 concurrent) can handle it, with a daily limit of approximately 50000 files; beyond that, use K8s cluster HPA with 3-10 replicas for horizontal scaling. In testing, a 500GB document library was compressed to 82GB in 4 hours, an 83.6% compression ratio.

Q2: How long to compress a 500GB document library to 82GB?

Testing took 4 hours using SmartSlim Enterprise with 12 concurrent tasks, deployed on an 8-core 32GB server. It was divided into 3 phases: scan and classify 40 min, parallel compression 2 hr 50 min, verify and archive 30 min. Compression ratio was 83.6% (500GB to 82GB), averaging about 29 seconds per GB. If increased to 24 concurrent tasks, estimated time could be reduced to 2.5 hours. The process was interrupted once, with checkpoint recovery costing only 12 additional minutes.

Q3: What if batch compression is interrupted?

SmartSlim Enterprise supports checkpoint recovery. Each file's pre- and post-compression status is written to MySQL, and the task queue records processing progress. After interruption and service restart, the system automatically reads the checkpoint and continues from unprocessed files without reprocessing completed files. In testing, a 500GB task was interrupted at the 2nd hour and resumed from the checkpoint, with a total time of 4 hr 12 min (including 12 min recovery overhead). This is an essential capability for enterprise scenarios.

Q4: How do enterprise compression solutions ensure data security?

Four layers of protection: first, private deployment — data never leaves the enterprise intranet and doesn't pass through any third-party servers; second, 5-level security (DISABLED/LOW/MEDIUM/HIGH/MAXIMUM), default MEDIUM; third, 7 security capabilities including command parameter validation, file integrity verification, malicious code scanning, audit logging, rate limiting, secure temporary file management, and file size limits; fourth, compression log auditing — each compression records operator, time, file hash, and compression parameters, meeting MLPS 2.0 compliance requirements.

Summary

The core of enterprise batch compression is the "task queue + parallel compression + checkpoint recovery" trio, solving scale, efficiency, and stability problems. SmartSlim Enterprise, based on Celery task queues and the Rust compression engine, compressed a 500GB document library to 82GB in 4 hours, an 83.6% compression ratio, with checkpoint recovery after interruption costing only 12 additional minutes. The solution is tiered by scale: small-medium enterprises use Server API, medium enterprises use Enterprise single-machine, large enterprises use K8s clusters, and classified units must use private deployment.

Three points for enterprise selection: first, check daily file volume to determine deployment form (single-machine/cluster/private); second, check security compliance requirements to determine security level (default MEDIUM, MAXIMUM for classified); third, check whether audit logging is needed (required for MLPS 2.0). Choose the right solution, and batch compression of 10000 files is no longer an operational nightmare.