Why Batch Compress Images
You have 1000 high-resolution photos exported from a camera, each around 8MB, and the entire directory consumes nearly 8GB. If the goal is uploading to a website, embedding into a presentation, or sending via email, that volume is entirely unacceptable. Compressing them one by one in a graphical editor is both inefficient and error-prone—this is exactly where scripted batch processing shines.
Python's Pillow library is the de facto standard for image processing. It provides read and write support for mainstream formats including JPEG, PNG, WebP, BMP, and TIFF, along with fine-grained control over quality factors, sampling rates, and color modes. With a few dozen lines of Python, you can traverse an entire directory tree, apply a uniform compression strategy, and preserve the original folder structure in the output.
This article starts from Pillow's basic usage and progressively builds a complete, runnable, multithreaded batch compression script. It also covers practical engineering details such as EXIF preservation, format conversion, and intelligent parameter selection.
Pillow Installation and Basic Usage
Installation
Pillow is the actively maintained fork of PIL (Python Imaging Library), but the import name remains PIL. Install it with pip:
pip install Pillow
If you need to handle a large volume of images or want faster decoding, install the full version with plugin support:
pip install "Pillow[all]"
After installation, verify the version:
from PIL import Image, __version__
print(__version__) # e.g., 10.4.0
Basic Operations
The core entry point of Pillow is the Image class. The three most common operations are open, modify, and save:
from PIL import Image
# Open an image (lazy loading; pixel data is read on first access)
img = Image.open("photo.jpg")
print(img.format, img.size, img.mode) # JPEG (4000, 3000) RGB
# Get a pixel value
pixel = img.getpixel((100, 100))
# Resize
resized = img.resize((1920, 1080))
# Save
resized.save("photo_small.jpg", quality=85)
A few details are worth noting: the object returned by Image.open() holds a file reference that is not released until explicitly closed or the with block exits. In batch processing, always use a context manager or call close() explicitly, otherwise file handles will leak.
Single Image Compression Examples
The foundation of batch compression is single-image processing. We first implement a function that supports JPEG quality adjustment, resolution downscaling, and format conversion.
JPEG Quality Adjustment
The quality parameter of JPEG ranges from 1 to 100 and directly controls the scaling of the quantization table. It is the primary knob for balancing file size and image quality:
from PIL import Image
def compress_jpeg(input_path, output_path, quality=75):
"""Compress a JPEG with the specified quality factor"""
with Image.open(input_path) as img:
# JPEG does not support transparency; RGBA/P modes must be converted first
if img.mode in ("RGBA", "P", "LA"):
img = img.convert("RGB")
elif img.mode != "RGB":
img = img.convert("RGB")
img.save(
output_path,
format="JPEG",
quality=quality,
optimize=True, # Enable Huffman encoding optimization
progressive=True, # Generate progressive JPEG; slightly larger but better loading experience
)
# Compare different quality factors
compress_jpeg("photo.jpg", "q95.jpg", quality=95)
compress_jpeg("photo.jpg", "q75.jpg", quality=75)
compress_jpeg("photo.jpg", "q50.jpg", quality=50)
optimize=True makes Pillow scan the data an extra time during saving to rebuild an optimal Huffman encoding table. This typically reduces the file size by another 2-5% at the cost of slightly slower saving.
Resolution Adjustment
For web display, a 4000x3000 original is far larger than necessary. Using thumbnail() to downscale proportionally is the most effective approach—halving the resolution reduces the file size by roughly 75%:
from PIL import Image
def resize_image(input_path, output_path, max_size=1920, quality=80):
"""Scale the longest edge to max_size while preserving the aspect ratio"""
with Image.open(input_path) as img:
# thumbnail modifies in place and never exceeds the specified dimensions
# Image.Resampling.LANCZOS is the best-quality downsampling algorithm
img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
if img.mode != "RGB":
img = img.convert("RGB")
img.save(output_path, format="JPEG", quality=quality, optimize=True)
resize_image("photo.jpg", "photo_1920.jpg", max_size=1920, quality=80)
The difference between thumbnail() and resize() is that thumbnail() preserves the aspect ratio and never enlarges the image—the final dimensions are always less than or equal to the specified values. resize() forces the image to the exact target dimensions. In batch processing, you almost always want thumbnail().
Format Conversion
Converting PNG to WebP is a common optimization—WebP is 26% smaller than PNG and 25-35% smaller than JPEG at equivalent quality, while also supporting transparency:
from PIL import Image
def convert_to_webp(input_path, output_path, quality=80, lossless=False):
"""Convert any format to WebP"""
with Image.open(input_path) as img:
img.save(
output_path,
format="WebP",
quality=quality,
lossless=lossless, # When True, uses lossless mode
method=6, # Compression effort 0-6; 6 is slowest but smallest
)
convert_to_webp("icon.png", "icon.webp", quality=85)
convert_to_webp("logo.png", "logo_lossless.webp", lossless=True)
The method parameter controls compression effort, ranging from 0 to 6. Higher values yield better compression ratios but take longer. For batch processing where ultimate file size matters, set it to 6; for speed, 4 is a good balance.
Complete Batch Compression Script
Now we assemble the single-image logic into a complete batch script. This script traverses all images in the input directory, preserves the original folder structure in the output, and displays processing progress along with the overall compression results.
The flowchart below illustrates the overall batch processing workflow:
The complete script:
"""
batch_compress.py - Batch image compression script
Usage: python batch_compress.py <input_dir> <output_dir> [--quality 75] [--max-size 1920]
"""
import os
import sys
import argparse
from pathlib import Path
from PIL import Image
SUPPORTED_FORMATS = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".tiff", ".tif"}
def compress_single(input_path, output_path, quality=75, max_size=1920,
target_format=None, keep_exif=True):
"""Compress a single image"""
with Image.open(input_path) as img:
# 1. Extract EXIF data (capture time, camera model, GPS, etc.)
exif_data = img.info.get("exif", b"") if keep_exif else b""
# 2. Color mode conversion: JPEG does not support transparency
out_format = (target_format or img.format or "JPEG").upper()
if out_format in ("JPEG", "JPG") and img.mode in ("RGBA", "P", "LA"):
# Composite onto a white background to avoid black regions after conversion
background = Image.new("RGB", img.size, (255, 255, 255))
if img.mode == "P":
img = img.convert("RGBA")
background.paste(img, mask=img.split()[-1] if img.mode == "RGBA" else None)
img = background
elif img.mode not in ("RGB", "RGBA", "L"):
img = img.convert("RGB")
# 3. Resolution adjustment: only downscale, never upscale
if max_size and max(img.size) > max_size:
img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
# 4. Save
save_kwargs = {"optimize": True}
if out_format in ("JPEG", "JPG"):
save_kwargs["quality"] = quality
save_kwargs["progressive"] = True
if exif_data:
save_kwargs["exif"] = exif_data
elif out_format == "WEBP":
save_kwargs["quality"] = quality
save_kwargs["method"] = 6
elif out_format == "PNG":
save_kwargs["optimize"] = True
img.save(output_path, format=out_format, **save_kwargs)
def batch_compress(input_dir, output_dir, quality=75, max_size=1920,
target_format=None, keep_exif=True):
"""Batch compress an entire directory"""
input_dir = Path(input_dir)
output_dir = Path(output_dir)
# Collect all files to process
tasks = []
for root, _, files in os.walk(input_dir):
for filename in files:
if Path(filename).suffix.lower() in SUPPORTED_FORMATS:
src = Path(root) / filename
rel = src.relative_to(input_dir)
dst = output_dir / rel
# Adjust extension based on target format
if target_format:
dst = dst.with_suffix(f".{target_format.lower()}")
tasks.append((src, dst))
total = len(tasks)
if total == 0:
print("No supported image files found")
return
print(f"Found {total} images. Starting compression...\n")
processed = 0
total_original = 0
total_compressed = 0
errors = []
for src, dst in tasks:
# Ensure output directory exists
dst.parent.mkdir(parents=True, exist_ok=True)
original_size = src.stat().st_size
try:
compress_single(src, dst, quality, max_size, target_format, keep_exif)
new_size = dst.stat().st_size
total_original += original_size
total_compressed += new_size
except Exception as e:
errors.append((src, str(e)))
new_size = original_size # Count as original size on failure
processed += 1
pct = processed / total * 100
saved = (1 - new_size / original_size) * 100 if original_size else 0
print(f"[{pct:5.1f}%] ({processed}/{total}) {src.relative_to(input_dir)} "
f"{original_size // 1024}KB -> {new_size // 1024}KB (-{saved:.1f}%)")
# Summary report
print(f"\n{'=' * 50}")
print(f"Done: {processed} succeeded, {len(errors)} failed")
if total_original > 0:
ratio = total_original / total_compressed if total_compressed else 0
saved_mb = (total_original - total_compressed) / 1024 / 1024
print(f"Original size: {total_original / 1024 / 1024:.2f} MB")
print(f"Compressed size: {total_compressed / 1024 / 1024:.2f} MB")
print(f"Space saved: {saved_mb:.2f} MB (ratio {ratio:.2f}x)")
for src, err in errors:
print(f" Failed: {src} - {err}")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Batch image compression tool")
parser.add_argument("input_dir", help="Input image directory")
parser.add_argument("output_dir", help="Output directory")
parser.add_argument("--quality", type=int, default=75, help="JPEG/WebP quality factor (1-100)")
parser.add_argument("--max-size", type=int, default=1920, help="Maximum longest edge in pixels")
parser.add_argument("--format", default=None, help="Target format (JPEG/WebP/PNG)")
parser.add_argument("--no-exif", action="store_true", help="Discard EXIF data")
args = parser.parse_args()
batch_compress(
args.input_dir,
args.output_dir,
quality=args.quality,
max_size=args.max_size,
target_format=args.format,
keep_exif=not args.no_exif,
)
Usage examples:
# Basic usage
python batch_compress.py ./photos ./output
# Convert to WebP, quality 80, max longest edge 2560
python batch_compress.py ./photos ./output --quality 80 --max-size 2560 --format webp
This script has three core design principles: it uses os.walk() to recursively traverse and preserve the directory structure; it uses relative_to() to compute relative paths ensuring the output layout matches the input; and it uses Path.stat() to obtain file sizes for calculating the actual compression effect.
Advanced Techniques
EXIF Information Preservation
EXIF data contains capture time, GPS coordinates, camera parameters, aperture, shutter speed, and other metadata. By default, img.save() discards this information. To preserve it, you must extract it manually and pass it when saving:
from PIL import Image
def compress_with_exif(input_path, output_path, quality=75):
with Image.open(input_path) as img:
exif = img.info.get("exif", b"") # Extract raw EXIF bytes
if img.mode != "RGB":
img = img.convert("RGB")
img.save(
output_path,
format="JPEG",
quality=quality,
exif=exif if exif else None, # Pass None when there is no EXIF
optimize=True,
)
If you only need to retain certain EXIF fields (for example, only the orientation tag), use the getexif() method to pick items individually:
from PIL import ExifTags
def keep_orientation_only(input_path, output_path, quality=75):
with Image.open(input_path) as img:
exif = img.getexif()
orientation = exif.get(0x0112) # Orientation tag
# Rotate the image based on the orientation tag
if orientation:
from PIL import ImageOps
img = ImageOps.exif_transpose(img)
img.save(output_path, format="JPEG", quality=quality, optimize=True)
ImageOps.exif_transpose() automatically rotates the image according to the EXIF orientation tag. This is especially useful when processing photos taken with smartphones—many phone sensors are mounted in landscape orientation and rely on the EXIF orientation tag to indicate the correct display direction.
PNG to WebP
When converting PNG to WebP, you need to choose between lossy and lossless modes based on the image characteristics. For images with text and sharp lines (icons, UI elements), lossless WebP preserves crisp edges. For photographic content, lossy WebP offers a significant size advantage:
from PIL import Image
def png_to_webp(input_path, output_path, lossless_threshold=0.3):
"""Intelligently choose lossy or lossless WebP based on image characteristics"""
with Image.open(input_path) as img:
# Count colors to determine if this is a simple graphic
colors = img.getcolors(maxcolors=65536)
is_simple = colors is not None and len(colors) < 256
if is_simple:
# Few colors, sharp edges: lossless mode
img.save(output_path, format="WebP", lossless=True, method=6)
print(f"Lossless WebP (color count: {len(colors)})")
else:
# Rich colors, photographic: lossy mode
if img.mode != "RGB":
img = img.convert("RGB")
img.save(output_path, format="WebP", quality=82, method=6)
print("Lossy WebP (photographic)")
getcolors() returns a list of colors or None (when the color count exceeds maxcolors). A small number of colors usually indicates a simple graphic such as an icon or screenshot, which is better suited for lossless compression.
Smart Parameter Selection
Different images are best suited for different compression strategies. The following function automatically selects the format and parameters based on resolution, transparency, and color count:
from PIL import Image
def smart_compress(input_path, output_path):
"""Automatically select the optimal compression strategy based on image characteristics"""
with Image.open(input_path) as img:
width, height = img.size
pixel_count = width * height
mode = img.mode
has_transparency = mode in ("RGBA", "LA") or (
mode == "P" and "transparency" in img.info
)
# Strategy 1: Has transparency -> WebP (balances transparency and size)
if has_transparency:
if pixel_count > 2_000_000:
q = 80
else:
q = 88
img.save(output_path, format="WebP", quality=q, method=6)
return "Lossy WebP (transparent)"
# Strategy 2: Very large photo -> downscale + JPEG medium quality
if pixel_count > 8_000_000 or width > 4000:
img = img.convert("RGB")
img.thumbnail((2560, 2560), Image.Resampling.LANCZOS)
img.save(output_path, format="JPEG", quality=75, optimize=True)
return "JPEG (large image downscaled)"
# Strategy 3: Normal photo -> JPEG higher quality
if pixel_count > 500_000:
img = img.convert("RGB")
img.save(output_path, format="JPEG", quality=82, optimize=True)
return "JPEG (high quality)"
# Strategy 4: Small image -> keep as PNG
img.save(output_path, format="PNG", optimize=True)
return "PNG (small image preserved)"
The core logic of this strategy is: images with transparency go to WebP, large images are downscaled before compression, normal photos use JPEG, and small images use PNG for fidelity. You can adjust the thresholds based on your actual requirements.
Performance Optimization
Multithreaded Processing
Image compression is a task that mixes I/O-bound and CPU-bound work. Disk read/write and encoding each account for a portion of the time. Python's GIL limits pure CPU parallelism, but Pillow releases the GIL when calling the underlying C libraries for JPEG/WebP encoding, so multithreading can bring substantial speedups:
import os
from pathlib import Path
from PIL import Image
from concurrent.futures import ThreadPoolExecutor, as_completed
import threading
SUPPORTED_FORMATS = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".tiff"}
_print_lock = threading.Lock()
def compress_single(input_path, output_path, quality=75, max_size=1920):
with Image.open(input_path) as img:
exif = img.info.get("exif", b"")
if img.mode != "RGB":
img = img.convert("RGB")
if max_size and max(img.size) > max_size:
img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
kwargs = {"quality": quality, "optimize": True, "progressive": True}
if exif:
kwargs["exif"] = exif
img.save(output_path, format="JPEG", **kwargs)
def batch_compress_threaded(input_dir, output_dir, quality=75, max_size=1920,
workers=4):
input_dir = Path(input_dir)
output_dir = Path(output_dir)
# Collect tasks
tasks = []
for root, _, files in os.walk(input_dir):
for f in files:
if Path(f).suffix.lower() in SUPPORTED_FORMATS:
src = Path(root) / f
rel = src.relative_to(input_dir)
dst = output_dir / rel
dst.parent.mkdir(parents=True, exist_ok=True)
tasks.append((src, dst))
total = len(tasks)
print(f"{total} images total, processing with {workers} threads\n")
completed = 0
total_saved = 0
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = {
executor.submit(compress_single, src, dst, quality, max_size): (src, dst)
for src, dst in tasks
}
for future in as_completed(futures):
src, dst = futures[future]
try:
future.result()
saved = src.stat().st_size - dst.stat().st_size
total_saved += saved
except Exception as e:
with _print_lock:
print(f"Failed: {src.name} - {e}")
completed += 1
with _print_lock:
print(f"[{completed / total * 100:5.1f}%] ({completed}/{total}) {src.name}")
print(f"\nDone. Space saved: {total_saved / 1024 / 1024:.2f} MB")
if __name__ == "__main__":
batch_compress_threaded("./photos", "./output", quality=75, max_size=1920, workers=4)
The empirical rule for thread count is the number of CPU logical cores. On an 8-core machine, 4-6 threads typically achieve near-optimal speedup. Too many threads can actually degrade performance due to disk I/O contention and memory pressure.
If you need to go further with parallelism, you can use ProcessPoolExecutor to bypass the GIL. The tradeoff is that processes cannot share memory, each process must independently load the Pillow library, and startup overhead is higher.
Memory Management
Memory consumption becomes a concern when processing high-resolution images. A single 6000x4000 RGB image occupies about 72MB of memory after decoding (6000x4000x3 bytes). If 10 are open simultaneously, that is 720MB.
Several key memory management principles:
from PIL import Image
import gc
# 1. Always use the with statement to ensure file handles and pixel buffers are released promptly
def safe_compress(input_path, output_path, quality=75):
with Image.open(input_path) as img:
# 2. Copy the image if you need to modify it, avoiding operations on the original
work = img.copy()
# img is released; work is still in scope
if work.mode != "RGB":
work = work.convert("RGB")
work.save(output_path, format="JPEG", quality=quality, optimize=True)
work.close() # 3. Close explicitly
# 4. Trigger garbage collection periodically during batch processing
def batch_with_gc(tasks, interval=100):
for i, task in enumerate(tasks):
process(task)
if i % interval == 0:
gc.collect()
Additionally, Image.MAX_IMAGE_PIXELS defaults to a limit of approximately 178 million pixels (about 8900x8900). Exceeding this raises a DecompressionBombError. If you genuinely need to process very large images, you can raise this threshold, but be mindful of memory consumption:
Image.MAX_IMAGE_PIXELS = None # Disable the limit (use with caution)
Compression Results Comparison
The table below shows measured data from a set of 50 smartphone photos (original average 6.5MB, 4000x3000 pixels). All tests ran on the same hardware, with resolution uniformly downscaled to a longest edge of 1920:
| Quality Factor | Format | Average Size | Ratio | Subjective Quality | Use Case |
|---|---|---|---|---|---|
| Original | JPEG | 6.5 MB | 1.0x | Baseline | — |
| 95 | JPEG | 1.8 MB | 3.6x | Virtually identical | High-quality archival |
| 85 | JPEG | 0.92 MB | 7.1x | Imperceptible difference | Website hero images |
| 75 | JPEG | 0.58 MB | 11.2x | Slight artifacts on close inspection | Thumbnails, listing pages |
| 65 | JPEG | 0.42 MB | 15.5x | Visible blocky artifacts | Not recommended |
| 50 | JPEG | 0.28 MB | 23.2x | Obvious distortion | Tiny previews only |
| 80 | WebP | 0.51 MB | 12.7x | Comparable to JPEG 85 | Modern websites |
| Lossless | WebP | 4.2 MB | 1.5x | Perfectly lossless | When transparency is needed |
Several patterns emerge from the data:
- Quality 85 is the cost-effectiveness inflection point: it saves nearly half the size compared to 95, with quality differences that are virtually imperceptible to the eye.
- Below quality 70, returns diminish rapidly: file size continues to decrease, but quality degradation accelerates noticeably.
- WebP quality factors are not directly equivalent to JPEG: WebP 80 produces quality comparable to JPEG 85 but at a smaller size.
- Lossless WebP only makes sense when transparency is required; for photographs, the size advantage is minimal.
FAQ
Q1: Why do images appear upside down or sideways after compression?
This happens because the EXIF orientation tag from the smartphone is discarded during compression, while the actual pixel data was stored in landscape orientation. The solution is to call ImageOps.exif_transpose(img) before saving. It rotates the pixel data according to the EXIF orientation tag so the image is correctly oriented before being saved. This way, even if downstream software does not read the EXIF orientation information, the image will display correctly.
Q2: Why does memory usage keep growing when processing 1000 images?
The most common cause is failing to close image objects promptly. Always use the with Image.open(...) as img: context manager to ensure each image is released immediately after processing. Second, in the multithreaded version, if all tasks are submitted at once, the future objects hold references to results until they are consumed, causing memory to accumulate. The solution is to submit tasks in batches using chunksize, or switch to a producer-consumer queue pattern to control concurrency. Finally, calling gc.collect() periodically can reclaim memory held by circular references.
Q3: Why do transparent areas turn black after converting PNG to JPEG?
The JPEG format does not support transparency. When Pillow converts RGBA to RGB by default, it fills the transparent background with black. The solution is to first create a white-background RGB image, then use paste() to overlay the original image with the alpha channel as a mask:
background = Image.new("RGB", img.size, (255, 255, 255))
background.paste(img, mask=img.split()[3]) # The 4th channel is alpha
Alternatively, convert directly to WebP, which natively supports transparency without any extra handling.
Q4: Why doesn't multithreading deliver a linear speedup?
Image compression involves both disk I/O and CPU encoding. Once the thread count increases beyond a certain point, disk bandwidth becomes the bottleneck—the CPU cannot go faster than the data can be read and written. Additionally, decoded pixel data for high-resolution images is large, and processing multiple large images simultaneously in different threads leads to memory bandwidth contention and reduced cache hit rates. In practice, 4-8 threads typically yield a 2-4x speedup, with diminishing returns beyond that. For mechanical hard drives, 4 threads is recommended; for NVMe SSDs, the count can be increased somewhat.
Summary
Batch compressing images with Python Pillow can be broken down into three layers:
- Foundation layer: Master the three methods
Image.open(),thumbnail(), andsave(), combined with thequalityandoptimizeparameters, and you can compress a single image. This is the building block for all batch processing. - Engineering layer: Use
os.walk()to traverse directories,Path.relative_to()to preserve structure, andconcurrent.futuresfor parallel processing to scale single-image logic to thousands of images. This layer focuses on correctness, robustness, and throughput. - Optimization layer: Intelligently select formats and parameters based on image characteristics—transparent images go to WebP, large images are downscaled first, small images stay as PNG. Details like EXIF preservation, orientation correction, and memory management determine whether the script can run reliably in a production environment.
There is an empirical rule for quality factor selection: 85 is the threshold of visually lossless quality, 75 is the balance point between size and quality, and below 65 is not recommended for display purposes. WebP is now fully supported across modern browsers and should be the preferred format for new projects.
The value of this script lies not in replacing professional tools but in its customizability. You can adjust every stage according to your actual needs—configure different parameters for specific directories, automatically upload to a CDN after compression, or integrate it into a CI/CD pipeline. Understanding the principles behind each line of code gives you the foundation to handle any image processing scenario.
Related Reading: