1. The Evolution of Video Codecs
The history of video coding stretches back to the late 1980s. From H.261, designed for video conferencing over ISDN lines, to AV1, engineered for ultra-high-definition streaming, codec standards have evolved over nearly three decades. Each generation pursues the same fundamental goal: reproduce the picture using fewer bits while preserving visual quality.
Video compression is essential because raw, uncompressed video is enormous. A ten-second 1080p clip in raw form can consume several gigabytes—untenable for either storage or transmission. Codecs exploit spatial and temporal redundancy within frames to shrink this data to a manageable size. Spatial redundancy refers to the correlation between neighboring pixels within a single frame; temporal redundancy refers to the similarity between consecutive frames. Virtually every modern codec standard revolves around these two kinds of redundancy.
Concretely, an encoder classifies pictures into three frame types: I-frames (intra-coded frames) record a full picture and can be decoded independently, offering the lowest compression but the strongest error resilience; P-frames (predictive frames) store only the differences from a preceding frame; B-frames (bidirectional frames) reference both preceding and following frames, achieving the highest compression. The group of frames between two I-frames forms a GOP (Group of Pictures)—a longer GOP yields better compression but weaker random access and error recovery. This inter-frame prediction framework has been in place since H.261; successive standards primarily refine block partitioning, prediction accuracy, and entropy coding.
H.261 established the block-based coding framework; MPEG-1 extended it to optical disc storage at roughly 1.5 Mbps, sufficient for VCD-quality video; H.263 optimized for low-bitrate communication, maintaining recognizable pictures below 64 kbps; H.264 opened the era of high-definition streaming; H.265 targeted 4K ultra HD; and AV1 represents the next-generation, royalty-free direction. Each generation attempts to improve compression efficiency over its predecessor while keeping encoding complexity within the reach of real-world hardware.
2. Technical Breakdown of Mainstream Codecs
H.264 (AVC)
Released in 2003, H.264 (Advanced Video Coding) uses macroblock partitioning with a 16×16 base block, supporting sub-blocks from 4×4 to 16×16. Motion estimation reaches quarter-pixel precision, and the standard introduces multi-reference frames, in-loop deblocking, Context-Adaptive Variable-Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC).
Compared to its predecessor H.263, H.264 roughly doubles compression efficiency—halving the bitrate at equivalent quality. CABAC saves 5–15% more bitrate than CAVLC but demands more computation, so low-end devices often opt for CAVLC. H.264 also supports multiple slices and Flexible Macroblock Ordering (FMO), facilitating parallel processing and error resilience.
H.264's strength is its mature ecosystem: virtually every modern device, browser, and player supports it natively. Its weakness is insufficient compression efficiency for 4K and beyond, resulting in higher bitrates. Thanks to ubiquitous hardware decoding, H.264 also excels in power efficiency on mobile devices, making it the most compatible codec available today.
H.265 (HEVC)
Released in 2013, H.265 (High Efficiency Video Coding) introduces the Coding Tree Unit (CTU), supporting blocks up to 64×64 with adaptive partitioning down to 8×8 based on picture complexity. At equivalent subjective quality, H.265 reduces bandwidth by roughly 40–50% compared to H.264.
Key improvements include larger CTU sizes that boost coding efficiency, especially for large flat regions; 35 intra prediction directions (versus 9 in H.264) for finer texture representation; advanced motion vector merging (Merge) and Advanced Motion Vector Prediction (AMVP) that reduce the encoding overhead of motion information; Sample Adaptive Offset (SAO) that further suppresses ringing artifacts after in-loop filtering; and parallel processing tools such as Tiles and WPP that ease multi-core encoding.
These features make H.265 well-suited for 4K and 8K content distribution, but they also bring significantly higher computational complexity—H.265 encoding can take several times longer than H.264. Larger block sizes can actually increase overhead at low resolutions, so H.265's advantages only fully emerge at higher resolutions.
VP9
Developed by Google and released in 2013 as part of the WebM project, VP9 uses 64×64 superblocks and supports 32 intra prediction modes, along with advanced inter prediction and adaptive quantization. Its compression efficiency is comparable to H.265, and it is completely royalty-free.
VP9 borrows ideas from H.265 but simplifies the syntax and toolset to lower implementation difficulty. It supports up to three reference frames and allows mixing references of different resolutions to handle variable-resolution scenarios. VP9's main advantage is openness: any vendor can implement it without patent concerns. However, its hardware decoding ecosystem is relatively limited, so it is most active on the web, particularly in Chromium-based browsers. VP9 has been widely adopted by major video platforms as an efficient alternative to H.264, performing reliably in web video distribution.
AV1
Released in 2018 by the Alliance for Open Media (AOMedia), AV1 extends previous-generation techniques with 128×128 superblocks and more sophisticated prediction and transform tools, including intra block copy, affine motion compensation, palette mode, and 1/8-pixel motion precision. Compared to H.265, AV1 saves an additional 20–30% in bitrate at equivalent quality—roughly 60% savings versus H.264.
AOMedia was founded jointly by several industry players specifically to circumvent the fragmented patent licensing of H.265. AV1 builds on VP9, incorporates selected advanced tools from H.265, and adds numerous experimental techniques, giving it a far larger toolset than H.265. It is designed to be entirely royalty-free and has been gradually adopted by major streaming platforms. Encoding complexity is extremely high, and real-time encoding still relies on hardware acceleration. The latest GPUs and mobile chips are beginning to integrate AV1 hardware decoding, and hardware-assisted encoding is progressing. Widespread deployment across both on-demand and live streaming is expected over the next few years.
3. Patent Licensing Comparison
Licensing is a decisive factor in codec adoption. The table below compares the licensing situation of four mainstream formats:
| Codec | Licensor | Royalty Cost | Complexity |
|---|---|---|---|
| H.264 | MPEG-LA unified pool | Tiered, with annual cap | Low |
| H.265 | Multiple pools (MPEG-LA, HEVC Advance, etc.) | High, no unified cap | Very high |
| VP9 | Open source (BSD license) | Free | None |
| AV1 | Alliance for Open Media | Free | None |
Licensing is the primary reason H.265 adoption has lagged. H.264 is managed by MPEG-LA with a clear fee structure and a cap, so content providers can deploy it with predictable costs; H.265, by contrast, involves multiple competing patent pools with patent holders spread across different organizations, uncertain fees, and no unified ceiling. According to public reports, some content providers delayed H.265 deployment because they could not estimate licensing costs. This uncertainty has driven some browser vendors and open-source communities toward VP9 and AV1, and it directly motivated the formation of AOMedia, giving rise to a royalty-free open-source codec ecosystem.
4. Scenario-Based Recommendations
Different use cases have very different codec requirements. The table below summarizes recommendations for common scenarios:
| Use Case | Recommended Codec | Reason |
|---|---|---|
| Email attachment | H.264 | Best compatibility, no extra decoder needed |
| Web embedding | VP9 / H.264 | Native browser support; VP9 saves bandwidth |
| Mobile playback | H.264 / H.265 | Hardware decoding ubiquitous, battery-friendly |
| Streaming delivery | AV1 / H.265 | High compression lowers bandwidth cost |
| Long-term archival | H.265 / AV1 | Saves storage while preserving quality |
| Live streaming | H.264 | Fastest encoding, controllable latency |
Always verify the decoding capabilities of the target playback environment first, then weigh compression efficiency against encoding cost. For content that must reach the widest possible audience, H.264 remains the fallback; for bandwidth-sensitive distribution, newer codecs should take priority. In practice, many platforms adopt a multi-bitrate, multi-codec strategy: serving H.264 to cover all clients while providing AV1 or H.265 versions for devices that support newer codecs, balancing compatibility and efficiency.
5. FFmpeg Compression Comparison
Using the same 1080p source video, compress with each codec. Lower CRF (Constant Rate Factor) values yield higher quality; similar values generally indicate comparable quality.
# H.264 (CRF 23, medium preset)
ffmpeg -i input.mp4 -c:v libx264 -crf 23 -preset medium -an output_h264.mp4
# H.265 (CRF 28, medium preset)
ffmpeg -i input.mp4 -c:v libx265 -crf 28 -preset medium -x265-params log-level=error -an output_h265.mp4
# VP9 (CRF 32, unrestricted bitrate)
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 32 -b:v 0 -an output_vp9.webm
# AV1 (CRF 30, unrestricted bitrate)
ffmpeg -i input.mp4 -c:v libaom-av1 -crf 30 -b:v 0 -cpu-used 4 -an output_av1.mkv
Parameter notes: -crf controls quality, -preset/-cpu-used balance speed against compression ratio, -b:v 0 lets the encoder allocate bitrate based on CRF, and -an discards audio to focus the comparison on video compression. Note that the CRF scales are not uniform across encoders: H.264's CRF 23 and H.265's CRF 28 are roughly comparable in subjective quality, but not an exact match—rigorous comparisons should rely on objective metrics such as VMAF or SSIM.
In practice, at equivalent subjective quality, H.265, VP9, and AV1 produce progressively smaller files—but encoding time increases accordingly. AV1 encoding can take tens of times longer than H.264 and requires hardware or parallel optimization to be practical. In production batch transcoding environments, encoding speed is often just as important as compression ratio, so many platforms use AV1 only for offline transcoding while keeping H.264 for real-time scenarios.
6. Frequently Asked Questions
Q1: H.265 is technically superior—why has adoption been slower than H.264?
Primarily due to licensing. H.265 involves multiple patent pools with complex, uncapped fee structures, deterring browser vendors and content platforms. Some browser makers explicitly declined to support H.265 natively because of licensing concerns. Limited early hardware decoding support also slowed mobile adoption. Only in recent years have mainstream chips generally integrated H.265 hardware decoding, but web-side support remains inconsistent. By contrast, H.264 licensing is handled centrally by MPEG-LA with predictable fees, making it the most widely deployed codec standard to date.
Q2: When will AV1 fully replace H.265?
AV1 decoding is supported in next-generation hardware and major browsers, but real-time encoding remains computationally expensive. On-demand delivery has begun scaling up, but live streaming demands low encoding latency and still relies mainly on H.264. Widespread use in live streaming is expected only after hardware encoders mature and encoding speed optimizations improve. In the short term, a more likely scenario is AV1 and H.265 coexisting, with adaptive switching based on client decoding capability. AV1 encoding speed is also improving steadily—open-source encoders such as SVT-AV1 have substantially reduced encoding times, and further gains are expected.
Q3: Why is VP9 mostly confined to the web?
VP9's hardware decoding coverage is relatively limited, relying mainly on Chromium-based browsers' software decoding. On desktop players and native mobile apps, H.264 and H.265 enjoy better hardware support and lower power consumption. VP9 never built a cross-platform hardware decoding ecosystem like H.264, so it remains primarily a web video format. However, VP9 paved the way for AV1—many of its technical ideas continue and are upgraded in AV1. Some platforms serve both VP9 and AV1 versions simultaneously, gradually transitioning from VP9 to AV1.
7. Conclusion
Video codec development is a constant balance between compression efficiency and implementation cost. H.264, with its comprehensive compatibility, remains the safest choice today; H.265 excels in bandwidth-sensitive UHD scenarios, though licensing constraints limit adoption; VP9 and AV1 represent the royalty-free direction, with AV1 emerging as the core of next-generation streaming.
When selecting a codec, consider the target playback environment, bandwidth cost, storage capacity, and computational overhead rather than chasing the newest standard. Finding the right balance between compatibility, efficiency, and cost is the engineering optimum. Understanding the applicable boundaries of each codec is more valuable than remembering which one is newest. As hardware support for AV1 matures and encoding tools continue to optimize, the cost gap between generations will narrow—yet the fundamental tradeoffs between compatibility, efficiency, and licensing will persist. Staying informed about these dynamics is the best way to make sound codec decisions over time.
Related Reading: