Sixth MPEG-4 AVC/H.264 Video Codecs Comparison - Short Version

MSU Graphics & Media Lab (Video Group)

Project head: Dr. Dmitriy Vatolin

Measurements, analysis: Dr. Dmitriy Kulikov,
Alexander Parshin

History of versions

25th May 2010: Public report is released

16th June 2010: Appendix with VP8 encoder analysis is released

Different Versions of Report

There are four different versions of H.264 Comparison 2010 report:

MPEG-4 AVC/H.264 Video Codecs Comparison - Short Version (this report)

MPEG-4 AVC/H.264 Video Codecs Comparison - Pro Version (Movies Edition)

MPEG-4 AVC/H.264 Video Codecs Comparison - Pro Version (HDTV Edition)

MPEG-4 AVC/H.264 Video Codecs Comparison - Pro Version (Enterprise Edition)

Here is the comparison of the versions:

	Short Version	Pro Version (Movies)	Pro Version (HDTV)	Pro Version (Enterprise)
Objective Metrics (SSIM)
Additional Metrics (PSNR)		For all results SSIM and PSNR metrics are used	For all results SSIM and PSNR metrics are used	For all results SSIM and PSNR metrics are used
Movie use case	Only Normal preset	High Speed, Normal, High Quality presets		High Speed, Normal, High Quality presets
HDTV use case			High Speed, Normal, High Quality presets	High Speed, Normal, High Quality presets
Subjective Comparison				SAMVIQ analysis including psycho-visual enhancements
Additional Analysis				Synthetic Motion Analyzer
Additional Analysis				Distortion in Tail Area Analyzer
Additional Analysis				Spatially Variable Noise Analyzer
Number of pages	25	116	114	177
Number of figures	50	131	131	226
Prices	Free	$349	$349	$699
Purchase
Purchase	Hint: You can remove "Extended download" service while purchasing to save money.
We can help you to analyze your codec

Pro versions of report will be available immediately after report purchasing.

Html-Version of Report

Video Codecs that Were Tested

DivX H.264

Elecard H.264

Intel® MediaSDK AVC/H.264

MainConcept H.264

Microsoft Expression Encoder

Theora

x264

XviD (MPEG-4 ASP codec)

Overview

Sequences

Table 1. Summary of video sequences

Sequence	Number of frames	Frame rate	Resolution
Ice Age	2014	24	720x480
Indiana Jones	5000	30	704x288
State Enemy	6500	24	720x304
Up	1920	24	720x480
Amazon	1200	24	1280x720
Iron Man	600	24	1920x1080
Mobile Calendar	504	50	1280x720
Troy	300	24	1920x1072

Brief descriptions of the sequences used in our comparison are given in Table 1. More detailed descriptions of these sequences can be found in Appendix 4. Test Set of Video Sequences

Codecs

Table 2. Short codec descriptions

Codec	Developer	Version
DivX AVC/H.264 Video Encoder	DivX, Inc.	version 1.1.1.6
Elecard AVC Video Encoder 8-bit edition	Elecard Ltd	2.1.022202.091207
Intel® MediaSDK AVC/H.264 transcoder sample application	Intel Corp.	1.10.1.15
MainConcept AVC/H.264 Video Encoder Console Application	MainConcept GmbH	8.5.0.12837
Microsoft Expression Encoder 3	Microsoft Corp.	3.0.1332.0
Theora encoder	Xiph.Org.	Provided by developers
x264	x264 Development Team	x264 core:85 r1442M 781d300
XviD raw mpeg4 bitstream encoder	XviD Development Team	xvid-1.3.0-dev

Brief descriptions of the codecs used in our comparison are given in Table 2. XviD was used as a good quality MPEG-4 ASP reference codec for comparison purposes. Detailed descriptions of all codecs used in our comparison can be found in Appendix 5. Tested Codecs and Presets.

Every codec developer could take private participation in this compasrison
to test and analyze its codec.

Objectives and Testing Rules

H.264 Codec Testing Objectives

The main goal of this report is the presentation of a comparative evaluation of the quality of new H.264 codecs using objective measures of assessment. The comparison was done using settings provided by the developers of each codec.

The main task of the comparison is to analyze different H.264 encoders for the task of transcoding video—e.g., compressing video for personal use. Speed requirements are given for a sufficiently fast PC; fast presets are analogous to real-time encoding for a typical home-use PC.

Testing Rules

The entire test set was divided into two primary types of applications. These applications differ by resolution, bitrate and encoding speed requirements:

Movies (bitrates of 500-2000 kbps)

High-definition television (“HDTV”; bitrates of 0.7-10 mbps)

There are special presets and speed limitations for every type of application:

Movies (speed requirements for 750 kbps 4CIF sequences):

Minimum 120 fps for "High Speed" preset (1 pass, no B-frames, 1 reference frame)

Minimum 80 fps for "Normal" preset)

Minimum 40 fps for "High Quality" preset

HDTV (speed requirements for 3 mbps 1280x720 sequences):

Minimum 100 fps for "High Speed" preset (1 pass, no B-frames, 1 reference frame)

Minimum 50 fps for "Normal" preset

Minimum 20 fps for "High Quality" preset

Each codec’s developer provided settings for each type of application. Each setting’s individual parameters were, to a large extent, chosen by the developers, except the following:

DivX H.264

Microsoft Expression Encoder

Theora

Each codec was tested for speed three times; the minimum score was then used as the representative time.

During the testing process, source video sequences were in the YV12 format (.yuv file extension) for all codecs except Theora (.y4m with YV12)

For all measurements the PRO version of the YUVsoft Video Codec Scoring System was used

The following computer configuration was used for the main tests:

4-cores processor: Intel Core i7 920, 2.67GHz

OS Name: Microsoft Windows 7 Professional 64-bit

Total Physical Memory: 6 GB

GPU: NVIDIA GeForce GTX 275

During the evaluation the following measures were used:

SSIM (Y components)

PSNR (Y components)

More detailed information about these measures may be found on the Internet at the Quality Measure page.

Results for Movies

Full results for Movies could be found in professional versions of this report:
H.264 Comparison Report Pro 2010 - Movie edition
H.264 Comparison Report Pro 2010 - Enterprise edition

This version contains only Y-SSIM results only for Normal preset.

RD Curves

The Normal preset results for each sequence are presented in next figures. The results do not depend strongly on the metric used. SSIM metric: The leader is x264; MainConcept placed second, and DivX H.264 placed third. For the "Ice Age" and "Up" sequences, the MediaSDK and Elecard encoders exhibited similar results. For the "Indiana Jones" and "State Enemy" sequences, the XviD encoder placed fourth.

Encoding Speed

Absolute speed results are presented in next figures. All the encoders except Microsoft Expression and Theora have a similar growth rate for encoding time versus increasing bitrate. Elecard is the fastest encoder. The encoding speed of the Microsoft Expression and Theora encoders exhibits almost no dependency on bitrate, and Microsoft Expression’s encoding speed decreases at 1,200 kbps for the “Ice Age” sequence. XviD’s encoding speed exhibits little dependency on bitrate for the “Up” sequence.

Speed/Quality tradeoff

Detailed descriptions of the speed/quality trade-off graphs can be found in Appendix 7. Sometimes, codec results are not present in the particular graph owing to the codec’s extremely poor performance. The codec’s RD curve has no intersection with the reference’s RD curve.

The speed/quality trade-off graphs simultaneously show relative quality and encoding speed for the encoders tested in this comparison. XviD is the reference codec, for which both quality and speed are normalized to unity for all of the graphs. The terms “better” and “worse” are used to compare codecs in the same manner as in previous portions of this comparison.

Please note that the method of averaging among all sequences assumes that all codecs produced results for each sequence. When this is not the case, only existing results are taken into account.

Next figures show results for the Normal preset. The results differ depending on the chosen metric.

Y-SSIM: The three best codecs (no codec performs faster with higher quality) in terms of speed and quality are XviD, Elecard and x264 on average, except for the "Ice Age" and "Indiana Jones" sequences. The x264 encoder is better on average than DivX H.264 and MainConcept.

Almost all encoders except Microsoft Expression exhibited better results on average than did Theora.

Bitrate Handling

Encoders with Normal presets, except the Microsoft Expression encoder, demonstrate good bitrate handling for all sequences. For the “Up” sequence, XviD showed less than stellar results, especially at high bitrates. For all sequences, MainConcept, x264, Elecard and DivX H.264 showed the best results compared with other encoders. DivX H.264’s bitrate decreases at 1,500 kbps. MediaSDK slightly decreases bitrates, but the results are stable. Theora has a good bitrate handling mechanism but is not very stable.

Relative Quality Analysis

Next table shows relative bitrates for a fixed-quality output for all codecs. Note that these tables do not include information about the speed of the encoder.

Note that each number in the tables below corresponds to some range of bitrates (see Appendix 7 for more details). Unfortunately, these ranges can differ significantly because of differences in the quality of compared encoders. This situation can lead to some inadequate results when three or more codecs are compared. This comparison technique will be improved in the future.

The leaders are x264 and MainConcept, depending on the quality metric (the bitrate difference is 18% for a fixed quality).

Table 3. Average bitrate ratio for a fixed quality—usage area “Movies,” Normal preset, Y-SSIM metric

	DivX H.264	Elecard	Expression	MediaSDK	MainConcept	Theora	x264	XviD
DivX H.264	100%	124%	127%	118%	87%	138%	76%	128%
Elecard	81%	100%	103%	95%	70%	115%	61%	104%
Expression	79%	97%	100%	93%	69%	114%	61%	101%
Media SDK	84%	105%	107%	100%	74%	120%	64%	108%
MainConcept	115%	142%	144%	136%	100%	157%	88%	145%
Theora	72%	87%	88%	84%	64%	100%	57%	88%
x264	131%	163%	164%	155%	114%	176%	100%	164%
XviD	78%	96%	99%	93%	69%	113%	61%	100%

Next figure depicts the data from the table above. Each line in the figures corresponds to one codec. Values on the vertical axis are the average relative bitrates compared with the codecs along the horizontal axis. A lower bitrate indicates better relative results.

Results for HDTV

Full results for HDTV could be found in professional versions of this report:
H.264 Comparison Report Pro 2010 - HDTV edition
H.264 Comparison Report Pro 2010 - Enterprise edition

Speed/Quality tradeoff

Please note that the method of averaging among all sequences assumes that all codecs produced results for each sequence. When this is not the case, only existing results are taken into account.

Conclusions

Movies

The leading encoders in this usage area are MainConcept and x264. The quality of the Theora encoder is rather low.

High Speed Preset

The x264 encoder demonstrates better quality on average, and MainConcept shows slightly lower quality. These codecs’ bitrate handling algorithm is acceptable for this usage area. The MediaSDK codec places third, along with the dicas encoder (MediaSDK has slightly higher quality, but its encoding speed is 25% lower). The top four codecs for this preset are the following:

x264
MainConcept
MediaSDK

Normal Preset

The results for the Normal preset differ from those for the High Speed presets only in third place. The x264 encoder demonstrates better quality on average, and MainConcept shows slightly lower quality. The DivX H.264 encoder holds third place. The top three codecs for this preset are the following:

x264
MainConcept
DivX H.264

High Quality Preset

The results for this preset are similar to those of the Normal preset. The leaders are the x264 and MainConcept codecs. DivX H.264 takes third place. The top three codecs for this preset are the following:

x264
MainConcept
DivX H.264

Average bitrate for movies for all presets

*The encoding speed of these encoders is not in the requirements range owing to our chosen presets or to a lack of options
**The MediaSDK encoder does not meet the requirements for High Speed presets–specifically, no B-frames and exactly one reference frame

HDTV

The leaders in the HDTV area are x264, DivX H.264 and MainConcept. The XviD encoder trails all other H.264 encoders. Theora demonstrates rather poor results compared with other encoders.

High Speed Preset

The x264 encoder demonstrates better quality on average, and MainConcept shows slightly lower quality. The top three codecs for this preset are the following:

x264
DivX H.264 (this preset does not meet speed requirements)
MainConcept

Normal Preset

The results for the Normal preset differ from those of the High Speed presets. MainConcept shows the best results. The difference between best codecs are 2-4% of bitrate for the same quality. The top three codecs for this preset are the following:

MainConcept
x264
DivX H.264

High Quality Preset

The results for the High Quality preset are very interesting: x264 held first place, MainConcept was in second place, and two codecs (DivX H.264 and Elecard) shared third place (their quality results are very close, but DivX H.264 is 1.2 times faster than Elecard). The top four codecs for this preset are the following

x264
MainConcept
DivX H.264 and Elecard

Average bitrate for HDTV for all presets

*The encoding speed of this encoder is not in the requirements range owing to our chosen presets or to a lack of options
**The DivX H.264 High Speed preset for HDTV does not meet the speed requirements

Overall Conclusions

Overall, the leader in this comparison is x264, followed by MainConcept and DivX H.264. The Theora encoder demonstrates the worst results among all codecs tested. Average bitrate for Movies and HDTV for all presets

Average bitrate for Movies and HDTV for all presets

*The encoding speed of this codec does not fall in the required range owing to our chosen presets or to a lack of options.

The overall ranking of the codecs tested in this comparison is as follows:

x264
MainConcept
DivX H.264
MediaSDK
Elecard
XviD
Theora

Microsoft Expression Encoder 3 could not be placed in this list because of its much longer encoding time compared with other encoders (except Theora).

Using the standard Theora interface, we could not find any multithreading options; Theora works only in single-thread mode. The leader in this comparison is x264—its quality difference (according to the SSIM metric) could be explained by the special encoding option ("--tune ssim"). Interestingly, using the PSNR metric for MainConcept yielded results comparable with or better than those of x264. This means that no encoder can achieve the best results for both SSIM and PSNR when using the same parameters.

The difference between the MainConcept and DivX H.264 encoders is not overly significant, so these encoders tied for second in this comparison. The developers of the Elecard encoder do not provide a High Speed preset, so its ranking is based solely on the results for the Normal and High Quality presets. The dicas encoder holds third or fourth place for some sequences.

This rank is based only on the encoders’ quality results. Encoding speed is not considered here.

Codec Conclusion

DivX H.264 – quite balanced encoder with not very big number of parameters, this fact could be comfortable for users. This encoder is designed as a free sample application for DivX Plus HD compliant video encoding, and is a feature-constrained, for-purpose application.

Elecard - very fast codec with good encoding quality and very flexible settings. Many adjustable encoding settings are provided. This encoder has a very good bitrate handling mechanism (especially for the “Movies” usage area).

Microsoft Expression Encoder – the low encoding speed could result from the GUI loading every time a video is encoded. Unfortunately, we could not automate the encoder to eliminate this GUI loading.

MediaSDK – balanced encoder with a limited number of parameters but several use cases (speed/quality presets), making the encoder very comfortable for users.

MainConcept – one of the best codecs by encoding quality; has many encoding settings that can be adjusted. This encoder has a very good bitrate handling mechanism.

Theora – not an H.264 codec. Using our testing methodology (objective metrics) and test set, this encoder yields lower quality than do H.264 codecs and even MPEG-4 ASP.

x264 – one of the best codecs by encoding quality; has very user-friendly predefined presets, as well as many adjustable encoding settings.

XviD – an MPEG-4 ASP codec; its quality could be very close to or even higher than that of some commercial H.264 standard implementations, especially for encoding “Movie” sequences, but not for “HDTV” sequences.

Professional Versions of Comparison Report

H.264 Comparison Report Pro 2010 versions contain:

Additional objective metrics (PSNR)

Full information on Movies and HDTV usage areas, including three speed/quality presets

Full subjective comparison results

Additional quality analyzers

Much more pages and figures

etc.

Acknowledgments

The Graphics & Media Lab Video Group would like to express its gratitude to the following companies for providing the codecs and settings used in this report:

Elecard Ltd

Intel Corporation

MainConcept GmbH

Theora Development Team

x264 Development Team

XviD Development Team

The Video Group would also like to thank these companies for their help and technical support during the tests.