Thanks

We would like to express our gratitude to the following companies for providing us with codecs and settings:

ArcSoft, Inc.

Ateme

ATI Technologies Inc.

Elecard LTD

Fraunhofer Institute for Integrated Circuits

Vanguard Software Solutions, Inc.

x264 Development Team

We would also like to thank these companies for their technical support and help during our tests.

 

Overview

Codecs

 

Note: DivX 6.0 is not H.264 codec. It is MPEG-4 ASP codec, it takes part in this testing only as one of the best representative of previous video codec standard MPEG-4

Sequences

Goal and testing rules

H.264 Codecs Testing Objectives

The main goal of this document is an evaluation of the quality of new H.264 codecs, using objective metrics for comparison. The comparison was done using the settings provided by the developers for each codec.

Testing rules

Every codecs developers provided settings for both "Best Quality" (high quality compression) and "Best Speed" (better processing speed) presets, except DivX.

Codec restrictions: Only codecs that supported MPEG-4 AVC/H.264 Main profile and single pass encoding were tested.

Each codec was tested for speed 3 times, after this the median score (the middle value of the three measurements) was used as the resulting time.

The following 10 bitrates were used (in Kilobits per second) 100, 225, 340, 460, 700, 938, 1140, 1340, 1840, 2340.

During the testing three types of video sequences were used:

Source sequences (.yuv extension) in the YUV 4:2:0 format.

(.avi extension) sequences in the YUV 4:2:0 format.

These sequences were made in the following way:
Sequence was opened in VirtualDub using AviSynth script and, after that, it was saved in AVI file format. VirtualDub parameters "Color depth - Decompression Format" and "Color depth - Output format for compressor" were set to "Autoselect" and "Same as decompression format" respectively.
AviSynth script for sequence opening:

RawSource("<source_file>.yuv", <wigth>, <height>, "I420")
AssumeFPS(<source_fps*1000>, 1000)

After that in obtained YUV file fourcc was manually changed from "DIB " to "YV12" using HexEditor from VirtualDub. This was required for correct work of Ateme codec.

*.yuv sequences with duplicated last frame. These sequences were used for testing x264 codec (see its description). Written for this case, special utility was used for generating such sequences.

In this testing VirtualDub version 1.6.10 was used.

In this testing AviSynth version 2.55 was used.

For all metrics' measurements in this test special build of MSU Video Quality Measure was used.

Two computers with following configuration were used for encoding:

Processor: Pentium 4, 2.8 GHz with Hyper Threading

Operating system: Windows 2000 Pro, SP4

Memory: 1Gb

Video Accelerator: ATI Radeon 9600 Series

Hard Drives: ATA100 2x80Gb

 

Metrics Used in Comparison

During testing following metrics were measured:

PSNR (Y, U, V, L, R, G, B components)

SSIM (Y component)

VQM (Y component)

Blurring (Y, U, V components)

Blocking (Y, U, V components)

Detailed description of these metrics including VQM and SSIM algorithms could be found here:

http://www.compression.ru/video/quality_measure/info_en.html

Number of graphs

Sequences

Foreman

This is one of the most famous sequences. It represents a face with very rich mimic. Motion is not very intensive here, but on the other hand it is disordered, not forward. Intricate character of motion creates problems for the motion compensation process. In addition camera is shaking that makes the image unsteady. In the end of the sequence camera suddenly turns to the building site and there follows an almost motionless scene. So this sequence also shows codec's behavior on a static scene after intensive motion.

Susi

This sequence is characterized by high-level noise and slow motion. In its first part the scene is almost static (the girl only blinks), then there is some motion (she abruptly moves her head) and then the scene becomes almost static again. Noise is suppressed on every second frame due to the B-frames option in MPEG-2 codec.

BBC

 

This sequence is characterized by pronounced rotary motion. It contains a rotating striped drum with different pictures and photos on it. Quality of the compressed sequence can be evaluated by observing details on these images.

Battle

This sequence is a fragment of the "Terminator-2" movie, which represents its very beginning. In terms of compression this sequence is the most difficult one among all other sequences that took part in the testing. That is because of three main reasons: constant brightness changes (explosions and laser flashes, see the picture above), very quick motion and frequent changes of the scene that make codecs often compress frames as I-frames.

Simpsons

This sequence is a fragment of "Simpsons" cartoon film. This is a classical representative of cartoon films: sketchy movement, great number of monochrome regions with abrupt junctions between them. Previously this sequence was compressed in MPEG-2 with rather low bitrate.

Matrix

This sequence is a fragment of "Matrix" movie. Relatively simple movement, quite dim colors and small resolution allows codec to treat this sequence in rather simple way.

Concert

This sequence is a part of HDTV broadcast of symphonic orchestra concert. Sequence's spatial resolution is very high. At the same time motion is rather simple and sometimes it completely disappears. There are two scene changes in this sequence.

Codecs

DivX 6.0

DivX 6.0 is not H.264 codec. It is MPEG-4 ASP codec, it takes part in this testing only as one of the best representative of previous video codec standard MPEG-4

This is a VfW (Video for Windows) codec.

Compression was performed using VirtualDub 1.6.10 video processing program.

Evaluation version of codec works for 6 months.

There were no presets from developers. All tests were performed using "Home Theater Profile". For different test presets only Codec Performance settings were changed:

For "Best Quality" mode preset "Insane quality" was used

For "Best Speed" mode preset "Balanced" was used

Remarks:

Codec shifts source video sequence by one frame while using its own decoder. With other decoders (specifically, XviD) this effect disappears. For removing such shift next AviSynth script was used:

clip = AVISource("source.avi")

clip = clip.DeleteFrame(0)

clip = clip.DuplicateFrame(clip.FrameCount()-1)

return clip

Codec generates an error on "concert" sequence's compression.

However, despite these drawbacks, codec significantly improved compression quality in comparison with previous version. On the next graph there is a comparison between different versions of DivX (metric is Y-PSNR):

 

ArcSoft H.264

Console encoding program.

DirectShow filters were used for decoding.

Codec (encoder and decoder) and presets were provided by ArcSoft, Inc company specially for this test.

Remarks:

Codec works without remarks.

 

Ateme H.264

Console encoding program.

Reference decoder JM9.8 was used for decoding.

Codec and presets were provided by Ateme company specially for testing.

Remarks:

Codec stores compressed sequence in MPEG4 container. Program mp4box version 0.4.0 was used for extracting H.264 stream from it.

 

ATI H.264

Console encoding program.

Reference decoder JM9.8 was used for decoding.

Codec and presets were provided by ATI Technologies Inc. company specially for this test.

Remarks:

Codec has only "Best Speed" preset.

 

Elecard H.264

Console encoding program.

Console decoder provided by developers was used for decoding.

Codec (encoder and decoder) and presets were provided by Elecard LTD company specially for this test.

Remarks:

Codec works without remarks.

 

Fraunhofer IIS H.264

Console encoding program.

Reference decoder JM9.8 was used for decoding.

Codec and presets were provided by Fraunhofer Institute for Integrated Circuits IIS specially for this test.

Remarks:

Codec works without remarks.

 

VSS H.264

Console encoding program.

Console decoder was used for decoding.

Codec (encoder and decoder) and presets were provided by Vanguard Software Solutions, Inc. company specially for this test.

Remarks:

Codec generates an error while compressing "foreman" sequence.

 

x264

Console encoding program.

Program ffmpeg 2005.09.09 was used for decoding.

This codec is an Open Source project.

Optimal settings were provided by codec's Development Team.

Remarks:

During decoding ffmpeg program does not write last frame of video sequence. Therefore we used sequences with duplicated last frame for this codec.