Second Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison

For professional users and codec authors

MSU Graphics & Media Lab (Video Group)

Return to second H.264/AVC comparison home page!

Part 4. Additional information and conclusions

Take a look at this article on the new site! Follow the link
https://videoprocessing.ai/codecs/mpeg4-avc-h264-2005-part4.html

Contents

  • Part1. Introduction
  • Part2. "Best quality" preset
  • Part3. "Best speed" preset
  • Part4. Additional information and conclusions

    • 2-pass mode and High profile

    Additionally to one-pass mode in Main profile of standard H.264 we tested two-pass mode and compression using possibilities of High profile. Only x264, Elecard, Ateme and Fraunhofer IIS codecs took part in this test. This test was held on "foreman" and "battle" sequences and without speed measuring.

    As it is seen on picture below, it is difficult to analyze results using PSNR/Bitrate graphs. That is why we use Delta Y-PSNR graphs instead of absolute Y-PSNR graphs in this part. In all graphs reference is Main Profile one pass results for current codec.

     

    Picture 1.   Different presets of codec x264, Absolute Y-PSNR

     

    Sequence "foreman", preset "Best quality"

    Picture 2.   Different presets of codec x264, Delta Y-PSNR

     

    Picture 3.   Different presets of codec Elecard H.264, Delta Y-PSNR

     

    Picture 4.   Different presets of codec Ateme H.264, Delta Y-PSNR

     

    Picture 5.   Different presets of codec Ateme H.264, Delta Y-VQM

     

    Picture 6.   Different presets of codec Ateme H.264, Delta-Y-SSIM

     

    Picture 7.   Different presets of codec Fraunhofer IIS H.264, Delta Y-PSNR

     

    Sequence "battle", preset "Best quality"

    Picture 8.   Different presets of codec x264, Delta Y-PSNR

     

    Picture 9.   Different presets of codec Elecard H.264, Delta Y-PSNR

     

    Picture 10. Different presets of codec Ateme H.264, Delta Y-PSNR

     

    Picture 11. Different presets of codec Ateme H.264, Delta Y-VQM

     

    Picture 12. Different presets of codec Ateme H.264, Delta Y-SSIM

     

    Picture 13. Different presets of codec Fraunhofer IIS H.264, Delta Y-PSNR

    Note:

  • Fraunhofer IIS H.264 codec showed some not very adequate results during this part of testing because of presets that were provided for us:

  • Main profile with two passes has quality parameter equal to 4 (good quality)

  • Main profile with one pass and High profile have quality parameter equal to 6 (best quality)

    •

    Per frame comparison

    In addition to average metrics' values we measured metrics' values for separate frames. Basing on this data, Rate Control of each codec could be analyzed. As a rule, on such graphs it is clearly seen whether codec uses B-frames or not ("striation" of graph), location of I-frames (abrupt peaks or collapses on graph), scene changing reaction, quality of R-D models, etc.

    On the next graphs there are frames on abscissa axis and bit rates on axis of ordinates (low bit rates are below). Color shows metric's value: the more red, the lesser is value. Thus, for Y-PSNR metric red color means bad quality, blue - good quality. All graphs of one sequence have the same colors (every color corresponds to only one value of metric on all graphs).

    Analyzing these graphics we should take into account bit rate handling of codecs since these graphs do not consider it.

    Sequence "foreman", Y-PSNR, Preset "Best quality"

     

    Picture 14. Codec DivX. Sequence "foreman".

     

    Picture 15. Codec x264. Sequence "foreman".

     

    Picture 16. Codec ArcSoft H.264. Sequence "foreman".

     

    Picture 17. Codec Elecard H.264. Sequence "foreman".

     

    Picture 18. Codec Ateme H.264. Sequence "foreman".

     

    Picture 19. Codec Fraunhofer IIS H.264. Sequence "foreman".

     

     

    Sequence "bbc", Y-PSNR, Preset "Best quality"

    Picture 20. Codec DivX. Sequence "bbc".

    On this graph compression quality does not significantly depend from bit rate, because DivX codec does not maintain low bit rates well, and works badly on high ones.

    Picture 21. Codec x264. Sequence "bbc".

     

    Picture 22. Codec ArcSoft H.264. Sequence "bbc".

     

    Picture 23. Codec VSS H.264. Sequence "bbc".

     

    Picture 24. Codec Elecard H.264. Sequence "bbc".

     

    Picture 25. Codec Ateme H.264. Sequence "bbc".

     

    Picture 26. Codec Fraunhofer IIS H.264. Sequence "bbc".

    Remarks:

  • These graphs show that x264, Elecard, Ateme and VSS codecs use B-frames.

  • DivX and Elecard codecs have very interesting B-frames placement algorithm. They change positions of each type of frames depending on bit rate value. This is shown on graphs as vertical wavy lines.

  • "bbc" sequence clearly shows problems in bit rate control of ArcSoft codec. It is represented as presence of "holes" on average bit rates. Considering complex circular motion in this sequence, one could suppose presence of problems with motion estimation in this codec.

  • Large quality overstating of I-frames in x264 codec (it is also typical for Ateme and Fraunhofer IIS codecs, but at smaller degree) may be caused by inaccurate prediction of current frame type. It is represented as distinct blue lines that appear several times in sequence.

    •

    Graph type

    Total number of graph

    Inserted in this document

    Per frame metrics

    1414

    13 (0.9%)

    Visual comparison

    Even in spite of appearance of new metrics, that more adequately reflect human perception of video, subjective video tests still do not lose their importance.

    We set ourselves a problem to compare video codecs using only existing objective metrics. But in addition to various graphs that impartially evaluate quality of obtained results, below we give several frames from "bbc" and "battle" sequences, compressed with different codecs.

    Sequence "bbc", frame 170, bitrate 1140 kbps

    Picture 27. Original

    Picture 28.   DivX

    Picture 29. x264

    Picture 30. Original. Image differences. Y-PSNR

    Picture 31.   DivX. Image differences. Y-PSNR

    Picture 32. x264. Image differences. Y-PSNR

     

    Sequence "bbc", frame 250, bitrate 1140 kbps

    Picture 33. Original

    Picture 34.   DivX

    Picture 35. x264

    Picture 36. Original. Image differences. Y-PSNR

    Picture 37.   DivX. Image differences. Y-PSNR

    Picture 38. x264. Image differences. Y-PSNR

     

    Sequence "battle", frame 527, bitrate 700 kbps

    Picture 39. Original

     

    Picture 40. DivX

     

    Picture 41. x264

     

    Picture 42. ArcSoft

     

    Picture 43. VSS

     

    Picture 44. Elecard

     

    Picture 45. Ateme

     

    Picture 46. Fraunhofer IIS

    Sequence "battle", frame 527, bitrate 700 kbps, Y-PSNR

    Picture 47. Y-PSNR ArcSoft

    Picture 48. Y-PSNR Ateme

    Picture 49. Y-PSNR DivX

    Picture 50. Y-PSNR Elecard

    Picture 51. Y-PSNR Fraunhofer IIS

    Picture 52. Y-PSNR VSS

    Picture 53. Y-PSNR Original

    Picture 54. Y-PSNR x264

    Informal codec comparison

     

    As it is clearly seen on the graphs, codecs show significantly different results on different sequences. Selected sequences have different types in terms of motion and noise, and this allows checking which codecs are tuned to all types of sequences, and which one - to only one. But also it is very interesting to evaluate codec behavior on all test data.

    We considered averaging of results between different sequences as incorrect, and used a method of assigning points to codecs for different measurements. Points are united in arbitrary informal rating in the end.

    We tried to reflect real situation on test data as much as possible. We are still developing methodology of our informal comparison. Therefore it is better to consider following results as subjective opinion of comparison authors.

    Informal comparison rules

    Separate comparisons were carried out for "Best Quality" and "Best Speed" presets. Several graphs were chosen for each preset, on which comparison was held. Obtained marks were summed with some predefined weights.

    Codecs received points in a following way:

  • If codec is one of the leaders on current graph then it received 3 points.

  • If codec is trailing on current graph then it received 1 point.

  • Otherwise it is received 2 points.

    •

    Following graphs were used for "Best Quality" preset with corresponding coefficients:

  • Y-PSNR graphs, coefficient 4.

  • U-PSNR graphs, coefficient 1.

  • V-PSNR graphs, coefficient 1.

  • SSIM graphs, coefficient 1.

  • VQM graphs, coefficient 1.

  • Bit rate handling graphs, coefficient 3 (denoted in tables as "BH").

  • Graphs of absolute speed, coefficient 1.

    •

    Following graphs were used for "Best Speed" preset with corresponding coefficients:

  • Y-PSNR graphs, coefficient 1.

  • Bit rate handling graphs, coefficient 2 (denoted in tables as "BH").

  • Graphs of absolute speed, coefficient 4.

    •

    Average values were computed as arithmetic average of all successfully encoded sequences (i.e. codecs' errors were not taken into account).

    Informal comparison results

    Preset "Best quality"

    Sequence "foreman"

     

    DivX

    x264

    ArcSoft

    VSS

    Elecard

    Ateme

    Fraunhofer IIS

    Y-PSNR

    1

    2

    1

    0

    2

    3

    2

    U-PSNR

    1

    3

    1

    0

    3

    3

    2

    V-PSNR

    1

    3

    2

    0

    3

    3

    2

    SSIM

    1

    3

    1

    0

    2

    2

    2

    VQM

    1

    2

    1

    0

    2

    3

    2

    BH

    1

    2

    2

    0

    2

    2

    2

    Time

    2

    2

    3

    0

    2

    3

    1

    Total

    13

    27

    18

    0

    26

    32

    23

     

    Sequence "susi"

     

    DivX

    x264

    ArcSoft

    VSS

    Elecard

    Ateme

    Fraunhofer IIS

    Y-PSNR

    2

    2

    1

    2

    2

    3

    2

    U-PSNR

    2

    3

    1

    2

    2

    2

    1

    V-PSNR

    2

    3

    1

    3

    2

    2

    1

    SSIM

    2

    2

    1

    2

    2

    3

    2

    VQM

    2

    2

    1

    2

    1

    3

    2

    BH

    1

    2

    3

    2

    2

    2

    1

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    21

    26

    20

    25

    23

    30

    18

     

    Sequence "bbc"

     

    DivX

    x264

    ArcSoft

    VSS

    Elecard

    Ateme

    Fraunhofer IIS

    Y-PSNR

    1

    2

    1

    2

    2

    3

    3

    U-PSNR

    1

    3

    1

    2

    2

    2

    3

    V-PSNR

    2

    3

    1

    2

    2

    3

    3

    SSIM

    2

    2

    1

    3

    2

    3

    2

    VQM

    2

    2

    1

    2

    2

    3

    3

    BH

    1

    3

    1

    2

    2

    2

    3

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    16

    29

    14

    25

    24

    31

    33

     

    Sequence "battle"

     

    DivX

    x264

    ArcSoft

    VSS

    Elecard

    Ateme

    Fraunhofer IIS

    Y-PSNR

    2

    2

    1

    1

    2

    3

    2

    U-PSNR

    1

    3

    1

    2

    3

    2

    3

    V-PSNR

    1

    3

    1

    2

    3

    2

    3

    SSIM

    2

    2

    1

    1

    2

    3

    2

    VQM

    2

    2

    1

    2

    2

    3

    3

    BH

    1

    3

    2

    2

    2

    2

    3

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    19

    29

    17

    19

    26

    30

    29

     

    Sequence "simpsons"

     

    DivX

    x264

    ArcSoft

    VSS

    Elecard

    Ateme

    Fraunhofer IIS

    Y-PSNR

    2

    3

    1

    3

    2

    3

    2

    U-PSNR

    1

    3

    1

    3

    2

    2

    2

    V-PSNR

    1

    3

    1

    3

    3

    2

    2

    SSIM

    2

    3

    1

    3

    2

    2

    2

    VQM

    2

    3

    1

    3

    2

    2

    2

    BH

    1

    2

    2

    2

    2

    2

    3

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    19

    32

    17

    32

    25

    28

    26

     

    Sequence "matrix"

     

    DivX

    x264

    ArcSoft

    VSS

    Elecard

    Ateme

    Fraunhofer IIS

    Y-PSNR

    2

    2

    1

    2

    2

    3

    1

    U-PSNR

    2

    3

    1

    3

    2

    2

    1

    V-PSNR

    2

    3

    1

    2

    3

    2

    1

    SSIM

    2

    2

    1

    2

    2

    3

    1

    VQM

    2

    2

    1

    2

    2

    3

    2

    BH

    1

    1

    2

    3

    2

    2

    2

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    21

    23

    17

    28

    25

    30

    16

     

    Sequence "concert"

     

    DivX

    x264

    ArcSoft

    VSS

    Elecard

    Ateme

    Fraunhofer IIS

    Y-PSNR

    0

    2

    1

    3

    1

    3

    3

    U-PSNR

    0

    3

    3

    2

    2

    2

    1

    V-PSNR

    0

    3

    3

    2

    2

    2

    1

    SSIM

    0

    2

    1

    3

    2

    1

    3

    VQM

    0

    2

    1

    3

    1

    2

    3

    BH

    0

    2

    2

    2

    1

    2

    3

    Time

    0

    2

    2

    2

    3

    2

    1

    Total

    0

    26

    20

    30

    17

    27

    30

     

    Results

    Codec

    Average points

    Place

    Ateme

    29.71

    1

    x264

    27.43

    2

    VSS

    26.5

    3

    Fraunhofer IIS

    25

    4

    Elecard

    23.71

    5

    DivX

    18.17

    6

    ArcSoft

    17.57

    7

    Preset "Best speed"

    Sequence "foreman"

     

    DivX

    x264

    ATI

    ArcSoft

    VSS

    Elecard

    Fraunhofer IIS

    Y-PSNR

    1

    2

    1

    2

    0

    3

    2

    BH

    1

    2

    3

    2

    0

    2

    1

    Time

    2

    2

    3

    2

    0

    2

    1

    Total

    11

    14

    19

    14

    0

    15

    8

     

    Sequence "susi"

     

    DivX

    x264

    ATI

    ArcSoft

    VSS

    Elecard

    Fraunhofer IIS

    Y-PSNR

    2

    2

    1

    2

    2

    3

    2

    BH

    1

    2

    2

    3

    3

    2

    1

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    12

    14

    17

    16

    16

    15

    8

     

    Sequence "bbc"

     

    DivX

    x264

    ATI

    ArcSoft

    VSS

    Elecard

    Fraunhofer IIS

    Y-PSNR

    1

    2

    2

    1

    2

    3

    3

    BH

    1

    2

    2

    1

    2

    2

    2

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    11

    14

    18

    11

    14

    15

    11

     

    Sequence "battle"

     

    DivX

    x264

    ATI

    ArcSoft

    VSS

    Elecard

    Fraunhofer IIS

    Y-PSNR

    2

    3

    1

    2

    1

    3

    2

    BH

    1

    2

    2

    2

    2

    2

    3

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    12

    15

    17

    14

    13

    15

    12

     

    Sequence "simpsons'

     

    DivX

    x264

    ATI

    ArcSoft

    VSS

    Elecard

    Fraunhofer IIS

    Y-PSNR

    2

    2

    1

    2

    1

    3

    3

    BH

    1

    2

    2

    2

    2

    2

    3

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    12

    14

    17

    14

    13

    15

    13

     

    Sequence "matrix"

     

    DivX

    x264

    ATI

    ArcSoft

    VSS

    Elecard

    Fraunhofer IIS

    Y-PSNR

    3

    2

    2

    2

    2

    2

    2

    BH

    1

    2

    2

    3

    3

    2

    2

    Time

    2

    2

    3

    2

    2

    2

    1

    Total

    13

    14

    18

    16

    16

    14

    10

     

    Sequence "concert"

     

    DivX

    x264

    ATI

    ArcSoft

    VSS

    Elecard

    Fraunhofer IIS

    Y-PSNR

    0

    3

    1

    2

    2

    3

    3

    BH

    0

    2

    1

    2

    2

    1

    3

    Time

    0

    2

    3

    2

    2

    2

    1

    Total

    0

    15

    15

    14

    14

    13

    13

     

    Results

    Codec

    Average points

    Place

    ATI

    17.29

    1

    Elecard

    14.57

    2

    VSS

    14.33

    3

    x264

    14.29

    4

    ArcSoft

    14.14

    5

    DivX

    11.83

    6

    Fraunhofer IIS

    10.71

    7

     

    General conclusions

  • Comparing to previous year comparison, the level of H.264 standard codecs has increased significantly. One year ago best codecs of old MPEG4-ASP standard were comparable to best codecs of H.264 standard, now this situation greatly changed. DivX codec is inferior to most codecs of new standard, even despite considerable growth of its quality comparing to previous version.

  • It is rather difficult to choose absolute leader by quality among codecs of new standard. x264 and Ateme codecs showed approximately equal best results.

  • Obvious leader by speed is codec from ATI company. It was significantly faster than the others while not always showing worst results.

    •

    Contents

  • Introduction
  • Thanks
  • Overview
  • Goal and testing rules
  • Sequences
  • Codecs
  • "Best quality" preset
  • Y-PSNR
  • U-PSNR, V-PSNR
  • SSIM, VQM
  • Blurring measure, blocking measure
  • Bitrate handling
  • Time
  • "Best speed" preset
  • Y-PSNR
  • Speed/Quality Tradeoff
  • Bitrate handling
  • Time
  • Final part
  • 2-pass mode and High profile
  • Per frame comparison
  • Visual comparison
  • Informal codec comparison
  • General conclusions

    • Download

    Contact Information

    Additional information

    See all MSU Video Codecs Comparisons

    MSU video codecs comparisons resources:

    Other materials

    Video resources:

    3D and stereo video
    Projects on 3D and stereo video processing and analysis
    MSU Video Quality Measurement tools
    Programs with different objective and subjective video quality metrics implementation     		Codecs comparisons
    Objective and subjective quality evaluation
    tests for video and image codecs
    Ext. link: x264 parameters efficiency comparison
    Public MSU video filters
    Here are available VirtualDub and AviSynth filters. For a given type of digital video filtration we typically develop a family of different algorithms and implementations. Generally there are also versions optimized for PC and hardware implementations (ASIC/FPGA/DSP). These optimized versions can be licensed to companies. Please contact us for details via video(at)graphics.cs.msu_ru.     		Filters for companies
    We are working with Intel, Samsung, RealNetworks and other companies on adapting our filters other video processing algorithms for specific video streams, applications and hardware like TV-sets, graphics cards, etc. Some of such projects are non-exclusive. Also we have internal researches. Please let us know via video(at)graphics.cs.msu_ru if you are interested in acquiring a license for such filters or making a custom R&D project on video processing, compression, computer vision.
    Video codecs projects
    Different research and development
    projects on video codecs     		Other
    Other information
    Last updated: 12-May-2022

    Server size: 8069 files, 1215Mb (Server statistics)

    Project updated by
    Server Team and MSU Video Group

    Project sponsored by YUVsoft Corp.

    Project supported by MSU Graphics & Media Lab