MSU Subjective Comparison of Modern Video Codecs
MSU Graphics & Media Lab (Video Group)Return to Subjective Comparison of Modern Video Codecs home page!
Part 4. Conclusions
Contents
Results of the comparison
Following graphs present MOS value for each codec after averaging among all sequences and all bitrates.
Picture 21. Average MOS for all codecs
Picture 22. Average MOS for all codecs and bitrates
Plots from previous part of report reflect average subjective opinion on codecs. To calculate rank of each codec on the whole test set, we apply following method: count rank of codec for each sequence and then count sum of ranks ("Overall" row). Sorting this array we get rank of codec in comparison (Ref. - uncompressed sequence, was evaluated by experts as well as others).
|
Battle |
Rancho |
Matrix sc.1 |
Matrix sc.2 |
Overall |
Place |
|
|
Ref. |
1 |
1 |
2 |
1 |
5 |
- |
|
x264 1024 |
2 |
3 |
1 |
2 |
8 |
1 |
|
DivX 1024 |
3 |
4 |
5 |
3 |
15 |
2 |
|
XviD 1024 |
7 |
2 |
3 |
5 |
17 |
3 |
|
x264 690 |
5 |
6 |
4 |
4 |
19 |
4 |
|
WMV 1024 |
4 |
5 |
6 |
6 |
21 |
5 |
|
DivX 690 |
6 |
8 |
7 |
7 |
28 |
6 |
|
WMV 690 |
8 |
7 |
9 |
9 |
33 |
7 |
|
XviD 690 |
9 |
9 |
8 |
8 |
34 |
8 |
x264 supremacy is obvious for both 690 kbps and 1024 kbps, it is interesting that its' average mark for 690 kbps is better than the mark of WMV on 1024 kbps. DivX is the second, and WMV and XviD are last. Last place of XviD can be explained by the lack of deblocking during the assessment.
Conclusions
- Open source x264 codec of the new H.264 compression standard was found out to be better than long-developed proprietary solutions.
- Subjective comparison is stable and productive method for video systems' assessment when all testing conditions are precisely adhered to.
- Subjective tests can not always be replaced by objective ones with sufficient precision. SSIM was the best objective metric in our comparison.
References
1. International Telecommunication Union, "Methodology for the subjective assessment of the quality of television pictures (ITU-R BT.500-11)" (2002)
2. European Broadcasting Union, "SAMVIQ - a New EBU Methodology for Video Quality Evaluations In Multimedia", web site www.ebu.ch
3. Feng Xiao, "DCT-based Video Quality Evaluation", Final Project for EE392J (2000)
4. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image Quality Assessment: From Error Visibility to Structural Similarity", IEEE Transactions on Image Processing, vol. 13, no. 4 (2004)
5. VQEG, "Final Report from the Video Quality Experts Group on the Validation of Objective Models of Video Quality Assessment" (2000), web site www.vqeg.org
6. MSU Graphics & Media Lab (Video Group), "MSU Perceptual Video Quality Tool" www.compression.ru/video/quality_measure/perceptual_video_quality_tool_en.html
7. MSU Graphics & Media Lab (Video Group), "MSU Video Quality Measurement Tool" www.compression.ru/video/quality_measure/video_measurement_tool_en.html
8. M.H. Prinson, S. Wolf, "The Impact of Monitor Resolution and Type on Subjective Video Quality Testing", NTIA TM-04-412
9. Avery Lee, "VirtualDub", web site www.virtualdub.org
