RUSSIAN VERSION

MSU Subjective Comparison of Modern Video Codecs

MSU Graphics & Media Lab (Video Group)

Take a look at this article on the new site! Follow the link
https://videoprocessing.ai/codecs/subjective-codecs-comparison-2006.html

Project head: Dr. Dmitriy Vatolin
Organization of assessment: Alexander Parshin, Petrov Oleg
Refinement, translation: Oleg Petrov
Verification: Alexander Parshin, Artem Titarenko

About comparison

Tested codecs:

  • DivX 6.0
  • Xvid 1.1.0
  • x264
  • WMV 9.0

Bitrates:

  • 690 kbps
  • 1024 kbps

Number of sequences: 4

Number of experts: 50

Download MSU Subjective Comparison of Modern Video Codecs (32 pages in PDF, 1.38 MB)

Goal of subjective comparison

During last few years many comparisons of video, audio and image codecs were carried out by our Graphics & Media Lab at Moscow State University. All of them used objective metrics like PSNR, VQM or SSIM. This fact has raised reasonable questions on adequacy of objective measures to the subjectively perceived quality, which is the main parameter of a codec's performance. Goals of our assessment are subjective comparison of new versions of popular videocodecs, comparison of results with objective metrics and subjective assessment technology testing.

Main comparison parts:

  1. Subjective comparison of videocodecs.
  2. Comparison of results with objective metrics.

Subjective comparison

Main quality parameter for a codec - subjective impressions of a viewer of a compressed video. Latter is the main idea of subjective comparison methods - quality score for a compressed sequence is the average opinion of a group of experts on its quality (MOS, Mean Opinion Score).
There are lots of methodologies of subjective assessment, many of them are implemented in MSU Perceptual Video Quality tool that was used for the assessment. Our comparison is using SAMVIQ - method that was recently developed by EBU (European Broadcasting Union), PDF with the comparison contains its description. On a graph below you can see average MOS for all codecs, the higher the better. Probably, XviD results could be improved by switching on deblocking algorithm (this algorithm isn't used by default).

average MOS
Average MOS for all codecs

Following graph illustrates average MOS for each codec and bitrate and its' 95% confidence intervals.

average MOS
Average MOS for all codecs and bitrates

Evaluation of objective metrics

PSNR, VQM and SSIM were measured for each sequence with MSU Video Quality Measurement tool, description of these metrics can be found on this page. Note, that "VQM" metric refers to algorithm of Feng Xiao, not proposed by VQEG objective quality metric. The following graphs illustrate relation between subjective opinion and an objective metric: if set of points is close to a straight line, then the metric is adequate to the subjectively perceived quality (values of PSNR and SSIM were mapped on MOS scale, see PDF with the comparison for more details).

PSNR VQM
SSIM
On our test set SSIM was far more adequate to the subjective opinion than PSNR and VQM

Content of HTML version

  • Introduction
  • Gratitude
  • Comparison abstract
  • Overview
  • Rules and goals
  • Sequences
  • Codecs
  • Results of subjective comparison
  • Analysis of objective and subjective results
  • MOS+PSNR/bitrate graphs
  • MOS+PSNR graphs, grouped by bitrate
  • Evaluation of objective metrics
  • Correlation of objective metrics and subjective scores
  • Conclusions
  • Average MOS for all codecs
  • General conclusions

    • Download

    E-mail: 

    See all MSU Video Codecs Comparisons

    MSU video codecs comparisons resources:

    Other resources

    Video resources:

    3D and stereo video
    Projects on 3D and stereo video processing and analysis
    MSU Video Quality Measurement tools
    Programs with different objective and subjective video quality metrics implementation     		Codecs comparisons
    Objective and subjective quality evaluation
    tests for video and image codecs
    Ext. link: x264 parameters efficiency comparison
    Public MSU video filters
    Here are available VirtualDub and AviSynth filters. For a given type of digital video filtration we typically develop a family of different algorithms and implementations. Generally there are also versions optimized for PC and hardware implementations (ASIC/FPGA/DSP). These optimized versions can be licensed to companies. Please contact us for details via video(at)graphics.cs.msu_ru.     		Filters for companies
    We are working with Intel, Samsung, RealNetworks and other companies on adapting our filters other video processing algorithms for specific video streams, applications and hardware like TV-sets, graphics cards, etc. Some of such projects are non-exclusive. Also we have internal researches. Please let us know via video(at)graphics.cs.msu_ru if you are interested in acquiring a license for such filters or making a custom R&D project on video processing, compression, computer vision.
    Video codecs projects
    Different research and development
    projects on video codecs     		Other
    Other information
    Last updated: 12-May-2022

    Server size: 8069 files, 1215Mb (Server statistics)

    Project updated by
    Server Team and MSU Video Group

    Project sponsored by YUVsoft Corp.

    Project supported by MSU Graphics & Media Lab