Hot news:

If you find a spelling error, please select an incorrect text and press Ctrl+Enter. Thank you!

Compression project >> Video Area Home
RUSSIAN

MSU Video Quality Measurement Tool

MSU Graphics & Media Lab (Video Group)

Projects, ideas: Dr. Dmitriy Vatolin
Implementation: Alexey Moskvin, Oleg Petrov, Sergey Putilin, Sergey Grishin, Arsaev Marat


Basic Information About VQMT



What's new in MSU VQMT 3.0?



Brief Description


Demo Video: Basic Use Case Overview

MSU Video Quality Measurement Tool (VQMT) is a program for objective video quality assessment. It provides functionality for both full-reference (two videos are examined) and single-reference (one video is analyzed) comparisons. Basic scheme of the tool is illustrated on Pic.1 below.

VQMT input and output
Pic.1. VQMT input and output

VQMT inputs:

VQMT outputs:

Video quality is a characteristic of a video passed through a video transmission/processing system, a formal or informal measure of perceived video degradation (typically, compared to the original video). Video processing systems may introduce some amounts of distortion or artifacts in the video signal, so video quality evaluation is an important problem. MSU VQMT is a program for video quality measurements. This application allows you to create objective comparison of video codecs and perform video processing filters analysis.

Visualization examples
Visualization examples for several metrics

Program allows to answer the next questions:

  1. One codec has more blurring than another. On what frames? Average mark? *
  2. One codec has more blocking than another. On what frames? Average mark?*
  3. One codec has lower quality than another. On what frames? Average mark in PSNR, VQM, SSIM?*
*Corresponding "bad frames" are saved automatically as BMP files.

Current version: 3.0 [Changelog]


Main Features


Here is the list of VQMT main features, some of them are available in PRO version only:
Table 1. VQMT Main Features

#

Feature

FREE version

PRO version

1 Flexible Functionality

he tool provides functionality for calculating:

  • metric values for each frame
  • average value for sequence
  • metric values for specific color component

YES YES
2 20 Objective Metrics Support:
  1. PSNR (Y,U,V,L,R,G,B) - most popular
  2. PSNR_256 (Y,U,V,L,R,G,B)
  3. APSNR (Y,U,V,L,R,G,B) - most popular
  4. APSNR_256 (Y,U,V,L,R,G,B)
  5. Delta (Y,U,V,L, R,G,B)
  6. MSAD (Y,U,V,L,R,G,B)
  7. MSE (Y,U,V,L,R,G,B)
  8. SSIM_fast (Y)
  9. SSIM_precise (Y)
  10. MS-SSIM_fast (Y)
  11. MS-SSIM_precise (Y)
  12. 3SSIM (Y)
  13. stSSIM (Y)
  14. VQM (Y)
  15. MSU Blurring Metric (Y,U,V)
  16. MSU Blocking Metric (Y,U,V)
  17. MSU Brightness Flicking Metric (Y) - plugin
  18. MSU Brightness Independent PSNR (Y) - plugin
  19. MSU Drop Frame Metric (Y) - plugin
  20. MSU Noise Estimation Metric (Y) - plugin
  21. MSU Scene Change Detector (Y) - plugin
  22. Accelerated SSIM, 3SSIM and MS-SSIM for CUDA-enabled GPUs
YES YES
3 Over 20 Video Formats Support:
  1. *.AVI
  2. *.AVS
  3. *.Y4M
  4. *.YUV (scientific) - 8 bits per color component:
    • YUV
    • YV12
    • IYUV
    • UYUV
    • Y
    • YUY2
  5. *.BMP (RGB24)
  6. via AviSynth:
    • *.MOV
    • *.VOB
    • *.WMV
    • *.MP4
    • *.MPG
    • *.MKV
    • *.FLV
    • and others
YES YES
4 Results File Format is *.CSV

Results are saved in *.CSV (comma-separated values) file which can be opened everywhere.

YES YES
5

Metrics Visualization

Video visualization file is generated for combination [metric, color component, reference]. Color and luminance of each pixel in visuzalization video is defined by metric value in that pixel (very useful for codec and filter debugging).

YES YES
6

ROI Support (via masking)

Allows to calculate metrics in Regions of Interest (ROI) only. ROI should be specified by additonal provided video with mask for each frame.

YES YES
7 Comparative Analysis

Two references can be compared against the original simultaneously.

YES YES
8 Professional Bit Depth Support
(over 8 bits per color component for *.YUV)
  • P010 (10 bit 4:2:0 yuv)
  • P014 (14 bit 4:2:0 yuv)
  • P016 (16 bit 4:2:0 yuv)
  • P210 (10 bit 4:2:2 yuv)
  • P214 (14 bit 4:2:2 yuv)
  • P216 (16 bit 4:2:2 yuv)
  • P410 (10 bit 4:4:4 yuv)
  • P414 (14 bit 4:4:4 yuv)
  • P416 (16 bit 4:4:4 yuv)
  • P410_RGB (10 bit 4:4:4 rgb)
  • P414_RGB (14 bit 4:4:4 rgb)
  • P416_RGB (16 bit 4:4:4 rgb)
NO YES
9 Batch Processing

VQMT provides GUI and command-line (CL) interfaces. It is possible using CL-tool to calculate several metrics for a given set of color components simultaneously (in parallel), thus achieveing significant speed advantage comparing to case of using GUI tool.

NO YES
10 HD Support

Processing of video with resolution higher than 1280x720.

NO YES


Plots Examples


These plots were built in MATLAB, based on measurement results from MSU VQMT.

Results (1)
Pic.2. Perframe Y-PSNR for several codecs

Results (2)
Pic.3. Y-PSNR dependency vs bitrate and frame number

Results (3)
Pic.4. Average Y-PSNR for several codecs and bitrates

Results (4)
Pic.5. Colored surface of Y-PSNR vs frame number and bitrate

Error (1)
Pic.6. Example of the error in codec, detected with VQMT.
Plots examples
Pic.7. Another examples of plots built based on VQMT output.

Comparison with Analogs


Table 2 presents the comparison of VQMT with analogs:
Table 2. Comparison with Analogs

MSU VQMT
(not free for companies)

Elecard Video QuEst (commercial)

Video Quality Studio (freeware)

PSNR Checker (freeware)

Avisynth plugins (SSIM, VQM) (freeware)

Supported objective metrics 16 + 5 (plugins) + 3 (GPU) 8 3 1 2
Plug-ins interface YES NO NO NO NO
Comparative analysis (comparison of two codecs in one pass) YES NO NO NO NO
Supported input formats *.AVI, AviSynth (*.AVS), *.BMP, *.YUV, *.Y4M and other media formats (*.MOV, *.VOB, *.WMV, *.MP4, *.MPG, *.MKV, *.FLV, etc.) using AviSynth and automatic scripts generation Supports any video stream if compatible decoder supporting the YV12 or UYVY output is registerd in the system AVI AVI Formats, supported by AviSynth
User interface GUI + command line (PRO) GUI GUI GUI Called from AviSynth scripts (you need to learn AviSynth scripting)
Automatically saving frames with maximum difference in quality YES NO NO NO NO
Plots with results of the comparison YES NO NO NO NO
Video with metric value visualization YES YES/NO NO NO NO


Contributors


Special thanks to following contributors.

Testimonials


We have testimonials from the following big companies: Small companies: And universities:

Changelog


3.0
[+] Added stSSIM metric
[+] Added .y4m raw video internal support
[+] Added Autoupdate feature for free version (our PRO customers receive updates automatically)
[+] Added GPU realization for SSIM-based metrics (SSIM, 3-SSIM, MS-SSIM. Requires CUDA-capable device. See metrics info page for additional info)
[*] Added subjective comparison for the most popular metrics (see metrics info)
[*] Added 64-bit version of MSU VQMT (up to 10% speedup)
[!] Program crashes due memory lack when -metr ALL specified with large (i.e. 1280x720) video frames.

2.7.3
[*] Fixed bug with MSSIM metric causing source frame change
[*] Fixed some metrics inaccuracy causing different metric values by enabling\disabling visualization
[!] Program crashes due memory lack when -metr ALL specified with large (i.e. 1280x720) video frames

2.7.2
[*] Fixed bug causing incorrect metric values, when using 3SSIM and MSSSIM simultaneously
[*] Fixed bug causing incorrect PSNR metric values in CSV files
[*] Fixed bug causing no metric calculation for large (>4gb) files
[*] Not existing directory specified in "-cod" parameter will be created now and processing will not cancel.
[!] Program crashes due memory lack when -metr ALL specified with large (i.e. 1280x720) video frames

2.7.1
[*] Fixed bug in CVS file generation. Sometimes first frame metric value was empty
[*] Fixed bug causing incorrect MSE metric values after calculating SSIM metric

2.7
[+] MSSSIM (fast and precise) metric implemented
[+] 3SSIM metric implemented
[*] Fixed bug in calculation of VQM metric under Windows 7
[*] Fixed bug during program launch on some machines

2.6 (Windows Vista & Windows 7 support)
[*] Fixed bug in Scene Change Detection plugin when working under Windows Vista or Windows 7
[*] Fixed bug in saving visualization video when running on Windows Vista or Windows 7
[*] Fixed dependency with vcomp.dll

2.5
[+] Video with any resolution is now supported by all metrics. Video with resolution which is not appropriate for some metric is now expanded (via data duplication, separately for each metric) to make resolution acceptable
[+] 1.95 times speed up of command line tool multiple metrics calculation on average (PRO version only)
[+] YUV files with size more than 2Gb are supported now
[+] Output directory for *.CSV and visualization files is automatically set to folder of last specified reference file
[*] Fixed bug in processing of *.YUV files with non-standard resolution
[*] Fixed bug in loading the mask from *.YUV files
[*] Fixed bug in masking of L (LUV colorspace) component
[*] Fixed bug in processing of non-standard resolution *.AVS files
[*] Fixed bug in calculation of SSIM (precise) for second reference file
[*] Fixed bug in conversion from RGB32 to YUV color spaces for video with non-standard resolutions (affects calculation of metric for *.AVI files)

2.01 beta
[+] 1.5 times speed up of command line tool multiple metrics calculation on average (PRO version only)
[+] Masking is added
[*] Fixed bug in 4:2:2 raw files with more than 8 bits per component support

2.0 beta
[+] *.MOV, *.VOB, *.WMV, *.MP4, *.MPG, *.MKV, *.FLV formats support simplified
[+] HDTV support (PRO version only)
[+] Raw files with more than 8 bits color depth per component are supported (PRO version only)
[+] Alternative SSIM and PSNR are added for compatibility with other implementations
[+] New version of *.CSV files with average metric values (PRO version only)
[+] Minor acceleration
[+] Preview buttons are added
[+] Options save is improved
[+] All MSU plugins are renamed (names are now more correct in GUI and simpler to call from PRO console)
[*] MSU Noise Estimation plugin bug with incorrect (identical) values for some videos is fixed
[*] MSU Noise Estimation and MSU BI-PSNR plugins provide correct information about their home pages now
[*] MSU BI-PSNR plugin crash during visualizing a metric for video sequences with dimensions less than 255 is fixed

1.52
[*] Error in saving CSV file for comparative analysis is fixed

1.51
[*] Error at the opening YUV-files is fixed

1.5
[*] Set of interface fixes

1.4
[*] Bugfixing in BMP processing (visualization saving, etc)

1.3
[*] Bug with YV12 yuv files fixed

1.2
[+] Now it is possible to compress visualization
[+] Plugin mechanism released
[!] Problem with some DV codecs

1.0
[+] More YUV filetypes are supported, including YV12, YUY2, YUV
[+] Supports unicode
[+] Visualization dialog was extensively reworked
[*] Interface is more user-friendly

0.81
[+] New AVI Reading system (support large AVI Files, VP 70)
[*] Bugfixing in final dialog

0.8
(This version was developed with Nicolay Trunichkin)
[+] New dialog with visualisation of the comparison and for comparison of the selected frames added.

0.75
[+] Improved codecs support (x264)
[+] Now data from YUV-AVI files is extracted without convertion
[*] Bugfixing: #NAN in VQM calculation, calculation of RGB-metrics for YUV-files
[*] MSU Blocking Measure changed

0.74
[+] MSE, MSAD metrics added
[+] Saving of averange value of the metric added
[+] Improved codecs support (DivX3, WMV, mjpeg2000)
[*] Delta metric was changed
[*] Bugfixing (AviSynth - wrong result for comparison of three files)

0.73
[+] All colorspaces from AVISynth are supported
[+] I420 (IYUV) support added
[+] XviD and B-frames support added
[*] Bugfixing (YUV-files, SSIM)

0.72
[+] AVS Support added
[*] Bugfixing
[!] Doesn't work with XVID

0.71
[+] First public beta


MSU Video Quality Measurement Tools



Contacts


e-mail: 


Other Resources


Video resources:

Fifth Report on S3D quality assessment (5 films) is released!
Appendix to Eighth MSU .264 codec comparison is released

Bookmark this page:   Add to Del.icio.us Add to Del.icio.us     Digg It Digg It     reddit reddit


 
Last updated: 25-August-2011

Search (Russian):
Server size: 8069 files, 1215Mb (Server statistics)

Project updated by
Server Team and MSU Video Group


Project sponsored by YUVsoft Corp.

Project supported by MSU Graphics & Media Lab

Rambler's Top100 @Mail.ru