MSU Quality Measurement Tool: Metrics information

MSU Graphics & Media Lab (Video Group)

---

Metrics Info

---

Peak signal-to-noise ratio (PSNR)

General info

Metric type:

full-reference image metric

Value range:

(completely different) 0..100 (similar to original)

Value interpretation:

bigger is better quality

MSU VQMT implementaions:

cpu multithreaded

MSU VQMT visualization:

pixel-wise

Available colorspaces:

R, G, B, Y, U, V, L, RGB, YUV

Output values:

metric value

Aggregated values:

standard set , total PSNR

MSU VQMT usages:

-metr psnr [over <color components>]

External references:

Wikipedia

Algorithm description

Metric depends only on difference of original and distorted, and more preciesly, only on $L_2$-norm of this difference (see MSE). Unlike MSE, metric has logrithmic scale and can be calculated using the following formula:

$$\mbox{PSNR} = 10\cdot\log_{10}\cfrac{\mbox{MaxErr}^2\cdot w\cdot h}{\sum\limits_{i=1,j=1}^{w,h}(x_{i,j}-y_{i,j})},$$

where MaxErr - maximum possible absolute value of color component (MaxErr=1 in VQMT), w - video width, h - video height.

Total PSNR is aggregated value, that considers all processed frames as a single huge image and then calculates PSNR. Total PSNR takes into account sum suqared distortion on all frames, and doesn't distinguish situation where all distorion on one frame from situation where it is distributed across all frames. While arithmetic mean aggregated value depends from geometric mean of MSE's of each frames, Total PSNR depends on arithmetic mean of them.

In MSU VQMT you can calculate PSNR for all YUV and RGB components and for L component of LUV color space. Also, since VQMT 12 you can calculate over all YUV space or all RGB space, achieving a single value for 3 components. PSNR metric is easy and fast to calculate, but sometimes it is not appropriate to human's perception.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	492.737	0.002
VQMT 13 default-Y	1 core enabled	HD 720p	477.477	0.002
VQMT 13 default-Y	8 cores	FullHD 1080p	213.148	0.005
VQMT 13 default-Y	1 core enabled	FullHD 1080p	213.664	0.005
VQMT 13 default-Y	8 cores	UHD 4k 2160p	48.946	0.020
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	50.437	0.020
VQMT 13 default-YUV	8 cores	HD 720p	365.428	0.003
VQMT 13 default-YUV	1 core enabled	HD 720p	276.738	0.004
VQMT 13 default-YUV	8 cores	FullHD 1080p	161.647	0.006
VQMT 13 default-YUV	1 core enabled	FullHD 1080p	134.382	0.007
VQMT 13 default-YUV	8 cores	UHD 4k 2160p	37.379	0.027
VQMT 13 default-YUV	1 core enabled	UHD 4k 2160p	33.964	0.029

Example

Legacy notes

Video Multimethod Assessment Fusion (Netflix VMAF)

General info

Metric type:

full-reference temporal metric

Value range:

(completely different) 0..100 (similar to original)
-∞..∞ if truncation to 0..100 is off
dependent on model in case of custom model

Value interpretation:

bigger better quality, value 100 does not mean that the images match pixel by pixel

MSU VQMT implementations:

CPU multithreaded
OpenCL (since VQMT 13)

MSU VQMT visualization:

block-wise (for VMAF visualization),
pixel-wise (for ADM, VIF, ANSNR visualisation)

Available colorspaces:

Y

Output values:

metric value,
bagging values (if on),
confidence intervals (if on),
values of elementary features (if on)

Aggregated values:

standard set

MSU VQMT usages:

-metr vmaf [-dev <OpenCL device>]

External links:

Original paper

Algorithm description

VMAF is modern reference metric developed by Netflix in cooperation with the University of Southern California. VQMT has full support of VMAF with multiple configuration switches. MSU VQMT support the following VMAF models: VMAF 0.60, VMAF 0.61 (2k, 4k), VMAF 0.62 (2k, 4k), VMAF 0.63 (2k), also, you can compute phone model and elementary features of VMAF. You can use custom model in pkl format with VQMT.

VMAF consist of 4 features (ADM, VIF, Motion, ANSNR) and 35 elementary features, but VMAF models uses only 6 of them: adm2, motion2, vif_scale0, vif_scale1, vif_scale2, vif_scale3. VMAF applies an SVM model to this set of features, which depends on current settings. After applying SVM, the value is clipped to interval 0..100 by default. Motion feature is the only temporal feature, it consider adjacent frames. To calculate VMAF value for current frame it is needed to use the previous frame and the next frame.

VMAF can also compute confidence intervals by applying multiple models and calculating standard deviation of result. VMAF has models, that aimed for 4k and 2k. By default, VQMT will automatically select the correct model by the resolution of input video.

Since VQMT 13 VMAF has a real block-wise visualization, which computes individual VMAF value for each 16x16 block of image. You also can visualize every feature besides motion (ADM, VIF, ANSNR).

MSU VQMT parameters description

Model preset

• Description. Choose built-in model or 'custom' for loading model from file. Built-in models:
  • default - VMAF default behaviour:
    1. VMAF v0.6.1 for running without confidence interval and per-model values
    2. VMAF v0.6.1 4k for previous case if applying 4k model
    3. VMAF v0.6.3 for running with confidence interval or per-model values
    4. VMAF v0.6.2 4k for previous case if applying 4k model (NOTE: no v0.6.3 for 4k)
  • vmaf_v061 - Netflix model VMAF v0.6.1 (2k or 4k)
  • vmaf_v062 - Netflix model VMAF v0.6.2 (2k or 4k), supports confidence interval
  • vmaf_v063 - Netflix model VMAF v0.6.3 (only 2k), supports confidence interval
  • all_models - vmaf_v061 - vmaf_v063 computed sumultaneously
  • basic_features - view only basic features from VMAF. Model will not be applied
  • standard_features - features that is used in VMAF v0.6.1 and VMAF score (2k or 4k)
  • all_features - view all features from VMAF. Model will not be applied
  • all - all feature and next models:
    1. VMAF v0.6.1 (2k or 4k)
    2. VMAF v0.6.2 (2k or 4k)
    3. VMAF v0.6.3
• Default value. default
• Usage. -set "model_preset=<value>", where <value> can be:
  default, vmaf_v061, vmaf_v062, vmaf_v063, vmaf_v060, all_models, basic_features, standard_features, all_features, all, custom.

Custom model (*.pkl)

• Description. You can specify path to *.pkl file here (or multiple ;-separated *.pkl files). Model file should be placed near pkl file. NOTE: this only means if preset is set to 'custom'
• Default value. empty
• Usage. -set "custom_model_files=<value>", where <value> can be:
  any string.

4k

• Description. Selection 4k model policy:
  • auto - select 4k if exists suitable model and input video is 4k
  • forced_2k - always 2k model
  • forced_4k - 4k if exsists: VMAF v0.6.1-2

  NOTE: this param does not affects custom model

• Default value. auto
• Usage. -set "4k=<value>", where <value> can be:
  auto, forced_2k, forced_4k

Confidence interval

• Description. Turn on additional VMAF features: 95%-confidence interval output and other statistical information
• Default value. false
• Usage. -set "confidence_interval=<value>", where <value> can be:
  true, false

Per-model values

• Description. Output values for all bootstrap models if confidence interval is on
• Default value. false
• Usage. -set "permodel_values=<value>", where <value> can be:
  true, false

Visualize algorithm (if on)

• Description. If visualization turned on you can select feature to visualize. It's impossible to calculate distribution of real VMAF value, so you can only visualize one of supposed features
• Default value. default
• Usage. -set "visualize_alg=<value>", where <value> can be: default, adm, ansnr, vif

Use phone model

• Description. Turn on postprocessing of metric value that produces more precise results for handheld devices. Select 'both' to see both results with and without postprocessing
• Default value. no
• Usage. -set "phone_model=<value>", where <value> can be:
  no, yes, both

Disable clipping values

• Description. Turn off clipping value to range set by model (0..100 for example)
• Default value. false
• Usage. -set "disable_clip=<value>", where <value> can be:
  true, false

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	492.737	0.002
VQMT 13 default-Y	1 core enabled	HD 720p	477.477	0.002
VQMT 13 default-Y	8 cores	FullHD 1080p	213.148	0.005
VQMT 13 default-Y	1 core enabled	FullHD 1080p	213.664	0.005
VQMT 13 default-Y	8 cores	UHD 4k 2160p	48.946	0.020
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	50.437	0.020
VQMT 13 default-YUV	8 cores	HD 720p	365.428	0.003
VQMT 13 default-YUV	1 core enabled	HD 720p	276.738	0.004
VQMT 13 default-YUV	8 cores	FullHD 1080p	161.647	0.006
VQMT 13 default-YUV	1 core enabled	FullHD 1080p	134.382	0.007
VQMT 13 default-YUV	8 cores	UHD 4k 2160p	37.379	0.027
VQMT 13 default-YUV	1 core enabled	UHD 4k 2160p	33.964	0.029

Example

Naturalness Image Quality Evaluator (NIQE)

General info

Metric type

no-reference image metric

Value range

(very natural) 0..∞ (not natural)

Value interpretation

bigger metric values - better quality (naturalness)
normal metric values are in range about 3..20. Also, there can be values NAN and 0, which are abnormal and should be considered as symptom of not natural image

MSU VQMT implementations

CPU multithreaded
OpenCL (since VQMT 13)

MSU VQMT visualization

block-wise

Available colorspaces

Y

Output values

metric value

Aggregated values

standard set , NIQE mean

MSU VQMT usages

-metr niqe [-dev <OpenCL device>]

External links

original paper (A. Mittal, R. Soundararajan and A. C. Bovik),
MSU paper

Algorithm description

NIQE performs feature extraction for every 96x96 block of image on 2 scales. Than it computes correlations of features between all blocks and applies leaned model. For more details, please refer to original paper. Since version 13 VQMT can build a block-wise visualization showing contribution of each block to the final result.

This metric has a special aggregated value NIQE mean, it takes a weighted mean filtering abnormal metric values and taking suspicies values with low weight. You can turn this mode using metric settings. Also, please see our paper.

This metric only applicable to filmed scenes. Metric can produce inadequate result if some graphics is on it (including credits, subtitles, etc.). Please, run NIQE excluding all rendered scenes, they can fatally spoil average value. Also, this metric can produce bad result on scenes containing noisy objects, like sand or grass, however on scenes with big constant areas, like monotonic sky. In common case normal metric results lies in the interval 3..20.

Sometimes, metric shows better result for the compressed image and this correlates with human perception. Compressed image not always is perceived as worse. It can occur for example in case of noisy images (if the noise has non-compression nature). This is only metric in VQMT now that can detect increasing of subjective quality in comparison to original.

Sometimes, codec can allow geometry transformation (like shift of heterogeneous objects in frame), that not critical for subjective perception. Objective-reference metrics are very perceptive to such transformation, and in this cases no-reference metric can show result closer to subjective score.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	HD 720p	92.981	0.011
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	FullHD 1080p	41.738	0.024
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	UHD 4k 2160p	10.481	0.095
VQMT 13 default-Y	8 cores	HD 720p	43.979	0.023
VQMT 13 default-Y	1 core enabled	HD 720p	10.670	0.094
VQMT 13 default-Y	8 cores	FullHD 1080p	19.645	0.051
VQMT 13 default-Y	1 core enabled	FullHD 1080p	4.573	0.219
VQMT 13 default-Y	8 cores	UHD 4k 2160p	4.808	0.208
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	1.136	0.880

MSU VQMT parameters description

Mean threshold

• Description. Values of metric greater than this value will be skipped during mean calculation. 0 for disable skipping
• Default value. 15.
• Usage. -set "mean_thresh=<value>", where <value> can be:
  any floating point number

Threshold smoothing

• Description. Values of metric greater than 'Mean threshold' + 'Threshold smoothing' will be skipped, values less than 'Mean threshold' - 'Threshold smoothing' will be assumed with weight 1. Intermediate values will be taken with intermediate weight
• Default value. 5.
• Usage. -set "mean_thresh_smoothing=<value>", where <value> can be:
  any floating point number

Type of normalization

• Description. Can be: fast - the fastest algorithm, low precision; native - like in native NIQE implementation. Slowest one; precise - the most precise algorithm
• Default value. native
• Usage. -set "norm_alg=<value>", where <value> can be:
  fast, native, precise

Example

SSIM-family

Structural Similarity (SSIM)

General info

Metric type:

full-reference image metric

Value range:

(images are different) -1..1 (images are same)

Value interpretation:

bigger is better quality

MSU VQMT implementations:

CPU multithreaded fast (default),
cpu multithreaded precice,
cpu multithreaded GPU identical,
OpenCL (recommended),
CUDA

MSU VQMT visualization:

pixel-wise

Available colorspaces:

Y, U, V

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr ssim [over <color components>]
-metr ssim_precise [over <color components>]
-metr ssim_gpu_id [over <color components>]
-metr ssim_cuda [over <color components>]
-metr ssim [over <color components>] -dev <OpenCL device>

External links:

original paper (Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli)

Algorithm description

Main idea of the structure similarity index (SSIM) is to compare distortion of three image components:

• Luminance comparison
• Contrast comparison
• Structure comparison

This algorithm uses some window function $W_{i,j}$, $i,j=0..N$ and performs convolution with this window, defined as follows:

$$\left<U,W\right> = \sum\limits_{i=-R, j=-R}^{R,R}U(x+i, y+j)W_{i+R,j+R},\ \mbox{where}\ R=\frac N2$$

In fast implementation, it uses box window: $W_{i,j}=\frac1{(N+1)^2}$, in other implementations (precise, CUDA, OpenCL, GPU identical) it uses Gaussian window with σ= 1.5, N=10. You can note, that formula above uses negative $U$ indexes, and indexes out of image area. We should define image values outside of it's edge. In VQMT we spread closest edge pixel to the desired position.

SSIM uses the following convolutions:

$$\begin{aligned} \mu_x &= \left< X, W\right>\\ \mu_y &= \left< y, W\right>\\ \sigma_x &= \left< (X-\mu_x)^2, W\right>\\ \sigma_y &= \left< (Y-\mu_y)^2, W\right>\\ \sigma_{xy} &= \left< (X-\mu_x)(Y-\mu_x), W\right>\\ \end{aligned}$$

And the computed SSIM in each pixel by the following formula:

$$\mbox{SSIM}\ = \frac{(2\mu_x\mu_y+C_1)(2\sigma_{xy}+C_2)}{(\mu_x^2+\mu_y^2+C_1)(\sigma_x+\sigma_y+C_2)},$$

where $C_1=0.01^2$, $C_2=0.03^2$. Destination metric value is arithmetic mean of SSIM values for each pixel. You also can see SSIM for each individual pixel on visualization.

GPU identical, CUDA, OpenCL implementations should produce very similar result. They use Gaussian window as presice implementation. Value of Presice implementation can differs from these ones.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	HD 720p	356.442	0.003
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	FullHD 1080p	162.801	0.006
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	UHD 4k 2160p	42.519	0.024
VQMT 13 Fast-Y	8 cores	HD 720p	228.253	0.004
VQMT 13 Fast-Y	1 core enabled	HD 720p	80.533	0.012
VQMT 13 Fast-Y	8 cores	FullHD 1080p	99.314	0.010
VQMT 13 Fast-Y	1 core enabled	FullHD 1080p	36.313	0.028
VQMT 13 Fast-Y	8 cores	UHD 4k 2160p	21.397	0.047
VQMT 13 Fast-Y	1 core enabled	UHD 4k 2160p	9.369	0.107
VQMT 13 Precise-Y	8 cores	HD 720p	100.199	0.010
VQMT 13 Precise-Y	1 core enabled	HD 720p	49.398	0.020
VQMT 13 Precise-Y	8 cores	FullHD 1080p	44.808	0.022
VQMT 13 Precise-Y	1 core enabled	FullHD 1080p	22.163	0.045
VQMT 13 Precise-Y	8 cores	UHD 4k 2160p	10.704	0.093
VQMT 13 Precise-Y	1 core enabled	UHD 4k 2160p	5.640	0.177

Example (SSIM Precise)

Example (SSIM Fast)

Multi-Scale Structural Similarity (MSSSIM)

General info

Metric type:

full-reference image metric

Value range:

(images are different) -1..1 (images are same)

Value interpretation:

bigger is better quality

MSU VQMT implementations:

CPU multithreaded fast (default),
cpu multithreaded precice,
cpu multithreaded GPU identical,
OpenCL (recommended),
CUDA

MSU VQMT visualization:

pixel-wise

Available colorspaces:

Y, U, V

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr msssim [over <color components>]
-metr msssim_precise [over <color components>]
-metr msssim_gpu_id [over <color components>]
-metr msssim_cuda [over <color components>]
-metr msssim [over <color components>] -dev <OpenCL device>

Other names:

MS-SSIM

External links:

original paper (Z. Wang, A. C. Bovik and E. P. Simoncelli)

Algorithm description

This metric performs SSIM calculation as described in SSIM paragraph for 5 scales of input images. Each next scale divides width and height by 2. The result SSIM values are producted with the following powers: 0.0448, 0.2856, 0.3001, 0.2363, 0.1333.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	HD 720p	292.609	0.003
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	FullHD 1080p	153.978	0.006
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	UHD 4k 2160p	41.315	0.024
VQMT 13 Fast-Y	8 cores	HD 720p	140.919	0.007
VQMT 13 Fast-Y	1 core enabled	HD 720p	59.057	0.017
VQMT 13 Fast-Y	8 cores	FullHD 1080p	62.080	0.016
VQMT 13 Fast-Y	1 core enabled	FullHD 1080p	26.435	0.038
VQMT 13 Fast-Y	8 cores	UHD 4k 2160p	13.583	0.074
VQMT 13 Fast-Y	1 core enabled	UHD 4k 2160p	6.681	0.150
VQMT 13 Precise-Y	8 cores	HD 720p	60.840	0.016
VQMT 13 Precise-Y	1 core enabled	HD 720p	28.119	0.036
VQMT 13 Precise-Y	8 cores	FullHD 1080p	27.262	0.037
VQMT 13 Precise-Y	1 core enabled	FullHD 1080p	12.176	0.082
VQMT 13 Precise-Y	8 cores	UHD 4k 2160p	6.669	0.150
VQMT 13 Precise-Y	1 core enabled	UHD 4k 2160p	3.132	0.319

Example (MSSSIM Precise)

Example (MSSSIM Fast)

Three-commponent Structural Similarity (3SSIM)

General info

Metric type:

full-reference image metric

Value range:

(images are different) -1..1 (images are same)

Value interpretation:

bigger is better quality

MSU VQMT implementations:

CPU multithreaded (default),
OpenCL (recommended),
CUDA

Available colorspaces:

Y, U, V

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr 3ssim [over <color components>]
-metr 3ssim_cuda [over <color components>]
-metr 3ssim [over <color components>] -dev <OpenCL device>

Other names:

3-SSIM

External links:

original paper (C. Li and A. C. Bovik)

Algorithm description

3-Component SSIM Index based on region division of source frames. There are 3 types of regions - edges, textures and smooth regions. Result metric calculated as weighted average of SSIM metric for those regions. In fact, human eye can see difference more precisely on textured or edge regions than on smooth regions. Division based on gradient magnitude is presented in every pixel of images.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	HD 720p	81.189	0.012
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	FullHD 1080p	86.813	0.012
VQMT 13 OpenCL-Y	NVidia GeForce GTX 660 Ti	UHD 4k 2160p	9.364	0.107
VQMT 13 default-Y	8 cores	HD 720p	52.110	0.019
VQMT 13 default-Y	1 core enabled	HD 720p	30.669	0.033
VQMT 13 default-Y	8 cores	FullHD 1080p	23.131	0.043
VQMT 13 default-Y	1 core enabled	FullHD 1080p	13.018	0.077
VQMT 13 default-Y	8 cores	UHD 4k 2160p	5.698	0.176
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	3.350	0.299

Example

Spatio-Temporal SSIM Index

General info

Metric type:

full-reference temporal metric

MSU VQMT implementations:

this metric was temporary excluded from VQMT due to unstability of results

External links:

original paper (A. K. Moorthy and A. C. Bovik)

Algorithm description

The idea of this algorithm is to use motion-oriented weighted windows for SSIM Index. MSU Motion Estimation algorithm is used to retrieve this information. Based on the ME results, weighting window is constructed for every pixel. This window can use up to 33 consecutive frames (16 + current frame + 16). Then SSIM Index is calculated for every window to take into account temporal distortions as well. In addition, another spooling technique is used in this implementation. We use only lower 6% of metric values for the frame to calculate frame metric value. This causes larger metric values difference for difference files.

Legacy notes

This metric was temporary removed in VQMT 10.

Norm calculation metrics

Identity

General info

Metric type:

full-reference image metric

Value range:

(images are different) 0..1 (images are same)

Value interpretation:

binary mode: 1 - images are same, 0 - images are not same;
pixel mode: proportion of same pixels

MSU VQMT implementations:

CPU multithreaded

MSU VQMT visualization:

pixel-wise

Available colorspaces:

R, G, B, Y, U, V, L, RGB, YUV

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr identity [over <color components>]

Other names:

inverse L∞-norm

Algorithm description

Metric cas to modes: binary and pixels. In binary mode only two values are possible: 1 if images are pixel-wise similar, 0 if images have at least 1 different value pixel.

In pixel mode the value is proporsion of similar pixels. 1 means all pixels are similar, 0 means all pixels are dirrefent.

MSU VQMT parameters description

Mode

• Description. In binary mode only two values are possible: 1 if images are pixel-wise similar, 0 if images have at least 1 different value pixel. In pixel mode the value is proporsion of similar pixels. 1 means all pixels are similar, 0 means all pixels are dirrefent.
• Default value. binary
• Usage. -set "mode=<value>", where <value> can be:
  binary, pixels

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	476.267	0.002
VQMT 13 default-Y	1 core enabled	HD 720p	517.130	0.002
VQMT 13 default-Y	8 cores	FullHD 1080p	236.593	0.004
VQMT 13 default-Y	1 core enabled	FullHD 1080p	236.156	0.004
VQMT 13 default-Y	8 cores	UHD 4k 2160p	53.023	0.019
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	53.498	0.019
VQMT 13 default-YUV	8 cores	HD 720p	425.065	0.002
VQMT 13 default-YUV	1 core enabled	HD 720p	328.717	0.003
VQMT 13 default-YUV	8 cores	FullHD 1080p	189.175	0.005
VQMT 13 default-YUV	1 core enabled	FullHD 1080p	161.241	0.006
VQMT 13 default-YUV	8 cores	UHD 4k 2160p	42.603	0.023
VQMT 13 default-YUV	1 core enabled	UHD 4k 2160p	39.692	0.025

Example

Legacy notes

This metric was introduce in VQMT 13.

Delta

General info

Metric type:

full-reference image metric

Value range:

(original is much brigther) -1..1 (distorted is much brigther)

Value interpretation:

bigger is darker original, 0 - same brightness

MSU VQMT implementations:

CPU multithreaded

MSU VQMT visualization:

pixel-wise

Available colorspaces:

R, G, B, Y, U, V, L

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr delta [over <color components>]

Other names:

mean difference

Algorithm description

The value of this metric is the mean difference of the color value in the corresponding points of image.

$$d(X,Y) = \frac{\sum\limits_{i=1,j=1}^{m,n}Y_{i,j}-X_{i,j}}{mn}$$

where $m$ - video width, $n$ - video height, image data are in range 0..1. This formula can be rewriten to the following way: $\mbox{mean distorted brightness} - \mbox{mean original brightness}$.

This metric doesn't show a quality loss, because it can be 0 for completely different images, but you can detect general brightness shifts using this metric if you are sure images have same structure.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	512.883	0.002
VQMT 13 default-Y	1 core enabled	HD 720p	495.231	0.002
VQMT 13 default-Y	8 cores	FullHD 1080p	221.637	0.005
VQMT 13 default-Y	1 core enabled	FullHD 1080p	225.761	0.004
VQMT 13 default-Y	8 cores	UHD 4k 2160p	50.758	0.020
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	51.687	0.019

Example

Legacy notes

Since VQMT 12 metric uses input range 0..1. In legacy mode is assumed input range to be 0..255 and 0..100 for L channel.

Mean Sum of Absolute Differences (MSAD)

General info

Metric type:

full-reference image metric

Value range:

(same images) 0..1 (completely different)

Value interpretation:

smaller is better quality

MSU VQMT implementations:

CPU multithreaded

MSU VQMT visualization:

pixel-wise

Available colorspaces:

R, G, B, Y, U, V, L

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr msad [over <color components>]

Other names:

SAD, $L_1$-norm

External links:

Wikipedia

Algorithm description

This metric has very similar formula to Delta, but has a modulo around the difference:

$$d(X,Y) = \frac{\sum\limits_{i=1,j=1}^{m,n}|Y_{i,j}-X_{i,j}|}{mn}$$

where $m$ - video width, $n$ - video height, image data are in range 0..1. Metric depends only on difference of original and distorted, it is $L_1$-norm of this difference.

Unlike Delta, this metric will show real difference between images, 0 means completely equivalent images.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	511.594	0.002
VQMT 13 default-Y	1 core enabled	HD 720p	497.523	0.002
VQMT 13 default-Y	8 cores	FullHD 1080p	224.310	0.004
VQMT 13 default-Y	1 core enabled	FullHD 1080p	225.746	0.004
VQMT 13 default-Y	8 cores	UHD 4k 2160p	50.950	0.020
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	51.451	0.019

Example

Legacy notes

Since VQMT 12 metric uses input range 0..1. In legacy mode is assumed input range to be 0..255 and 0..100 for L channel.

Mean Squared Error (MSE)

General info

Metric type:

full-reference image metric

Value range:

(same images) 0..1 (completely different)

Value interpretation:

smaller is better quality

MSU VQMT implementations:

CPU multithreaded

MSU VQMT visualization:

pixel-wise

Available colorspaces:

R, G, B, Y, U, V, L

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr mse [over <color components>]

Other names:

$L_2$-norm

External links:

Wikipedia

Algorithm description

Metric depends only on difference of original and distorted, it is L2-norm of this difference. Metric could be computed using the following formula:

$$d(X,Y) = \frac{\sum\limits_{i=1,j=1}^{m,n}(Y_{i,j}-X_{i,j})^2}{mn}$$

where $m$ - video width, $n$ - video height, image data are in range 0..1.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	511.686	0.002
VQMT 13 default-Y	1 core enabled	HD 720p	494.866	0.002
VQMT 13 default-Y	8 cores	FullHD 1080p	224.004	0.004
VQMT 13 default-Y	1 core enabled	FullHD 1080p	225.775	0.004
VQMT 13 default-Y	8 cores	UHD 4k 2160p	51.577	0.019
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	49.594	0.020

Example

Legacy notes

Since VQMT 12 metric uses input range 0..1. In legacy mode is assumed input range to be 0..255 and 0..100 for L channel.

Other metrics

DCT-based Video Quality Metric (VQM)

General info

Metric type:

full-reference image metric

Value range:

(images are same) 0..∞ (bad quality)

Value interpretation:

smaller is better quality

MSU VQMT implementations:

CPU multithreaded

MSU VQMT visualization:

pixel-wise

Available colorspaces:

Y

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr vqm

External links:

original paper (Feng Xiao)

Algorithm description

This metric uses discrete cosine transform (DCT) to predict human rank. It is different from widely spreaded VQM metric by ITU, that currently not implemented in VQMT. Following calculations are processed to get value of metric:

• Color transform. YUV color space is used for metric calculation.
• DCT transform of blocks 8x8. It is used to separate images into different frequencies.
• Conversion from DCT coefficients to local contrast (LC) using following equation:

$$LC_{i,j}=\frac{DCT_{i,j}\cdot DC}{2^{20}}$$

where DC is the DCT coefficient with indexes (0, 0).

• Conversion from LC to just-noticeable difference:

$$JND_{i,j}=LC_{i,j}\cdot CSF_{i,j}$$

where CSF is Contrast Sensitivity Function. Inverse MPEG-4 default quantization matrix is used as CSF in original article.

• Weighted pooling of mean and maximum distortions. First, absolute difference D is calculated for JND coefficients following by VQM value construction:

$$\mbox{VQM}=\mbox{mean}(|D|)+0.005\cdot\mbox{max}(|D|)$$

Please, refer the original paper for the details.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	103.883	0.010
VQMT 13 default-Y	1 core enabled	HD 720p	24.360	0.041
VQMT 13 default-Y	8 cores	FullHD 1080p	47.902	0.021
VQMT 13 default-Y	1 core enabled	FullHD 1080p	11.205	0.089
VQMT 13 default-Y	8 cores	UHD 4k 2160p	12.045	0.083
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	2.848	0.351

Example

Legacy notes

In VQMT 12 we removed input aligning, also metric is optimized.

MSU Time shift

General info

Metric type:

full-reference temporal metric

Value range:

depending on settings

Value interpretation:

Metric value is shift in frames of distorted sequence relative to original. If n-th value is x, then n+x'th frame of distorted corresponds to n'th frame of original

MSU VQMT implementations:

CPU multithreaded

MSU VQMT visualization:

block-wise

Available colorspaces:

Y

Output values:

metric value
multiple individual measument results if on

Aggregated values:

standard set

MSU VQMT usages:

-metr time-shift

Algorithm description

This metric performes calculation of another metric (base metric: PSNR or SSIM) between each frame of original image and several frames of distorted image. You can choose what interval will be used in settings. This metric can detect such artifacts as skipped frame, duplicated frame, small fps mismatch.

Metric considers, that the best metric value for specific original frame is the correct shift. Metric can prefer smaller shift with base metric value X to bigger shift with metric value Y if $X > Y \cdot \mbox{threshold}$, where threshold can be set in metric settings. This helps to avoid random fluctuations.

Also, values of base metric will be smothed over adjacent frames if smoothing is on. This can help in case of very there are very similar adjacent frames in seuqence or the desired frame is absent (for example, negative shift on first frame).

MSU VQMT parameters description

Max. shift

• Description. Maximum shift, that can be detected. Note: large values leads big memory consumption.
• Default value. 5.
• Usage. -set "max-shift=<value>", where <value> can be:
  integer in range 0..25

Direction

• Description. Detect only positive shifts (frame dups), negatives (frame drops) or both.
• Default value. both
• Usage. -set "direction=<value>", where <value> can be:
  positive, negative, both

Destination metric

• Description. This metric will be used to measure similarity between frames
• Default value. psnr
• Usage. -set "metric=<value>", where <value> can be:
  psnr, ssim

Show metric values

• Description. Metric will output now only shift, but destination metric values
• Default value. false
• Usage. -set "show-metric=<value>", where <value> can be:
  true, false

Threshold

• Description. We will consider shift only if metric for neighbour frame better than this thresold multiplied to metric for similar frame
• Default value. false
• Usage. -set "threshold=<value>", where <value> can be:
  any floating point number

Smoothing

• Description. Will smooth metric values over time. If equal n, than smoothing will be in the interval frame-n..frame+n
• Default value. 1
• Usage. -set "smoothing=<value>", where <value> can be:
  integer in range 0..25

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	341.820	0.003
VQMT 13 default-Y	1 core enabled	HD 720p	276.412	0.004
VQMT 13 default-Y	8 cores	FullHD 1080p	125.327	0.008
VQMT 13 default-Y	1 core enabled	FullHD 1080p	98.154	0.010
VQMT 13 default-Y	8 cores	UHD 4k 2160p	30.121	0.033
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	24.025	0.042

Example

Legacy notes

This metric was introduced in VQMT 13.

No-reference informational metrics

MSU Blurring Metric

General info

Metric type:

no-reference image metric

Value range:

(constant image) 0..1 (very noisy)

Value interpretation:

bigger is more noise

MSU VQMT implementations:

CPU multithreaded sigma (default),
CPU multithreaded delta

MSU VQMT visualization:

pixel-wise

Available colorspaces:

Y, R, G, B for sigma
Y, U, V, R, G, B for delta

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr blurring [over <color components>]
-metr blurring_delta [over <color components>]

Algorithm description

This metric allows you to compare power of blurring of two images. If value of the metric for first picture is greater than for second, it means that second picture is more blurred, than first.

Main features: this metric is fast and doesn't require source video.

This method estimates color variance in the neighborhood of a pixel and computes average variance. This metric has 2 variations:

• Sigma (default since VQMT 11). It uses 3-pixel radius neighborhood and normalized Gaussian kernel
• Delta (the only before VQMT 11). It uses 1-pixel radius neighborhood

Notes:

• You can't measure blurriness on constant or gradient areas of input image. So, the value of metric is very dependent on amount of edges in images.
• This metric will detect not only compression artifacts, but natural not-infocus areas, so the value of metric is very dependent on area of focused objects in the frame. You shouldn't use value of this metric as final blurrness index, use is only to compare blurrness of images with similar structure.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 Delta-Y	8 cores	HD 720p	1062.426	0.001
VQMT 13 Delta-Y	1 core enabled	HD 720p	998.102	0.001
VQMT 13 Delta-Y	8 cores	FullHD 1080p	451.868	0.002
VQMT 13 Delta-Y	1 core enabled	FullHD 1080p	449.640	0.002
VQMT 13 Delta-Y	8 cores	UHD 4k 2160p	105.745	0.009
VQMT 13 Delta-Y	1 core enabled	UHD 4k 2160p	107.198	0.009
VQMT 13 Sigma-Y	8 cores	HD 720p	101.870	0.010
VQMT 13 Sigma-Y	1 core enabled	HD 720p	38.839	0.026
VQMT 13 Sigma-Y	8 cores	FullHD 1080p	45.462	0.022
VQMT 13 Sigma-Y	1 core enabled	FullHD 1080p	17.164	0.058
VQMT 13 Sigma-Y	8 cores	UHD 4k 2160p	11.145	0.090
VQMT 13 Sigma-Y	1 core enabled	UHD 4k 2160p	4.211	0.237

Example (Blurring sigma)

Example (Blurring delta)

Legacy notes

In VQMT 12 metric was optimized, consider correct range, added legacy mode

MSU Blocking Metric

General info

Metric type:

no-reference image metric

Value range:

(no blocks) 0..∞ (a lot of blocks)

Value interpretation:

bigger is more blocks

MSU VQMT implementations:

CPU multithreaded

MSU VQMT visualization:

pixel-wise

Available colorspaces:

Y

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr blocking

Algorithm description

This metric contains heuristic method for detecting objects edges, which are placed to the edge of the block. In this case metric value is pulled down, allowing to measure blocking more precisely. This metric also considers image contrast around of block and use it as weight for obtained value.

Notes:

• This algirithm considers 8x8 blocks of image, so it applicable only for I frames of video and some of encoders.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	403.223	0.002
VQMT 13 default-Y	1 core enabled	HD 720p	117.668	0.008
VQMT 13 default-Y	8 cores	FullHD 1080p	182.505	0.005
VQMT 13 default-Y	1 core enabled	FullHD 1080p	56.020	0.018
VQMT 13 default-Y	8 cores	UHD 4k 2160p	44.448	0.022
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	14.029	0.071

Example

Legacy notes

In VQMT 12 metric was optimized, consider correct range, added legacy mode.

Spatial Information (SI)

General info

Metric type:

no-reference image metric

Value range:

(simple, monotone frame) 0..1 (very complex frame)

Value interpretation:

bigger more complex frame

MSU VQMT implementations:

CPU multithreaded

MSU VQMT visualization:

pixel-wise, sobel transformation

Available colorspaces:

Y

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr si

External links:

ITU-T Recommendation P.910: Subjective video quality assessment methods for multimedia applications, 1999. - 37 p.

Algorithm description

This metric measures complexity (entropy) of an input image. This metric represents simplest SI realization that takes standard deviation of sequence of pixel values (Y-component) of Sobel transformation of input image:

$$\mbox{SI}(X) = \mbox{std}[\mbox{Sobel}(X(x,y))]$$

$\mbox{Sobel}$ is length of vector $(\mbox{Sobel}_h, \mbox{Sobel}_v)$, where $\mbox{Sobel}_h$ and $\mbox{Sobel}_v$ are horizontal and vertical Sobel transformation. While calculating Sobel, the edge pixels will be excluded from calculation.

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	339.146	0.003
VQMT 13 default-Y	1 core enabled	HD 720p	95.112	0.011
VQMT 13 default-Y	8 cores	FullHD 1080p	149.358	0.007
VQMT 13 default-Y	1 core enabled	FullHD 1080p	44.072	0.023
VQMT 13 default-Y	8 cores	UHD 4k 2160p	37.605	0.027
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	11.157	0.090

Example

Legacy notes

This metric firstly imlemented in VQMT 11, then excluded in VQMT 12 and returned in VQMT 13

Spatial Information (TI)

General info

Metric type:

no-reference temporal metric

Value range:

(simple, static video) 0..1 (very diverse frames)

Value interpretation:

bigger more diverse frames

MSU VQMT implementations:

CPU multithreaded

MSU VQMT visualization:

pixel-wise, difference between adjacent frames

Available colorspaces:

Y

Output values:

metric value

Aggregated values:

standard set

MSU VQMT usages:

-metr ti

External links:

ITU-T Recommendation P.910: Subjective video quality assessment methods for multimedia applications, 1999. - 37 p.

Algorithm description

This metric measures complexity (entropy) of defference between consequent frames of input video. This metric represents simplest TI realization that takes standard deviation of sequence of differences of corresponding pixel values of a frame and previous frame:

$$\mbox{TI}(V) = \frac{\mbox{std}[V_n(X(x,y))-V_{n-1}(X(x,y))]}{\mbox{max}}$$

Benchmark

Implementation	System & settings	Resulution	FPS	Sec. per frame
VQMT 13 default-Y	8 cores	HD 720p	691.497	0.001
VQMT 13 default-Y	1 core enabled	HD 720p	252.300	0.004
VQMT 13 default-Y	8 cores	FullHD 1080p	271.375	0.004
VQMT 13 default-Y	1 core enabled	FullHD 1080p	98.760	0.010
VQMT 13 default-Y	8 cores	UHD 4k 2160p	67.874	0.015
VQMT 13 default-Y	1 core enabled	UHD 4k 2160p	25.194	0.040

Example

Legacy notes

This metric firstly imlemented in VQMT 11, then excluded in VQMT 12 and returned in VQMT 13

License (Creative Commons Attribution)

These images by Compression.RU Team and MSU Video Group distributed under the terms of Creative Commons Attribution Lincense. In the examples we used materials from the following sources:

• https://vimeo.com/162777344 by AJA Video Systems (CC Attribution)
• https://vimeo.com/311282786 by Harold Aune (CC Attribution)
• https://vimeo.com/123907536 by Glenn Ng (CC Attribution)

---

MSU Video Quality Measurement Tools

---

• MSU Video Quality Measurement Tool (program for objective comparison)
  About Metrics Python Online help Changelog Screenshots Plugins & SDK
• MSU Perceptual Quality Tool (program for subjective comparison)
  • Basic info
  • Download
  • MSU PVQT Documentation

Other resources

Video resources:

3D and stereo video Projects on 3D and stereo video processing and analysis MSU S3D-video analysis reports MSU 3D Devices Testing 3D Displays Video Generation 3D Displays Video Capturing Stereo Video Depth Map Generation SAVAM Saliensy-Aware Video Compression & Dataset Video Matting Benchmark Video Inpainting Benchmark MSU Video Quality Measurement tools Programs with different objective and subjective video quality metrics implementation MSU Video Quality Measurement Tool - objective metrics for codecs and filters comparison Implemented metrics short info: PSNR, Delta, MSAD, MSE, SSIM, VQM, MSU Blurring Metric, MSU Blocking Metric Why update Download & Purchase Documentation MSU Human Perceptual Quality Metric - several metrics for exact visual tests Methods info: ITU-R BT.500-11: DSIS, DSCQS I & II, SCACJ; EBU: SAMVIQ; MSUCQE; Custom Download Documentation	Codecs comparisons Objective and subjective quality evaluation tests for video and image codecs MSU Video Codecs Comparison 2023-2024 MSU Video Codecs Comparison 2022 MSU Video Codecs Comparison 2021 MSU Video Codecs Comparison 2020 MSU Cloud Benchmark 2020 Cloud Encoding Services Comparison 2019 HEVC/AV1 Codec Comparison 2019 HEVC/AV1 Codec Comparison 2018 HEVC/AV1 Codec Comparison 2017 HEVC Codec Comparison 2016 HEVC Codec Comparison 2015 9-th MPEG4-AVC/H.264 Comparison 8-th MPEG4-AVC/H.264 Comparison 7-th MPEG4-AVC/H.264 Comparison 6-th MPEG4-AVC/H.264 Comparison 5-th MPEG4-AVC/H.264 Comparison Codec Analysis for Companies: Private Video Codec Comparison MSU Video Quality Measurement Tool Options Analysis of Codec x264 4-th MPEG4-AVC/H.264 Comparison Lossless Video Codecs Comparison 2007 3-rd MPEG4-AVC/H.264 Comparison HD Photo and JPEG 2000 Comparison MPEG-2 Decoders Crash Test Subjective Compar. of 4 Modern Codecs 2-nd MPEG-4 AVC/H.264 Comparison FAQ of 2-nd H.264 Comparison 1-st MPEG-4 AVC/H.264 Codecs Comparison FAQ of 1-st H.264 Comparison JPEG 2000 Image Codecs Comparison Lossless Video Codecs Comparison 2004 MPEG-4 SP/ASP Codecs Comparison Video codecs comparison (old) Ext. link: x264 parameters efficiency comparison
Public MSU video filters Here are available VirtualDub and AviSynth filters. For a given type of digital video filtration we typically develop a family of different algorithms and implementations. Generally there are also versions optimized for PC and hardware implementations (ASIC/FPGA/DSP). These optimized versions can be licensed to companies. Please contact us for details via video(at)graphics.cs.msu_ru. MSU Cartoon Restore MSU Noise Estimation MSU Frame Rate Conversion MSU Image Restoration MSU Denoising MSU Old Cinema MSU Deblocking MSU Smart Brightness and Contrast MSU Smart Sharpen MSU Noise generation MSU Noise estimation MSU Motion Estimation Information MSU Subtitles removal MSU Logo removal MSU Deflicker MSU Field Shift Fixer AviSynth plug-in MSU StegoVideo MSU Cartoonizer MSU SmartDeblocking MSU Color Enhancement MSU Old Color Restoration MSU TV Commercial Detector MSU filters FAQ MSU filters statistics	Filters for companies We are working with Intel, Samsung, RealNetworks and other companies on adapting our filters other video processing algorithms for specific video streams, applications and hardware like TV-sets, graphics cards, etc. Some of such projects are non-exclusive. Also we have internal researches. Please let us know via video(at)graphics.cs.msu_ru if you are interested in acquiring a license for such filters or making a custom R&D project on video processing, compression, computer vision. 3D Displays Video Generation 3D Displays Video Capturing Stereo Video Depth Map Generation Automatic Objects Segmentation Semiautomatic Objects Segmentation New Frame Rate Conversion New Deinterlacer MSU-Samsung Deinterlacing Project Digital TV Signal Enhancement Old Film Recovery Tuner TV Restore Panorama Video2Photo SuperResolution SuperPrecision MSU-Samsung image and video resampling MSU-Samsung Frame Rate Conversion Motion Phase filter Deshaker (video stabilization) Film Grain/Degrain filter Deblurring filter Video Content Search
Video codecs projects Different research and development projects on video codecs MSU Lossless Video Codec (Top!) MSU Screen Capture Lossless Codec (Top!) MSU MPEG-2 Video Codec x264 Codec Improvement	Other Other information Crazy gallery (filters screams :) License for commercial usage of MSU VideoGroup Public Software (please be careful: some soft like metrics has another license!)

Last updated: 12-May-2022

---

Server size: 8069 files, 1215Mb (Server statistics)

Project updated by
Server Team and MSU Video Group

Project sponsored by YUVsoft Corp.

Project supported by MSU Graphics & Media Lab