| RUSSIAN |
MSU Video Quality Measurement tools:
subjective quality comparison
MSU Graphics & Media Lab (Video Group)
Subjective quality metric comparison
There are a lot of full reference objective quality metrics and each of them somehow represents difference between two video sequences.
We performed comparisons with subjective quality for each metrics to understand which metric is better.
Such comparison provided for the most popular metrics: PSNR, SSIM, 3-SSIM, MS-SSIM and new stSSIM.
Subjective quality of video sequence is represented as MOS value (Mean Opinion Score) - mean quality score of a sequence estimated by a group of people.
More about standards of subjective quality measurement you can read on the VQEG website or in the description of our Perceptual Video Quality Tool.
Two indices of similarity between subjective quality and objective metric values for these metrics provided:
Pearson Linear Correlation Coefficient (LCC)
Pearson product-moment correlation coefficient is a measure of the correlation (linear dependence) between two variables X and Y, giving a value between +1 and -1 inclusive.
It is widely used in the sciences as a measure of the strength of linear dependence between two variables. Wikipedia
So, correlation value of +1 means that the metric values have a perfect linear dependence on a subjective quality.
Graphically it means that pairs of (MOS, metric value) for all of the videos in the database lie on the strict line.
LCC index can answer the question: how well metric represents how much one video is visually better than another?
Automatically this means that higher metric values means better subjective visual quality.
Also we perform non-linear regression of metric values with logistic function over MOS values, to fit metric model to the MOS scores.
We use following four-parameters function for the regression:
- β1 = max(Q)
- β2 = min(Q)
- β3 = mean(MOS)
- β4 = 1.0
Spearmen Rank Order Correlation Coefficient (SROCC)
Spearman's rank correlation coefficient is a non-parametric measure of statistical dependence between two variables. It assesses how well the relationship between two variables can be described using a monotonic function. If there are no repeated data values, a perfect Spearman correlation of +1 or -1 occurs when each of the variables is a perfect monotone function of the other. (Wikipedia)So, correlation value of +1 means that metric values are the perfect monotone function of subjective quality: two metric values X1 and X2 where X1 is greater than X2 means that average human will rate first video higher than other. Otherwise for correltion value -1. Correlation value 0 means there is no dependency at all. Usually objective quality metrics are non-linear functions of subjective quality. In this case SROCC index can answer the question: how well metric values correspond that one video is visually better than another? (higher metric value corresponds th the higher subjective quality?)
To calculate SROCC you need to convert pairs of metric values and MOS (Xi, Yj) for every video in the video database to the ranks (xi, yj) - both values from a pair are assigned a rank equal to the average of their positions in the ascending order of the values separately. Then you need to perform Pearson Linear Correlation calculation over the results.
Comparison results
Following coefficients provided for public video bases with subjective quality values:- Laboratory for Image & Video Engineering Video Quality Database
- Video Quality Experts Group Phase I video sequences database
Correlation coefficients for LIVE video quality database
Correlation coefficients for VQEG Phase-I video quality database
| E-mail: |  |
Other Resources
Video resources:
Server size: 8069 files, 1215Mb (Server statistics)
Project updated by
Server Team and
MSU Video Group
Project sponsored by YUVsoft Corp.
Project supported by MSU Graphics & Media Lab