MSU PVQT–Easy subjective video testing organization for companies and universities

MSU Perceptual Video Quality Tool

Easy subjective video testing organization for companies and universities

MSU Graphics & Media Lab (Video Group)

Take a look at this article on the new site! Follow the link
https://videoprocessing.ai/vqmt/pvqt.html

Projects, ideas: Dr. Dmitriy Vatolin
Implementation: Oleg Petrov, Georgy Osipov

How to make subjective video quality comparison with MSU PVQT:

  1. Collect materials for the test

    We support more than 300 video formats and all main image formats

    See Recommendations on materials.

  2. Choose method and create test with PVQT Editor

    Now we support DSIS, DSCQS I and II, SCACJ, SAMVIQ from ITU-R BT.500 and EBU

    See Methods' brief.

  3. Invite observers and organize tests with PVQT Player

    Observers can pass test in local network, in specially prepared room or individually. ITU-R BT.500 recommends at least 15 observers to complete the test. If you need observers to test your videos, please, mail us

    See Test organization variants.

  4. Collect and process results

    You see results in real-time in most cases. Statistical estimates as well as Bradley-Terry model results for pairwise comparison are available in PVQT

    See Result inspection section in documentation.

PVQT 2.1 FREE licenses period expanded: Get now!

remains to the end of BETA-testing

Table of contents:

Download FREE BETA-testing version with limitations or Buy PRO

How to use PVQT

PVQT main window

What is PVQT?

PVQT is a tool, developed by MSU Graphics & Media Lab. Video Group for evaluating of subjective quality (with reference or without one) of a video and image files.

The tool was developed according to ITU-R BT.500 “Methodology for the subjective assessment of the quality of television pictures”. The program does not exactly repeat the standard, as it is focused on the measurement of quality for television, but PVQT is aimed for quality measure of digital materials. Also, it has some additional features not described in the standard.

PVQT structure

PVQT consists of 2 basic parts:

  • Editor (MSU PVQT) — for creating a test and collecting results (see section Creating a test in Documentation),
  • Player (MSU PVQT Player) — for performing (playing) test by observer or operator in case of Network test (see section Passing a test in Documentation).

Materials selection

To create PVQT test you need one or more groups of source files. The source files can be videos or images: PVQT supports all basic video and image formats. Each group consists of different versions of one picture, which will be compared with another picture of that group. Group can have a reference. In that case all pictures will be compared to the reference. Without reference there can be pairwise comparison with SCACJ method.

ITU-R BT.500 recommends videos to be at most 10 s length. In case of images, you can tune duration that the image displays. Whole test should be no longer 30 m including voting, training questions and period between videos. You can have several sessions with breaks, each for 30 minutes. See ITU-R BT.500 for details.

Main Features overview

Main ITU-R and BTU methods
DSIS, DSCQS I and II, SCACJ, SAMVIQ
300+ Video and all Main Image Formats
All popular video codecs, including HEVC
Real-Time result observation
Convenient control of all test passes in real-time
No-reference pairwise methods
Use SCACJ to collect pairwise results, you don't need to have references for this method
Confidence interval calculation
Automatic calculation of mean, standard deviation, confidence interval
Results table
View breaf results with information about each observer or detailed results in summary table
Bradley-Terry model
Automatic estimation of Bradley-Terry ranks for pairwise comparison
Several techniques of test organization
Observers can view test in classroom, individually, or simultaneously
LAN test passing
Use shared folders to pass test on different computers without any copying operations
Video playback bufferization
Playback has a tunable bufferization to avoid video lags
Smart diagnostic
PVQT performs checks for standard compatibility and for possible problems, all you see in message window
Fast checking of test
PVQT performs test checking before each run. This prevents test failure in middle of progress
Continuation of test
You can continue a test interrupted due to tech problem, power failure or by request without loss of progress
Training questions
Adjust the number of training questions, which helps observer to stabilize opinion and understand the task
See on big screen, TV or projector
Control test on one monitor, see on big screen, TV or projector. PVQT has built-in multiscreen support
Fast debug
Debug test to check it, run test immediately from Editor
Test estimates
PVQT calculates estimated time of test and number of questions
Grouping
Use multiple sequences in one task. Observer shouldn't see variants of only one sequence
Result collecting via E-Mail
In case of distributed observers test you can collect results via E-Mails
JSON support
PVQT saves a detailed protocol in JSON format for each test pass
Online test
Any number of observers take the test simultaneously, each on their mobile device. They're looking at the material on the big screen. No additional mobile applications needed to install
Custom tests
Customize the wording of the question, instructions, way of passing the test. Add any number of questions. You can use custom tests for paired, reference comparison, or to view single videos
Result collecting via FTP
In case of distributed observers, results can be automatically uploaded to FTP server. Test manager chooses a folder and login settings
* available in experimental version

Purchase & Download

File Edition Content Price Type
Purchase PRO version Unlimited Player & Editor $490 Windows Installer 32 & 64-Bit
MSU_PVQT_2.1r12323BETA_pro_64_Nov_1.exe
(recommended)		 Temporary, till Jun 01 2020 Player & Editor FREE Windows Installer 64-Bit
MSU_PVQT_Player_2.1r12323BETA_pro_64_Nov_1.exe Temporary, till Jun 01 2020 Player FREE Windows Installer 64-Bit
MSU_PVQT_Documentation.pdf – Documentation FREE PDF document
we have great discounts for educational or scientific use. Please, contact us using email:
video-measure@compression.ru

Methods' brief

DSIS — Double Stimulus Impairment Scale Videos are shown consequently in pairs: first video is the reference (observer is informed about it), second video is an impaired.
DSCQS I — Double Stimulus Continuous Quality Scale Type I In one playback window observer is free to switch between two videos. One of videos is the reference one, but an observer is not informed about it.
DSCQS II — Double Stimulus Continuous Quality Scale Type II Videos are played simultaneously (side-by-side) a given amount of times (repetitions), one is the reference one, but an observer is not informed about it.
SCACJ — Quality evaluation using categorical scale

Two videos are shown simultaneously, after playback observer is asked about his opinion.

You do not need to have references for this test. Pairwise results will be processed according to Bradley-Terry model.

SAMVIQ — Subjective Assessment Method for Video Quality evaluation Observer is able to play any sequence from the test set and give it mark, plus observer is able to play reference video.

Organization of testing

Single computer testing All observers pass a test on the same computer. No additional preparations are needed. It can be the computer on which test was created, or project correctly transmitted to another computer. We recommend install PVQT player (or full PVQT) to that computer and run test from Windows Explorer.
Testing via shared folder

In this method, the test can be run by an arbitrary number of observers simultaneously or not on computers on the same local network, for example, in a computer class. The test manager can view the results in real time.

Edit and save project to location available to another computers via shared folder. You should use “Relative paths” and all videos should be also available via same shared folder or be referred in project by network path.

Multiple observers can pass test simultaneously (for example, in computer class). If Editor for this project file is opened you can see progress and results of each observer in Real-time. Also, it will display overall progress in summary.

Testing with distributed observers

In this method, you are sending test and videos to observers, located anywhere, who will pass the test at any time. After test is complete, observers will send you back results of testing.

To use this method, specify email in field “Result collecting e-mail”. After test is finished, observer will be asked to send a special archive (*.pvqtres) file to that email.

After you collect some archives from observers, you can import them into PVQT using “Import results from *.pvqtres file” command (project should be saved). You can import multiple files at once.

Testing using mass view

This method will be implemented in the next version of PVQT.

You choose to show the video on a large screen, for example, through a projector that is viewed by several observers. Each of them votes on their device.

Changelog

  • Version 2.0 BETA
    • Completely new utility
    • A lot of new features
  • Version 1.0
    • DSCQS type II added
    • Bugfixing
  • Version 0.9
    • DSCQS, DSIS and SAMVIQ added
    • AviSynth support

MSU Video Quality Measurement Tools

Contacts

E-mail: video-measure@compression.ru

Other Resources

Video resources:

3D and stereo video
Projects on 3D and stereo video processing and analysis
MSU Video Quality Measurement tools
Programs with different objective and subjective video quality metrics implementation 		Codecs comparisons
Objective and subjective quality evaluation
tests for video and image codecs
Ext. link: x264 parameters efficiency comparison
Public MSU video filters
Here are available VirtualDub and AviSynth filters. For a given type of digital video filtration we typically develop a family of different algorithms and implementations. Generally there are also versions optimized for PC and hardware implementations (ASIC/FPGA/DSP). These optimized versions can be licensed to companies. Please contact us for details via video(at)graphics.cs.msu_ru. 		Filters for companies
We are working with Intel, Samsung, RealNetworks and other companies on adapting our filters other video processing algorithms for specific video streams, applications and hardware like TV-sets, graphics cards, etc. Some of such projects are non-exclusive. Also we have internal researches. Please let us know via video(at)graphics.cs.msu_ru if you are interested in acquiring a license for such filters or making a custom R&D project on video processing, compression, computer vision.
Video codecs projects
Different research and development
projects on video codecs 		Other
Other information
Last updated: 12-May-2022

Server size: 8069 files, 1215Mb (Server statistics)

Project updated by
Server Team and MSU Video Group

Project sponsored by YUVsoft Corp.

Project supported by MSU Graphics & Media Lab