MSU Annual Video Codecs Comparison 2020:
Call for codecs

All industry professionals will know about your codec success!

compression.ru	in cooperation with	Lomonosov MSU Graphics & Media Lab (Video Group)	Fifteen modern video codecs comparison For real researchers, developers and professional users in field of high-end video compression
		Dubna State University
		Institute for Information Transmission Problems, Russian Academy of Science

Important update on comparison use-cases

Due to many requests, in this year encoders will be compared in slightly different conditions. Constant quality encoding mode (CRF, QP) is preferable (but it's still possible to participate with VBR mode). Participants are to provide encoding presets which produce bitrates and encoding speed similar to reference (x265) encoder: instruction for choosing encoding presets (PDF)

Encoding use cases to be compared in 2020:

1. Main comparison (FullHD videos, objective quality metrics)

"Real-time encoding" - 30fps, constant quality mode
"Offline encoding" - 1fps, constant quality mode

2. Subjective comparison (FullHD videos, subjective quality)

"Real-time encoding" - 30fps, constant quality mode
"Offline encoding" - 1fps, constant quality mode

3. Comparison at 4K videos (4K(UHD)&HDR videos, objective quality metrics)

"Fast encoding" - 1fps, constant quality mode
"Universal encoding" - 20fps, constant quality mode

Important Dates

May, 14 ~~April, 30~~

Application deadline

Participants are to provide codec name, name of encoding standard, developer's (company) name, and list of comparison parts in which they want to participate

May, 15	Deadline for settling technical problems with codec functioning
August, 3	Draft version of report that will be sent to all participants
August, 10	Reception of comments to the draft
August-September	Main and Subjective reports release
November-December	4K report release

About Annual MSU Video Codecs Comparisons

MSU team has up to 21 years of experience in video codec analysis, testing and optimization. Some facts about previous MSU Video Codecs Comparisons:

There were more than 400.000 downloads of previous H.264/MPEG-4 AVC and H.265/HEVC video codec comparisons reports
Many codecs' bugs were found and reported to developers
210+ encoders were tested
More than 35 private reports for codec developers (description of codec's weak and strong points) after public report versions

Page Outline

New in 2020 MSU Video Codecs Comparison
Task of the Comparison
Typical Scope of Test
- Summary report topics
- Comparison Rules and Methodology
Participate in 2020 Video Codecs Comparison!
What would you like to see in comparison reports?
Special thanks to our sponsors
Contact information

New features in MSU Video Codecs Comparison

Since 2020	Open voting for comparison videos selection (for participants and industry experts) Automatic codecs submission system Open to compare Lunix-based solutions Open to compare cloud-based solutions Enlarging of video collection
Since 2019	Increasing the number of test videos up to 100+, adding UGC (user-generated) videos (vlogs, noise videos, etc.) and HDR videos Open to compare cloud-based solutions Enlarging of video collection Going to use new no-reference metrics
Since 2018	Increased target bitrates for comparison on 4K videos Started to use VMAF objective metric Upgraded testing hardware to Coffee Lake Enlarged video collection (15833 videos)
Since 2017	Started to perform subjective comparisons with hundreds of observers Started to compare AV1 Enlarged video collection (9299 videos)
Since 2016	New methodology of videos selection New video collection (2909 videos)

Task of the Comparison

We perform comparative unbiased analysis for

software implementations
hardware (GPU-based) implementations
cloud-based implementations

of H.265/HEVC video coding standard and compare it to the best implementations of other video coding standards (H.264/MPEG-4 AVC, AV1, VP9, VP8, MPEG-4 and other) using objective metrics (SSIM, VMAF, PSNR and other).

With MSU Codecs Comparison developers can verify the performance of their codec. We share test sequences, encoding parameters and codecs versions so all developers can reproduce the results of the comparison. Participation with publishing of all results is for FREE.

Scope of Test

Summary report topics:

Objective measurements + Subjective analysis
Encoding time
Bitrate keeping for evaluating rate-control mechanism
Speed/Quality trade-off analysis
Averaged objective results analysis
Leaders in different use-cases

Comparison methodology main points:

50-100 HD video sequences (main report) + 10-12 4K&HDR video sequences (report appendix).
Number of videos may be increased and depends on the number of participating codecs
SSIM, PSNR, VMAF objective metrics & subjective evaluation
Bitstream analysis; profile, latency and other restrictions control
3 color-planes (Y,U,V) and integral metric values
2 use-cases for comparison on FullHD videos: Fast and Offline encoding (differ by encoding speed) + 2 use-cases for subjective comparison on FullHD videos + 2 special use-cases for comparison on 4K videos
8-10 different target bitrates (1-12 Mbps for HD and 2-20 Mbps for 4K)
Prosumer-level modern hardware
Fully automatic testing system
7000+ result figures, PDF and HTML reports

Comparison Rules and Methodology

Please pay attention that we will use multi-core CPU for encoding, so you can use multi-threading

Decoding is performed with reference decoder (for H.264/AVC and H.265/HEVC only). For other standards participants should provide decoder
We don't limit GOP size and intra-period
We don’t limit number of passes in the encoding. Total encoding time should fulfill use case speed requirements
Constant quality mode is used during all tests (other modes are discussed individually)
Before results publishing each developer will receive the results of its codec and competitive free open-source codecs. Developers of each codec can validate the results and write a comment (one paragraph) about the comparison results which will be included in the report
Participation is free with publication of the results
- You can join comparison for free if you agree that your codec results will be published
- Private participation. Compare your codec with world leaders staying incognito! If your company wants to know results of your codec's testing with possibility to exclude them from publication and information disclosure, you should pay for measurements and report preparing before comparison begins
Enterprise version of comparison report is available for direct participants for free
All participants will receive following deliverables to verify the results for free:
- video sequences used in comparison
- binaries of all free encoders used in comparison to verify the results
- all raw video quality metric and encoding speed data for its encoder and for all of free encoders used in comparison
We are willing to completely or partially delete information about some codec in the public version of comparison report only in exceptional cases (e.g. critical errors in the codec)

Test Hardware Characteristics

The following hardware is used for codecs testing:

CPU: Intel Socket 1151 Core i7 8700K (Coffee Lake) (3.7Ghz, 6C12T, TDP 95W)
Mainboard: ASRock Z370M Pro4
RAM: Crucial CT16G4DFD824A 2x16GB (totally 32 GB) DIMM DDR4 2400MHz CL15
OS: Windows 10 x64 and Linux (Ubuntu)

Encoding speed requirements

For encoder alignment selected presets should provide following encoding speed.
All speed requirements are checked at video sequence encoded at 6Mbps:

For objective quality comparison:
- Real-time – 1080p@30fps
- Offline – 1080p@1fps and SSIM-RD curve better than x264-veryslow
For subjective quality comparison:
- Real-time – 1080p@30fps
- Offline – 1080p@1fps
For objective quality comparison on 4K(UHD)&HDR videos:
- Fast – 20fps
- Universal – 1fps

Codec Requirements

Presets for different speed requirements should be provided by the developers

Codec should allow to set arbitrary bitrate of resulting stream in constant quality mode

Preferable codec interface - console codec version (with batch processing support — bitrate and file names must be possible to assign from the command line)

Encoder should be compatible with reference decoder

Developers Deliverables

The following deliverables should be provided by each developer:

Codec files (CLI executable file is preferable)

Codec's presets

Full comparison methodology

Take part in 2020 Video Codecs Comparison!

Deepest codecs review: 5 reports, including subjective with 1000+ viewers and 7000+ charts

Applying for participating, you agree with comparison rules.

Contact for participation.

Please list the following:

use cases you want to participate in (Real-time, Offline, Real-time subjective, Offline Subjective, 4K Universal, 4K Fast)
codec name
standard
which codec version you want to submit (for Windows or for Linux)
company name
kind of participation (Free/public - we test your encoder and publish all results; or Private - you pay a participation fee, we test your encoder and send you the results, then you make the decision to publish results or not)