Video Area Home >> Video-Codec Comparisons >> HEVC/AV1 Video Codecs Comparison 2020

Subjective MSU Video Codecs Comparison 2020

Video group head: Dr. Dmitriy Vatolin
Project head: Dr. Dmitriy Kulikov
Measurements, analysis: Dr. Mikhail Erofeev,
Egor Sklyarov,
Nickolay Safonov,
Anastasia Antsiferova

Navigation


Summary

Goal: This report presents the results of video codecs comparison using subjective quality measurement.

Main points of this test:

  • 11 codecs of HEVC/AV1/AVC and other standards were evaluated in winter 2020–2021
  • For subjective quality measurements we used Subjectify.us crowdsourcing platform. We involved more than 6,100 participants
Offline (1 fps) Online (30 fps)
Best quality
(YUV-Subjective)
1st: BVC2
2nd: Alibaba S265
3rd: KAV1_v2
1st: Alibaba S265
2nd: XCCZM265
3rd: x265
Best speed/quality trade-off
(YUV-Subjective)
1st: x264
2nd: QAV1
3rd: SIF Codec
1st: XCCZM265
2nd: SIF Codec
Figure 1 Overall subjective quality — Offline (1 fps)

Download

Subjective Report
Subjective comparison conducted on Subjectify.us platform
Released on March, 23
11 codecs
Alibaba S265, BVC2, KAV1_v1, KAV1_v2, QAV1, rav1e, Reference x265, SIF Codec, XCCZM265, x264, x265
Free version


HTML report (800 KB)


PDF & HTML reports (ZIP, 5.3 MB)


Enterprise version

Buy

You will receive enterprise versions of all reports (FullHD, Subjective, 4K and High Quality)
Contact us if you want to buy only Enterprise Subjective Report
6,100+ unique observers
236,736 valid answers
8 video sequences
Short fragments from Xiph, Vimeo, YouTube UGC
2 Encoding Use Cases
Offline (1 fps), Online (30 fps)
14 metrics
Subjective score and 13 objective, including: YUV-SSIM, YUV-PSNR (avg. log), YUV-PSNR (avg. MSE), Y-VMAF (v0.6.3)
HTML and PDF documents
540+ interactive charts and 69 pages

Participated codecs

Codec name Use cases Standard Version
1 Alibaba S265
Alibaba Taobao codec Team
Offline (1 fps),
Online (30 fps)
HEVC S265 v5, Windows
2 BVC2
Bytedance Inc.
Offline (1 fps) Other V1, Windows
3 KAV1_v1
CKL DXX ZWJ XQQ CC ZHB ZZW
Offline (1 fps) AV1 8ec71bcc, Windows
4 KAV1_v2
CKL DXX ZWJ XQQ CC ZHB ZZW
Offline (1 fps) AV1 04ea2aa4, Windows
5 QAV1
iQIYI Inc.
Offline (1 fps) AV1 Linux
6 rav1e
The rav1e contributors
Offline (1 fps) AV1 Windows
7 Reference x265
MulticoreWare, Inc., presets by MSU
Offline (1 fps),
Online (30 fps)
HEVC Windows
8 SIF Codec
SIF Codec LLC
Offline (1 fps),
Online (30 fps)
Other Windows
9 XCCZM265
XCCZM Codec Team
Offline (1 fps),
Online (30 fps)
HEVC Linux
10 x264
x264 project
Offline (1 fps),
Online (30 fps)
AVC Windows
11 x265
MulticoreWare, Inc.
Offline (1 fps),
Online (30 fps)
HEVC Windows

Subjective Comparison Methodology

For subjective quality measurements we used Subjectify.us crowdsourcing platform. We involved more than 6,100 participants. After deleting replies from bots we got 236,736 pairwise answers. Bradley-Terry model was used to compute global rank.

  1. To conduct an online crowdsourced comparison, we uploaded encoded streams to Subjectify.us. For better browser compatibility we performed transcoding with x264 and CRF=16.
  2. The platform hired study participants and showed the upload streams to them in pairs. Each pair consisted of two variants of the same test video sequence encoded by various codecs at various bitrates. Videos from each pair were presented to study participant sequentially (i.e., one after another) in full-screen mode. After viewing each pair, participants were asked to choose the video with the best visual quality. They also had the option to play the videos again or to indicate that the videos have equal visual quality. We assigned each study participant 12 pairs, including 2 hidden quality-control pairs, and each received money reward after successfully completing the task. The quality-control pairs consisted of test videos compressed by the x264 encoder at 1 Mbps and 4 Mbps. Responses from participants who failed to choose the 4 Mbps sequence for one or more quality-control questions were excluded from further consideration.
  3. In total we collected 236,736 valid answers from 6,100+ unique participants. To convert the collected pairwise results to subjective scores, we used the Bradley-Terry model [1]. Thus, each codec run received a quality score. We then linearly interpolated these scores to get continuous rate-distortion (RD) curves, which show the relationship between the real bitrate (i.e., the actual bitrate of the encoded stream) and the quality score. Section “RD Curves” shows these curves.

Test Hardware Characteristics

  • CPU: Intel Socket 1151 Core i7 8700K (Coffee Lake) (3.7Ghz, 6C12T, TDP 95W)
  • Mainboard: ASRock Z370M Pro4
  • RAM: Crucial CT16G4DFD824A 2x16GB (totally 32 GB) DIMM DDR4 2400MHz CL15
  • OS: Windows 10 x64

Codec Analysis and Tuning for Codec Developers and Codec Users

Computer Graphics and Multimedia Laboratory of Moscow State University:

  • 15+ years working in the area of video codec analysis and tuning using objective quality metrics and subjective comparisons.
  • 27+ reports of video codec comparisons and analysis (H.265, H.264, AV1, VP9, MPEG-4, MPEG-2, decoders' error recovery).
  • Methods and algorithms for codec comparison and analysis development, separate codec's features and codec's options analysis.

We could perform next tasks for codec developers and codec users.

Strong and Weak Points of Your Codec

  • Deep encoder parts analysis (ME, RC on GOP, mode decision, etc).
  • Weak and strong points for your encoder and complete information about encoding quality on different content types.
  • Encoding Quality improvement by the pre and post filtering (including technologies licensing).

Independent Codec Estimation Comparing to Other Codecs for Different Use-cases

  • Comparative analysis of your encoder and other encoders.
  • We have direct contact with many codec developers.
  • You will know place of your encoder between other newest well-known encoders (compare encoding quality, speed, bitrate handling, etc.).

Encoder Features Implementation Optimality Analysis

We perform encoder features effectiveness (speed/quality trade-off) analysis that could lead up to 30% increase in the speed/quality characteristics of your codec. We can help you to tune your codec and find best encoding parameters.

Thanks


Special thanks to the following contributors of our previous comparisons
Google Intel AMD NVidia
ATI Adobe ISPhone dicas
KDDI R&D labs Dolby Tata Elxsi Octasic
Qualcomm Voceweb Elgato Telecast
Huawei MainConcept Vitec Tencent

Leave a feedback



Contact Information



compression.ru
in cooperation
with
Lomonosov MSU
Graphics & Media Lab
(Video Group)
Dubna State University
Institute for Information
Transmission Problems RAS

Subscribe to report updates



Materials about MSU Codec Comparison


See all MSU Video Codecs Comparisons

MSU video codecs comparisons resources:


Other Materials


Video resources:

Last updated: 09-September-2021


Server size: 8069 files, 1215Mb (Server statistics)

Project updated by
Server Team and MSU Video Group

Project sponsored by YUVsoft Corp.

Project supported by MSU Graphics & Media Lab