Video Area Home >> Video Codecs Comparisons >> MSU Video Codecs Comparison 2022

MSU Video Codecs Comparison 2022 Part 2: Cloud

Seventeen Annual Video-Codecs Comparison by MSU

Video group head: Dr. Dmitriy Vatolin
Project head: Dr. Dmitriy Kulikov
Measurements, analysis: Ekaterina Shumitskaya,
Nickolay Safonov,
Nikita Alutis,
Mikhail Voronin
compression.ru Lomonosov
Moscow State University (MSU)
Graphics and Media Lab
Dubna International
State University
Institute for Information
Transmission Problems,
Russian Academy of Science

News

  • 18.04.2023 Additional charts added
  • 14.04.2023 Release of the comparison

Navigation


Objective Results


  • The results below are based solely on quality scores and do not take into account encoding speed
  • Services with scores closer than ~1% share one place
480p Comparison
H264 HEVC/AV1
Best quality
YUV-SSIM 6:1:1
1st: Alibaba Media Processing Service
2nd: Tencent MPS-O264
3nd: Bitmovin
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265, Alibaba Media Processing Service
3nd: Zencoder
Best quality
YUV-PSNR avg. MSE 6:1:1
1st: Alibaba Media Processing Service
2nd: Tencent MPS-O264
3nd: Bitmovin, Qencode
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265
3nd: Alibaba Media Processing Service
Best quality
Y-VMAF 0.6.3
1st: Alibaba Media Processing Service
2nd: Tencent MPS-O264
3nd: Bitmovin
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265
3nd: Alibaba Media Processing Service
Best quality
YUV-VMAF 0.6.3 6:1:1
1st: Alibaba Media Processing Service, Tencent MPS-O264
2nd: Bitmovin
3nd: Qencode
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265
3nd: Alibaba Media Processing Service


720p Comparison
H264 HEVC/AV1
Best quality
YUV-SSIM 6:1:1
1st: Alibaba Media Processing Service
2nd: Tencent MPS-O264
3nd: Bitmovin, Qencode
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265, Alibaba Media Processing Service
3nd: Zencoder
Best quality
YUV-PSNR avg. MSE 6:1:1
1st: Alibaba Media Processing Service, Tencent MPS-O264
2nd: Bitmovin, Qencode
3nd: Azure
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265
3nd: Alibaba Media Processing Service
Best quality
Y-VMAF 0.6.3
1st: Alibaba Media Processing Service
2nd: Tencent MPS-O264
3nd: Bitmovin, Qencode
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265
3nd: Alibaba Media Processing Service
Best quality
YUV-VMAF 0.6.3 6:1:1
1st: Alibaba Media Processing Service, Tencent MPS-O264
2nd: Bitmovin, Qencode
3nd: Azure
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265
3nd: Alibaba Media Processing Service


1080p Comparison
H264 HEVC/AV1
Best quality
YUV-SSIM 6:1:1
1st: Alibaba Media Processing Service
2nd: Tencent MPS-O264
3nd: Qencode
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265, Alibaba Media Processing Service
3nd: Zencoder
Best quality
YUV-PSNR avg. MSE 6:1:1
1st: Tencent MPS-O264
2nd: Alibaba Media Processing Service
3nd: Qencode
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265
3nd: Alibaba Media Processing Service
Best quality
Y-VMAF 0.6.3
1st: Alibaba Media Processing Service
2nd: Tencent MPS-O264
3nd: Qencode
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265
3nd: Alibaba Media Processing Service
Best quality
YUV-VMAF 0.6.3 6:1:1
1st: Tencent MPS-O264, Alibaba Media Processing Service
2nd: Qencode
3nd: Bitmovin
1st: Tencent MPS-TXAV1
2nd: Tencent MPS-V265
3nd: Alibaba Media Processing Service


The winners may vary for different objective quality metrics. The participants were rated using BSQ-rate (enhanced BD-rate) scores [1].




[1] A. Zvezdakova, D. Kulikov, S. Zvezdakov, D. Vatolin, "BSQ-rate: a new approach for video-codec performance comparison and drawbacks of current solutions," 2020.

Subjective Results


  • The results below are based solely on quality scores and do not take into account encoding speed
  • Services with scores closer than ~1% share one place
1080p Subjective Comparison
H264 HEVC/AV1
Best quality
YUV-Subjective
1st: Alibaba Media Processing Service
1st: Tencent MPS-O264
1st: Bitmovin
1st: Tencent MPS-TXAV1
1st: Tencent MPS-V265
1st: Alibaba Media Processing Service

For subjective quality measurements we used Subjectify.us crowdsourcing platform. We involved more than 10,800+ participants


Encoding time deviation


We performed three encodes for each sequence on different days and day times. The chart below shows a deviation of encoding time among all videos for each iteration. Big delays could be caused by high load of service resources (and big queues) or long time of accessing our videos from storage (the table with services description above shows which storage was used for each service).



Download and buy report


Cloud Report
Objective and subjective comparison of cloud encoding services
Released on April, 14



Full version for free
10 cloud encoding services
Alibaba Media Processing Service, AWS Elemental MediaConvert, Azure, Bitmovin, Coconut, Quencode, Tencent MPS-V265, Tencent MPS-O264, Tencent MPS-TXAV1, Zencoder
15 videos with different resolutions
1080p, 720p, 480p
28+ objective metrics
VMAF, SSIM, MS-SSIM, PSNR of different variants
HTML report (ZIP)
5400+ interactive charts

Participated services


Codec names Use cases Storage
1 Alibaba Media Processing Service
H264, HEVC Alibaba Cloud OSS
2 Tencent MPS-O264
H264 Tencent Cloud COS
3 Tencent MPS-V265
HEVC Tencent Cloud COS
4 Tencent MPS-XAV1
AV1 Tencent Cloud COS
5 AWS Elemental MediaConvert
H264, HEVC, AV1 Amazon S3 (us-east-1)
6 Azure
H264, HEVC Amazon S3 (us-east-1)
7 Bitmovin
H264, HEVC Amazon S3 (us-east-1)
8 Coconut
H264, HEVC Amazon S3 (us-east-1)
9 Quencode
H264, HEVC Amazon S3 (us-east-1)
10 Zencoder
H264, HEVC, AV1 Amazon S3 (us-east-1)

Subjective Comparison Methodology


For subjective quality measurements we used Subjectify.us crowdsourcing platform. We involved 10,800+ participants. After deleting replies from bots we got 529,171 pairwise answers. Bradley-Terry model was used to compute global rank.

To conduct an online crowdsourced comparison, we uploaded encoded streams to Subjectify.us. For better browser compatibility we performed transcoding with x264 and CRF=16.

The platform hired study participants and showed the upload streams to them in pairs. Each pair consisted of two variants of the same test video sequence encoded by various codecs at various bitrates. Videos from each pair were presented to study participant sequentially (i.e., one after another) in full-screen mode. After viewing each pair, participants were asked to choose the video with the best visual quality. They also had the option to play the videos again or to indicate that the videos have equal visual quality. We assigned each study participant 12 pairs, including 2 hidden quality-control pairs, and each received money reward after successfully completing the task. The quality-control pairs consisted of test videos compressed by the x264 encoder at 1 Mbps and 4 Mbps. Responses from participants who failed to choose the 4 Mbps sequence for one or more quality-control questions were excluded from further consideration.

In total we collected 529,171 valid answers from 10,800+ unique participants. To convert the collected pairwise results to subjective scores, we used the Bradley-Terry model [1]. Thus, each codec run received a quality score. We then linearly interpolated these scores to get continuous rate-distortion (RD) curves, which show the relationship between the real bitrate (i.e., the actual bitrate of the encoded stream) and the quality score. Section "RD Curves" shows these curves.

We obtained the subjective scores for this study using Subjectify.us. This platform enables researchers and developers to conduct subjective comparisons of image and video processing methods (e.g., compression, inpainting, denoising, matting, etc.) and carry out studies of human quality perception.

To conduct a subjective study, researchers must apply the methods under comparison to a set of test videos (images), upload the results to Subjectify.us and write a task description for study participants. Subjectify.us handles all the laborious steps of a crowdsourced study: it recruits participants, presents uploaded content in a pairwise fashion, filters out responses from participants who cheat or are careless, analyzes collected results, and generates a study report with interactive plots. Thanks to the pairwise presentation, researchers need not invent a quality scale, as study participants just select the best option of the two.

The platform is optimized for comparison of large video files: it prefetches all videos assigned to a study participant and loads them into his or her device before asking the first question. Thus, even participants with a slow Internet connection wont experience buffering events that might affect quality perception.
To try the platform in your research project, reach out to www.subjectify.us. This demo video shows an overview of the Subjectify.us workflow.


Codec Analysis and Tuning for Codec Developers and Codec Users


Computer Graphics and Multimedia Laboratory of Moscow State University:

  • 17+ years working in the area of video codec analysis and tuning using objective quality metrics and subjective comparisons.
  • 30+ reports of video codec comparisons and analysis (H.265, H.264, AV1, VP9, MPEG-4, MPEG-2, decoders' error recovery).
  • Methods and algorithms for codec comparison and analysis development, separate codec's features and codec's options analysis.

Strong and Weak Points of Your Codec

  • Deep encoder parts analysis (ME, RC on GOP, mode decision, etc).
  • Weak and strong points for your encoder and complete information about encoding quality on different content types.
  • Encoding Quality improvement by the pre and post filtering (including technologies licensing).

Independent Codec Estimation Comparing to Other Codecs for Different Use-cases

  • Comparative analysis of your encoder and other encoders.
  • We have direct contact with many codec developers.
  • You will know place of your encoder between other newest well-known encoders (compare encoding quality, speed, bitrate handling, etc.).

Encoder Features Implementation Optimality Analysis

We perform encoder features effectiveness (speed/quality trade-off) analysis that could lead up to 30% increase in the speed/quality characteristics of your codec. We can help you to tune your codec and find best encoding parameters.

Thanks


Special thanks to the following contributors of our previous comparisons
Apple Google Intel NVidia
Huawei AMD Adobe Tencent
Zoom video communications Facebook Inc. Netflix Alibaba
KDDI R&D labs Dolby Tata Elxsi Octasic
Qualcomm Voceweb Elgato Telecast
ATI MainConcept Vitec dicas

Contact Information

We appreciate any feedback on our comparison


Subscribe to report updates




Other Materials


Video resources:

Last updated: 12-May-2022


Server size: 8069 files, 1215Mb (Server statistics)

Project updated by
Server Team and MSU Video Group

Project sponsored by YUVsoft Corp.

Project supported by MSU Graphics & Media Lab