MSU Cloud Video Transcoding Benchmark 2020
conducted by Moscow State University Graphics & Media Lab Video Group
| Video group head: | Dr. Dmitriy Vatolin |
| Project head: | Dr. Dmitriy Kulikov |
| Measurements, analysis: |
Dr. Mikhail Erofeev,
Alexander Yakovenko, Egor Sklyarov, Anastasia Antsiferova |
Contents
- About
- List of compared services
- Results
- Download HTML report
- Methodology
- List of test videos
- Contact
- Subscribe to receive notifications about our reports
About
We perform an independent comparison of cloud-based video-encoding services. This year test included encoding performance comparison using services' default settings. The results of our previous comparison showed that in the case of cloud services that provide the same quality, the file-size difference can reach 100% and the cost per minute difference is 700%.
Comparison main points
-
Encoding time and objective videos quality are measured
We perform a series of measurements to estimate average time needed for transcoding. Objective algorithms are used for estimating the resulting video quality -
Two encoding use cases: default encoding options and automatic sequence-based options
This year we performed two types of tests: encode test videos using service default encoding settings and using automatic sequence-based settings (per-title encoding). If a service does not have automatic sequence-based settings, it is not be presented in this test -
FullHD videos of different type and content (including UGC)
15 FullHD videos were used for tests. We performed two types of measurements: the main case involved encoding into FullHD, and the additional case included transcoding into several resolutions -
Anonymous accounts were used for testing
For fair comparison, we used accounts which were not connected to us as the comparison organizers. This was done to prevent the services reserving separate servers/rules for comparison test videos processing. If the account was provided by the owner of the service — it is noted in list of compared services
List of compared services
| Service name | Sequence-based encoding | Anonymous account | Storage | |
|---|---|---|---|---|
| 1 | Alibaba ApsaraVideo for Media Processing | Yes | Yes | Alibaba Cloud OSS |
| 2 | AWS Elemental MediaConvert | Partial2 4 | Yes | Amazon S3 (us-east-1) |
| 3 | Coconut | Partial4 | Yes | Amazon S3 (us-east-1) |
| 4 | Kingsoft Cloud | No | No1 | Kingsoft Cloud |
| 5 | Qencode | Partial4 | Yes | Amazon S3 (us-east-1) |
| 6 | Tencent Media Processing Service | Partial3 | No1 | Tencent Cloud COS |
| 7 | Zencoder | Yes | No1 | Amazon S3 (us-east-1) |
8*. FFMpeg was also used for measurements just as an example to be compared with Cloud-encoding solutions.
Notes
- 1 — account was provided by service team
- 2 — we didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started
- 3 — service provides parameterless sequence-based bitrate selection but doesn't calculate output resolutions. The following resolution ladder was used: 1080p, 720p, 480p, 360p, 240p
-
4 — service provides sequence-based bitrate selection but doesn't calculate output resolutions. The following resolution-quality parameters were used:
Resoultion AWS QVBR Coconut quality Qencode min–max CRFs 1080p 9 4 11–20 720p 8 4 11–20 720p 7 3 21–30 480p 7 3 21–30 360p 7 3 21–30 240p 7 3 21–30
Overall conclusions
Default services options were used for comparison, so the leaders are judged not only for quality score, but also for speed and cost trade-off. Below are the winners by different categories: speed/quality trade-off shows how frequently a service was pareto-optimal for encoding speed and quality among all test videos, cost/quality trade-off shows how frequently a service was pareto-optimal for video encoding cost and quality among all test videos, and ease of use shows our subjective estimation on service convenience.
| Category | Metric | HEVC | H.264 | |
|---|---|---|---|---|
| Best speed/quality trade-off | YUV-SSIM |
|
|
|
| YUV-PSNR |
|
|||
| Y-VMAF |
|
|||
| Best cost/quality trade-off | All metrics (YUV-SSIM, YUV-PSNR, Y-VMAF) |
|
|
|
| Ease of use |
|
|||
Encoding time
We performed three encodes for each sequence on different days and day times. The chart below shows a deviation of encoding time among all videos for each iteration.
Big delays could be caused by high load of service resources (and big queues) or long time of accessing our videos from storage (the table with services description above shows which storage was used for each service).
Alibaba ApsaraVideo for Media Processing showed the least average encoding time and deviation for both HEVC and H.264 encoding
AWS Elemental MediaConvert showed small encoding time and deviation for H.264 encoding
Kingsoft Cloud showed small time deviation for both HEVC and H.264 encoding
Different resolution test
Three resolutions were used to transcode source videos: FullHD(1920x1080), HD(1280x720), SD(854x480). The chart below shows RD-curves for different resolutions at pyranha_rafting video sequence.
|
Resolution: 480p |
Resolution: 720 |
|
Resolution: 1080 |
Resolution: all on one chart |
Merged resolutions chart shows all RD-curves constructed into one curve by the rule: it contains a point from each curve if this points is not covered by quality of a point from another resolution curve.
Quality/encoding speed trade-off
Integrating RD-curves by bitrate and calculating relative scores (AWS Elemental MediaConvert was takes as a reference), we got relative quality/relative speed plots (detailed methodology).
The chart below shows relative quality and speed scores averaged for all videos (HEVC encoding, SSIM metric).
Tencent Media Processing Service, Zencoder and Alibaba ApsaraVideo for Media Processing were most frequently optimal among all services.
Cost/quality trade-off
The following charts show relative bitrate and encoding cost for same quality (SSIM) averaged for all videos.
Kingsoft Cloud, Qencode (for H.264 and HEVC encoding) and Coconut (for HEVC encoding) showed good quality for the least prices.
| Service name | HEVC encoding | H.264 encoding |
|---|---|---|
| Alibaba ApsaraVideo for Media Processing | SD: 4.22 HD: 5.63 FullHD: 11.25 | SD: 0.84 HD: 1.13 FullHD: 2.25 |
| AWS Elemental MediaConvert | SD: 2.72 HD: 5.44 FullHD: 5.44 | SD: 0.85 HD: 1.7 FullHD: 1.7 |
| Coconut | SD: 1.5 HD: 1.5 FullHD: 1.5 | SD: 1.5 HD: 1.5 FullHD: 1.5 |
| Kingsoft Cloud | SD: 0.72 HD: 1.4 FullHD: 2.8 | SD: 0.18 HD: 0.37 FullHD: 0.71 |
| Qencode | SD: 1 HD: 2 FullHD: 2 | SD: 0.5 HD: 1 FullHD: 1 |
| Tencent Media Processing Service | SD: 1.27 HD: 2.48 FullHD: 4.94 | SD: 0.79 HD: 1.09 FullHD: 2.15 |
| Zencoder | SD: 5 HD: 10 FullHD: 10 | SD: 5 HD: 10 FullHD: 10 |
Overall comparison of HEVC and H.264 encoding
The plot below shows encoding speed and relative quality scores for HEVC and H.264 averaged for all test videos, SSIM metric. The results for different metrics are almost the same (see all charts in full report). H.264 encoding results of AWS Elemental MediaConvert was used a reference to get quality scores.
Tencent Media Processing Service, Kingsoft Cloud and Zencoder showed best quality for HEVC encoding, Alibaba ApsaraVideo for Media Processing showed also good quality for higher encoding speed.
Per-title encoding test
In addition to default encoding settings, per-sequence encoding optimization was tested for services, which supported this feature at the time we did the tests. This option optimizes encoding settings to each input video to get better encoding performance (better quality of less bitrate).
We didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started, and Qencode sequence-based encoding, because it produces only one output without any bitrate or resolution variation.
According to our tests, Tencent Media Processing Service showed the best quality for per-title encoding, Alibaba ApsaraVideo for Media Processing is at the second place.
Download full HTML report
|
MSU Cloud Benchmark 2020
Download free full version 
|
|
|
Video Transcoding Clouds Comparison 2019
Download free full version 
|
|
Methodology
Brief methodology review is given in the presentation:
1. Preprocessing
We performed lossless compression with x264 (x265 for AWS MediaConvert) and passed compressed stream as input, as most of services don't support raw YUV420p.2. Storage
To equalize network latency in speed measurements we used Amazon S3 storage (us-east-1 region) for most of the services as widely used solution. Some services (Alibaba Cloud, Tencent Cloud) don't support Amazon S3, so we used corresponding storage analogs (Alibaba Cloud OSS, Kingsoft Cloud, Tencent Cloud COS).3. Encoding
We used services' API to run encoding tasks. We performed H.264 and HEVC transcoding with multiple bitrates for each resolution. In the first use case we used default options everywhere it was possible.| Bitrate, kbps | 480p | 720p | 1080p |
|---|---|---|---|
| 75 | + | ||
| 120 | + | ||
| 250 | + | + | |
| 500 | + | + | |
| 750 | + | + | |
| 1000 | + | + | + |
| 2000 | + | + | + |
| 4000 | + | + | |
| 6000 | + | + | |
| 8000 | + | ||
| 10000 | + | ||
| 12000 | + |
4. Speed measurement
This year we tried to measure encoding time including files transmission. We used services' callbacks to get jobs finish time. Also, we encoded one job multiple times to estimate deviation and minimize single time delays.5. Objective metrics calculation
We used MSU VQMT to calculate objective metrics (SSIM, PSNR, VMAF) against original YUV files.6. Plots
We merge RD curves of different resolutions to evaluate overall service performance. After that we made all our plots same way as in the Main report. You can learn more about our base methodology in presentation.Videos
| Name | Preview | Resolution | FPS | Frames | Description | Source |
|---|---|---|---|---|---|---|
| blue_hair | 
|
1920x1080 | 30 | 600 | Shaky handheld vlog video. ID: Vlog_1080P-35cd | YouTube UGC |
| christmas_cats | 
|
1920x1080 | 25 | 1500 | Concert record with superimposed complicated translucent CG effects | Vimeo |
| construction_site | 
|
1920x1080 | 30 | 1043 | Shots of building under construction | Vimeo |
| crowd_run | 
|
1920x1080 | 50 | 500 | A crowd of sportsmen runs while the camera slowly moves left and right | Xiph |
| football |  |
1920x1080 | 30 | 599 | People are playing football. ID: Sports_1080P-15d1 | YouTube UGC |
| hard_rock | 
|
1920x1080 | 25 | 500 | Poorly lit very noisy scene of people. ID: Musicvideo_1080P-6260 | YouTube UGC |
| kindergarten_interview | 
|
1920x1080 | 30 | 1016 | One man is interviewing other | Vimeo |
| park_mobile | 
|
1920x1080 | 24 | 359 | Information video with people speaking in front of camera and old fashioned scenes | Vimeo |
| pyranha_rafting | 
|
1920x1080 | 24 | 1203 | Panning shots of whitewater rafting | Vimeo |
| stone |  |
1920x1080 | 30 | 598 | Shots of stone in front of a camera. ID: HowTo_1080P-7cf2 | YouTube UGC |
| street_musician | 
|
1920x1080 | 24 | 974 | Handheld video of a musician performing and people listening. Heavy grain and black and white sections | Vimeo |
| summer_of_adventure | 
|
1920x1080 | 30 | 994 | Summer camp commercial, consists of nature scenes, POV shots and a slideshow | Vimeo |
| tennis_vlog | 
|
1440x1080 | 30 | 599 | Selfie video followed by a girl practicing hitting a baseball. ID: Vlog_1080P-19cc | YouTube UGC |
| the_forest | 
|
1920x1080 | 30 | 600 | The Forest gameplay. ID: Gaming_1080P-72c8 | YouTube UGC |
| wedding_party | 
|
1920x1080 | 24 | 1757 | Bride and groom dancing. Colorful illumination with flashes of light | Vimeo |
Contact Information
 |
compression.ru |
in cooperation with |
|
Lomonosov MSU
Graphics & Media Lab (Video Group) |
|
Dubna State University | ||
|
Institute for Information Transmission Problems RAS |
Subscribe to report updates
Materials about MSU Codec Comparison
See all MSU Video Codecs Comparisons
MSU video codecs comparisons resources:
- Introduction to Video Codecs Comparison
- Lossless Video Codecs Comparison 2004 (October 2004)
- MPEG-4 SP/ASP Video Codecs Comparison (March 2005)
- JPEG 2000 Image Codecs Comparison (September 2005)
- First Annual MPEG-4 AVC/ H.264 Video Codecs Comparison (January 2005)
- Second Annual MPEG-4 AVC/H.264 Video Codec Comparison (December 2005)
- Subjective Comparison of Modern Video Codecs (February 2006)
- MPEG-2 Video Decoders Comparison (May 2006)
- WMP and JPEG2000 Comparison (October 2006)
- Third Annual MPEG-4 AVC/H.264 Comparison (December 2006) (All versions for free!)
- Lossless Video Codecs Comparison 2007 (March 2007)
- Fourth Annual MPEG-4 AVC/H.264 Comparison (December 2007) (All versions for free!)
- Options Analysis of MPEG-4 AVC/H.264 Codec x264 (December 2008)
- Fifth MPEG-4 AVC/H.264 Comparison (May 2009) (All versions for free!)
- Sixth MPEG-4 AVC/H.264 Comparison (May 2010)
- Seventh MPEG-4 AVC/H.264 Comparison (May 2011)
- Eighth MPEG-4 AVC/H.264 Comparison (May 2012)
- Ninth MPEG-4 AVC/H.264 Comparison (Dec 2013)
- Tenth Video Codec Comparison (HEVC) (Oct 2015)
- Eleventh Video Codec Comparison (HEVC) (Aug 2016)
- Twelfth Video Codec Comparison (HEVC) (Aug 2017)
- Thirteen Video Codec Comparison (HEVC) (Aug 2018)
- Fourteen Video Codec Comparison (HEVC) (Sept 2019)
- Cloud Encoding Servoces Comparison 2019 (Dec 2019)
- Fifteen Video Codec Comparison (HEVC) (Dec 2020)
- Sixteen Video Codec Comparison (Dec 2021)
- Seventeen Video Codecs Comparisons (Nov 2022)
- Eighteen Video Codecs Comparisons
- Codec Analysis for Companies:
Other Materials
Video resources:
Server size: 8069 files, 1215Mb (Server statistics)
Project updated by
Server Team and
MSU Video Group
Project sponsored by YUVsoft Corp.
Project supported by MSU Graphics & Media Lab