MSU Cloud Video Transcoding Benchmark 2020

Cloud Video Transcoding Services Comparison
conducted by Moscow State University Graphics & Media Lab Video Group

Video group head: Dr. Dmitriy Vatolin
Project head: Dr. Dmitriy Kulikov
Measurements, analysis: Dr. Mikhail Erofeev,
Alexander Yakovenko,
Egor Sklyarov,
Anastasia Antsiferova


Contents


About

We perform an independent comparison of cloud-based video-encoding services. This year test included encoding performance comparison using services' default settings. The results of our previous comparison showed that in the case of cloud services that provide the same quality, the file-size difference can reach 100% and the cost per minute difference is 700%.

Comparison main points

  • Encoding time and objective videos quality are measured
    We perform a series of measurements to estimate average time needed for transcoding. Objective algorithms are used for estimating the resulting video quality
  • Two encoding use cases: default encoding options and automatic sequence-based options
    This year we performed two types of tests: encode test videos using service default encoding settings and using automatic sequence-based settings (per-title encoding). If a service does not have automatic sequence-based settings, it is not be presented in this test
  • FullHD videos of different type and content (including UGC)
    15 FullHD videos were used for tests. We performed two types of measurements: the main case involved encoding into FullHD, and the additional case included transcoding into several resolutions
  • Anonymous accounts were used for testing
    For fair comparison, we used accounts which were not connected to us as the comparison organizers. This was done to prevent the services reserving separate servers/rules for comparison test videos processing. If the account was provided by the owner of the service — it is noted in list of compared services

List of compared services

Service name Sequence-based encoding Anonymous account Storage
1 Alibaba ApsaraVideo for Media Processing Yes Yes Alibaba Cloud OSS
2 AWS Elemental MediaConvert Partial2 4 Yes Amazon S3
(us-east-1)
3 Coconut Partial4 Yes Amazon S3
(us-east-1)
4 Kingsoft Cloud No No1 Kingsoft Cloud
5 Qencode Partial4 Yes Amazon S3
(us-east-1)
6 Tencent Media Processing Service Partial3 No1 Tencent Cloud COS
7 Zencoder Yes No1 Amazon S3
(us-east-1)

8*. FFMpeg was also used for measurements just as an example to be compared with Cloud-encoding solutions.

Notes

  • 1 — account was provided by service team
  • 2 — we didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started
  • 3 — service provides parameterless sequence-based bitrate selection but doesn't calculate output resolutions. The following resolution ladder was used: 1080p, 720p, 480p, 360p, 240p
  • 4 — service provides sequence-based bitrate selection but doesn't calculate output resolutions. The following resolution-quality parameters were used:
    Resoultion AWS QVBR Coconut quality Qencode min–max CRFs
    1080p 9 4 11–20
    720p 8 4 11–20
    720p 7 3 21–30
    480p 7 3 21–30
    360p 7 3 21–30
    240p 7 3 21–30

Overall conclusions

Default services options were used for comparison, so the leaders are judged not only for quality score, but also for speed and cost trade-off. Below are the winners by different categories: speed/quality trade-off shows how frequently a service was pareto-optimal for encoding speed and quality among all test videos, cost/quality trade-off shows how frequently a service was pareto-optimal for video encoding cost and quality among all test videos, and ease of use shows our subjective estimation on service convenience.

Category Metric HEVC H.264
Best speed/quality trade-off YUV-SSIM
  1. Tencent Media Processing Service
  2. Zencoder
  3. Alibaba ApsaraVideo for Media Processing
  1. Tencent Media Processing Service
  2. Alibaba ApsaraVideo for Media Processing
YUV-PSNR
  1. Tencent Media Processing Service
  2. Zencoder
  3. Alibaba ApsaraVideo for Media Processing
Y-VMAF
  1. Tencent Media Processing Service
  2. Alibaba ApsaraVideo for Media Processing
  3. AWS Elemental MediaConvert
Best cost/quality trade-off All metrics (YUV-SSIM, YUV-PSNR, Y-VMAF)
  1. Tencent Media Processing Service
  2. Kingsoft Cloud
  3. Coconut
  1. Tencent Media Processing Service
  2. Kingsoft Cloud
Ease of use
  1. Coconut

Encoding time

We performed three encodes for each sequence on different days and day times. The chart below shows a deviation of encoding time among all videos for each iteration.
Big delays could be caused by high load of service resources (and big queues) or long time of accessing our videos from storage (the table with services description above shows which storage was used for each service).

Alibaba ApsaraVideo for Media Processing showed the least average encoding time and deviation for both HEVC and H.264 encoding
AWS Elemental MediaConvert showed small encoding time and deviation for H.264 encoding Kingsoft Cloud showed small time deviation for both HEVC and H.264 encoding

Figure 1

Different resolution test

Three resolutions were used to transcode source videos: FullHD(1920x1080), HD(1280x720), SD(854x480). The chart below shows RD-curves for different resolutions at pyranha_rafting video sequence.

Resolution: 480p

Figure 2. RD-curves for tennis_vlog, 480p

Resolution: 720

Figure 3. RD-curves for tennis_vlog, 720p

Resolution: 1080

Figure 4. RD-curves for tennis_vlog, 1080p

Resolution: all on one chart

Figure 5. All resolutions RD-curves for tennis_vlog

Merged resolutions chart shows all RD-curves constructed into one curve by the rule: it contains a point from each curve if this points is not covered by quality of a point from another resolution curve.

Figure 6. Merged resolutions curves for tennis_vlog

Quality/encoding speed trade-off

Integrating RD-curves by bitrate and calculating relative scores (AWS Elemental MediaConvert was takes as a reference), we got relative quality/relative speed plots (detailed methodology).
The chart below shows relative quality and speed scores averaged for all videos (HEVC encoding, SSIM metric).
Tencent Media Processing Service, Zencoder and Alibaba ApsaraVideo for Media Processing were most frequently optimal among all services.

Figure 7. Merged resolutions speed/bitrate trade-off, HEVC encoding, average for all videos, SSIM.

Cost/quality trade-off

The following charts show relative bitrate and encoding cost for same quality (SSIM) averaged for all videos.
Kingsoft Cloud, Qencode (for H.264 and HEVC encoding) and Coconut (for HEVC encoding) showed good quality for the least prices.

Figure 8. Merged resolutions cost/bitrate trade-off, HEVC encoding, average for all videos, SSIM.
Service name HEVC encoding H.264 encoding
Alibaba ApsaraVideo for Media ProcessingSD: 4.22
HD: 5.63
FullHD: 11.25
SD: 0.84
HD: 1.13
FullHD: 2.25
AWS Elemental MediaConvertSD: 2.72
HD: 5.44
FullHD: 5.44
SD: 0.85
HD: 1.7
FullHD: 1.7
CoconutSD: 1.5
HD: 1.5
FullHD: 1.5
SD: 1.5
HD: 1.5
FullHD: 1.5
Kingsoft CloudSD: 0.72
HD: 1.4
FullHD: 2.8
SD: 0.18
HD: 0.37
FullHD: 0.71
QencodeSD: 1
HD: 2
FullHD: 2
SD: 0.5
HD: 1
FullHD: 1
Tencent Media Processing ServiceSD: 1.27
HD: 2.48
FullHD: 4.94
SD: 0.79
HD: 1.09
FullHD: 2.15
ZencoderSD: 5
HD: 10
FullHD: 10
SD: 5
HD: 10
FullHD: 10
Encoding cost per minute, 0.01$ (September 2020)

Overall comparison of HEVC and H.264 encoding

The plot below shows encoding speed and relative quality scores for HEVC and H.264 averaged for all test videos, SSIM metric. The results for different metrics are almost the same (see all charts in full report). H.264 encoding results of AWS Elemental MediaConvert was used a reference to get quality scores.
Tencent Media Processing Service, Kingsoft Cloud and Zencoder showed best quality for HEVC encoding, Alibaba ApsaraVideo for Media Processing showed also good quality for higher encoding speed.

Figure 9. HEVC and H.264 encoding speed/bitrate comparison, SSIM metric
FFMpeg was used for measurements just as an example to be compared with Cloud-encoding solutions

Per-title encoding test

In addition to default encoding settings, per-sequence encoding optimization was tested for services, which supported this feature at the time we did the tests. This option optimizes encoding settings to each input video to get better encoding performance (better quality of less bitrate).

We didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started, and Qencode sequence-based encoding, because it produces only one output without any bitrate or resolution variation.

According to our tests, Tencent Media Processing Service showed the best quality for per-title encoding, Alibaba ApsaraVideo for Media Processing is at the second place.

Figure 10. Per-title encoding quality scores (avg. for HEVC and H.264 encoding), SSIM metric

Download full HTML report

MSU Cloud Benchmark 2020
Download free full version
  • Cloud encoding services
    Alibaba Cloud: ApsaraVideo for Media Processing, AWS Elemental MediaConvert, Coconut, Kingsoft Cloud, Qencode, Tencent Cloud: Media Processing Service, Zencoder
  • 15 FullHD video sequences
    From Vimeo and YouTube UGC
  • Three resolutions
    FullHD, HD, SD
  • Two encoding use cases
    Encoding with default options of tested services and with automatic sequence-based options
  • HTML report
    5153 interactive charts
Video Transcoding Clouds Comparison 2019
Download free full version
  • 6 cloud encoding services
    Alibaba, Amazon Elastic Transcoder, AWS Elemental MediaConvert, Coconut, Qencode, Zencoder
  • 4 FullHD video sequences
    epson, fountains, hawk and hockey from Vimeo website
  • Three resolutions
    FullHD, HD, SD
  • Three encoding use cases (presets)
    Encoding with similar options, default options of tested services and with tuned options for our test videos
  • HTML and PDF documents
    118 interactive charts and 33 pages

Methodology

Brief methodology review is given in the presentation:

1. Preprocessing

We performed lossless compression with x264 (x265 for AWS MediaConvert) and passed compressed stream as input, as most of services don't support raw YUV420p.

2. Storage

To equalize network latency in speed measurements we used Amazon S3 storage (us-east-1 region) for most of the services as widely used solution. Some services (Alibaba Cloud, Tencent Cloud) don't support Amazon S3, so we used corresponding storage analogs (Alibaba Cloud OSS, Kingsoft Cloud, Tencent Cloud COS).

3. Encoding

We used services' API to run encoding tasks. We performed H.264 and HEVC transcoding with multiple bitrates for each resolution. In the first use case we used default options everywhere it was possible.
Bitrate, kbps 480p 720p 1080p
75+
120+
250++
500++
750++
1000+++
2000+++
4000++
6000++
8000+
10000+
12000+
Encoding bitrates and resolutions

4. Speed measurement

This year we tried to measure encoding time including files transmission. We used services' callbacks to get jobs finish time. Also, we encoded one job multiple times to estimate deviation and minimize single time delays.

5. Objective metrics calculation

We used MSU VQMT to calculate objective metrics (SSIM, PSNR, VMAF) against original YUV files.

6. Plots

We merge RD curves of different resolutions to evaluate overall service performance. After that we made all our plots same way as in the Main report. You can learn more about our base methodology in presentation.

Videos

Name Preview Resolution FPS Frames Description Source
blue_hair blue_hair 1920x1080 30 600 Shaky handheld vlog video. ID: Vlog_1080P-35cd YouTube UGC
christmas_cats christmas_cats 1920x1080 25 1500 Concert record with superimposed complicated translucent CG effects Vimeo
construction_site construction_site 1920x1080 30 1043 Shots of building under construction Vimeo
crowd_run crowd_run 1920x1080 50 500 A crowd of sportsmen runs while the camera slowly moves left and right Xiph
football football 1920x1080 30 599 People are playing football. ID: Sports_1080P-15d1 YouTube UGC
hard_rock hard_rock 1920x1080 25 500 Poorly lit very noisy scene of people. ID: Musicvideo_1080P-6260 YouTube UGC
kindergarten_interview kindergarten_interview 1920x1080 30 1016 One man is interviewing other Vimeo
park_mobile park_mobile 1920x1080 24 359 Information video with people speaking in front of camera and old fashioned scenes Vimeo
pyranha_rafting pyranha_rafting 1920x1080 24 1203 Panning shots of whitewater rafting Vimeo
stone stone 1920x1080 30 598 Shots of stone in front of a camera. ID: HowTo_1080P-7cf2 YouTube UGC
street_musician street_musician 1920x1080 24 974 Handheld video of a musician performing and people listening. Heavy grain and black and white sections Vimeo
summer_of_adventure summer_of_adventure 1920x1080 30 994 Summer camp commercial, consists of nature scenes, POV shots and a slideshow Vimeo
tennis_vlog tennis_vlog 1440x1080 30 599 Selfie video followed by a girl practicing hitting a baseball. ID: Vlog_1080P-19cc YouTube UGC
the_forest the_forest 1920x1080 30 600 The Forest gameplay. ID: Gaming_1080P-72c8 YouTube UGC
wedding_party wedding_party 1920x1080 24 1757 Bride and groom dancing. Colorful illumination with flashes of light Vimeo

Contact Information


compression.ru
in cooperation
with
Lomonosov MSU
Graphics & Media Lab
(Video Group)
Dubna State University
Institute for Information
Transmission Problems RAS

Subscribe to report updates


Materials about MSU Codec Comparison


See all MSU Video Codecs Comparisons

MSU video codecs comparisons resources:


Other Materials


Video resources:

Last updated: 08-May-2021


Server size: 8069 files, 1215Mb (Server statistics)

Project updated by
Server Team and MSU Video Group

Project sponsored by YUVsoft Corp.

Project supported by MSU Graphics & Media Lab