MSU Cloud Video Transcoding Benchmark 2020

If you want to receive notifications about our reports, please

Contents

• About
• List of compared services
• Results
• Download HTML report
• Methodology
• List of test videos
• Contact
• Subscribe to receive notifications about our reports

About

We perform an independent comparison of cloud-based video-encoding services. This year test included encoding performance comparison using services' default settings. The results of our previous comparison showed that in the case of cloud services that provide the same quality, the file-size difference can reach 100% and the cost per minute difference is 700%.

Comparison main points

• Encoding time and objective videos quality are measured
  We perform a series of measurements to estimate average time needed for transcoding. Objective algorithms are used for estimating the resulting video quality
• Two encoding use cases: default encoding options and automatic sequence-based options
  This year we performed two types of tests: encode test videos using service default encoding settings and using automatic sequence-based settings (per-title encoding). If a service does not have automatic sequence-based settings, it is not be presented in this test
• FullHD videos of different type and content (including UGC)
  15 FullHD videos were used for tests. We performed two types of measurements: the main case involved encoding into FullHD, and the additional case included transcoding into several resolutions
• Anonymous accounts were used for testing
  For fair comparison, we used accounts which were not connected to us as the comparison organizers. This was done to prevent the services reserving separate servers/rules for comparison test videos processing. If the account was provided by the owner of the service — it is noted in list of compared services

List of compared services

8^*. FFMpeg was also used for measurements just as an example to be compared with Cloud-encoding solutions.

Notes

• ¹ — account was provided by service team
• ² — we didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started
• ³ — service provides parameterless sequence-based bitrate selection but doesn't calculate output resolutions. The following resolution ladder was used: 1080p, 720p, 480p, 360p, 240p
• ⁴ — service provides sequence-based bitrate selection but doesn't calculate output resolutions. The following resolution-quality parameters were used:

  Resoultion	AWS QVBR	Coconut quality	Qencode min–max CRFs
  1080p	9	4	11–20
  720p	8	4	11–20
  720p	7	3	21–30
  480p	7	3	21–30
  360p	7	3	21–30
  240p	7	3	21–30

Overall conclusions

Default services options were used for comparison, so the leaders are judged not only for quality score, but also for speed and cost trade-off. Below are the winners by different categories: speed/quality trade-off shows how frequently a service was pareto-optimal for encoding speed and quality among all test videos, cost/quality trade-off shows how frequently a service was pareto-optimal for video encoding cost and quality among all test videos, and ease of use shows our subjective estimation on service convenience.

Encoding time

We performed three encodes for each sequence on different days and day times. The chart below shows a deviation of encoding time among all videos for each iteration.
Big delays could be caused by high load of service resources (and big queues) or long time of accessing our videos from storage (the table with services description above shows which storage was used for each service).

Alibaba ApsaraVideo for Media Processing showed the least average encoding time and deviation for both HEVC and H.264 encoding
AWS Elemental MediaConvert showed small encoding time and deviation for H.264 encoding Kingsoft Cloud showed small time deviation for both HEVC and H.264 encoding

Figure 1

Different resolution test

Three resolutions were used to transcode source videos: FullHD(1920x1080), HD(1280x720), SD(854x480). The chart below shows RD-curves for different resolutions at pyranha_rafting video sequence.

Resolution: 480p Figure 2. RD-curves for tennis_vlog, 480p	Resolution: 720 Figure 3. RD-curves for tennis_vlog, 720p
Resolution: 1080 Figure 4. RD-curves for tennis_vlog, 1080p	Resolution: all on one chart Figure 5. All resolutions RD-curves for tennis_vlog

Merged resolutions chart shows all RD-curves constructed into one curve by the rule: it contains a point from each curve if this points is not covered by quality of a point from another resolution curve.

Figure 6. Merged resolutions curves for tennis_vlog

Quality/encoding speed trade-off

Integrating RD-curves by bitrate and calculating relative scores (AWS Elemental MediaConvert was takes as a reference), we got relative quality/relative speed plots (detailed methodology).
The chart below shows relative quality and speed scores averaged for all videos (HEVC encoding, SSIM metric).
Tencent Media Processing Service, Zencoder and Alibaba ApsaraVideo for Media Processing were most frequently optimal among all services.

Figure 7. Merged resolutions speed/bitrate trade-off, HEVC encoding, average for all videos, SSIM.

Cost/quality trade-off

The following charts show relative bitrate and encoding cost for same quality (SSIM) averaged for all videos.
Kingsoft Cloud, Qencode (for H.264 and HEVC encoding) and Coconut (for HEVC encoding) showed good quality for the least prices.

Figure 8. Merged resolutions cost/bitrate trade-off, HEVC encoding, average for all videos, SSIM.

Overall comparison of HEVC and H.264 encoding

The plot below shows encoding speed and relative quality scores for HEVC and H.264 averaged for all test videos, SSIM metric. The results for different metrics are almost the same (see all charts in full report). H.264 encoding results of AWS Elemental MediaConvert was used a reference to get quality scores.
Tencent Media Processing Service, Kingsoft Cloud and Zencoder showed best quality for HEVC encoding, Alibaba ApsaraVideo for Media Processing showed also good quality for higher encoding speed.

Figure 9. HEVC and H.264 encoding speed/bitrate comparison, SSIM metric
FFMpeg was used for measurements just as an example to be compared with Cloud-encoding solutions

Per-title encoding test

In addition to default encoding settings, per-sequence encoding optimization was tested for services, which supported this feature at the time we did the tests. This option optimizes encoding settings to each input video to get better encoding performance (better quality of less bitrate).

We didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started, and Qencode sequence-based encoding, because it produces only one output without any bitrate or resolution variation.

According to our tests, Tencent Media Processing Service showed the best quality for per-title encoding, Alibaba ApsaraVideo for Media Processing is at the second place.

Figure 10. Per-title encoding quality scores (avg. for HEVC and H.264 encoding), SSIM metric

Download full HTML report

MSU Cloud Benchmark 2020 Download free full version

Cloud encoding services Alibaba Cloud: ApsaraVideo for Media Processing, AWS Elemental MediaConvert, Coconut, Kingsoft Cloud, Qencode, Tencent Cloud: Media Processing Service, Zencoder 15 FullHD video sequences From Vimeo and YouTube UGC Three resolutions FullHD, HD, SD Two encoding use cases Encoding with default options of tested services and with automatic sequence-based options HTML report 5153 interactive charts

Video Transcoding Clouds Comparison 2019 Download free full version

6 cloud encoding services Alibaba, Amazon Elastic Transcoder, AWS Elemental MediaConvert, Coconut, Qencode, Zencoder 4 FullHD video sequences epson, fountains, hawk and hockey from Vimeo website Three resolutions FullHD, HD, SD Three encoding use cases (presets) Encoding with similar options, default options of tested services and with tuned options for our test videos HTML and PDF documents 118 interactive charts and 33 pages

Methodology

Brief methodology review is given in the presentation:

1. Preprocessing

2. Storage

3. Encoding

4. Speed measurement

5. Objective metrics calculation

6. Plots

Videos

Name	Preview	Resolution	FPS	Frames	Description	Source
blue_hair		1920x1080	30	600	Shaky handheld vlog video. ID: Vlog_1080P-35cd	YouTube UGC
christmas_cats		1920x1080	25	1500	Concert record with superimposed complicated translucent CG effects	Vimeo
construction_site		1920x1080	30	1043	Shots of building under construction	Vimeo
crowd_run		1920x1080	50	500	A crowd of sportsmen runs while the camera slowly moves left and right	Xiph
football		1920x1080	30	599	People are playing football. ID: Sports_1080P-15d1	YouTube UGC
hard_rock		1920x1080	25	500	Poorly lit very noisy scene of people. ID: Musicvideo_1080P-6260	YouTube UGC
kindergarten_interview		1920x1080	30	1016	One man is interviewing other	Vimeo
park_mobile		1920x1080	24	359	Information video with people speaking in front of camera and old fashioned scenes	Vimeo
pyranha_rafting		1920x1080	24	1203	Panning shots of whitewater rafting	Vimeo
stone		1920x1080	30	598	Shots of stone in front of a camera. ID: HowTo_1080P-7cf2	YouTube UGC
street_musician		1920x1080	24	974	Handheld video of a musician performing and people listening. Heavy grain and black and white sections	Vimeo
summer_of_adventure		1920x1080	30	994	Summer camp commercial, consists of nature scenes, POV shots and a slideshow	Vimeo
tennis_vlog		1440x1080	30	599	Selfie video followed by a girl practicing hitting a baseball. ID: Vlog_1080P-19cc	YouTube UGC
the_forest		1920x1080	30	600	The Forest gameplay. ID: Gaming_1080P-72c8	YouTube UGC
wedding_party		1920x1080	24	1757	Bride and groom dancing. Colorful illumination with flashes of light	Vimeo

compression.ru	in cooperation with		Lomonosov MSU Graphics & Media Lab (Video Group)
			Dubna State University
			Institute for Information Transmission Problems RAS

Materials about MSU Codec Comparison

Other Materials