MSU Video Codecs Comparison 2021:
Call for codecs

All industry professionals will know about your codec success!


compression.ru
in cooperation
with
Lomonosov MSU
Graphics & Media Lab
(Video Group)

Fifteen modern video codecs comparison
For real researchers, developers and professional users in field of high-end video compression

Dubna State University
Institute for Information
Transmission Problems, Russian Academy of Science

Important Dates

May, 31 FullHD comparison application deadline

Participants are to provide codec name, name of encoding standard, developer's (company) name, and list of comparison parts in which they want to participate


Navigation


2021 Comparison parts and main points

  1. FullHD Comparison
    Comparison using FullHD videos and objective video quality measurement (SSIM, PSNR, VMAF)
    • Fast FullHD - 1080p@30fps, ABR/VBR
    • Slow FullHD - 1080p@1fps, CRF
    • (optional) Adaptive streaming: 1 fps, VBV/HRD enabled
  2. Subjective Comparison
    Comparison using FullHD videos and subjective quality measurement (Subjectify.us)
    • Fast Subjective - 1080p@30fps, variable bitrate encoding mode (VBR of other)
    • Slow Subjective - 1080p@1fps, constant-quality encoding mode (CRF or other)
  3. 4K Comparison
    Comparison using 4K and 10-bit videos and objective quality measurement (Subjectify.us)
    • Fast 4K - UHD@20fps
    • Slow 4K - UHD@1fps
  4. Ultra-fast (FPGA, hardware-accelerates and ultra-fast encoders) comparison
    • Ultra-fast - from 60fps
  5. Cloud comparison
    No speed limits, offline encoding scenario, several resolutions

Comparison methodology main points:

  • 50 FullHD video sequences (FullHD report) + 10-12 4K 8- and 10-bit videos.
  • SSIM, PSNR, VMAF objective metrics & subjective evaluation
  • 3 color-planes (Y,U,V) and integral metric values
  • 2 encoding use cases for comparison on FullHD videos: Fast and Slow encoding (differ by encoding speed) + 2 encoding use cases for subjective comparison + 2 special encoding use cases for comparison on 4K videos
  • Special encoding use case: adaptive streaming (3 resolutions: 480p, 720p, 1080p; VBV-bufsize, maxrate and HRD enabled) in report appendix
  • 8-10 different target bitrates (1-12 Mbps for FullHD and 2-20 Mbps for 4K)
  • Prosumer-level modern hardware
  • Fully automatic testing system
  • 7000+ result figures, PDF and HTML reports


About Annual MSU Video Codecs Comparisons

We perform comparative unbiased analysis for

  • software implementations
  • hardware (GPU-based) implementations
  • cloud-based implementations
of different video coding standards (H.265/HEVC, H.264/MPEG-4 AVC, AV1, VP9, VP8, MPEG-4 and other) using objective metrics (SSIM, VMAF, PSNR and other).

With MSU Codecs Comparison developers can verify the performance of their codec. We share test sequences, encoding parameters and codecs versions so all developers can reproduce the results of the comparison. Participation with publishing of all results is for FREE. Private participation is for fee (contact us for the details).

MSU codecs analysis team has been conducting video codec analysis, testing and optimization since 2004. Some facts about previous MSU Video Codecs Comparisons:
  • There were more than 400.000 downloads of previous H.264/MPEG-4 AVC and H.265/HEVC video codec comparisons reports
  • Many codecs' bugs were found and reported to developers
  • 210+ encoders were tested
  • More than 35 private reports for codec developers (description of codec's weak and strong points) after public report versions
Summary report topics:
  • Objective measurements
  • Subjective analysis
  • Encoding time
  • Bitrate keeping for evaluating rate-control mechanism
  • Speed/Quality trade-off analysis
  • Averaged objective results analysis
  • Leaders in different use cases

New features in MSU Video Codecs Comparison

Since 2020
  • Automatic codecs submission system
  • Open to compare Linux-based solutions
  • Open to compare cloud-based solutions
  • Enlarging of video collection
Since 2019
  • Increasing the number of test videos up to 100+, adding UGC (user-generated) videos (vlogs, noise videos, etc.) and HDR videos
  • Open to compare cloud-based solutions
  • Enlarging of video collection
  • Going to use new no-reference metrics
Since 2018
  • Increased target bitrates for comparison on 4K videos
  • Started to use VMAF objective metric
  • Upgraded testing hardware to Coffee Lake
  • Enlarged video collection (15833 videos)

Comparison Rules and Methodology


Please pay attention that we will use multi-core CPU for encoding, so you can use multi-threading
  • Decoding is performed with reference decoder (H.264/AVC, H.265/HEVC, VP9, AV1). For other standards participants should provide decoder
  • We don't limit GOP size and intra-period
  • We don't limit number of passes in the encoding. Total encoding time should fulfill use case speed requirements
  • Before results publishing each developer will receive the results of its codec and competitive free open-source codecs. Developers of each codec can validate the results and write a comment (one paragraph) about the comparison results which will be included in the report
  • Participation is for free with publication of the results
    • You can join comparison for free if you agree that your codec results will be published
    • Private participation. Compare your codec with world leaders staying incognito! If your company wants to know the results of your encoder testing with possibility to exclude them from publication and information disclosure, you should pay for measurements and report preparing before comparison begins
  • Enterprise version of comparison report is available for all participants for free
  • All participants will receive following deliverables to verify the results for free:
    • video sequences used in comparison
    • binaries of all open-source encoders used in comparison to verify the results
    • all raw video quality metric and encoding speed data for their encoder and for all of open-source encoders used in comparison
  • We are willing to completely or partially delete information about some codec in the public version of comparison report only in exceptional cases (e.g. critical errors in the codec)

Test Hardware Characteristics

The following hardware is used for codecs testing:
  • CPU: Intel Socket 1151 Core i7 8700K (Coffee Lake) (3.7Ghz, 6C12T, TDP 95W)
  • Mainboard: ASRock Z370M Pro4
  • RAM: Crucial CT16G4DFD824A 2x16GB (totally 32 GB) DIMM DDR4 2400MHz CL15
  • OS: Windows 10 x64 and Linux (Ubuntu)

Encoding speed requirements

For encoder alignment selected presets should provide following encoding speed.
    • FullHD Comparison:
      • Fast - 1080p@30fps, variable bitrate encoding mode (VBR of other)
      • Slow - 1080p@1fps, constant-quality encoding mode (CRF or other)
    • Subjective Comparison:
      • Fast - 1080p@30fps, variable bitrate encoding mode (VBR of other)
      • Slow - 1080p@1fps, constant-quality encoding mode (CRF or other)
    • 4K Comparison:
      • Fast - UHD@20fps
      • Slow - UHD@1fps

Codec Requirements

  • Presets for different speed requirements should be provided by the developers
  • Codec should allow to set arbitrary bitrate of resulting stream in constant quality mode
  • Preferable codec interface - console codec version (with batch processing support — bitrate and file names must be possible to assign from the command line)
  • Encoder should be compatible with reference decoder
  • Developers Deliverables

    The following deliverables should be provided by each developer:
  • Codec files (CLI executable file is preferable)
  • Codec's presets
  • Full comparison methodology


    Take part in 2021 Video Codecs Comparison!

    Deepest codecs review: 5 reports, including subjective with 1000+ viewers and 7000+ charts

    Applying for participating, you agree with comparison rules.

    Contact for participation.

    Please list the following:

    • use cases you want to participate in (Fast FullHD, Slow FullHD, Fast Subjective, Slow Subjective, Fast 4K, Slow 4K)
    • codec name
    • standard
    • which codec version you want to submit (for Windows or for Linux)
    • company name
    • kind of participation (Free/public - we test your encoder and publish all results; or Private - you pay a participation fee, we test your encoder and send you the results, then you make the decision to publish results or not)

    We are open to any suggestions for 2021 comparison: use this form for any proposals. Your contribution is very important for comparison improvement!


    Thanks


    Special thanks to the following contributors of our previous comparisons
    Google Intel AMD NVidia
    ATI Adobe ISPhone dicas
    KDDI R&D labs Dolby Tata Elxsi Octasic
    Qualcomm Voceweb Elgato Telecast
    Huawei MainConcept Vitec Tencent

    Contact Information

    See all MSU Video Codecs Comparisons

    MSU video codecs comparisons resources:


    Other Materials

    Video resources:

    Last updated: 08-May-2021


    Server size: 8069 files, 1215Mb (Server statistics)

    Project updated by
    Server Team and MSU Video Group

    Project sponsored by YUVsoft Corp.

    Project supported by MSU Graphics & Media Lab