Video Codec Testing and Quality Measurement
draft-ietf-netvc-testing-05

Document Type Active Internet-Draft (netvc WG)
Last updated 2017-03-27
Replaces draft-daede-netvc-testing
Stream IETF
Intended RFC status (None)
Formats plain text xml pdf html bibtex
Stream WG state WG Document
Document shepherd No shepherd assigned
IESG IESG state I-D Exists
Consensus Boilerplate Unknown
Telechat date
Responsible AD (None)
Send notices to (None)
Network Working Group                                           T. Daede
Internet-Draft                                                   Mozilla
Intended status: Informational                                 A. Norkin
Expires: September 28, 2017                                      Netflix
                                                          I. Brailovskiy
                                                           Amazon Lab126
                                                          March 27, 2017

              Video Codec Testing and Quality Measurement
                      draft-ietf-netvc-testing-05

Abstract

   This document describes guidelines and procedures for evaluating a
   video codec.  This covers subjective and objective tests, test
   conditions, and materials used for the test.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on September 28, 2017.

Copyright Notice

   Copyright (c) 2017 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of

Daede, et al.          Expires September 28, 2017               [Page 1]
Internet-Draft Video Codec Testing and Quality Measurement    March 2017

   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Subjective quality tests  . . . . . . . . . . . . . . . . . .   3
     2.1.  Still Image Pair Comparison . . . . . . . . . . . . . . .   3
     2.2.  Video Pair Comparison . . . . . . . . . . . . . . . . . .   3
     2.3.  Subjective viewing test . . . . . . . . . . . . . . . . .   4
   3.  Objective Metrics . . . . . . . . . . . . . . . . . . . . . .   4
     3.1.  Overall PSNR  . . . . . . . . . . . . . . . . . . . . . .   4
     3.2.  Frame-averaged PSNR . . . . . . . . . . . . . . . . . . .   5
     3.3.  PSNR-HVS-M  . . . . . . . . . . . . . . . . . . . . . . .   5
     3.4.  SSIM  . . . . . . . . . . . . . . . . . . . . . . . . . .   5
     3.5.  Multi-Scale SSIM  . . . . . . . . . . . . . . . . . . . .   5
     3.6.  CIEDE2000 . . . . . . . . . . . . . . . . . . . . . . . .   5
     3.7.  VMAF  . . . . . . . . . . . . . . . . . . . . . . . . . .   6
   4.  Comparing and Interpreting Results  . . . . . . . . . . . . .   6
     4.1.  Graphing  . . . . . . . . . . . . . . . . . . . . . . . .   6
     4.2.  BD-Rate . . . . . . . . . . . . . . . . . . . . . . . . .   6
     4.3.  Ranges  . . . . . . . . . . . . . . . . . . . . . . . . .   7
   5.  Test Sequences  . . . . . . . . . . . . . . . . . . . . . . .   7
     5.1.  Sources . . . . . . . . . . . . . . . . . . . . . . . . .   7
     5.2.  Test Sets . . . . . . . . . . . . . . . . . . . . . . . .   8
       5.2.1.  regression-1  . . . . . . . . . . . . . . . . . . . .   8
       5.2.2.  objective-2-slow  . . . . . . . . . . . . . . . . . .   8
       5.2.3.  objective-2-fast  . . . . . . . . . . . . . . . . . .  12
       5.2.4.  objective-1.1 . . . . . . . . . . . . . . . . . . . .  14
       5.2.5.  objective-1-fast  . . . . . . . . . . . . . . . . . .  17
     5.3.  Operating Points  . . . . . . . . . . . . . . . . . . . .  18
       5.3.1.  Common settings . . . . . . . . . . . . . . . . . . .  18
       5.3.2.  High Latency CQP  . . . . . . . . . . . . . . . . . .  19
       5.3.3.  Low Latency CQP . . . . . . . . . . . . . . . . . . .  19
       5.3.4.  Unconstrained High Latency  . . . . . . . . . . . . .  19
       5.3.5.  Unconstrained Low Latency . . . . . . . . . . . . . .  19
   6.  Automation  . . . . . . . . . . . . . . . . . . . . . . . . .  20
     6.1.  Regression tests  . . . . . . . . . . . . . . . . . . . .  20
     6.2.  Objective performance tests . . . . . . . . . . . . . . .  20
Show full document text