Video Codec Testing and Quality Measurement
draft-daede-netvc-testing-01

The information below is for an old version of the document
Document Type Active Internet-Draft (individual)
Last updated 2015-07-06
Replaced by draft-ietf-netvc-testing
Stream (None)
Intended RFC status (None)
Formats pdf htmlized bibtex
Stream Stream state (No stream defined)
Consensus Boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date
Responsible AD (None)
Send notices to (None)
Network Working Group                                           T. Daede
Internet-Draft                                                J. Moffitt
Intended status: Informational                                   Mozilla
Expires: January 07, 2016                                  July 06, 2015

              Video Codec Testing and Quality Measurement
                      draft-daede-netvc-testing-01

Abstract

   This document describes guidelines and procedures for evaluating an
   internet video codec specified at the IETF.  This covers subjective
   and objective tests, test conditions, and materials used for the
   test.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on January 07, 2016.

Copyright Notice

   Copyright (c) 2015 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Daede & Moffitt         Expires January 07, 2016                [Page 1]
Internet-Draft Video Codec Testing and Quality Measurement     July 2015

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Subjective Metrics  . . . . . . . . . . . . . . . . . . . . .   2
   3.  Objective Metrics . . . . . . . . . . . . . . . . . . . . . .   2
     3.1.  PSNR  . . . . . . . . . . . . . . . . . . . . . . . . . .   3
     3.2.  PSNR-HVS-M  . . . . . . . . . . . . . . . . . . . . . . .   3
     3.3.  SSIM  . . . . . . . . . . . . . . . . . . . . . . . . . .   3
     3.4.  Fast Multi-Scale SSIM . . . . . . . . . . . . . . . . . .   4
   4.  Comparing and Interpreting Results  . . . . . . . . . . . . .   4
     4.1.  Graphing  . . . . . . . . . . . . . . . . . . . . . . . .   4
     4.2.  Bjontegaard . . . . . . . . . . . . . . . . . . . . . . .   4
     4.3.  Ranges  . . . . . . . . . . . . . . . . . . . . . . . . .   4
   5.  Test Sequences  . . . . . . . . . . . . . . . . . . . . . . .   5
     5.1.  Sources . . . . . . . . . . . . . . . . . . . . . . . . .   5
     5.2.  Test Sets . . . . . . . . . . . . . . . . . . . . . . . .   5
     5.3.  Operating Points  . . . . . . . . . . . . . . . . . . . .   7
   6.  Automation  . . . . . . . . . . . . . . . . . . . . . . . . .   7
   7.  Informative References  . . . . . . . . . . . . . . . . . . .   7
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   8

1.  Introduction

   When developing an internet video codec, changes and additions to the
   codec need to be decided based on their performance tradeoffs.  In
   addition, measurements are needed to determine when the codec has met
   its performance goals.  This document specifies how the tests are to
   be carried about to ensure valid comparisons and good decisions.

2.  Subjective Metrics

   Subjective testing is the preferable method of testing video codecs.

   Because the IETF does not have testing resources of its own, it has
   to rely on the resources of its participants.  For this reason, even
   if the group agrees that a particular test is important, if no one
   volunteers to do it, or if volunteers do not complete it in a timely
   fashion, then that test should be discarded.  This ensures that only
   important tests be done in particular, the tests that are important
   to participants.

3.  Objective Metrics

   Objective metrics are used in place of subjective metrics for easy
   and repeatable experiments.  Most objective metrics have been
   designed to correlate with subjective scores.

Daede & Moffitt         Expires January 07, 2016                [Page 2]
Internet-Draft Video Codec Testing and Quality Measurement     July 2015

   The following descriptions give an overview of the operation of each
   of the metrics.  Because implementation details can sometimes vary,
   the exact implementation is specified in C in the Daala tools
Show full document text