Shared Brotli Compressed Data Format
draft-vandevenne-shared-brotli-format-01

• Status
• IESG evaluation record
• IESG writeups
• Email expansions
• History

• Versions
• 00
• 01

Network Working Group                                      J. Alakuijala
Internet-Draft                                                  T. Duong
Intended Status: Informational                            E. Kliuchnikov
Expires: Feb 04, 2019                                           R. Obryk
                                                             Z. Szabadka
                                                           L. Vandevenne
                                                             Google, Inc
                                                                Aug 2018

                  Shared Brotli Compressed Data Format
                draft-vandevenne-shared-brotli-format-01

Abstract

   This specification defines a data format for shared brotli
   compression, which adds support for shared dictionaries, large
   window, patching and a container format to brotli [RFC7932].

   Shared dictionaries and large window support allow significant
   compression gains compared to regular brotli, and patching allows
   smaller patches of binary files.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79. Internet-Drafts are working
   documents of the Internet Engineering Task Force (IETF). Note that
   other groups may also distribute working documents as Internet-
   Drafts. The list of current Internet-Drafts is at
   http://datatracker.ietf.org/drafts/current.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on Feb 04, 2019.

Copyright Notice

   Copyright (c) 2018 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents

Alakuijala et. al.        Expires Feb 04, 2019                  [Page 1]
Internet-Draft          Shared Brotli Data Format               Mar 2018

   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . . . 3
      1.1.  Purpose  . . . . . . . . . . . . . . . . . . . . . . . . . 3
      1.2.  Intended audience  . . . . . . . . . . . . . . . . . . . . 3
      1.3.  Scope  . . . . . . . . . . . . . . . . . . . . . . . . . . 3
      1.4.  Compliance . . . . . . . . . . . . . . . . . . . . . . . . 4
      1.5.  Definitions of terms and conventions used  . . . . . . . . 4
         1.5.1.  Packing into bytes  . . . . . . . . . . . . . . . . . 4
   2.  Shared Brotli Overview  . . . . . . . . . . . . . . . . . . . . 5
   3.  Shared Dictionaries . . . . . . . . . . . . . . . . . . . . . . 6
      3.1.  Custom Static Dictionaries . . . . . . . . . . . . . . . . 6
         3.1.1.  Transform Operations  . . . . . . . . . . . . . . . . 7
      3.2.  LZ77 Dictionaries  . . . . . . . . . . . . . . . . . . . . 9
   4.  Varint Encoding  . . . . . . . . . . . . . . . . . . . . . . . 10
   5.  Shared Dictionary Stream . . . . . . . . . . . . . . . . . . . 10
   6.  Large Window Brotli Compressed Data Stream . . . . . . . . . . 13
   7.  Patching Format Compressed Data Stream . . . . . . . . . . . . 13
   8.  Shared Brotli Compressed Data Stream . . . . . . . . . . . . . 13
   9.  Shared Brotli Framing Format Stream  . . . . . . . . . . . . . 14
      9.1.  Main Format . . . . . . . . . . . . . . . . . . . . . . . 14
      9.2.  Chunk Format  . . . . . . . . . . . . . . . . . . . . . . 14
      9.3.  Metadata Format . . . . . . . . . . . . . . . . . . . . . 16
      9.4.  Chunk Specifications  . . . . . . . . . . . . . . . . . . 17
         9.4.1.  Padding Chunk  . . . . . . . . . . . . . . . . . . . 17
         9.4.2.  Metadata Chunk . . . . . . . . . . . . . . . . . . . 17
         9.4.3.  Data Chunk . . . . . . . . . . . . . . . . . . . . . 18
         9.4.4.  First Partial Data Chunk . . . . . . . . . . . . . . 18
         9.4.5.  Middle Partial Data Chunk  . . . . . . . . . . . . . 19
         9.4.6.  Last Partial Data Chunk  . . . . . . . . . . . . . . 19
         9.4.7.  Footer Metadata Chunk  . . . . . . . . . . . . . . . 19
         9.4.8.  Global Metadata Chunk  . . . . . . . . . . . . . . . 19
         9.4.9.  Repeat Metadata Chunk  . . . . . . . . . . . . . . . 19
         9.4.10.  Central Directory Chunk . . . . . . . . . . . . . . 20
         9.4.11.  Final Footer Chunk  . . . . . . . . . . . . . . . . 21