Zstandard Compression and the application/zstd Media Type
RFC 8478

Document Type RFC - Informational (October 2018; No errata)
Was draft-kucherawy-dispatch-zstd (individual in art area)
Last updated 2018-10-02
Stream IETF
Formats plain text pdf html bibtex
Reviews
Stream WG state Submitted to IESG for Publication
Document shepherd No shepherd assigned
Shepherd write-up Show (last changed 2018-05-21)
IESG IESG state RFC 8478 (Informational)
Consensus Boilerplate Yes
Telechat date
Responsible AD Alexey Melnikov
Send notices to (None)
IANA IANA review state Version Changed - Review Needed
IANA action state RFC-Ed-Ack
Internet Engineering Task Force (IETF)                         Y. Collet
Request for Comments: 8478                             M. Kucherawy, Ed.
Category: Informational                                         Facebook
ISSN: 2070-1721                                             October 2018

       Zstandard Compression and the application/zstd Media Type

Abstract

   Zstandard, or "zstd" (pronounced "zee standard"), is a data
   compression mechanism.  This document describes the mechanism and
   registers a media type and content encoding to be used when
   transporting zstd-compressed content via Multipurpose Internet Mail
   Extensions (MIME).

   Despite use of the word "standard" as part of its name, readers are
   advised that this document is not an Internet Standards Track
   specification; it is being published for informational purposes only.

Status of This Memo

   This document is not an Internet Standards Track specification; it is
   published for informational purposes.

   This document is a product of the Internet Engineering Task Force
   (IETF).  It represents the consensus of the IETF community.  It has
   received public review and has been approved for publication by the
   Internet Engineering Steering Group (IESG).  Not all documents
   approved by the IESG are candidates for any level of Internet
   Standard; see Section 2 of RFC 7841.

   Information about the current status of this document, any errata,
   and how to provide feedback on it may be obtained at
   https://www.rfc-editor.org/info/rfc8478.

Collet & Kucherawy            Informational                     [Page 1]
RFC 8478                    application/zstd                October 2018

Copyright Notice

   Copyright (c) 2018 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Collet & Kucherawy            Informational                     [Page 2]
RFC 8478                    application/zstd                October 2018

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   4
   2.  Definitions . . . . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Compression Algorithm . . . . . . . . . . . . . . . . . . . .   5
     3.1.  Frames  . . . . . . . . . . . . . . . . . . . . . . . . .   6
       3.1.1.  Zstandard Frames  . . . . . . . . . . . . . . . . . .   6
         3.1.1.1.  Frame Header  . . . . . . . . . . . . . . . . . .   7
         3.1.1.2.  Blocks  . . . . . . . . . . . . . . . . . . . . .  12
         3.1.1.3.  Compressed Blocks . . . . . . . . . . . . . . . .  14
         3.1.1.4.  Sequence Execution  . . . . . . . . . . . . . . .  28
         3.1.1.5.  Repeat Offsets  . . . . . . . . . . . . . . . . .  29
       3.1.2.  Skippable Frames  . . . . . . . . . . . . . . . . . .  30
   4.  Entropy Encoding  . . . . . . . . . . . . . . . . . . . . . .  30
     4.1.  FSE . . . . . . . . . . . . . . . . . . . . . . . . . . .  31
       4.1.1.  FSE Table Description . . . . . . . . . . . . . . . .  31
     4.2.  Huffman Coding  . . . . . . . . . . . . . . . . . . . . .  34
       4.2.1.  Huffman Tree Description  . . . . . . . . . . . . . .  35
         4.2.1.1.  Huffman Tree Header . . . . . . . . . . . . . . .  36
         4.2.1.2.  FSE Compression of Huffman Weights  . . . . . . .  37
         4.2.1.3.  Conversion from Weights to Huffman Prefix Codes .  38
       4.2.2.  Huffman-Coded Streams . . . . . . . . . . . . . . . .  39
   5.  Dictionary Format . . . . . . . . . . . . . . . . . . . . . .  40
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  42
     6.1.  The 'application/zstd' Media Type . . . . . . . . . . . .  42
     6.2.  Content Encoding  . . . . . . . . . . . . . . . . . . . .  43
     6.3.  Dictionaries  . . . . . . . . . . . . . . . . . . . . . .  43
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  43
   8.  Implementation Status . . . . . . . . . . . . . . . . . . . .  44
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  45
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .  45
     9.2.  Informative References  . . . . . . . . . . . . . . . . .  45
   Appendix A.  Decoding Tables for Predefined Codes . . . . . . . .  46
     A.1.  Literal Length Code Table . . . . . . . . . . . . . . . .  46
     A.2.  Match Length Code Table . . . . . . . . . . . . . . . . .  49
Show full document text