Skip to main content

Constrained Low Pass Filter
draft-midtskogen-netvc-clpf-03

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft whose latest revision state is "Expired".
Authors Steinar Midtskogen , Arild Fuldseth , Mo Zanaty
Last updated 2016-10-31
RFC stream (None)
Formats
Additional resources
Stream Stream state (No stream defined)
Consensus boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date (None)
Responsible AD (None)
Send notices to (None)
draft-midtskogen-netvc-clpf-03
Network Working Group                                      S. Midtskogen
Internet-Draft                                               A. Fuldseth
Intended status: Standards Track                               M. Zanaty
Expires: May 4, 2017                                               Cisco
                                                        October 31, 2016

                      Constrained Low Pass Filter
                     draft-midtskogen-netvc-clpf-03

Abstract

   This document describes a low complexity filtering technique which is
   being used as a low pass loop filter in the Thor video codec.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on May 4, 2017.

Copyright Notice

   Copyright (c) 2016 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Midtskogen, et al.         Expires May 4, 2017                  [Page 1]
Internet-Draft         Constrained Low Pass Filter          October 2016

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Definitions . . . . . . . . . . . . . . . . . . . . . . . . .   2
     2.1.  Requirements Language . . . . . . . . . . . . . . . . . .   2
     2.2.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   2
   3.  Filtering Process . . . . . . . . . . . . . . . . . . . . . .   3
   4.  Further complexity considerations . . . . . . . . . . . . . .   4
   5.  Performance . . . . . . . . . . . . . . . . . . . . . . . . .   5
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   7
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .   7
   8.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   7
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   7
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .   7
     9.2.  Informative References  . . . . . . . . . . . . . . . . .   8
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   8

1.  Introduction

   Modern video coding standards such as Thor [I-D.fuldseth-netvc-thor]
   include in-loop filters which correct artifacts introduced in the
   encoding process.  Thor includes a deblocking filter which corrects
   artifacts introduced by the block based nature of the encoding
   process, and a low pass filter correcting artifacts not corrected by
   the deblocking filter, in particular artifacts introduced by
   quantisation errors of transform coefficients and by the
   interpolation filter.  Since in-loop filters have to be applied in
   both the encoder and decoder, it is highly desirable that these
   filters have low computational complexity.

2.  Definitions

2.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

2.2.  Terminology

   This document will refer to a pixel X and six of its neighbouring
   pixels A, B, C, D, E, F ordered in the following pattern.

Midtskogen, et al.         Expires May 4, 2017                  [Page 2]
Internet-Draft         Constrained Low Pass Filter          October 2016

                           +---+---+---+---+---+
                           |   |   | A |   |   |
                           +---+---+---+---+---+
                           | B | C | X | D | E |
                           +---+---+---+---+---+
                           |   |   | F |   |   |
                           +---+---+---+---+---+

                     Figure 1: Filter pixel positions

   In Thor the frames are divided into filter blocks (FB) of 128x128,
   64x64 or 32x32 pixels, which is signalled for each frame to be
   filtered.  Also, each frame is divided into coding blocks (CB) which
   range from 8x8 to 128x128 independent of the FB size.  The filter
   described in this draft can be switched on or off for the entire
   frame or optionally on or off for each FB.  CB's that have been coded
   using the skip mode are not filtered, and if a FB only contains CB's
   that have been coded in skip mode, the FB will not be filtered and no
   signal will be transmitted for this FB.

   If the frame can't fit a whole number of FB's, the FB's at the right
   and bottom edges are clipped to fit.  For instance, if the frame
   resolution is 1920x1080 and the FB size is 128x128, the size of the
   FB's at the bottom of the frame becomes 128x56.

3.  Filtering Process

   Given a pixel X and its neighbouring pixels described above we can
   define a general non-linear filter as:

X' = X + clip(a*clip(A-X,-s,s) + b*clip(B-X,-s,s) + c*clip(C-X,-s,s) +
              d*clip(D-X,-s,s) + e*clip(E-X,-s,s) + f*clip(F-X,-s,s),-g,g)

                           Figure 2: Equation 1

   If a neighbour pixel is outside the image frame, it is given the same
   value as the closest pixel within the frame.  To avoid dependencies
   prohibiting parallel processing, all neighbour pixels must be the
   unfiltered pixels of the frame being filtered.

   Experiments in Thor have shown that a good compromise between
   complexity and performance is a=f=1/4, b=e=1/16, c=d=3/16 and the
   filter strength s being 1, 2 or 4 signalled at frame level when the
   bitdepth is 8.  The strengths are scaled according to the bitdepth,
   so they become 4, 8 and 16 when the bitdepth is 10, and 16, 32 and 64

Midtskogen, et al.         Expires May 4, 2017                  [Page 3]
Internet-Draft         Constrained Low Pass Filter          October 2016

   when the bitdepth is 12.  The values for a, b, c, d, e and f
   eliminate the need for the outer clipping to +/-g.  The rounding is
   to the nearest integer.

   This gives us the equation:

   X' = X + (4*clip(A-X,-s,s) + clip(B-X,-s,s) + 3*clip(C-X,-s,s) +
             3*clip(D-X,-s,s) + clip(E-X,-s,s) + 4*clip(F-X,-s,s)) / 16

                           Figure 3: Equation 2

   It can be noted that a=c=d=f=1/4, b=e=0 and s=1 give a slighly
   simpler filter which is very similar to the one described in the
   first version of this draft.

   The filter leaves the encoder 13 different choices for a frame.  The
   filter can be disabled for the entire frame, or the frame is filtered
   using all distinct combinations of strength (1, 2 or 4 scaled for
   bitdepth), non-skip FB signal (enabled/disabled) and FB size (32x32,
   64x64 or 128x128).  Note that the FB size only matters when FB
   signalling is in use.

   The decisions at both frame level and FB level may be based on rate-
   distortion optimisation (RDO), but an encoder running in a low-
   complexity mode, or possibly a low-delay mode, may instead assume
   that a fixed mode will be beneficial.  In general, using s=2, a QP
   dependent FB size and RDO only at the FB level gives good results.

   However, because of the low complexity of the filter, fully RDO based
   decisions are not costly.  The distortion of the 13 configurations of
   the filter can easily be computed in a single pass by keeping track
   of the distortions of the three different strengths and the bit costs
   for different FB sizes.

   The filter is applied after the deblocking filter.

4.  Further complexity considerations

   The filter has been designed to offer the best compromise between low
   complexity and performance.  A single pixel can be filtered with
   simple operations as illustrated by this C function:

Midtskogen, et al.         Expires May 4, 2017                  [Page 4]
Internet-Draft         Constrained Low Pass Filter          October 2016

 int clpf_sample(int X, int A, int B, int C, int D, int E, int F, int s)
 {
   int delta =
     4*clip(A - X, -s, s) + clip(B - X, -s, s) + 3*clip(C - X, -s, s) +
     3*clip(D - X, -s, s) + clip(E - X, -s, s) + 4*clip(F - X, -s, s);
   return (8 + delta - (delta < 0)) >> 4;  // Assumes arithmetic shift
 }

                             Figure 4: C code

   Also, these operations are easily vectorised in architectures
   supporting SIMD instructions, such as x86/SSE4 and ARM/NEON.  The
   pixel difference is 9 bit, but it can be computed using adding an 8
   bit offset and the use of 8 bit saturated signed subtraction.  This
   means that 16 pixels per core can be filtered in parallel on these
   architectures.  Clipping at frame borders can be implemented using
   shuffle instructions.

   A C implementation using x86/SSE4 intrinsics required 6.8
   instructions per pixel to filter a single 8x8 block.  The
   corresponding number for ARM/NEON (armv7) was 4.9.  The compiler was
   gcc 4.8.4 in both cases.

   Since the filter only needs to look up pixels in the line directly
   above and below the pixel to be filtered, the line buffer requirement
   in hardware implementations is very low.

5.  Performance

   The table below shows filters effect on the bandwidth for a selection
   of 10 second video sequences encoded in Thor with uni-prediction
   only.  The numbers have been computed using the Bjontegaard Delta
   Rate (BDR).  BDR-low and BDR-high indicate the effect at low and high
   bitrates, respectively, as described in BDR [BDR].

   The effect of the filter was tested in two encoder low-delay
   configurations: high complexity in which the encoder strongly favours
   compression efficiency over CPU usage, and medium complexity which is
   more suited for real-time applications.  The bandwidth reduction is
   somewhat less in the high complexity configuration.

Midtskogen, et al.         Expires May 4, 2017                  [Page 5]
Internet-Draft         Constrained Low Pass Filter          October 2016

       +----------------+--------------------+--------------------+
       |                | MEDIUM COMPLEXITY  |  HIGH COMPLEXITY   |
       +----------------+------+------+------+--------------------+
       |                |      | BDR- | BDR- |      | BDR- | BDR- |
       |Sequence        |  BDR | low  | high |  BDR | low  | high |
       +----------------+------+------+------+------+------+------+
       |Kimono          | -2.7%| -2.3%| -3.4%| -1.9%| -1.8%| -2.0%|
       |BasketballDrive | -3.3%| -2.5%| -4.5%| -2.1%| -1.6%| -3.0%|
       |BQTerrace       | -7.2%| -4.9%| -9.1%| -5.5%| -3.7%| -6.7%|
       |FourPeople      | -5.7%| -3.9%| -8.6%| -4.0%| -2.8%| -6.0%|
       |Johnny          | -5.9%| -4.0%| -9.0%| -4.7%| -4.0%| -5.8%|
       |ChangeSeats     | -6.4%| -3.4%|-10.8%| -4.5%| -2.8%| -6.8%|
       |HeadAndShoulder | -8.6%| -2.6%|-18.8%| -5.8%| -2.2%|-11.1%|
       |TelePresence    | -5.9%| -3.1%|-10.7%| -4.0%| -2.0%| -7.0%|
       +----------------+------+------+------+--------------------+
       |Average         | -5.7%| -3.3%| -9.4%| -4.0%| -2.6%| -6.0%|
       +----------------+------+------+------+--------------------+

          Figure 5: Compression Performance without Biprediction

   While the filter objectively performs better at relatively high
   bitrates, the subjective effect seems better at relatively low
   bitrates, and overall the subjective effect seems better than what
   the objective numbers suggest.

   If biprediction is allowed, there is generally less bandwidth
   reduction as the table below shows.  These results reflect low-delay
   biprediction without frame reordering.

Midtskogen, et al.         Expires May 4, 2017                  [Page 6]
Internet-Draft         Constrained Low Pass Filter          October 2016

       +----------------+--------------------+--------------------+
       |                | MEDIUM COMPLEXITY  |  HIGH COMPLEXITY   |
       +----------------+------+------+------+--------------------+
       |                |      | BDR- | BDR- |      | BDR- | BDR- |
       |Sequence        |  BDR | low  | high |  BDR | low  | high |
       +----------------+------+------+------+------+------+------+
       |Kimono          | -2.2%| -1.8%| -2.7%| -1.4%| -1.3%| -1.5%|
       |BasketballDrive | -2.6%| -2.5%| -2.7%| -1.4%| -1.6%| -1.1%|
       |BQTerrace       | -4.1%| -3.1%| -4.7%| -2.7%| -2.7%| -2.5%|
       |FourPeople      | -4.0%| -2.9%| -5.3%| -2.7%| -1.9%| -3.4%|
       |Johnny          | -3.5%| -2.7%| -4.6%| -2.2%| -1.6%| -3.1%|
       |ChangeSeats     | -4.2%| -3.0%| -6.1%| -2.6%| -2.0%| -3.2%|
       |HeadAndShoulder | -4.1%| -2.9%| -6.1%| -2.3%| -1.8%| -2.8%|
       |TelePresence    | -2.8%| -1.9%| -4.3%| -1.6%| -1.2%| -2.1%|
       +----------------+------+------+------+------+------+------+
       |Average         | -3.4%| -2.6%| -4.6%| -2.1%| -1.9%| -2.5%|
       +----------------+------+------+------+------+------+------+

            Figure 6: Compression Performance with Biprediction

6.  IANA Considerations

   This document has no IANA considerations yet.  TBD

7.  Security Considerations

   This document has no security considerations yet.  TBD

8.  Acknowledgements

   The authors would like to thank Gisle Bjontegaard for reviewing this
   document and design, and providing constructive feedback and
   direction.

9.  References

9.1.  Normative References

   [I-D.fuldseth-netvc-thor]
              Fuldseth, A., Bjontegaard, G., Midtskogen, S., Davies, T.,
              and M. Zanaty, "Thor Video Codec", draft-fuldseth-netvc-
              thor-02 (work in progress), March 2016.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <http://www.rfc-editor.org/info/rfc2119>.

Midtskogen, et al.         Expires May 4, 2017                  [Page 7]
Internet-Draft         Constrained Low Pass Filter          October 2016

9.2.  Informative References

   [BDR]      Bjontegaard, G., "Calculation of average PSNR differences
              between RD-curves", ITU-T SG16 Q6 VCEG-M33 , April 2001.

Authors' Addresses

   Steinar Midtskogen
   Cisco
   Lysaker
   Norway

   Email: stemidts@cisco.com

   Arild Fuldseth
   Cisco
   Lysaker
   Norway

   Email: arilfuld@cisco.com

   Mo Zanaty
   Cisco
   RTP,NC
   USA

   Email: mzanaty@cisco.com

Midtskogen, et al.         Expires May 4, 2017                  [Page 8]