Skip to main content

An Analysis of Lightweight Virtualization Technologies for NFV
draft-natarajan-nfvrg-containers-for-nfv-02

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft whose latest revision state is "Expired".
Authors natarajan.sriram@gmail.com , Ramki Krishnan , Anoop Ghanwani , Dilip Krishnaswamy, Peter Willis , Ashay Chaudhary , Felipe Huici
Last updated 2016-07-04
Replaces draft-natarajan-containers-for-nfv
RFC stream (None)
Formats
Additional resources
Stream Stream state (No stream defined)
Consensus boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date (None)
Responsible AD (None)
Send notices to (None)
draft-natarajan-nfvrg-containers-for-nfv-02
NFVRG                                                      S. Natarajan
Internet Draft                                                   Google
Category: Informational                                     R. Krishnan
                                                           A. Ghanwani
                                                                   Dell
                                                        D. Krishnaswamy
                                                           IBM Research
                                                              P. Willis
                                                                     BT
                                                           A. Chaudhary
                                                                Verizon
                                                               F. Huici
                                                                    NEC

Expires: March 2016                                        July 4, 2016

      An Analysis of Lightweight Virtualization Technologies for NFV

               draft-natarajan-nfvrg-containers-for-nfv-02

Abstract

   Traditionally, NFV platforms were limited to using standard
   virtualization technologies (e.g., Xen, KVM, VMWare, Hyper-V, etc.)
   running guests based on general-purpose operating systems such as
   Windows, Linux or FreeBSD. More recently, a number of light-weight
   virtualization technologies including containers, unikernels
   (specialized VMs) and minimalistic distributions of general-purpose
   OSes have widened the spectrum of possibilities when constructing an
   NFV platform. This draft describes the challenges in building such a
   platform and discusses to what extent these technologies, as well as
   traditional VMs, are able to address them.

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with
   the provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents

Krishnan et al.          Expires January 2017                  [Page 1]
Internet-Draft    Container-based Platforms for NFV     September 2015

   at any time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire in January 2017.

Copyright Notice

   Copyright (c) 2015 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document.

Conventions used in this document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119.

Table of Contents

   1. Introduction...................................................3
   2. Lightweight Virtualization Background..........................3
      2.1. Containers................................................3
      2.2. OS Tinyfication...........................................3
      2.3. Unikernels................................................4
   3. Challenges in Building NFV Platforms...........................4
      3.1. Performance (SLA).........................................4
         3.1.1. Challenges...........................................4
      3.2. Continuity, Elasticity and Portability....................5
         3.2.1. Challenges:..........................................5
      3.3. Security..................................................6
         3.3.1. Challenges...........................................6
      3.4. Management................................................7
         3.4.1. Challenges...........................................8
   4. Benchmarking Experiments.......................................8
      4.1. Experimental Setup........................................8

Krishnan et al.          Expires January 2017                  [Page 2]
Internet-Draft    Container-based Platforms for NFV     September 2015

      4.2. Instantiation Times.......................................9
      4.3. Throughput................................................9
      4.4. RTT......................................................10
      4.5. Image Size...............................................11
      4.6. Memory Usage.............................................11
   5. Discussion....................................................12
   6. Conclusion....................................................13
   7. Future Work...................................................14
   8. IANA Considerations...........................................14
   9. Security Considerations.......................................14
   10. Contributors.................................................14
   11. Acknowledgements.............................................14
   12. References...................................................14
      12.1. Normative References....................................14
      12.2. Informative References..................................14
   Authors' Addresses...............................................16

1. Introduction

   This draft describes the challenges when building an NFV platform by
   describing to what extent different types of lightweight
   virtualization technologies, such as VMs based on minimalistic
   distributions, unikernels and containers, are able to address them.

2. Lightweight Virtualization Background

2.1. Containers

   Containers are a form of operating-system virtualization. To provide
   isolation, containers such as Docker rely on features of the Linux
   kernel such as cgroups, namespaces and a union-capable file system
   such as aufs and others [AUFS]. Because they run within a single OS
   instance, they avoid the overheads typically associated with
   hypervisors and virtual machines.

2.2. OS Tinyfication

   OS tinyfication consists of creating a minimalistic distribution of
   a general-purpose operating system such as Linux or FreeBSD. This
   involves two parts: (1) configuring the kernel so that only needed
   features and modules are enabled/included (e.g., removing extraneous
   drivers); and (2) including only the required user-level libraries
   and applications needed for the task at hand, and running only the
   minimum amount of required processes. The most notable example of a
   tinyfied OS is the work on Linux tinyfication [LINUX-TINY].

Krishnan et al.          Expires January 2017                  [Page 3]
Internet-Draft    Container-based Platforms for NFV     September 2015

2.3. Unikernels

   Unikernels are essentially single-application virtual machines based
   on minimalistic OSes. Such minimalistic OSes have minimum overhead
   and are typically single-address space (so no user/kernel space
   divide and no expensive system calls) and have a co-operative
   scheduler (so reducing context switch costs). Examples of such
   minimalistic OSes are MiniOS [MINIOS] which runs on Xen and OSv
   [OSV] which runs on KVM, Xen and VMWare.

3. Challenges in Building NFV Platforms

   In this section, we outline the set of main challenges for an NFV
   platform in the context of lightweight virtualization technologies
   as well as traditional VMs.

3.1. Performance (SLA)
   Performance requirements vary with each VNF type and configuration.
   The platform should support the specification, realization and
   runtime adaptation of different performance metrics. Achievable
   performance can vary depending on several factors such as the
   workload type, the size of the workload, the set of virtual machines
   sharing the underlying infrastructure, etc. Here we highlight some
   of the challenges based on potential deployment considerations.

     3.1.1. Challenges

   .  VNF provisioning time (including up/down/update) constitutes the
     time it takes to spin-up the VNF process, its application-specific
     dependencies, and additional system dependencies. The resource
     choices such as the hypervisor type, the guest and host OS flavor
     and the need for hardware and software accelerators, etc.,
     constitute a significant portion of this processing time
     (instantiation or down time) when compared to just bringing up the
     actual VNF process.

   .  The runtime performance (achievable throughput, line rate speed,
     maximum concurrent sessions that can be maintained, number of new
     sessions that can be added per second) for each VNF is directly
     dependent on the amount of resources (e.g., virtual CPUs, RAM)
     allocated to individual VMs. Choosing the right resource setting
     is a tricky task. If VM resources are over-provisioned, we end up
     under-utilizing the physical resources. On the contrary if we
     under-provision the VM resources, then upgrading the resource to
     an advanced system setting might require scaling out or scaling up
     of the resources and re-directing traffic to the new VM; scaling
     up/down operations consume time and add to the latency. This

Krishnan et al.          Expires January 2017                  [Page 4]
Internet-Draft    Container-based Platforms for NFV     September 2015

     overhead stems from the need to account resources of components
     other than the actual VNF process (e.g., guest OS requirements).

   .  If each network function is hosted in individual VMs/containers,
     then an efficient inter-VM networking solution is required for
     performance.

3.2. Continuity, Elasticity and Portability

   VNF service continuity can be interrupted due to several factors:
   undesired state of the VNF (e.g., VNF upgrade progress), underlying
   hardware failure, unavailability of virtualized resources, VNF SW
   failure, etc.  Some of the requirements that need consideration are:

     3.2.1. Challenges:

   o  VNF's are not completely decoupled from the underlying
     infrastructure. As discussed in the previous section, most VNFs
     have a dependency on the guest OS, hypervisor type, accelerator
     used, and the host OS (this last one applies to containers too).
     Therefore porting VNFs to a new platform might require identifying
     equivalent resources (e.g., hypervisor support, new hardware
     model, understanding resource capabilities) and repeating the
     provisioning steps to bring back the VNF to a working state.

   o  Service continuity requirements can be classified as follows:
     seamless (with zero impact) or non-seamless continuity (accepts
     measurable impacts to offered services). To achieve this, the
     virtualization technology needs to provide an efficient high
     availability solution or a quick restoration mechanism that can
     bring back the VNF to an operational state. For example, an
     anomaly caused by a hardware failure can impact all VNFs hosted on
     that infrastructure resource. To restore the VNF to a working
     state, the user should first provision the VM/container, spin-up
     and configure the VNF process inside the VM, setup the
     interconnects to forward network traffic, manage the VNF-related
     state, and update any dependent runtime agents.

   o  Addressing the service elasticity challenges require a holistic
     view of the underlying resources. The challenges for presenting a
     holistic view include the following

        o Performing Scalable Monitoring: Scalable continuous
          monitoring of the individual resource's current state is
          needed to spin-up additional resources (auto-scale or auto-

Krishnan et al.          Expires January 2017                  [Page 5]
Internet-Draft    Container-based Platforms for NFV     September 2015

          heal) when the system encounters performance degradation or
          spin-down idle resources to optimize resource usage.

        o Handling CPU-intensive vs I/O-intensive VNFs: For CPU-
          intensive VNFs the degradation can primarily depend on the
          VNF processing functionality. On the other hand, for I/O
          intense workloads, the overhead is significantly impacted by
          to the hypervisor/host features, its type, the number of
          VMs/contaiers it manages, the modules loaded in the guest OS,
          etc.

3.3. Security

   Broadly speaking, security can be classified into:

   o  Security features provided by the VNFs to manage the state, and

   o  Security of the VNFs and its resources.

   Some considerations on the security of the VNF infrastructure are
   listed here.

     3.3.1. Challenges

   o  The adoption of virtualization techniques (e.g., para-
     virtualization, OS-level) for hosting network functions and the
     deployment need to support multi-tenancy requires secure slicing
     of the infrastructure resources. In this regard, it is critical to
     provide a solution that can ensure the following:

        o Provision the network functions by guaranteeing complete
          isolation across resource entities (hardware units,
          hypervisor, virtual networks, etc.). This includes secure
          access between VM/container and host interface, VM-VM or
          container-to-container communication, etc. For maximizing
          overall resource utilization and improving service
          agility/elasticity, sharing of resources across network
          functions must be possible.

        o When a resource component is compromised, quarantine the
          compromised entity but ensure service continuity for other
          resources.

        o Securely recover from runtime vulnerabilities or attacks and
          restore the network functions to an operational state.
          Achieving this with minimal or no downtime is important.

Krishnan et al.          Expires January 2017                  [Page 6]
Internet-Draft    Container-based Platforms for NFV     September 2015

   Realizing the above requirements is a complex task in any type of
   virtualization option (virtual machines, containers, etc.)

   o  Resource starvation / Availability: Applications hosted in
     VMs/containers can starve the underlying physical resources such
     that co-hosted entities become unavailable. Ideally,
     countermeasures are required to monitor the usage patterns of
     individual VMs/containers and ensure fair use of individual
     resources.

3.4. Management

   The management and operational aspects are primarily focused on the
   VNF lifecycle management and its related functionalities. In
   addition, the solution is required to handle the management of
   failures, resource usage, state processing, smooth rollouts, and
   security as discussed in the previous sections.  Some features of
   management solutions include:

     oCentralized control and visibility: Support for web client,
       multi-hypervisor management, single sign-on, inventory search,
       alerts & notifications.

     oProactive Management: Creating host profiles, resource management
       of VMs/containers, dynamic resource allocation, auto-restart in
       HA model, audit trails, patch management.

     oExtensible platform: Define roles, permissions and licenses
       across resources and use of APIs to integrate with other
       solutions.

   Thus, the key requirements for a management solution

     o  Simple to operate and deploy VNFs.

     o  Uses well-defined standard interfaces to integrate seamlessly
        with different vendor implementations.

     o  Creates functional automation to handle VNF lifecycle
        requirements.

     o  Provide APIs that abstracts the complex low-level information
        from external components.

     o  Is secure.

Krishnan et al.          Expires January 2017                  [Page 7]
Internet-Draft    Container-based Platforms for NFV     September 2015

     3.4.1. Challenges

   The key challenge is addressing the aforementioned requirements for
   a management solution while dealing with the multi-dimensional
   complexity introduced by the hypervisor, guest OS, VNF
   functionality, and the state of network.

4. Benchmarking Experiments

   Having considered the basic requirements and challenges of building
   an NFV platform, we now provide a benchmark of a number of
   lightweight virtualization technologies to quantify to what extent
   they can be used to build such a platform.

4.1. Experimental Setup

   In terms of hardware, all tests are run on an x86-64 server with an
   Intel Xeon E5-1630 v3 3.7GHz CPU (4 cores) and 32GB RAM.

   For the hypervisors we use KVM running on Linux 4.4.1 and Xen
   version 4.6.0. The virtual machines running on KVM and Xen are of
   three types:

   (1)Unikernels, on top of the minimalistic operating systems OSv and
        MiniOS for KVM and Xen, respectively. The only application
        built into them is iperf. To denote them we use the shorthand
        unikernel.osv.kvm or unikernels.minios.xen.

   (2)Tinyfied Linux (a.k.a. Tinyx), consisting of a Linux kernel
        version 4.4.1 with only a reduced set of drivers (ext4, and
        netfront/blkfront for Xen), and a distribution containing only
        busybox, an ssh server for convenience, and iperf. We use the
        shorthand tinyx.kvm and tinyx.xen.

   (3)Standard VM, consisting of a Debian distribution including iperf
        and Linux version 4.4.1. We use the shorthand standardvm.kvm
        and standardvm.xen for it.

   (4)For containers, we use Docker version 1.11 running on Linux
        4.4.1.

   It is worth noting that the numbers reported here for virtual
   machines (whether standard, Tinyx or unikernels) include the
   following optimizations to the underlying virtualization
   technologies. For Xen, we use the optimized Xenstore, toolstack and
   hotplug scripts reported in [SUPERFLUIDITY] as well as the
   accelerated packet I/O derived from persistent grants (for Tx)

Krishnan et al.          Expires January 2017                  [Page 8]
Internet-Draft    Container-based Platforms for NFV     September 2015

   [PGRANTS]. For KVM, we remove the creation of a tap device from the
   VM's boot process and use a pre-created tap device instead.

4.2. Instantiation Times

   We begin by measuring how long it takes to create and boot a
   container or VM. The beginning time is when we issue the create
   operation. To measure the end time, we carry out a SYN flood from an
   external server and measure the time it takes for the container/VM
   to respond with a RST packet. The reason for a SYN flood is that it
   guarantees the shortest reply time after the unikernels/container is
   booted. It is just to measure boot time, nothing to do with real-
   world deployments and DoS attacks.

                +-----------------------+--------------+
                | Technology Type       | Time (msecs) |
                |--------------------------------------+
                | standardvm.xen        |     6500     |
                | standardvm.kvm        |     2988     |
                | container             |     1711     |
                | tinyx.kvm             |     1081     |
                | tinyx.xen             |     431      |
                | unikernel.osv.kvm     |     330      |
                | unikernels.minios.xen |     31       |
                +-----------------------+--------------+

   The table above shows the results. Unsurprisingly, standard VMs with
   a regular distribution (in this case Debian) fare the worst, with
   times in the seconds: 6.5s on Xen and almost 3s on KVM. The Docker
   container with iperf comes next, clocking in at 1.7s. The next best
   times are from Tinyx: 1s approximately on KVM and 431ms on Xen.
   Finally, the best numbers come from unikernels, with 330ms for OSv
   on KVM and 31ms for MiniOS on Xen. These results show that at least
   when compared to unoptimized containers, minimalistic VMs or
   unikernels can have instantiation times comparable to or better than
   containers.

4.3. Throughput

   To measure throughput we use the iperf application that is built in
   to the unikernels, included as an application in Tinyx and the
   Debian-based VMs, and containerized for Docker. The experiments in

Krishnan et al.          Expires January 2017                  [Page 9]
Internet-Draft    Container-based Platforms for NFV     September 2015

   this section are for TCP traffic between the guest and the host
   where the guest resides: there are no NICs involved so that rates
   are not bound by physical medium limitations.

   +-----------------------+-------------------+-------------------+
   |     Technology        | Throughput (Gb/s) | Throughput (Gb/s) |
   |       Type            |        Tx         |        Rx         |
   |-----------------------+-------------------+-------------------+
   | standardvm.xen        |        23.1       |        24.5       |
   | standardvm.kvm        |        20.1       |        38.9       |
   | container             |        45.1       |        43.8       |
   | tinyx.kvm             |        21.5       |        37.9       |
   | tinyx.xen             |        28.6       |        24.9       |
   | unikernel.osv.kvm     |        47.9       |        47.7       |
   | unikernels.minios.xen |        49.5       |        32.6       |
   +-----------------------+-------------------+-------------------+

   The table above shows the results for Tx and Rx. The first thing to
   note is that throughput is not only dependent on the guest's
   efficiency, but also on the host's packet I/O framework (e.g., see
   [CLICKOS] for an example of how optimizing Xen's packet I/O
   subsystem can lead to large performance gains). This is evident from
   the Xen numbers, where Tx has been optimized and Rx not. Having said
   that, the guest also matters, which is why, for example, Tinyx
   scores somewhat higher throughput than standard VMs. Containers and
   unikernels (at least for Tx and for Tx/Rx for KVM) are fairly
   equally matched and perform best, with unikernels having a slight
   edge.

4.4. RTT

   To measure round-trip time (RTT) from an external server to the
   VM/container we carry out a ping flood and report the average RTT.

                +-----------------------+--------------+
                | Technology Type       | Time (msecs) |
                |--------------------------------------+
                | standardvm.xen        |      34      |
                | standardvm.kvm        |      18      |
                | container             |       4      |
                | tinyx.kvm             |      19      |
                | tinyx.xen             |      15      |
                | unikernel.osv.kvm     |       9      |
                | unikernels.minios.xen |       5      |

Krishnan et al.          Expires January 2017                 [Page 10]
Internet-Draft    Container-based Platforms for NFV     September 2015

                +-----------------------+--------------+

   As shown in the table above, the Docker container comes out on top
   with 4ms, but unikernels achieve for all practical intents and
   purposes the same RTT (5ms on MiniOS/Xen and 9ms on OSv/KVM). Tinyx
   fares slightly better than the standard VMs.

4.5. Image Size

   We measure image size using the standard "ls" tool.

                +-----------------------+------------+
                | Technology Type       | Size (MBs) |
                |------------------------------------+
                | standardvm.xen        |    913     |
                | standardvm.kvm        |    913     |
                | container             |     61     |
                | tinyx.kvm             |    3.5     |
                | tinyx.xen             |    3.7     |
                | unikernel.osv.kvm     |     12     |
                | unikernels.minios.xen |      2     |
                +-----------------------+------------+

   The table shows the standard VMs to be unsurprisingly the largest
   and, followed by the Docker/iperf container. OSv-based unikernels
   are next with about 12MB, followed by Tinyx (3.5MB or 3.7MB on KVM
   and Xen respectively). The smallest image is the one based on
   MiniOS/Xen with 2MB.

4.6. Memory Usage

   For the final experiment we measure memory usage for the various
   VMs/container. To do so we use standard tools such as "top" and "xl"
   (Xen's management tool).

                +-----------------------+-------------+
                | Technology Type       | Usage (MBs) |
                |-------------------------------------+
                | standardvm.xen        |     112     |
                | standardvm.kvm        |     82      |
                | container             |     3.8     |
                | tinyx.kvm             |     30      |
                | tinyx.xen             |     31      |
                | unikernel.osv.kvm     |     52      |

Krishnan et al.          Expires January 2017                 [Page 11]
Internet-Draft    Container-based Platforms for NFV     September 2015

                | unikernels.minios.xen |     8       |
                +-----------------------+------------+

   The largest memory consumption, as shown in the table above, comes
   from the standard VMs. The OSv-based unikernels comes next due to
   the fact that OSv pre-allocates memory for buffers, among other
   things. Tinyx is next with about 30MB. From there there's a big jump
   to the MiniOS-based unikernels with 8MB. The best result comes from
   the Docker container, which is expected given that it relies on the
   host and its memory allocations to function.

5. Discussion

   In this section we provide a discussion comparing and contrasting
   the various lightweight virtualization technologies in view of the
   reported benchmarks. There are a number of issues at stake:

     .  Service agility/elasticity: this is largely dependent on the
        ability to quickly spin up/down VMs/containers and migrate
        them. Clearly the best numbers in this category come from
        unikernels and containers.

     .  Memory consumption: containers use and share resources from
        the common host they use and so each container instance uses up
        less memory than VMs, as shown in the previous section
        (although unikernels are not far behind). Note: VMs also have a
        common host (or dom0 in the case of Xen) but they incur the
        overhead of each having its own guest OS.

     .  Security/Isolation: an NFV platform needs to provide good
        isolation for its tenants. Generally speaking, VM-based
        technologies have been around for longer and so have had time
        to iron out most of the security issues they had. Type-1
        hypervisors (e.g., Xen), in addition, provide a smaller attack
        surface than Type-2 ones (e.g., KVM) so should in principle be
        more robust. Containers are relatively newcomers and as such
        still have a number of open issues [CONTAINER-SECURITY]. Use of
        kernel security modules like SELinux [SELINUX], AppArmor
        [APPARMOR] along with containers can provide at least some of
        the required features for a secure VNF deployment. Use of
        resource quota techniques such as those in Kubernetes
        [KUBERNETES-RESOURCE-QUOTA] can provide at least some of the
        resource guarantees for a VNF deployment.

Krishnan et al.          Expires January 2017                 [Page 12]
Internet-Draft    Container-based Platforms for NFV     September 2015

     .  Management frameworks: both virtual machines and containers
        have fully-featured management frameworks with large open
        source communities continually improving them. Unikernels might
        need a bit of "glue" to adapt them to an existing framework
        (e.g., OpenStack).

     .  Compatibility with applications. Both containers and standard
        VMs can run any application that is able to run on the general-
        purpose OS those VMs/containers are based on (typically Linux).
        Unikernels, on the hand, use minimalistic OSes, which might
        present a problem. OSv, for example, is able to build a
        unikernels as long as the application can be recompiled as a
        shared library. MiniOS requires that the application be
        directly compiled with it (c/c++ is the default, but MiniOS
        unikernels based on OCaml, Haskell and other languages exist).

   Overall, the choice between standard virtual machines, tinyfied
   ones, unikernels or containers is often not a black and white one.
   Rather, these technologies present points in a spectrum where
   criteria such as security/isolation, performance, and compatibility
   with existing applications and frameworks may point NFV operators,
   and their clients, towards a particular solution. For instance, an
   operator for whom excellent isolation and multi-tenancy is a must
   might lean towards hypervisor-based solutions. If that operator
   values ease of application deployment he will further choose guests
   based on a general-purpose OS (whether tinfyied or not). Another
   operator might put a prime on performance and so might prefer
   unikernels. Yet another one might not have a need for multi-tenancy
   (e.g., Google, Edge use cases such as CPE) and so would lean towards
   enjoying the benefits of containers. Hybrid solutions, where
   containers are run within VMs, are also possible. In short, choosing
   a virtualization technology for an NFV platform is (no longer) as
   simple as choosing VMs or containers.

6. Conclusion

   In this draft we presented the challenges when building an NFV
   platform. We further introduced a set of benchmark results to
   quantify to what extent a number of virtualization technologies
   (standard VMs, tinfyied VMs, unikernels and containers) can meet
   those challenges. We conclude that choosing a solution is nuanced,
   and depends on how much value different NFV operators place on
   criteria such as strong isolation, performance and compatibility
   with applications and management frameworks.

Krishnan et al.          Expires January 2017                 [Page 13]
Internet-Draft    Container-based Platforms for NFV     September 2015

7. Future Work

   Opportunistic areas for future work include but not limited to
   developing solutions to address the VNF challenges described in
   Section 3, distributed micro-service network functions, etc.

8. IANA Considerations

   This draft does not have any IANA considerations.

9. Security Considerations

   VM-based VNFs can offer a greater degree of isolation and security
   due to technology maturity as well as hardware support. Light-weight
   virtualization technologies such as unikernels (specialized VMs) and
   tinyfied VMs which were discussed enjoyed the security benefits of a
   standard VM. Since container-based VNFs provide abstraction at the
   OS level, it can introduce potential vulnerabilities in the system
   when deployed without proper OS-level security features. This is one
   of the key implementation/deployment challenges that needs to be
   further investigated.

   In addition, as containerization technologies evolve to leverage the
   virtualization capabilities provided by hardware, they can provide
   isolation and security assurances similar to VMs.

10. Contributors

11. Acknowledgements

   The authors would like to thank Vineed Konkoth for the Virtual
   Customer CPE Container Performance white paper.

12. References

12.1. Normative References

12.2. Informative References

   [AUFS] "Advanced Multi-layered Unification Filesystem,"
   https://en.wikipedia.org/wiki/Aufs

   [CONTAINER-SECURITY] "Container Security article,"
   http://www.itworld.com/article/2920349/security/for-containers-
   security-is-problem-1.html

Krishnan et al.          Expires January 2017                 [Page 14]
Internet-Draft    Container-based Platforms for NFV     September 2015

   [ETSI-NFV-WHITE]  "ETSI NFV White Paper,"
   http://portal.etsi.org/NFV/NFV_White_Paper.pdf

   [ETSI-NFV-USE-CASES] "ETSI NFV Use Cases,"
   http://www.etsi.org/deliver/etsi_gs/NFV/001_099/001/01.01.01_60/gs_N
   FV001v010101p.pdf

   [ETSI-NFV-REQ]   "ETSI NFV Virtualization Requirements,"
   http://www.etsi.org/deliver/etsi_gs/NFV/001_099/004/01.01.01_60/gs_N
   FV004v010101p.pdf

   [ETSI-NFV-ARCH]   "ETSI NFV Architectural Framework,"
   http://www.etsi.org/deliver/etsi_gs/NFV/001_099/002/01.01.01_60/gs_N
   FV002v010101p.pdf

   [ETSI-NFV-TERM] "Terminology for Main Concepts in NFV,"
   http://www.etsi.org/deliver/etsi_gs/NFV/001_099/003/01.01.01_60/gs_n
   fv003v010101p.pdf

   [KUBERNETES-RESOURCE-QUOTA] "Kubernetes Resource Quota,"
   http://kubernetes.io/v1.0/docs/admin/resource-quota.html

   [KUBERNETES-SELF-HEALING] "Kubernetes Design Overview,"
   http://kubernetes.io/v1.0/docs/design/README.html

   [LINUX-TINY] "Linux Kernel Tinification,"
   https://tiny.wiki.kernel.org/

   [MINIOS] "Mini-OS - Xen," http://wiki.xenproject.org/wiki/Mini-OS

   [OSV] "OSv - The Operating System Designed for the Cloud,"
   http://osv.io/

   [PGRANTS] http://lists.xenproject.org/archives/ html/xen-
   devel/2015- 05/msg01498.html

   [SELINUX] "Security Enhanced Linux (SELinux) project,"
   http://selinuxproject.org/

   [SUPERFLUIDITY] "The Case for the Suplerfluid Cloud," F. Manco, J.
   Martins, K. Yasukata, J. Mendes, S. Kuenzer, and F. Huici. USENIX
   HotCloud 2015

   [APPARMOR] "Mandatory Access Control Framework,"
   https://wiki.debian.org/AppArmor

Krishnan et al.          Expires January 2017                 [Page 15]
Internet-Draft    Container-based Platforms for NFV     September 2015

   [VCPE-CONTAINER-PERF] "Virtual Customer CPE Container Performance
   White Paper," http://info.ixiacom.com/rs/098-FRB-840/images/Calsoft-
   Labs-CaseStudy2015.pdf

Authors' Addresses

   Sriram Natarajan
   Google
   natarajan.sriram@gmail.com

   Ram (Ramki) Krishnan
   Dell
   ramki_krishnan@dell.com

   Anoop Ghanwani
   Dell
   anoop@alumni.duke.edu

   Dilip Krishnaswamy
   IBM Research
   dilikris@in.ibm.com

   Peter Willis
   BT
   peter.j.willis@bt.com

   Ashay Chaudhary
   Verizon
   the.ashay@gmail.com

   Felipe Huici
   NEC
   felipe.huici@neclab.eu

Krishnan et al.          Expires January 2017                 [Page 16]