End Marker in 5G Data Network
draft-zzhang-dmm-5gdn-end-marker-01

dmm                                                             Z. Zhang
Internet-Draft                                          Juniper Networks
Intended status: Standards Track                              M. Liebsch
Expires: 9 August 2024                                           NEC Lab
                                                                T. Jiang
                                                            China Mobile
                                                         6 February 2024

                     End Marker in 5G Data Network
                  draft-zzhang-dmm-5gdn-end-marker-01

Abstract

   In a mobile network, when a mobile device (referred to as User
   Equipment or UE) moves from one Access Node (source AN) to another
   (target AN), the anchoring node that connects the UE to the Data
   Network sends an End Marker to the source AN after it starts sending
   UE traffic towards the target AN.  The target AN buffers the data
   until it receives the End Marker forwarded by the source AN.  This is
   to preserve the order of packets during the handover between ANs.

   The anchoring node is referred to as User Plane Function (UPF) in 5G
   and it is a Network Function of the mobile network.  The UPFs are
   traditionally centrally deployed but are more and more distributed
   closer to the edge.  With distributed UPFs, handover becomes
   necessary among UPFs, and the End Marker mechanism may need to be
   extended to Data Network devices that are not mobile network
   functions.  This document discusses the problem and proposes a
   solution based on ICMP messages if packet ordering needs to be
   preserved during handover between UPFs.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 9 August 2024.

Zhang, et al.             Expires 9 August 2024                 [Page 1]
Internet-Draft              5G-DN End Marker               February 2024

Copyright Notice

   Copyright (c) 2024 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  End Marker with Centralized UPF . . . . . . . . . . . . .   2
     1.2.  End Marker with Distributed UPF/ANUP  . . . . . . . . . .   4
       1.2.1.  Traffic via a Hub Router  . . . . . . . . . . . . . .   5
       1.2.2.  Traffic from Anywhere . . . . . . . . . . . . . . . .   5
       1.2.3.  Remote UE Routes on Source UPF  . . . . . . . . . . .   6
       1.2.4.  Is End Marker Mechanism Really Needed?  . . . . . . .   6
   2.  Specification . . . . . . . . . . . . . . . . . . . . . . . .   7
   3.  Security Considerations . . . . . . . . . . . . . . . . . . .   7
   4.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   7
   5.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   7
   6.  Informative References  . . . . . . . . . . . . . . . . . . .   7
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   8

1.  Introduction

   In a mobile network like 5G [_3GPP-23.501], when a mobile device
   (referred to as User Equipment or UE) moves from one Access Node
   (source AN) to another (target AN), the anchoring node User Plane
   Function (UPF) that connects the UE to the Data Network sends an End
   Marker to the source AN after it starts sending UE traffic towards
   the target AN.  The target AN buffers the data until it receives the
   End Marker forwarded by the source AN.  This is to preserve the order
   of packets during the handover between ANs.

1.1.  End Marker with Centralized UPF

   The End Marker mechanism in case of centralized UPF is described with
   the following diagram.

Zhang, et al.             Expires 9 August 2024                 [Page 2]
Internet-Draft              5G-DN End Marker               February 2024

                    Internet
                    /
                   /N6
               UPF
             /     \
            /       \
            |  N3   |
            |       |
            |       |
           AN1     AN2
            |  ->   |
             \     /
               UE1

   UE1 is initially connected to AN1.  There is an N3 tunnel between AN1
   and UPF for the UE.  The UE1-AN1 connection and the AN1-UPF N3 tunnel
   together provide a Packet Data Unit Session (PDU Session) that
   connects UE1 to the Data Network (DN, which is Internet in this case)
   via the UPF.  The UPF does IP routing between the DN and UE (among
   other things).

   When UE1 moves from AN1 (source) to AN2 (target), a new N3 tunnel
   between AN2 and UPF is established as instructed by 5G control plane
   functions.  For Down Link (DL) traffic from the Internet, UPF starts
   sending towards AN2 via the new N3 tunnel.  There could be inflight
   packets towards AN1, which will be redirected by AN1 to AN2.  The
   redirected packets were sent by the UPF before it switches over to
   the new tunnel, yet they could arrive on AN2 after the packets that
   the UPF sends over the new tunnel to AN2.

   To preserve the packet ordering during this handover, the following
   sequence of actions are taken:

   *  The UPF sends an End Marker to the source AN1 over the old tunnel
      after it sends the last packet over the old tunnel and switches to
      the new tunnel for future packets.

   *  The source AN1 redirects inflight packets to the target AN2, who
      forwards the redirected traffic to UE1, while buffering data
      arriving on the new tunnel.

   *  The source AN1 redirects the received End Marker (which comes
      after the inflight packets) to the target AN2, who will now start
      sending buffered and new arrival data to UE1.

Zhang, et al.             Expires 9 August 2024                 [Page 3]
Internet-Draft              5G-DN End Marker               February 2024

1.2.  End Marker with Distributed UPF/ANUP

   With distributed UPF, the diagram becomes the following:

              Hub PE --- Internet
             /     \
            |  DN   |
            |       |
           PE1     PE2
            |       |
           UPF1    UPF2
            |       |
            |  N3   |
           AN1     AN2
            |  ->   |
             \     /
               UE1

   UPF1 and UPF2 are close to the ANs.  The DN is an IP VPN (referred to
   as DNVPN) provided by PE1/PE2 and a hub PE (where the previous
   central UPF was) connected to the Internet.  The UPFs are VPN CEs.

   [I-D.zzhang-dmm-mup-evolution] proposes an Integrated AN/UPF (ANUP)
   concept for co-located AN and UPFs.  Whether AN and UPFs are separate
   or integrated, the problem discussion and proposed solution in this
   document apply.

   As described in [I-D.zzhang-dmm-mup-evolution], when UE1 moves from
   AN1/UPF1 to AN2/UPF2, its IP address may be retained if that is
   desired.  A UPF has UE specific routes/state for its local UEs so
   that DL traffic can be forwarded accordingly.  The hub PE has UE
   specific routes for all UEs that have persistent addresses so that DL
   traffic may be sent to the appropriate UPFs.  While some PEs may also
   have UE specific routes for UEs attached to other UPFs, some may just
   have a default route to send all Up Link (UL) traffic (traffic not
   destined to its local UEs) towards the hub PE.

   This is achieved by standard hub-and-spoke VPN mechanisms: some
   routes are advertised with Route Targets that allow them to be
   installed everywhere, while some other routes are advertised with
   different Route Targets that allow them to be installed only on hub
   routers.

Zhang, et al.             Expires 9 August 2024                 [Page 4]
Internet-Draft              5G-DN End Marker               February 2024

1.2.1.  Traffic via a Hub Router

   For DL traffic from the Internet via the hub router, re-ordering may
   happen when UE1 moves from AN1/UPF1 to AN2/UPF2 - after the hub PE
   updates its UE1 route to send traffic to UPF2, there may be inflight
   UE1 packets towards PE/UPF1 but then rerouted to PE2/UPF2 (either
   directly or via the hub PE) in the DN, and they may arrive after the
   traffic that the hub sends to UPF2 directly.

   Notice that during this process, UPF1 will not trigger an End Marker
   to AN1 because there is no N3 tunnel changing on UPF1 as UE1 is now
   moving to gNB2/UPF2 and UPF1 does not use N3 tunneling to gNB2/UPF2.
   Instead, the hub PE needs to trigger UPF1 to send an End Marker to
   AN1 after the hub router changes its UE1 route from PE1 to PE2.

   Because PEs are just DN devices not 5G function, an ICMP message can
   be sent instead by the hub PE towards UPF1's address in the DN.  The
   ICMP message is of a new type/code and carries the UE route so that
   UPF1 can match the PDU session and send End Marker to AN1.  In
   certain cases, e.g., with [I-D.mhkk-dmm-srv6mup-architecture], the
   hub PE may know the PDU session information associated with a UE
   route and it can include the session information in the ICMP message.

   Note that, if PE1 also maintains UE routes received from other PEs,
   it prefers its own received from UPF1.  This is typical IPVPN
   behavior that local routes are preferred.  This ensures that inflight
   traffic (before the ICMP trigger for End Marker) from the hub PE is
   sent to UPF1 instead of routed to PE2/UPF2.  Otherwise, reordering
   may happen (inflight traffic before the End Markter trigger arrives
   after the traffic that is sent to UPF2 after the trigger).

   After UPF1 sends End Marker, it withdraws the UE1 route from PE1.

1.2.2.  Traffic from Anywhere

   In the previous section, only traffic from the hub PE is considered.
   However, with distributed UPFs, DL traffic may be from anywhere
   directly without going through a hub.  For example, there could be a
   UPF3/PE3 connecting to a local DN site and traffic from that site via
   PE3 or UE traffic via UPF3 to UE1 also needs to preserve order when
   UE1 moves to UPF2/AN2.  If PE3 has UE1 specific route (previously
   through PE1/UPF1 then through PE2/UPF2 directly instead of just a
   default route via the hub PE), then UPF3 also needs to trigger UPF1
   to send End Marker.

   In this case, the source UPF1 needs to know how many triggers it
   needs to receive before sending the End Marker.  The number is
   dependent on how many PEs are configured with the Route Target that

Zhang, et al.             Expires 9 August 2024                 [Page 5]
Internet-Draft              5G-DN End Marker               February 2024

   the UE1 route is advertised with (those PEs will install the UE1
   router and send End Marker trigger when their UE1 route changes from
   PE1/UPF1 to PE2/UPF2).  Since the trigger could be lost, and some PEs
   may be down or slow in sending the trigger, UPF1 needs to send End
   Marker to its AN when one of the following conditions are met:

   *  The expected number of triggers are received, or,

   *  A preset timer expires

   Alternatively, the source UPF1 can send the End Marker upon receipt
   of a master trigger from a controller.

1.2.3.  Remote UE Routes on Source UPF

   In both Section 1.2.1 and Section 1.2.2 scenarios, the source PE1 may
   have the specific UE1 route that is advertised from the target PE2.
   After the hub PE or PE3 switches to the target PE2/UPF2, inflight
   traffic sent before the switch, arriving on PE1 and then following
   the UE1 route on PE1, may arrive on UPF2 later than the traffic sent
   to UPF2 directly after the switch, if on PE1 the UE1 route through
   PE2/UPF2 is preferred over the UE1 route through UPF1.

   With VPN the CE routes through a PE-CE interface are typically
   preferred over those learned from another PE - so this should work
   just fine

1.2.4.  Is End Marker Mechanism Really Needed?

   One may wonder if the it is really necessary to extend the End Marker
   mechanism to the DN, given the following considerations:

   *  Complexity especially when traffic is not just from a hub PE.

   *  With higher speed of 5G/6G, buffering data may become more and
      more difficult.

   *  IP network does not guarantee packet ordering.  While routers do
      try to send traffic of the same flow out of the same ECMP branch,
      topology change will likely cause reordering and routers do not
      buffer data for ordering purposes.

Zhang, et al.             Expires 9 August 2024                 [Page 6]
Internet-Draft              5G-DN End Marker               February 2024

   On the other hand, mobile network is unique in that mobility handover
   happens more frequently than topology changes in traditional IP
   networks, and in the discussions related to session continuity in the
   context of distributed UPF and ANUP, preserving packet ordering is
   often brought up (at least rhetorically).  Therefore, this document
   does discuss the problem and propose a solution in case it is indeed
   needed.

2.  Specification

   Normative specification will be provided in future revisions, if it
   is agreed that End Marker mechanism is indeed needed in the DN.

3.  Security Considerations

   To be provided.

4.  IANA Considerations

   To be added.

5.  Acknowledgements

   The authors thank Constantine Polychronopoulos, Arda Akman and Jari
   Mutikainen for their review, comments and suggestions to make this
   document and solution more complete.

6.  Informative References

   [I-D.mhkk-dmm-srv6mup-architecture]
              Matsushima, S., Horiba, K., Khan, A., Kawakami, Y.,
              Murakami, T., Patel, K., Kohno, M., Kamata, T., Camarillo,
              P., Horn, J., Voyer, D., Zadok, S., Meilik, I., Agrawal,
              A., and K. Perumal, "Mobile User Plane Architecture using
              Segment Routing for Distributed Mobility Management", Work
              in Progress, Internet-Draft, draft-mhkk-dmm-srv6mup-
              architecture-06, 23 October 2023,
              <https://datatracker.ietf.org/doc/html/draft-mhkk-dmm-
              srv6mup-architecture-06>.

   [I-D.zzhang-dmm-mup-evolution]
              Zhang, Z. J., Patel, K., Contreras, L. M., Islam, K.,
              Mutikainen, J., Jiang, T., Jalil, L., Sejati, O. P., and
              S. Zadok, "Mobile User Plane Evolution", Work in Progress,
              Internet-Draft, draft-zzhang-dmm-mup-evolution-06, 10 July
              2023, <https://datatracker.ietf.org/doc/html/draft-zzhang-
              dmm-mup-evolution-06>.

Zhang, et al.             Expires 9 August 2024                 [Page 7]
Internet-Draft              5G-DN End Marker               February 2024

   [_3GPP-23.501]
              "System architecture for the 5G System (5GS), V18.2.1",
              June 2023.

Authors' Addresses

   Zhaohui Zhang
   Juniper Networks
   Email: zzhang@juniper.net

   Marco Liebsch
   NEC Lab
   Email: Marco.Liebsch@neclab.eu

   Tianji Jiang
   China Mobile
   Email: tianjijiang@chinamobile.com

Zhang, et al.             Expires 9 August 2024                 [Page 8]