MPT Network Layer Multipath Library
draft-lencse-tsvwg-mpt-08

lt;= N), the vector of the weights of the paths.

     (Note: We have N paths with indices (1, ... , N)

   OUTPUT: O[j] ( 1 <= j <= M ), the sending vector containing the
   indices of the paths; where M is the length of the sending cycle.

     lcm := Least Common Multiple for  (W[1], ... , W[N])

Lencse et al.         Expires December 14, 2021              [Page 17]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

     M := 0

     s[i] := 0, for all i (1 <= i <= N)

   (Note: s[i] will store the sum of the increments for path i, where
   the increment is lcm/W[i])

     WHILE TRUE DO

        z := min(s[1]+lcm/W[1], ... , s[N]+lcm/W[N])

        k := The smallest index i, for which z == s[i]+lcm/W[i]

        M := M+1

        s[k] := z

        O[M] := k

        IF s[i] == z for all i (1 <= i <= N) THEN RETURN

     DONE

   END

   A sample C code can be found in the Appendix.

5.2. Flow Based Mapping

   The aim of the flow based mapping is to be able to distinguish the
   packets belong to different network flows and map them to the path
   that was set for them. (E.g. WiFi is used for Torrent traffic and
   LTE is used for VoIP calls.)

   Our current implementation realizes a port-based flow mapping. It is
   possible to select the interface for the outgoing traffic based on
   transport protocol and port. For communication between two MPT
   servers, you can precisely specify which flow mapped to which path.

   The configuration of the mechanism is simple. Four new values can be
   added for the definition of paths:

   tcp_dst - TCP destination port matches

   tcp_src - TCP source port matches

Lencse et al.         Expires December 14, 2021              [Page 18]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

   udp_dst - UDP destination port matches

   udp_src - UDP source port matches

   All of these are optional, they can be listed in many ports. Ports
   that are not defined will continue to be per-packet based. The
   current implementation of the MPT with flow based mapping can be
   found on [MptFlow]

   In the example below, each outgoing TCP packet with destination port
   80, 443, and 8080 and UDP packet with destination port 5901 will be
   sent on path_1. TCP packets with source ports 7880 and 56000 will be
   sent on path_2.

   Example (flow based mapping configuration snippet)

   [path_1]
   ...
   tcp_dst         = 80 443 8080
   udp_dst         = 5901

   [path_2]

   ...

   tcp_src         = 7880 56000

5.3. Combined Mapping

   TBD

6. Packet Reordering

   As the delay of the different paths can be different, packet
   reordering may appear in a packet sequence transmission. The MPT
   environment offers an optional feature to ensure the right ordered
   packet transmission for the tunnel communication. If this feature is
   enabled, the receiver uses a buffer-array to store the incoming
   (unordered) packets. Then the packets are sorted according to the
   GRE sequence numbers, so ensuring the ordered transmission to the
   receiver's tunnel interface.

Lencse et al.         Expires December 14, 2021              [Page 19]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

   There are two parameters aimed to control the reordering. The
   reorder window parameter specifies the length of the buffer array
   used for reordering. The maximum buffer delay parameter specifies
   the maximum time (in milliseconds) while the packet is stored in the
   buffer-array. If the packet is delayed in the buffer-array for the
   specified time, it will be transmitted to the tunnel interface, even
   in the case, when some packets are missing before the considered
   packet. The missing packets are considered as lost packets (i.e. we
   will not wait more for a lost packet). The arrived packets are
   transferred to the tunnel interface according to their GRE sequence
   number, so the ordered delivery will be kept also in the case of
   packet loss.

   How to set the values of these parameters?

   As for maximum buffer delay, if its value is too small, then MPT may
   incorrectly consider a sequence number as lost, and if it arrives
   later, MPT has to drop it to keep on the order-right delivery.  If
   its value is too large, then the packet loss will be determined too
   late, and thus the communication performance may decrease. Our
   experience shows that a feasible choice could be: a few times the
   RTT (Round-Trip Time) of the slowest path.

   As for reorder window, it MUST be large enough to store packets
   arriving at maximum line rate from all the active paths of the given
   connection during a maximum buffer delay interval.

   The appropriate choice of these parameters is still subject of
   research.

7. Why MPT is Considered Experimental?

   We view MPT as a research area rather than a solution which is ready
   for deployment. We have an MPT implementation, which is workable,
   but it contains only the "per packet based" mapping of the tunnel
   traffic to the paths. One of our aims of writing this Internet Draft
   is to enable others to write MPT implementations. It is our hope
   that the experience gained with preparing other implementations as
   well as the results of their testing and performance analysis will
   lead to a better MPT specification, which may then serve as a
   standard track specification of an improved MPT, which will be ready
   for deployment.

   In this section, we summarize the most important results as well as
   the open questions of the MPT related research.

Lencse et al.         Expires December 14, 2021              [Page 20]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

7.1. Published Results

7.1.1. MPT Concept and First Implementation

   The conceptual architecture of MPT, comparison with other multipath
   solutions, some details of the first implementation and some test
   results are available in [Alm2017].

   The user manual of the first MPT implementation and the precompiled
   MPT libraries for Linux (both i386 and amd64) and Raspbian are
   available from [Mpt2017].

7.1.2. Estimation of the Channel Aggregation Capabilities

   The channel aggregation capabilities of an early MPT implementation,
   which did not use GRE-in-UDP, were analyzed up to twelve 100Mbps
   links in [Len2015].

   Some of the above tests were repeated with the current GRE-in-UDP
   based MPT implementation, and the path aggregation capabilities of
   MPT were compared to that of MPTCP in [Kov2016] and [Szi2018].

   Measurements were performed also using two 1Gbps links [Szi2018b]
   and four 1Gpbs links [Szi2019].

   The performance of MPT and MPTCP was compared using two 10Gbps links
   [Szi2019b].

   To test MPT in a more realistic environment than the previous simple
   laboratory testbeds, an emulated WAN environment was used for
   throughput aggregation of two 100Mbps links [Szi2021].

7.1.3. Demonstrating the Resilience of an MPT Connection

   The resilience property of the early MPT implementation, which did
   not use GRE-in-UDP, was demonstrated in [Alm2014] and in [Alm2015].

   The fast connection recovery of the GRE-in-UDP based MPT
   implementation was demonstrated in [Fej2016].

   Playout buffer length triggered path switching algorithm was
   developed for the GRE-in-UDP based MPT, and its effectiveness was
   demonstrated by the elimination of the stalling events on YouTube
   video playback [Fej2017].

Lencse et al.         Expires December 14, 2021              [Page 21]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

7.1.4. Examining the Effect of Different Congestion Control Algorithms

   The throughput of MPT and MPTCP using various TCP congestion control
   algorithms (TCP Cubic, Highspeed, Illinois, Reno, Scalable, Vegas,
   Veno) were compared in a test environment containing four 100Mbps
   communication channels in [Szi2020].

7.2. Open questions

7.2.1. Parameters

   The optimal (or good enough) choice of the reorder window size and
   maximum buffer delay parameters are important questions, which
   should be solved before MPT can be deployed.

7.2.2. Development of Further Mapping Algorithms

   The current MPT implementation [Mpt2017] includes only the per
   packet base mapping. For a precise specification of the further two
   mapping algorithms, we would like to use or experiences with them.
   There are some open questions e.g. how to handle the traffic that is
   neither TCP nor UDP?

7.2.3. Performance Issues

   The current MPT implementation [Mpt2017] works in user space. Thus,
   it is not surprising, that multipath transmission of the same amount
   of traffic by MPT results in higher CPU load than its multipath
   transmission by MPTCP [Kov2019]. How much CPU power could a kernel
   space MPT implementation save?

   It was also pointed out by [Kov2019], that MPT is not able to
   utilize the computing power of more than two CPU cores. It is so,
   because MPT uses only two threads (one for each direction). This is
   not a serious issue, when MPT is used on personal computers.
   However, when MPT is used to connect several networks, it is an
   important question, how MPT could utilize the computing power of the
   modern CPUs with several cores.

7.3. Implementation

   A sample implementation of the MPT software is available from
   [MptSrc] under GPLv3 license. It is intended for research and
   experimentation purposes only, as it has not been sufficiently
   tested to be used for commercial purposes.

Lencse et al.         Expires December 14, 2021              [Page 22]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

8. Security Considerations

   Threats that apply to GRE-in-UDP tunneling, apply here, too. For the
   security considerations of GRE-in-UDP, please refer to Section 11 of
   [RFC8086].

   If an MPT server is configured so, its peer is allowed to build up
   connections. It may lead to resource exhaustion and thus successful
   DoS (Denial of Service) attacks.

   Authentication between MPT servers is optional, which may lead to
   security issues.

9. IANA Considerations

   Port numbers may be reserved for local command port and remote
   command port.

10. Conclusions

   Hereby we publish the specifications of the MPT network layer
   multipath library in the hope, that it can be made better by the
   review and comments of the WG members and, after answering several
   open questions, one day MPT can mature to be a production tool. We
   seek for interested volunteers for a different implementation and we
   would be happy to take part in research cooperation. We welcome all
   kinds of feedback from anyone to make MPT better.

11. References

11.1. Normative References

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC8086] Young, L. (Editor), Crabbe, E., Xu, X., and T. Herbert,
             "GRE-in-UDP Encapsulation", RFC 8086, DOI:
             10.17487/RFC8086, March 2017.

11.2. Informative References

   [Alm2014] Almasi, B., "A solution for changing the communication
             interfaces between WiFi and 3G without packet loss", in
             Proc. 37th Int. Conf. on Telecommunications and Signal
             Processing (TSP 2014), Berlin, Germany, Jul. 1-3, 2014,
             pp. 73-77

Lencse et al.         Expires December 14, 2021              [Page 23]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

   [Alm2015] Almasi, B., Kosa, M., Fejes, F., Katona, R., and L. Pusok,
             "MPT: a solution for eliminating the effect of network
             breakdowns in case of HD video stream transmission", in:
             Proc. 6th IEEE Conf. on Cognitive Infocommunications
             (CogInfoCom 2015), Gyor, Hungary, 2015, pp. 121-126, doi:
             10.1109/CogInfoCom.2015.7390576 .

   [Alm2017] Almasi, B., Lencse, G., and Sz. Szilagyi, "Investigating
             the Multipath Extension of the GRE in UDP Technology",
             Computer Communications (Elsevier), vol. 103, no. 1,
             (2017.) pp. 29-38, DOI: 10.1016/j.comcom.2017.02.002

   [Fej2016] Fejes, F., Katona, R., and L. Pusok, "Multipath strategies
             and solutions in multihomed mobile environments", in:
             Proc. 7th IEEE Conf. on Cognitive Infocommunications
             (CogInfoCom 2016), Wroclaw, Poland, 2016, pp. 79-84, doi:
             10.1109/CogInfoCom.2016.7804529

   [Fej2017] Fejes, F., Racz, S., and G. Szabo, "Application agnostic
             QoE triggered multipath switching for Android devices",
             In: Proc. 2017 IEEE International Conference on
             Communications (IEEE ICC 2017), Paris, France, 21-25 May
             21-25, 2017. pp. 1585-1591.

   [Kov2016] Kovacs, A., "Comparing the aggregation capability of the
             MPT communications library and multipath TCP", in: Proc.
             7th IEEE Conf. on Cognitive Infocommunications (CogInfoCom
             2016), Wroclaw, Poland, 2016, pp. 157-162, doi:
             10.1109/CogInfoCom.2016.7804542

   [Kov2019] Kovacs, A., "Evaluation of the Aggregation Capability of
             the MPT Communications Library and Multipath TCP", Acta
             Polytechnica Hungarica, vol. 16, no. 6, 2019, pp. 129-147.
             DOI: 10.12700/APH.16.6.2019.6.9

   [Len2015] Lencse, G. and A. Kovacs, "Advanced Measurements of the
             Aggregation Capability of the MPT Multipath Communication
             Library", International Journal of Advances in
             Telecommunications, Electrotechnics, Signals and Systems,
             vol. 4. no. 2. (2015.) pp 41-48. DOI:
             10.11601/ijates.v4i2.112

   [Szi2018] Szilagyi, Sz., Fejes, F. and R. Katona, "Throughput
             Performance Comparison of MPT-GRE and MPTCP in the Fast
             Ethernet IPv4/IPv6 Environment", Journal of
             Telecommunications and Information Technology, vol. 3. no.
             2. (2018.) pp 53-59. DOI: 10.26636/jtit.2018.122817

Lencse et al.         Expires December 14, 2021              [Page 24]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

   [Szi2018b] Szilagyi, Sz., Bordan, I., Harangi, L. and B. Kiss,
             "MPT-GRE: A Novel Multipath Communication Technology for
             the Cloud", in: Proc. 9th IEEE Conf. on Cognitive
             Infocommunications (CogInfoCom 2018), Budapest, Hungary,
             2018, pp. 81-86, doi: 10.1109/CogInfoCom.2018.8639941

   [Szi2019] Szilagyi, Sz., Bordan, I., Harangi, L. and B. Kiss,
             "Throughput Performance Comparison of MPT-GRE and MPTCP in
             the Gigabit Ethernet IPv4/IPv6 Environment", Journal of
             Electrical and Electronics Engineering, vol. 12. no. 1.
             (2019.), pp. 57-60. ISSN: 1844-6035

   [Szi2019b] Szilagyi, Sz., Bordan, I., Harangi, L. and B. Kiss,
             "Throughput Performance Analysis of the Multipath
             Communication Technologies for the Cloud", Journal of
             Electrical and Electronics Engineering, Vol. 12, No. 2,
             (2019.), pp. 69-72, ISSN: 1844-6035

   [Szi2020] Szilagyi, Sz., Bordan, I., "The Effects of Different
             Congestion Control Algorithms over Multipath Fast Ethernet
             IPv4/IPv6 Environments", In: Proceedings of the 11th
             International Conference on Applied Informatics (ICAI
             2020), Eger, Hungary, 29-31 January, 2020. pp. 341-349.
             http://ceur-ws.org/Vol-2650/paper35.pdf

   [Szi2021] Szilagyi, Sz., Bordan, I., "Throughput Performance
             Measurement of the MPT-GRE Multipath Technology in
             Emulated WAN Environment", In: Proceedings of the 1st
             Conference on Information Technology and Data Science
             (CITDS 2020), Debrecen, Hungary, 6-8 November, 2020. pp.
             187-195.
             http://ceur-ws.org/Vol-2874/short17.pdf

   [Mpt2017] MPT - Multipath Communication Library,
             https://irh.inf.unideb.hu/~szilagyi/index.php/en/mpt/

   [MptFlow] "MPT - Multi Path Tunnel", source code version with flow
             based packet to path mapping feature,
             https://github.com/spyff/mpt/tree/flow_mapping

   [MptSrc]  "MPT - Multi Path Tunnel", source code,
             https://github.com/spyff/mpt

   [RFC6824] Ford, A., Raiciu, C, Handley, M., and O Bonaventure, "TCP
             Extensions for Multipath Operation with Multiple
             Addresses", RFC 6824, DOI: 10.17487/RFC6824, January,
             2013.

Lencse et al.         Expires December 14, 2021              [Page 25]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

   [RFC8157] Leymann, N., Heidemann, C., Zhang, M., Sarikaya, B, and M.
             Cullen, "Huawei's GRE Tunnel Bonding Protocol", RFC 8157,
             DOI: 10.17487/RFC8157, May, 2017

12. Acknowledgments

   The MPT Network Layer Multipath Library was invented by Bela Almasi,
   the organizer and original leader of the MPT development team.

   This document was prepared using 2-Word-v2.0.template.dot.

Lencse et al.         Expires December 14, 2021              [Page 26]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

Appendix A. Sample C code for calculating the packet sending order

   <CODE BEGINS>
   void calculate_pathselection(connection_type *con) {
       long long lcm;
       long gcd, min_inc, cinc;
       int i,j, min_idx;
       path_type *p;

       con->pathselectionlength = 0;
       gcd = con->mpath[0].weight_out;
       lcm = gcd;
       for (i = 0; i < M; i++)
         O[i] = NULL;

       for (i = 0; i < con->path_count; i++) {
           gcd = CALCULATE_GCD(gcd, con->mpath[i].weight_out);
           lcm = (lcm * con->mpath[i].weight_out ) / gcd;
           con->mpath[i].selection_increment = 0;
       }

       for (j = 0; j < M; j++) {
           min_idx = 0;
           min_inc = lcm + 1;
           for (i = 0; i < con->path_count; i++) {
               p = &con->mpath[i];
               cinc = p->selection_increment + (lcm / p->weight_out);
               if ((p->weight_out) && (cinc < min_inc)) {
                   min_idx = i;
                   min_inc = cinc;
               }
           }
           O[j] = &con->mpath[min_idx];
           con->mpath[min_idx].selection_increment = min_inc;

           for (i = 0; i < con->path_count; i++) // check if ready
               if (con->mpath[i].selection_increment != min_inc)
                   goto NEXT_SELECTION;
           break;

   NEXT_SELECTION:
           continue;
       }
       con->path_index = 0;
       con->pathselectionlength = j + 1;
   }
   <CODE ENDS>

Lencse et al.         Expires December 14, 2021              [Page 27]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

   Copyright (c) 2021 IETF Trust and the persons identified as authors
   of the code. All rights reserved.

   Redistribution and use in source and binary forms, with or without
   modification, is permitted pursuant to, and subject to the license
   terms contained in, the Simplified BSD License set forth in Section
   4.c of the IETF Trust's Legal Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info).

Lencse et al.         Expires December 14, 2021              [Page 28]
Internet-Draft   MPT Network Layer Multipath Library         June 2021

Authors' Addresses

   Gabor Lencse
   Szechenyi Istvan University
   Egyetem ter 1
   H-9026 Gyor
   Hungary

   Phone: +36 1 613 665
   Email: lencse@sze.hu

   Szabolcs Szilagyi
   University of Debrecen
   Egyetem ter 1.
   H-4032 Debrecen
   Hungary

   Phone: +36 52 512 900 / 75013
   Email: szilagyi.szabolcs@inf.unideb.hu

   Ferenc Fejes
   Eotvos Lorand University
   Egyetem ter 1-3.
   H-1053 Budapest
   Hungary

   Phone: +36 70 545 48 07
   Email: fejes@inf.elte.hu

   Marius Georgescu
   RCS&RDS
   Strada Dr. Nicolae D. Staicovici 71-75
   Bucharest 030167
   Romania

   Phone: +40 31 005 0979
   Email: marius.georgescu@rcs-rds.ro

Lencse et al.         Expires December 14, 2021              [Page 29]