INTERNET-DRAFT                                                   T. Faber
Expires: February 10, 1998                                       J. Touch
draft-faber-time-wait-avoidance-00.txt                             W. Yue
                                                                 USC/ISI
                                                             August 1997



            Avoiding the TCP TIME_WAIT state at Busy Servers



Status of this Memo

   This document is an Internet-Draft.  Internet-Drafts are working doc-
   uments of the Internet Engineering Task Force (IETF), its areas, and
   its working groups.  Note that other groups may also distribute work-
   ing documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet- Drafts as reference mate-
   rial or to cite them other than as "work in progress."

   To view the entire list of current Internet-Drafts, please check the
   "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow
   Directories on ftp.is.co.za (Africa), ftp.nordu.net (Europe),
   munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or
   ftp.isi.edu (US West Coast).

Abstract

   This document describes the problems associated with the accumulation
   of TCP TIME_WAIT states at a network server, such as a web server,
   and details two methods for avoiding that accumulation.  Servers that
   have many TCP connections in TIME_WAIT state experience performance
   degradation, and can collapse.  One solution is a TCP modification
   that causes clients to enter TIME_WAIT state rather than servers.
   The other is an HTTP modification that allows the client to close the
   transport connection, maintaining the TIME_WAIT state at the client.
   The goal of both approaches is ensure that TIME_WAIT states accumu-
   late at the less loaded endpoint.

   The document also presents initial performance data from reference
   implementations of these solutions, which suggest that the modifica-
   tions improve HTTP connection rates at the server by as much as 50%,
   and allow servers to operate at small transaction throughputs that
   they cannot sustain their default configuration.



Faber, Touch, Yue                                               [Page 1]


INTERNET-DRAFT                                            September 1997


Introduction

   This draft describes the causes and effects of TIME_WAIT TCP protocol
   control block (TCB) accumulation at servers and proposes independent
   application and transport level modifications that remove that
   buildup.  We present experimental results showing a 50% improvement
   in HTTP connection rates, as measured by WebSTONE[1], as well as evi-
   dence that modified servers function at higher loads than unmodified
   servers can.

TIME_WAIT state and its effects

   TCP includes a mechanism to ensure that packets associated with one
   connection that are delayed in the network are not accepted by later
   connections between the same hosts[2].  The mechanism is implemented
   by the TIME_WAIT state of the TCP protocol.  When an endpoint closes
   a TCP connection, it keeps state about that connection, usually a
   copy of the TCB, for twice the maximum segment lifetime (MSL).  A
   connection in this state is in TIME_WAIT, and the endpoint holding
   the TIME_WAIT TCB rejects any packets addressed to the TIME_WAIT con-
   nection from the other endpoint.

   Keeping this TIME_WAIT TCB at either of the hosts prevents a new con-
   nection with the same combination of source address, source port,
   destination address, destination port from being created.  Either
   endpoint being in TIME_WAIT prevents data transfer on the connection,
   so protocol correctness is unaffected by which host holds the
   TIME_WAIT TCB.  Our modifications center on ensuring that the
   TIME_WAIT TCB is on the less loaded endpoint.

   Heavily loaded servers potentially keep thousands of TIME_WAIT TCBs,
   which consume memory and can slow active connections.  In BSD-based
   TCP implementations, TCBs are kept in mbufs, the memory allocation
   unit of the networking subsystem.  There are a finite number of mbufs
   available in the system, and mbufs consumed by TCBs cannot be used
   for other purposes, e.g., to move data.  Certain systems on high
   speed networks run out of mbufs due to TIME_WAIT buildup under high
   connection load.  A SPARCStation 20/71 under SunOS 4.1.3 on a 640
   Mb/sec Myrinet[3] cannot support more than 60 connections/sec.

   Incoming packets must be demultiplexed by finding the receiving con-
   nection in the host's TCB list.  This process can be slowed when the
   TCB list is full of TIME_WAIT TCBs.  In the simplest implementation,
   the TCB list is searched linearly to demultiplex the incoming packet
   to the appropriate connection, which can make TCB lookup a bottle-
   neck.  The additional search overhead can cut throughput between two
   SunOS 4.1.3 SPARCStations on a Myrinet in half.




Faber, Touch, Yue                                               [Page 2]


INTERNET-DRAFT                                            September 1997


Other Proposed Solutions

   There are other solutions to the increased lookup overhead problem,
   e.g., storing all TIME_WAIT TCBs at the end of the list and using
   them as a search terminator as BSDI's BSD/OS does[4], or hashing TCBs
   rather than keeping them in a list[5].  These solutions do not
   address the loss of memory due to accumulation of TIME_WAIT states,
   so servers may still be unable to serve a high client load.  These
   approaches improve system response until the server collapses due to
   lack of free mbufs; our approach of removing the TIME_WAIT state from
   the server eliminates this cause of server collapse.

   Allocating more memory to system mbufs or reducing the amount of data
   cached per connection allows servers to function under a higher load
   before collapsing.  The servers' performance will continue to
   degrade.  Moving TIME_WAIT to clients removes this cause of system
   degradation and collapse without changing resource allocations.

   The costs of accumulating TIME_WAIT TCBs have become more apparent as
   HTTP becomes more prevalent.  Under HTTP 1.1, servers terminate con-
   nections by closing the underlying TCP connection[6], which results
   in accumulation of TCBs at servers[7].

   HTTP 1.1 reduces the number of connections per transaction using per-
   sistent connections; however, with respect to TIME_WAIT buildup, the
   use of persistent connections[6] is similar to adding more memory to
   servers: servers can support a larger load before the effect becomes
   noticeable, but performance eventually degrades.  Servers supporting
   persistent connections can support more transactions per connection,
   and will benefit from our modifications by being able to support more
   connections.

Our Proposed Solutions

   Because the accumulation of TIME_WAIT TCBs is caused by the interac-
   tion between transport and application protocols, modifications can
   be made to either protocol to to alleviate it.  Changing the trans-
   port protocol confers the benefits to more applications, but there
   may be more resistance to changing a protocol on which many applica-
   tions depend.  Application level changes restrict the benefits (and
   drawbacks) to the application for which the solution is implemented.
   Furthermore, application solutions are not always possible; for exam-
   ple, protocols that use the closing of a transport connection to
   indicate end-of-file are not good candidates for removing TIME_WAIT
   TCBs at the application layer.

   This document proposes distinct extensions to TCP and to HTTP that
   allow hosts to control which end of the connection remains in



Faber, Touch, Yue                                               [Page 3]


INTERNET-DRAFT                                            September 1997


   TIME_WAIT state. A solution needs to be implemented at only one
   level, transport or application. We describe and measure both to have
   a basis for comparison.  Preliminary experiments indicate that both
   systems reduce the memory usage of web servers due to TIME_WAIT
   states to negligible levels, with accompanying performance improve-
   ments.  The TCP modifications require only client side changes, and
   can be deployed incrementally.  The HTTP changes affect client and
   server, but are compatible with HTTP 1.1 behavior, and can also be
   incrementally deployed.

   The remainder of this document is organized as follows: it presents
   the two proposed solutions, compares them, discusses the results of
   initial experiments with the solutions, and draws conclusions and
   outlines future work.

Transport Level (TCP) Solution

   The TCP solution exchanges the TIME_WAIT state between the server and
   client.  We modify the client's TCP implementation so that after it
   has completed a passive close of a transport connection, it sends an
   RST packet to the server and puts itself into a TIME_WAIT state.  The
   RST packet removes the TCB in TIME_WAIT state from the server; the
   explicit transition to a TIME_WAIT state in the client preserves cor-
   rect TCP behavior.  If the client RST is lost, both server and client
   remain in TIME_WAIT state, which also ensures correct behavior and is
   equivalent to a simultaneous close in the current protocol.  If
   either host reboots during the RST exchange, the behavior is the same
   as if a host running unmodified TCP fails with connections in
   TIME_WAIT state: packets will not be erroneously accepted if the host
   recovers and refuses connections until a 2 MSL period has elapsed[2].

   More formally, the change to the TCP state machine results in chang-
   ing the arc from LAST_ACK to CLOSED to an arc from LAST_ACK to
   TIME_WAIT and sending an RST when the arc is traversed.  These modi-
   fications need to be made only to clients.

   Hosts that act primarily as clients may be configured with the new
   behavior for all connections; clients that serve as both client and
   server, for example proxies, may be configured to support both behav-
   iors.  The implementation of both behaviors is straightforward,
   although it requires a more extensive modification of the TCP state
   machine.

   Allowing both behaviors on the same host requires splitting the
   LAST_ACK state into two states, one that represents the current
   behavior (LAST_ACK) and one which represents the modified behavior
   (LAST_ACK_SWAP).  These states may both be reported as LAST_ACK to
   monitoring tools. The state machine determines which state to enter



Faber, Touch, Yue                                               [Page 4]


INTERNET-DRAFT                                            September 1997


   from CLOSE_WAIT based on whether the application issues a close or a
   close_swap.

   The current passive close path is:


      server                            client
      -----------------------------------------------------------
      ESTABLISHED                       ESTABLISHED
      (get application close)
      goto FIN_WAIT_1
      send FIN           ---FIN--->
                                        goto CLOSE_WAIT
                        <---ACK---      send ACK
      goto FIN_WAIT_2
                                        (get application close)
                                        goto LAST_ACK

                        <---FIN---      send FIN
      goto TIME_WAIT
      send ACK           ---ACK--->
                                        goto CLOSED


   This solution adds this branch from CLOSE_WAIT on the client side:


      server                            client
      -----------------------------------------------------------
      ESTABLISHED                       ESTABLISHED
      (get application close)
      goto FIN_WAIT_1
      send FIN           ---FIN--->
                                        goto CLOSE_WAIT
                        <---ACK---      send ACK
      goto FIN_WAIT_2
                                        (get application close_swap)
                                        goto LAST_ACK_SWAP
                        <---FIN---      send FIN
      goto TIME_WAIT
      send ACK           ---ACK--->
                                        goto TIME_WAIT
                        <---RST---      send RST
      goto CLOSED







Faber, Touch, Yue                                               [Page 5]


INTERNET-DRAFT                                            September 1997


   Strictly speaking, the transition of the client to TIME_WAIT is
   extraneous, because any host sending an RST is obligated not to allow
   a connection between the same pair of addresses and ports for a time
   of at least 2 MSL.

   Distinguishing between close and close_swap does not require changing
   the application interface.  For example, a per-connection flag can be
   added to change the default behavior, where the default behavior is
   chosen based on whether the host is primarily a client or a server.
   Hosts that are primarily clients will follow the close_swap path
   unless overridden and servers the close path.

   Implementations of this system do not change the API at all if all
   connections from the same host have the same semantics; hosts which
   are primarily clients will see no change.  Only hosts that support
   both semantics will have a change to the API, and this will be an
   additional socket option or similar small change.

   The solution we propose is designed to interoperate with the existing
   TCP specification.  A cleaner implementation of our solution would be
   to change both endpoint implementations to negotiate which endpoint
   maintains the TIME_WAIT TCB.  However this would require changing all
   TCP implementations, which ours does not.

   A SunOS 4.1.3 patch is available from the authors.

Application Level Solution for HTTP

   Protocols that use the state of the transport connection as sig-
   nalling dictate which endpoint closes a connection, and therefore
   which incurs the cost of the TIME_WAIT TCB.  For example, early HTTP
   servers used the state of the transport connection as an implicit
   indicator of both transaction lifetime and request length.  The
   server closing the TCP connection indicated to the client that the
   whole response had arrived, and, because there were no persistent
   connections, that the HTTP exchange was over.  Because the server was
   using the close to mark the end of both the transaction and the
   exchange, it was required to initiate the close.

   HTTP 1.1 has sufficient framing to allow a modification to shifting
   TIME_WAIT TCBs to the clients[6].  Responses are self-delineating;
   all responses include the size of the response either in the headers
   or via the chunking mechanism.  When using persistent connections,
   which is the default behavior in HTTP/1.1, requests have fields which
   can be used to control the transport connection.  The server is no
   longer required by the protocol to close the transport connection.





Faber, Touch, Yue                                               [Page 6]


INTERNET-DRAFT                                            September 1997


   To control the distribution of TIME_WAIT TCBs from the application
   level, our HTTP modifications arrange that the client closes the TCP
   connection.  This requires the client to be able to detect the end of
   a response.  Under HTTP 1.1, this information is available to the
   client as a side effect of persistent connections.  We advocate a
   change in client behavior which requires them to close the transport
   connection underlying an HTTP connection, and an extension of the
   request format which allows the client to notify the server that it
   is breaking the TCP connection.

   We propose adding a CLIENT_CLOSE request to HTTP that indicates that
   a client is ending the HTTP exchange by closing the underlying TCP
   connection.  A CLIENT_CLOSE request requires no reply.  It terminates
   a series of requests on a persistent connection, and indicates to the
   server that the client has closed the TCP connection.  A client will
   initiate an active close on the TCP connection immediately after
   sending the CLIENT_CLOSE request to the server.

   A CLIENT_CLOSE request differs from including a "Connection: close"
   in the header of a request because a request that includes "connec-
   tion: close" still requires a reply from the server, and the server
   will (passively) close the connection[6].  A CLIENT_CLOSE request
   indicates that the client has severed the TCP connection, and that
   the server should close its end without replying.

   Incorporating the CLIENT_CLOSE into the transaction is a minor exten-
   sion to the HTTP protocol.  Current HTTP clients conduct an HTTP
   transaction by opening the TCP connection, making a series of
   requests with a "Connection: close" line in the final request header,
   and collecting the responses.  The server closes the connection after
   sending the final byte of the final request.  Modified clients open a
   connection to the server, make a series of requests, collect the
   responses, and send a CLIENT_CLOSE request to the server after the
   end of the last response.  The client closes the connection immedi-
   ately after sending the CLIENT_CLOSE.

   Modified clients are compatible with the HTTP 1.1 specification[6].
   A server that does not understand CLIENT_CLOSE will see a conven-
   tional HTTP exchange, followed by a request that it does not imple-
   ment, and a closed connection when it tries to send an error
   response.  A conformant server must be able to handle the client
   closing the TCP connection at any point.  The client has gotten its
   data, closed the connection and holds the TIME_WAIT TCB.

   Modifying servers to recognize CLIENT_CLOSE can make parts of their
   implementation easier.  Mogul et al. note that detecting closed con-
   nections can be difficult for servers[6].  CLIENT_CLOSE marks closing
   connections, which simplifies the server code that detects and closes



Faber, Touch, Yue                                               [Page 7]


INTERNET-DRAFT                                            September 1997


   connections that clients have intentionally closed.

   The CLIENT_CLOSE request has been implemented directly in the
   apache-1.24[8] server and test programs from the WebSTONE performance
   suite.  Patches are available from the authors.

Initial Implementation

   In this section we present experiments that demonstrate the problem
   and show our solutions effectiveness.  The proposed solutions have
   been implemented under SunOS 4.1.3 and initial evaluations of their
   performance have been made using both custom benchmark programs and
   the WebSTONE benchmark.  The tests were run on hosts connected to the
   640 Mb/sec Myrinet LAN.

   We performed two experiments. The first experiment shows that TCB
   load degrades server performance and that our modifications reduce
   that degradation.  The second illustrates that both our TCP and HTTP
   solutions improve server performance under the WebSTONE benchmark,
   which simulates typical HTTP traffic.  The last experiment shows that
   our modifications enable a server to support HTTP loads that it can-
   not in their default configurations.

   The first experiment was designed to determine if TCB load reduces
   server performance and if our modifications alleviate it.  This
   experiment used four Sparc 20/71's across the Myrinet using a user-
   level data transfer program over TCP.  The throughput is the average
   of each of two client hosts doing a simultaneous bulk transfer to the
   server host.  We vary the number of TIME_WAIT TCBs at the server by
   adding dummy TIME_WAIT states.

   The experiment was:

   1. Two client machines establish connections to the server

   2. The server is loaded with extraneous TIME-WAIT TCBs state by a
      fourth host.

   3. The two bulk transport connections transfer data.  (Throughput
      timing begins when the data transfer begins, not when the connec-
      tion is established.  TIME_WAIT TCBs may expire during the trans-
      fer.)

   4. Between runs, The server is allowed to idle and remove TCBs, to
      control conditions for all runs.

   Each result is the average of ten runs.




Faber, Touch, Yue                                               [Page 8]


INTERNET-DRAFT                                            September 1997


     Connections      Throughput (Mb/sec)     Throughput (Mb/sec)
     in TIME_WAIT     (Unmodified)            (with TCP Modification)
                       avg.        std.         avg.          std.
                                   dev.                       dev.
    -------------------------------------------------------------------
            0          66.8         3.3         66.8           3.3
          500          49.6         3.9         66.8           3.3
         1000          41.9         4.1         66.5           3.1
         1500          35.3         2.8         64.6           3.0
         2000          31.2         4.9         64.3           3.0
         2500          30.5         3.0         64.3           2.9

                 Table 1: Worst case throughput experiment

   The experimental procedure is designed to isolate a worst case at the
   server.  The client connections are established first to put them at
   the end of the list of TCBs in the server kernel, which will maximize
   the time needed to find them using SunOS's linear search.  Two
   clients are used to neutralize the simple caching behavior in the
   SunOS kernel, which consists of keeping a single pointer to the most
   recently accessed TCB.  Two distinct clients are used to allow for
   bursts from the two clients to interleave; two client programs on the
   same host send bursts in lock-step, which reduces the cost of the TCB
   list scans.

   The experiment shows that under worst case conditions, TCB load can
   reduce throughput by as much as 50%, and that our TCP modifications
   improve performance under those conditions.

   While it is useful that our modifications perform well in the worst
   case, it is important to asses the worth of the modifications under
   expected conditions.  The previous experiment constructed a worst
   case scenario; the following experiment uses WebSTONE to test our
   modifications under more typical HTTP load.

   WebSTONE is a standard benchmark used to measure web server perfor-
   mance in terms of connection rate and per connection throughput.  To
   measure server performance, several workstations make HTTP requests
   of a server and monitor the response time and throughput.  A central
   process collects and combines the information from the individual web
   clients. The benchmark has been augmented to measure the amount of
   memory consumed by TCBs on the server machine.  We used WebSTONE ver-
   sion 2 for these experiments.

   WebSTONE models a heavy load that simulates HTTP traffic.  Two hosts
   run multiple web clients which continuously request files ranging
   from 9KB to 5MB from the server.  Each host runs 20 web clients.




Faber, Touch, Yue                                               [Page 9]


INTERNET-DRAFT                                            September 1997


   Results from a typical run using clients are summarized:

    System                 Throughput     Connections     TCB Memory Use
    Type                   (Mb/sec)       per second      (Kbytes)
   -----------------------------------------------------------------------
    Unmodified               20.97           49.09            722.7
    TCP Modification         26.40           62.02             24.1
    HTTP Modifications       31.73           74.70             24.4

              Table 2: WebSTONE benchmark with large fileset

   Both modifications show marked improvements in throughput and connec-
   tion rate.  TCP modifications increase connection rate by 25% and
   HTTP modifications increase connection rate by 50%.  We believe the
   TCP modification is less effective than the HTTP modification because
   it adds another packet exchange.  Packet traces are being used to
   confirm this. [note: more will be included on this in later drafts]

   When more clients request smaller files, unmodified systems fail com-
   pletely because they run out of memory; systems using our modifica-
   tions can support much higher connection rates than unmodified sys-
   tems.  The following table reports data from a typical WebSTONE run
   using 8 clients on 4 hosts connecting to a dedicated server.  All
   clients request only 500 byte files.

    System                 Throughput     Connections     TCB Memory Use
    Type                   (Mb/sec)       per second      (Kbytes)
   -----------------------------------------------------------------------
    Unmodified               fails           fails            fails
    TCP Modification          1.14           223.8             16.1
    HTTP Modifications        1.14           222.4             16.1

               Table 3: WebSTONE benchmark with small files

   The experiments support the hypothesis that the proposed solutions
   reduce the memory load on servers.  The custom benchmark shows that
   the system with a modified transport performs much better in the
   worst case, and that server bandwidth loss can be considerable.  The
   WebSTONE benchmark shows that both systems reduce memory usage, and
   that this leads to performance gains.  Finally modified systems are
   able to handle workloads that unmodified systems cannot.

   This is a challenging test environment because the TCB load of the
   server host is spread across only two client hosts rather than the
   hundreds that would share the load in a real system.  The clients
   suffer some performance degradation due to the accumulating TCBs,
   much as the server does in the unmodified system.




Faber, Touch, Yue                                              [Page 10]


INTERNET-DRAFT                                            September 1997


Comparison of Methods

   The primary contrast between the TCP solution and the HTTP solution
   is that they are implemented at different levels of the protocol
   hierarchy.  The TCP solution has the benefits and drawbacks of a
   transport level solution: it applies the fix transparently to all
   application protocols running over TCP, but may be difficult to adopt
   for the same reason.  A change to TCP affects many applications, and
   many resist changes to TCP to avoid unintended consequences.  The
   HTTP solution has the trade-offs of an application modification: only
   HTTP will exhibit the new behavior, and the behavior of other appli-
   cations will be limited.  If another protocol causes a TIME_WAIT
   state buildup, an HTTP fix will not prevent it.

   The performance of our TCP modification will also be limited by how
   efficiently hosts process RST packets.  Hosts that incur a high over-
   head to handling RSTs, or delay processing them, will not perform as
   well.  This may be one reason that the TCP solution shows less
   improvement than the HTTP solution in the small file experiment
   above.  [note this will be expanded upon]

   The meaning of the RST packet is also changed by out TCP solution.
   An RST packet is intended to indicate an unusual condition or error
   in the connection.  We are proposing making it part of standard oper-
   ating procedure.  The change in semantics of the RST packet is a
   result of maintaining compatibility with current TCP.  Some browsers
   are currently using RST in unintended ways as well.

   Ideally, the two TCP endpoints would negotiate which would hold the
   TIME_WAIT TCB during connection establishment, but this would require
   changing the TCP packet format to allow room for that negotiation,
   and further changes to the state machine.  We believe such a system
   is the best solution to the TIME_WAIT TCB accumulation problem, but
   recognize that such a large change to TCP would be difficult to get
   adopted.

   Adopting the HTTP solution is effective if HTTP connections are the
   source of TIME_WAIT loading;  however, if another protocol begins
   loading servers with TIME_WAIT states, that protocol will have to be
   fixed as well.  Currently, we believe HTTP causes the bulk of
   TIME_WAIT loading, which is why we chose to implement our solution
   under HTTP; in the future other protocols may be the source.

   Not adopting a TCP fix means that future protocols should be designed
   to control TIME_WAIT loading, which will constrain their semantics.
   Specifically, application protocols will not be able to use the state
   of the transport connection as implicit signalling; application layer
   protocols will be constrained to include framing and connection



Faber, Touch, Yue                                              [Page 11]


INTERNET-DRAFT                                            September 1997


   control information or run the risk of TIME_WAIT loading servers.
   For example, streaming real-time transmission systems may make use of
   such implicit signalling.

   Some existing protocols, such as FTP[9], make use of implicit sig-
   nalling, and cannot be retrofitted with TIME_WAIT controls.  As these
   protocols are currently used, they do not appear to be major sources
   of TIME_WAIT loading.  They could become important to TIME_WAIT load
   if a protocol has a resurgence or is used in new ways, or if its
   smaller loading characteristics become significant after the HTTP
   load is reduced.  If that happens, a backward-compatible solution may
   not be possible.

   Both the TCP and the HTTP solutions are incrementally deployable and
   solve the problem at hand.  Which to deploy in the Internet depends
   on how the community weighs the changing the semantics of the exist-
   ing transport protocol versus restricting the semantics of future
   application protocols.

Conclusions

   This document has discussed the problem of server memory load due to
   terminated connections remaining in TIME_WAIT state.  Servers can
   become so memory poor at high connection rates that they are unable
   to transfer data at all.  Even if servers can continue to function,
   their performance can suffer.

   Two solutions to the memory load problem have been presented at the
   transport (TCP) level and the application (HTTP).  Both solutions
   allow a client to take on its share of the server memory load.  The
   transport level solution adds a new state and operation to the TCP
   state machine that explicitly moves the TIME_WAIT state from active
   close initiator to passive closer.  The application level solution
   adds an access method to HTTP that allows a client to notify a server
   that it is actively closing the connection, and maintaining the
   TIME_WAIT state.

   Both solutions will interoperate with existing systems, allowing for
   easy deployment.  Patches are available from the authors for both
   solutions: TCP modifications are available for SunOS 4.1.3 and HTTP
   modifications are available for apache-1.24.

   Although there are certainly other methods of dealing with TIME_WAIT
   state accumulation, the methods presented here have the benefits that
   they preserve current TCP behavior, are incrementally deployable, and
   are small simple changes to existing systems.  Most other solutions,
   such as ending connections with an RST or moving TIME_WAIT TCBs to
   other internal queues at the server, either break transport behavior,



Faber, Touch, Yue                                              [Page 12]


INTERNET-DRAFT                                            September 1997


   or do not address the memory load problem directly.

Security Considerations

   The practices advocated in this paper do not seem to affect the secu-
   rity of either the HTTP or TCP protocols.

   The increased use and change in semantics of RST packets may cause
   false alarms in systems that monitor them.

Authors' Addresses

        Ted Faber
        Joseph Touch
        Wei Yue
        University of Southern California/Information Sciences Institute
        4676 Admiralty Way
        Marina del Rey, CA 90292-6695
        USA
        Phone: +1 310 822 1511
        Fax:   +1 310 823 6714
        EMail: faber@isi.edu
               touch@isi.edu
               wyue@isi.edu


   This draft expires March 20, 1998.

References

1. Gene Trent and Mark Sake, "WebSTONE: The First Generation in HTTP
   Server Benchmarking," white paper, Silicon Graphics International
   (February 1995), available electronically at
   <http://www.sgi.com/Products/WebFORCE/WebStone/paper.html>.

2. Jon Postel, "Transmission Control Protocol," RFC-793/STD-7 (Septem-
   ber, 1981).

3. Myricom, Inc., Nannette J. Boden, Danny Cohen, Robert E. Felderman,
   Alan E Kulawik, Charles L. Seitz, Jakov N. Selovic, and Wen-King Su,
   "Myrinet: A Gigabit-per-second Local Area Network," IEEE Micro, pp.
   29-36, IEEE (February 1995).

4. Mike Karels and David Borman, Personal Communication (July 1997).

5. Paul E. McKenney and Ken F. Dove, "Efficient Demultiplexing of Incom-
   ing TCP Packets," Proceedings of SIGCOMM 1992, vol. 22, no. 4, pp.
   269-279, Baltimore, MD (August 17-20, 1992).



Faber, Touch, Yue                                              [Page 13]


INTERNET-DRAFT                                            September 1997


6. R. Fielding, J. Gettys, J. Mogul, H. Frystyk, and T. Berners-Lee,
   "Hypertext Transport Protocol - HTTP/1.1," RFC-2068 (January, 1997).

7. Robert G. Moskowitz, "Why in the World Is the Web So Slow," Network
   Computing, pp. 22-24 (March 15, 1996).

8. Roy T. Fielding and Gail Kaiser, "Collaborative Work: The Apache
   Server Project," IEEE Internet Computing, vol. 1, no. 4, pp. 88-90,
   IEEE (July/August 1997), available electronically at
   <http://www.computer.org/internet/ic1997/pdf/w4088.pdf>.

9. J. Postel and J. K. Reynolds, "File Transfer Protocol," RFC-959
   (October, 1985).






































Faber, Touch, Yue                                              [Page 14]