UNIX Unleashed, Internet Edition - 21 - Introducing HyperText Transfer Protocol (HTTP) by James Edwards Without a doubt, the explosive growth of the Internet has been driven by the enormous popularity of the World Wide Web (the Web). The Web's continuing success is attributable to the simplicity with which it provides users an ability to locate and retrieve dispersed information from within the Internet. A major part of this success is due to the effectiveness of the hypertext transfer protocol (HTTP). HTTP is an application protocol that provides a mechanism for moving information between Web servers and clients (known as browsers). To be clear, HTTP is not a communication protocol; HTTP is an application. HTTP functions much like the other standard UNIX-based applications, such as Telnet, ftp, and SMTP. Like these applications HTTP makes use of its own well-known port address and the services of underlying reliable communication protocols, such as TCP/IP. This chapter details how the HTTP application operates. This will be done through examination of the protocols message formats and return codes, but also through examples that offer a step-by-step examination of how the protocol functions. This chapter highlights a number of performance problems attributable to the operation of HTTP. I will attempt to provide insight as to the how and why of these problems; as well as to indicate how re-configuration of certain server parameters may help to alleviate some situations. What HTTP Does The Internet is a huge mass of information. The development of the Web was driven by a desire for a simple and effective method of searching through this wealth of information for particular ideas or interests. Within the Web, information is stored in such a way that it can be referenced through the use of a standard address format called a Uniform Resource Locator (URL). Each URL points to a data object such that it has an uniquely identifiable location within the Internet. In addition, through the use of a standard data representation format known as Hypertext Markup Language (HTML), it becomes possible to include URLs along side actual data. These URLs can then provide reference to other related information located either on the same or remote Web server. Users are then able to creating their own discreet and distinct paths through the Web, using URLs to help them maneuver. Users access the Web through a client application called a browser. This is a special program that can interpret both the format information, such as font size and colors, as well as the URLs that are embedded within the HTML documents. The HTTP application provides the final piece in the puzzle; it provides a simple and effective method of transferring identified data objects between the Web server and the client. NOTE: Until the advent of the Web, the traditional method for moving files around the Internet was the ftp application. Why wouldn't this program be effective for transfers within the Web? Well, the performance overhead involved in using the ftp program would be too great. This is because ftp requires the use of a separate control connection for the file download request. After the download request has been made, the server would initiate the setup of another connection over which the selected file could be downloaded. HTTP provides a streamlining of this process through the use of a single connection that is used for both the file request and the data transfer components. Later in this chapter we outline in some detail the defined HTTP message types and protocol header formats. By way of an introduction to that section, a useful first step will be to examine the logical operation of HTTP and its interaction with the other components parts found within the Web. Figure 21.1 outlines how the HTTP application protocol relates to both the Web server and client programs. As indicated, the browser has been designed to interpret format information contained within HTML pages. It is possible for an HTML page to also contain URLs as references to other pages located elsewhere on the Internet. These links are referred to as hyperlinks and are often colored or underlined within the HTML pages. Figure 21.1. Logical organization of the Web. The main function of the HTTP application is to request the pages referenced by these hyperlinks. This is accomplished through the following steps: 1. When a browser is pointed at a Web server, HTTP will initiate the setup of a TCP connection between the client and port 80 on the indicated server--being the well-known port number reserved for the HTTP server process. 2. After this TCP connection has been established, the selected HTML document will be transferred over this connection to be displayed by the client. 3. The server will automatically terminate the TCP connection immediately after the HTML document has been transferred to the client. NOTE: The TCP connection will be terminated even if the client wishes to transfer more HTML pages from the same server within the same session. Each new request requires that another TCP connection is established. This method of operation raises some grave performance concerns, which are covered later in this chapter. One noted strength of HTML is that additional references to content can be encoded within each document. As indicated earlier, these could be pointers to content that could be located either on the same Web server, or alternatively on a different Web server. In addition to hyperlinks, HTML pages may often contain other information objects--particularly graphics. As the browser encounter references for embedded information it will ask HTTP to request the download these files. HTTP will initiate a separate TCP connection for each file that needs to be downloaded. Some HTML pages may contain a number data object, and to help speed up the overall download process some browsers will allow multiple TCP connection to be initiated simultaneously. NOTE: The Netscape Navigator browser allows the user to specify any number of possible simultaneous tcp connections. It is possible to notice some slight performance improvements by incrementally increasing the maximum number of connections; however, any improvement gains appear to flatten out after four connections. This is caused by the fact that the browser sets a hardcoded maximum of four concurrent connections even though it is possible for the client to request more. Protocol Definition The HTTP application uses a version numbering scheme consisting of a major and a minor number. These numbers are arranged using the following format: <Major Version> . <Minor Version> NOTE: The intention is to highlight changes within the HTTP base messaging system with changes to the major version number. In contrast, a change in the minor version number indicates the addition of features and functionality to the protocol. Current HTTP implementations are based upon the design specifications that have been outlined within the Request for Comments (RFC) 1945. These implementations are classified as HTTP version 1.0 (represented as HTTP/1.1). RFC 1945 is known as an informational RFC, as such it does not represent a validated application standard. This has resulted in a number of HTTP implementations from different vendors exhibiting some degree of variation in available functionality. A first draft of an updated version of the HTTP, termed version 1.1. (or HTTP/1.1), has been completed. The details of this upgraded specification have been published within the RFC 2068. The indicated changes contained within RFC 2068, classify as minor upgrades of the HTTP application. However, of greater significance is that HTTP/1.1 is being developed for ratification as an Internet Engineering Task Force (IETF) Standard. The objective being to set clear guidelines for the development of a common HTTP implementations in addition to providing some areas of enhanced functionality. So far we have only provided an overview of the general operation of the HTTP application by describing, at a somewhat high level, how HTML pages are transferred within the Web. The following section will extend this discussion by investigating the syntax of the HTTP application through the use of a real-world example. In addition to the example, an examination of available protocol messages, header fields, and return codes is undertaken, with summary tables provided for additional reference. HTTP Example Operation A more detailed study of the operation of HTTP can be made through an examination of the output from a standard packet analyzer. Table 21.1 provides an example of this operation through the use of trace data taken from a Linux server running the tcpdump application. The tcpdump application operates by placing the Ethernet card in a promiscuous mode, such that it can see and record each packet on the network. HTTP session information can then be collected by pointing a Web browser at an HTML page maintained on a local Web server. In order to simplify examination of the recorded session, I have made some minor adjustments to the recorded tcpdump output. These modifications consist of rearranging the presented order of some recorded packets as well as removing some superfluous detail. Table 21.1. Packet trace data illustrating HTTP operation. The Web client initiates a connection to the Web server. 1 client.22248 > server.80 S 1427079:1427079(0) win 4096 2 server.80 > client.22248 S 32179213:32179213(0) ack 1427079 win 4096 3 client.22248 > server.80 ack 1 win 4096 The client sends an HTTP request message to the Web server--requesting an HTML page. 4 client.22248 > server.80 1:537(536) ack 1 5 server.80 > client.22248 ack 537 6 client.22248 > server.80 537:1073(536) 7 client.22248 > server.80 P 1073:1515(536) The Web server sends back the requested page and status information. 8 server.80 > client.22248 1:537(536) ack 1516 9 server.80 > client.22248 537:1073(536) ack 1516 10 server.80 > client.22248 1073:1609(536) ack 1516 11 client.22248 > server.80 ack 1609 12 server.80 > client.22248 1609:2145(536) ack 1516 13 server.80 > client.22248 2145:2681(536) ack 1516 14 server.80 > client.22248 2681:3217(536) ack 1516 The requested page contains an embedded graphic. To transport this a second TCP connection is established. 15 client.22249 > server.80 S 21200132: 21200132(0) win 4096 16 server.80 > client.22249 S 13420003: 13420003(0) ack 21200132 win 4096 17 client.22249 > server.80 ack 1 win 4096 The client passes a requested to download the graphic to the Web server. 18 client.22249 > server.80 1:537(536) ack 1 19 server.80 > client.22249 ack 537 20 client.22249 > server.80 537:1073(536) ack 537 The Web server responses sends the graphic to the client 21 server.80 > client.22249 1:537(536) ack 537 22 server.80 > client.22249 537:1073(536) ack 537 The server completes sending the graphic and closes the TCP connection. 23 server.80 > client.22249 F 1073:1395(322) ack 537 24 client.22249 > server.80 ack 1396 25 client.22249 > server.80 F 537:537(0) ack 1395 26 server.80 > client.22249 ack 538 The server completes sending the original HTML page and closes the first TCP connection. 27 server.80 > client.22248 F 3217:3438(221) ack 1516 28 client.22248 > server.80 ack 3439 29 client.22248 > server.80 F 1516:1516(0) ack 3439 30 server.80 > client.22248 ack 1517 This table provides an example of a Web browser requesting an information page from a Web server. Remember that HTTP functions as an application that operates above a reliable communication services provided by TCP/IP. To this end, the initial connectivity between client and server involves the setup of a TCP connection between the hosts. The establishment of a TCP connection is accomplished through the completion of something known as the three-way handshake. This process is illustrated within the example. The client sends a TCP packet to the server, requesting a new TCP connection by setting the SYN option flag and supplying an initial sequence number (ISN). The server responds, by sending an acknowledgment (ack) back to the client along with its own ISN for this connection, which it highlights by also setting the SYN flag. The client responds, with the final part of the three-way exchange, by acknowledging the server's response and ISN. Following the three-way handshake, the TCP connection is opened and ready for data transfer. This begins with the client sending an HTTP request message to retrieve an indicated HTML page (packets four through seven in the example). The server responds with an HTTP response message that contains the requested HTML page as the message body. This data is transferred over the TCP connection with the client sending acknowledgment packets back to the server as required. While this HTML page is being transferred, a reference to an embedded object is encountered. The HTTP application on the server will automatically transfer this object to the client over a separately established TCP connection. In the table, packets 15, 16, and 17 illustrate the establishment of another TCP connection, which again involves the outlined three-way handshake. The client now has two active TCP connections with the Web server. NOTE: In the example, the packets relating to each connection will be intermingled and not be neatly separated as the table illustrates. The session trace output was rearranged to clarify the overall operation of the Web session. After the server has completed the transfer of each data item, it automatically closes the corresponding TCP connection. This process involves the transfer of four additional packets. First, the server will send a TCP packet with the FIN option flag set, indicating that it wishes to close its end of the active connection. This packet is acknowledged by the client, which in turn closes its end of the connection by sending a similar packet to the server. The server sends an acknowledgment, and the connection is closed. The table illustrates how both separate TCP connections are independently terminated following the completion of data transfer. It should be noted that the establishment and termination of each TCP connection involves the exchange of a minimum of seven packets. This can represent a significant amount of protocol overhead. Messages, Headers, and Return Codes HTTP messaging is formatted using standard ASCII requests that are terminated by a carriage return and a line feed. The HTTP application makes use of two defined messages types: message requests and message responses. Requests are made from Web clients to Web servers, and are used to request either the retrieval of data objects (such as HTML pages) or to return information to the server (such as a completed electronic form). The Web server uses response messages to deliver requested data to the client. Each response contains a status line that indicates some detail about the client request. This might be an indication that an error occurred or simply that the request was successful. Both request and response messages can be accompanied by one or more message headers. These headers allow the client or server to pass additional information along with its message. Before investigating the use of available headers fields, we consider the operation of the standard message formats. HTTP Request Messages The Listing 21.1 provides an outline of the general format for making HTTP data requests. Listing 21.1. HTTP data request syntax. Request method headers <blank line> (Carriage Return /Line Feed) message body NOTE: Older versions of HTTP, such as version 0.9, only allow for what the standard refers to as `simple' request and response formats. What this means is that the HTTP request messages are not able to include any header information. The use and availability of headers is summarized in Table 21.3 below. Request Methods The general syntax for a request methods is as follows: <request method> <requested-URL> <HTTP-Version> HTTP version 1.0 defines three request methods: GET, POST, and HEAD. Table 21.2 summarizes the functions of each support method and outlines a specific example. Table 21.2. Request method syntax and examples. Request Description GET Used to retrieve object identified within the URI. The use of defined headers can make the retrieval conditional. Example: GET HTTP://www.dttus.com/home.html HTTP/1.0 Result: The Web server will return the identified HTML page to the client. POST Used to request that the destination Web server accept the enclosed message body; this is generally used for returning completed electronic forms or for posting electronic news or e-mail messages. Example: POST HTTP://www.dttus.com/survey/completed.HTML HTTP/1.0 From: jamedwards@dttus.com Result: message body placed here HEAD This method is identical to GET except that the Web server does not return an enclosed message body--only the relating header information. This method is often used to test validity, accessibility, or for any recent changes. Example: HEAD HTTP://www.dttus.com/home.html HTTP/1.0 Result: The Web server will return a result code to the client. NOTE: A Uniform Resource Identifier (URI) is a generic reference that HTTP uses to identify any resource. The resource could be identified through its location, by using a Uniform Resource Location (URL), or by a name, using a Uniform Resource Name (URN). Defined Header Values Header values are used to relay additional information about an HTTP message. A single HTTP message may have multiple headers associated with it. Generally, it is possible to separate headers into the following four distinct groups: Those that relate to message requests Those that relate to responses Those that relate to the message content Those that can be applied to both message requests and message responses The operation and use of message headers can best be seen through the following simple example: GET HTTP://www.dttus.com/home.html HTTP/1.0 - GET request If-Modified-Since: Sun, 16 Mar 1997 01:43:31 GMT - Conditional Header - CR/line feed In this example, a Web client has forwarded an HTTP request to a Web server asking to retrieve a specified HTML page. This accomplished using the GET request method. This HTTP request has been supplemented with a single header field. This header asks that a conditional test be performed, asking that the indicated HTML page only be returned if it has been modified since the indicated date. The following tables provide a summary of the defined header values for each of the four groups. Note that within each grouping a large number of the headers are only available within the pending HTTP version 1.1 specification (they have been included for completeness). General header values are applicable to both request and response messages, but are independent of the message body. The following table summarizes the available values providing a short description of the related function. Table 21.3. Defined general header values. Header Name Header Description HTTP/1.1 Only Cache-Control Provides standard control for caching algorithms X Connection Forces a close on a persistent connection X Date Specifies data and time field Pragma Specifies the use of a cache (HTTP/1.0 specific) Transfer-Encoding Specifies whether any transformation has been applied to the message X Upgrade Allows a client to signal a request to use an upgraded version of a protocol X Via Used with trace method to determine paths X Some available header values relate specifically to client browser request messages--either GET, POST, or HEAD methods (also applicable to the new request methods introduced within HTTP/1.1). Table 21.4 provides a summary of the headers applicable to request messages. Table 21.4. Defined request header values. Header Name Header Description HTTP/1.1 Only Accept Indicates data formats acceptable for responses X Accept-Charset Indicates what character sets are acceptable X Accept-Coding Indicates what encoding schemes are acceptable X Accept-Language Indicates what languages are acceptable X Authorization Contains user credentials for authentication From E-mail address of client Host Host name and port of the requested resource X If-Modified-Since Conditional GET request If-Match Conditional GET request X If-None-Match Conditional GET request X If-Range Conditional GET request X If-Unmodified-Since Conditional GET request X Max-Forwards Used with TRACE to limit loop testing ranges X Proxy-Authorization Credentials for next proxy in service chain only X Range GET on a range of bytes within message body X Referer [sic] Address of URL where object was obtained User-Agent Details user agent making the request Web server generated responses to client requests may be supplement to a number of optional header values. Table 21.5 provides a summary of those header values specifically relating to response messages. Table 21.5. Defined response header values. Header Name Header Description HTTP/1.1 Only Age Indication of the "freshness" of a cached entry X Location Allows redirection of a location Proxy-Authenticate Provides authentication challenge for browser Public Lists capabilities and supported methods of server X Retry-After Used with 503 status to indicate a duration X Server Indicates software product and version on server Vary Listing of the selected option in request message X Warning Arbitrary information relayed to user X WWW-Authenticate Used with 401 status, contains challenge X Message body headers define optional meta-information about the data object, or, if a data object is not present, about the resource identified within the request. Table 21.6 outlines the available header values. Table 21.6. Defined message body header values. Header Name Header Description HTTP/1.1 Only Allow Lists the set of supported methods with that object Content-Base The base for resolving any specified relative URI's X Content-Encoding Indicates what coding has occurred--use of zip files Content-Language Natural language of specified object X Content-Length Size of transferred message body Content-Location URL of provided message X Content-MD5 MD-5 integrity check X Content-Range Partial message body references X Content-Type Media type of message sent Etag Entity tag for object Xcomparisons Expires The stale date Last-Modified Date and time of last modification Response Messages The general syntax for a Web servers response message is as follows: Status Line headers <blank line> (CR/LF) message body The first line of the Web server response consists of something known as the status line. This is a general syntax for this information is: <HTTP-Version> <Status-Code> <Status Code Description> Table 21.7 provides a complete listing of the defined status codes and their corresponding descriptions. As with HTTP requests it is possible to include one or more headers within Web servers responses. Listing 21.2 outlines an example of how this might occur. Table 21.7. HTTP response message status line descriptions. Status Line Response Description HTTP/1.1 Only 1xx Informational X 100 Continue--interim server response, client should continue sending X 101 Switching Protocol--ability to switch between older and new HTTP versions X 2xx Success--action was received and understood 200 Okay--the request message was successful 201 Created--the POST request was successful 202 Accepted 204 No Content 205 Reset Content--reset client view that cause request to be sent X 206 Partial Content--server completed a part of the GET request X 3xx Redirection--further action required to complete request 301 Object moved permanently 302 Object moved temporarily 304 Object not modified X 305 Use proxy--the client request must be via the indicated proxy X 4xx Client error--the request cannot be fulfilled 400 Bad request 401 Unauthorized, authentication issue 403 Forbidden, request not allowed 404 Not found 405 Method is not allowed X 406 Request is not acceptable X 407 Proxy authentication required X 408 Request time-out X 409 Conflict X 410 Gone--and no forwarding address is known X 411 Length required X 412 Precondition failed X 413 Request entity too large X 414 Request-URI too large X 415 Unsupported media type X 5xx Server error--the server failed to fulfill a valid request 500 Internal server error 501 Not implemented 502 Bad gateway 503 The service is unavailable Listing 21.2. Using Web server response headers. workstation> telnet www.dttus.com 80 trying 207.134.34.23 Connected to 207.134.34.23 Escape character is [^ GET /pub/images/dttus/mapimage.gif request line entered request terminated by CR/LF HTTP/1.0 200 OK response starts with Status Line Date: Friday, 14-Feb-97 22:23:11 EST header details are here Content-type: image/gif Last-Modified: Thursday, 13-Mar-97 17:17:22 EST Content-length: 5568 headers terminated by CR/LF content is transferred here Connection closed by foreign host tcp connection is terminated after transfer is complete workstation> workstation> In the preceding example, the Telnet program is used to create a TCP connection to the remote Web servers HTTP application port--port 80. After a connection has been established, an HTTP GET request is made. Listing 21.2 outlines how the requested image is returned to the Web client along with a number of header values directly after the status line. Note the established connection is automatically terminated by the Web server after the requested image has been transferred. The previous table lists a summary of the possible status line return codes and their description values. This table includes return codes for both HTTP/1.0 as well as for the pending version upgrade--HTTP/1.1. The table provides an indication where a code is supported only within the later version--which has full backward-compatibility support. NOTE: HTTP implementations do not have to be able to understand all existing return codes. However, they should be aware of each major code group. For example, if a Web server returned a value 412, the client must only be aware that something was wrong with its request and be able to interpret the return code as such. Identifying and Overcoming Some Performance Problems This section will look at some real world implementations and uses of the HTTP application. In particular, we will focus on some recognized limitations of the application and provide, where possible, some alternative configurations to help ease recognized problems. There are there main areas within the operation of HTTP that have the potential to cause performance problems--connection establishment, connection termination, and communication protocol operation--we discuss some of the main issues of each. Connection Establishment--The Backlog Queue There is an upper-limit to the number of outstanding TCP connection requests a given application server process can handle. Outstanding TCP connection requests are placed upon a queue known as the backlog. This queue has a predefined limit to the number of requests it can handle at any one time. When a servers backlog queue reaches its defined limits, any new connection requests will be ignored until space on the queue becomes available. If you have ever seen the message, "Web Site Found. Waiting for ReplyÉ", chances are that you have fallen victim to the problem of a complete backlog queue. During this time your Web browser will be resending the unanswered connection request in the hope that space on the backlog queue becomes available after a few seconds wait. To understand why HTTP is so susceptible to this problem, it is necessary to understand why a server would have any outstanding TCP connection requests. A server process will queue a connection request for two reasons: Either it is waiting for the completion of the connection request hand-shake, or it is waiting for the server process to execute the accept() system call. In order to establish a TCP connection, the client and the server must complete the three-way handshake. This process is initiated by the client, which sends a TCP connection request packet with the SYN flag set. The server will respond by sending a TCP packet with both the SYN and ACK flags set. The server will then wait for the client to acknowledge receipt of this packet by sending a final TCP packet acknowledging the server's response. While the server is waiting for this final response from the client the outstanding connection request will be placed in the backlog queue. Once a TCP connection has been established, the server process will execute the accept() system call in order to remove the connection from the backlog queue. It is possible on a busy server that the execution of this system call may be delayed and the connection request remain on the queue for an extended period of time. Generally, the server process will make the accept() call promptly and this will not cause the backlog queue to fill up. However, the required completion of the three-way handshake can and does cause the queue to fill and outstanding requests to be dropped. Why is this a particular problem of HTTP within the Internet? Well, it is possible for a Web server to fill its backlog queue if it receives a large volume of connection requests from clients facing a particularly large round trip time. Figure 21.2 illustrates that the Web server receives a number of connection requests in a very short period of time. The server responds back to each client--placing the outstanding request upon the backlog queue. Figure 21.2. Filling up a Web servers backlog queue. As the figure illustrates, the backlog queue fills up and there is a period of time during which the Web server will ignore any new connection request. During this time the Web client will display the "Web Site Found. Waiting for ReplyÉ" message as it attempts to successfully complete a connection request. Default sizes of the backlog queue differ between UNIX flavors and implementations. As an example, BSD flavors of UNIX define the size of the backlog queue through the SOMAXCONN constant. Calculation of the size of the queue size is then derived using the following formula: backlog queue size = (SOMAXCONN * 3) / 2 + 1 By default, the SOMAXCONN constant is set to a value of five--providing for a maximum of eight outstanding TCP connections to be held on the backlog queue. Other UNIX flavors or TCP/IP implementations use other parameters and parameter values to limit the size of the backlog queues. For example, Solaris 2.4 uses a default value set to five and allows this to be incremented to 32; Solaris 2.5 uses a default of 32 and allows a maximum of 1024; SGI's IRIX implementation provides default queue size of eight connections; Linux release 1.2 defaults to 10 connections; and Microsoft Windows NT 4.0 provides for a backlog maximum size of six. For busy Web servers, system administrators should look to increasing the backlog size from the default values. Failure to do so can effectively limit the overall availability of their Web servers with the Internet. NOTE: filling up the backlog queue for a given process has been used in a number of well-publicized denial of service attacks within the Internet. An attacker would send a Web server TCP connection requests containing spoofed source IP addresses that were unreachable. The Web server would send out its SYN and ACK packet to the spoofed address--placing the request upon the backlog queue. As the spoofed address could not be reached, the server would never receive a response, eventually timing out the request after a 75 second period. In order to deny service to the Web server, all the attacker would need to do is to send enough of these messages (10 or so) every 60 to 70 seconds. The backlog queue would then always be full and no access would be possible. Connection Termination The operation of HTTP on a busy server can become a major drain on available system resources. The previous section outlined how a server could potentially fill its backlog queue and prevent the creation of new TCP connections until system resources can be reassigned. In this section, we examine how a Web server could run out of available system resources as a result of how it handles the termination of established TCP connections. The operation of the HTTP protocol causes the established TCP connections to be terminated immediately following the completed data transfer. The termination of the TCP connection is initiated by the Web server, which completes what the TCP protocol specification outline as an active close. This is done by sending the client a TCP packet with the FIN flag set. Upon receipt, the client will return an acknowledgment packet and then complete what the specification outlines as a passive close--involving sending the Web server a TCP packet with the FIN flag and awaiting a server acknowledgment. The TCP protocol specifications allow either the client or the server to perform the active close on any connection. However, the end that performs this operation must place the connection in a state known as TIME-WAIT. This is done in order to ensure that any straggler packets can be properly handled. During the TIME-WAIT duration the connection information--stored within a structure known as a Transaction Control Block (TCB)--must be saved. NOTE: The TIME-WAIT value is calculated to be equal to twice the maximum segment lifetime (which is why it is often referred to as the 2msl value). The segment lifetime is related to the time-to-live value of IP datagrams within the network. Waiting for the TIME-WAIT duration allows TCP to ensure connections are correctly closed--the host performing the active close needs to stick around long enough so that the other end can transmit its FINAL packet--even if some acknowledgments are lost. Normally, it will be the client end of an established TCP connection that performs the active close. The operation of the HTTP application reverses this--with the Web servers initiating the close. The net effect is that some server resources cannot be reallocated for the TIME-WAIT duration. UNIX servers allocate a fixed number of TCBs--with values typically configurable up to a maximum of 1024. It is possible on a busy Web server for all available TCBs to become temporarily used up resulting "Web Site Found. Waiting for ReplyÉ" message being displayed to clients. The netstat program can be used to determine the number of outstanding TCBs currently in the TIME-WAIT state. Communication Protocol Operation--TCP and Congestion Management Some of the major criticism leveled at HTTP relate to its inefficient use of the underlying communication protocols. The TCP protocol was designed as a windowing communication protocol. Such protocols allow the receiving station to enforce a degree of flow control by providing an indication of how much data they are able to currently accept from the sender. The objective being that the receiving station can indicate to the sender how quickly it is able to process the sent data. Importantly, the use of windowing can fail to consider the capability of the connecting network to transfer the amounts of data within the advertised window sizes. Consider a data exchange between two network hosts: The receiving station, will advertise to the sender an amount of data it is able to receive up to a maximum value. The sending station will transmit the requested amount of data, which the receive will place within its buffers. The receiving station will read data from these buffers, freeing up space for it to accept more of the senders data. The receiver will request the sender transmit a window of data sufficient to fill the available space within its receive buffers. Such an operation will work fine--assuming that the network between the two hosts is capable of transmitting the requested volumes of data. Consider the case if the two hosts were connected a routed wide area link. If the router or the link became congested, it may not be able to process all of the data that the receiving station is advertising it can accept. The network needs to be able to indicate to the sending station to slow down its transmission! The receiving station might be advertising it is able to receive a given amount of data based on the fact that its buffers are empty, but the network in between cannot. The slow start algorithm provides this functionality. It does this through the operation of something called the congestion window. This value is incremented based upon the number of acknowledgment packets returned from the receiving station. The sending station will transmit an amount of data, which is the minimum of either the congestion window or the window size advertised by the receiver. In such a way, the sending station can now limit its transmissions based upon both the receiving station as well as the networks capability to handle different amounts of data. This relates to HTTP because slow start will add additional round trip delays to the transfer of data over established TCP connections. This is due to the fact that the congestion window size under the slow start algorithm is initialized to one packet--meaning that a sender will only transmit a single packet then await an acknowledgment from the receiving station. Upon receipt of each acknowledgment, the congestion window size will double in size until it matches the receivers advertised window size. Figure 21.3 shows how this additional round trip delay will affect the transmission under HTTP. This figure contrasts the effects of the slow start algorithm against that of a long lived connection. As the diagram indicates, the long lived connection transfers the receivers maximum window size of data enabling a fast transmission time. In contrast, under the slow start algorithm, additional delays are introduced while the sender waits for the receiver's acknowledgment packets. Figure 21.3. Illustrating the effects of the slow start algorithm. Under the TCP protocol, efficient transfers require long lived connection times, which reduce the overall impacts of the slow start algorithm. This is contrary to the operation of the HTTP application, which uses a number of ephemeral TCP connections, each of which being dedicated to transferring only a small amount of data. Each of these connections will invoke the slow start algorithm and, as such, reduce the overall efficiency of data transfers under HTTP. Increasingly, there is a growing desire to make greater use of persistent or long lived TCP connections within the operation of HTTP. These ideas are further examined in the following section. NOTE: This chapter relates to the operation of HTTP and not TCP. For a full and complete analysis of TCP and the use configuration of window sizes and other operating parameters refer to TCP/IP Unleashed by SAMS Publishing. Providing Multiple Links within an HTML Page The previous section examined the operation of the HTTP application protocol. This demonstration outlined how a Web client would make use of a separately established TCP connection to download each data object specified within a single HTML page. As previously mentioned, the use of multiple TCP connections between the client and server will reduce the total elapse time required to transport all the data objects specified within the page. However, there is a price to be paid. Each TCP connection requires the additional use of Web server resources, and it is very possible for servers to have difficulty in keeping pace with these requirements. This section outlines some simple HTML page design principles that will help to combat these effects and allow for more efficient use of resources. Figure 21.4 and figure 21.5 provide a comparison of two different Web pages. Figure 21.4 makes use of a number of individual icons to outline its contents. Each icon provides a reference to a URL guiding the user through the site. This approach is in contrast to that contained in figure 21.5. The web page in figure 21.5 uses a single graphic with URLs embedded behind different parts of the graphic to provide a map of the site. Figure 21.4. Comparing map and icon HTML designs--Poor design. Figure 21.5. Comparing map and icon HTML designs--Good design. From the Web client's perspective, it is possible not even to notice a difference in download time between each of the pages. This is especially true if the browser has been configured to operate multiple simultaneous TCP connections. Even though the Web page on the right contains a number of individual graphic images, each of those images could be downloaded simultaneously. The main issues relate to the effective use of resources on the Web server. The map approach as indicated in Figure 21.5 indicates how this might be configured. The map would be a single graphic that would embed hyperlinks defined as URLs. This could then be downloaded over a single TCP connection as opposed to the four separate connections required within the icon approach. This allows the Web server to potentially support a larger number of uses for the same amount of available resources. Using a Cache to Reduce Downloads The outline specification of HTTP within the informational RFC1945 introduces the idea of a response chain providing a link between Web server and client. The RFC indicates that there may be a number of intermediate devices upon this response chain which will act as forwarders of any messages. It is possible for any intermediary devices to act as a cache--the benefits being both a greatly reduced response time and the preservation of available bandwidth within the Internet. Figure 21.6 illustrates how this might occur. It outlines that the communication path between the Web server and client is effectively shortened with the use of a cache on an intermediate server. The intermediate server is able to return requested web pages to the client if those pages exist within its cache. In order to ensure that up-to-date information is returned, the client will send a conditional GET request. Figure 21.6. Improving HTTP performance through the use of a cache. The conditional GET request is made through the use of message request headers available within version 1.0 of HTTP. Listing 21.3 provides an example of how such requests would be structured. Listing 21.3. Making a conditional GET request. workstation> telnet www.dttus.com 80 trying 207.134.34.23 Connected to 207.134.34.23 Escape character is [^ GET /pub/images/dttus/mapimage.gif request line entered If-Modified-Since:Thursday,13-Feb-97 22:23:01 EST conditional GET Header request terminated by CR/LF HTTP/1.0 304 Object Not Modified response starts with Status Line which indicates that the object was not modified. (Single Blank line ends server response headers). Connection closed by foreign host TCP connection is terminated. workstation> workstation> Listing 21.3 indicates that the client formats a conditional GET request using the If-Modified-Since request header value. This request asks the Web server only to send the image if it has changed since the indicated date. The Web server returns a response status line indicating that the image has not changed. The server closes the TCP connection and the Web client displays the image from within its cache. The use of caches and intermediary servers enable the HTTP application protocol to operate more efficiently. It should be noted that even with the use of a cache, a TCP connection between the server and the client would need to be established; however, the successful use of a cache greatly reduces the total required data transfer. Looking to the Future HTTP version 1.1 is classed as a minor upgrade of the existing application protocol implementation. RFC 1945 tells us that minor upgrades provide enhancements in key areas of functionality--without major changes in the operation of the underlying operation of the application. Two of the key areas of improvement HTTP 1.1 focus on providing the support of persistent TCP connections and more effective use of caching techniques. More importantly, HTTP version 1.1 is being developed as a standard to be ratified by the Internet Engineering Task Force (IETF). Previous implementation of HTTP never underwent standards acceptance. This resulted in a proliferation of applications that called themselves HTTP/1.0 without implementing all the recommendations outlined within the published RFC. Given the diversity within existing implementations, a protocol version change was determined the most effective way to enable a basis for a standard solution. In this section, we focus on some of the main recommended changes to the existing protocol implementations. Refer to the previous reference tables for HTTP/1.1 included message formats and return codes. Supporting Persistent TCP Connections The standard operation within HTTP version 1.1 is for the client to request a single TCP connection with the remote server and use this for all the required transfers within a sessions. This directly contrasts the existing requirement to setup a separate TCP connection for the transfer of any single object within a given HTTP session. This single change has the potential to save both Web server resources and available network bandwidth. In addition, the use of a persistent connection will provide for greater operational efficiencies found within existing implementations of TCP. New Request Methods Supported In addition to the request methods supported within HTTP/1.0, version 1.1 has added the following new methods outlined in Table 21.8. Table 21.8. HTTP version 1.1 Additionally Supported Request Methods. New Methods Description OPTIONS This is a newly defined request method that provides the HTTP client with the capability to query a remote Web server as to its ability to its communication and protocol support. New message body information is passed using this method. PUT Enables a Web client to deliver a new object to a Web server. This object will be identified using a URL specified within the method. The PUT method differs from the HTTP/1.0 POST method (which is still supported). The POST request provides a URL reference to the object that will be used to interpret the supplied message; in contrast, the PUT method provides a URL as a reference it its message body contents. DELETE This is used to remove a specified object--referenced using an enclosed URL. TRACE This method provides an effective means for troubleshooting connections and/or performance. The TRACE method provides an application layer loop-back of messages--the final recipient of the message sending a 200 (OK) response back to the initiating client. This response will include header information that details the route the request and the response has taken. HTTP version 1.1 provides for support of a number of additional header values. These newly supported headers and their main functionality additions are summarized in Table 21.9. Table 21.9. HTTP version 1.1 Additionally Supported Header Values. General Header Most important addition within Version HTTP/1.1 is the support for persistent connections. The new header value "connection" allows a single TCP connection to be utilized for all data transfers between client and server. Response Header Several additional header values offer improved communication control between browser and server. Providing the server a capability to signal its features and available functions to the browser. In addition, some authentication controls are provided including challenge/response controls. Request Header The major addition to the request headers is the provision of an increased number of tests for a conditional download. These new tests allow a greater control over the download of Web server data. In addition, HTTP/1.1 request headers also allow browsers to flag to servers a list of acceptable data formats they are willing to receive. Message Header The most exciting addition is the inclusion of the "content-range" header. This allows for the partial transfer of data objects reflected only changes that might have occurred. This provides a far more efficient mechanism for providing data updates from Web servers. In addition, a number of new headers have been added that allow the server to relay more information about the actual content, such as content encoding type, language, and message size. Summary There is little doubt as to the importance of HTTP within the Internet. The application provides a simple solution for dynamically moving files between Web clients and servers--without the overhead or user intervention that is associated with the more standard file transfer programs, such as NFS and FTP. HTTP has some major strengths, but also some significant weaknesses. A number of serious performance problems have been associated with HTTP's operation. At the head of this list would probably be the requirement of having to fire up a separate connection to transfer each page or data object--. The use of server- and client-based file caches gets us part of the way to streamlining communications. However, we still require setup and tear down of TCP connections, which uses limited resources and slows down overall performance. The answer is to make use of persistent connections. This would involve establishing a single TCP connection and maintaining it until all data transfer between the client and web server has been completed. In such a way it would be possible to take advantage of useful performance features found within TCP that would aid the download of multiple web pages. HTTP version 1.1 attempts to provide this functionality through the use of additional protocol header values. Finally, existing vendor interoperability issues will effectively be over with the expected standardization of HTTP/1.1 in the later part of 1997. It is hoped that with HTTP/1.1 ratified as an Internet Standard it will be possible to improve the integration between different vendor solutions.

©Copyright, Macmillan Computer Publishing. All rights reserved.