Proxy Redirection Methods
So far, we have talked about general redirection methods. Content also may need to be accessed through various proxies (potentially for security reasons), or there might be a proxy cache in the network that a client should take advantage of (because it likely will be much faster to retrieve the cached content than it would be to go directly to the origin server).
But how do clients such as web browsers know to go to a proxy? There are three ways to determine this: by explicit browser configuration, by dynamic automatic configuration, and by transparent interception. We will discuss these three techniques in this section.
A proxy can, in turn, redirect client requests to a different proxy. For example, a proxy cache that does not have the content in its cache may choose to redirect the client to another cache. As this results in the response coming from a location different from the one from which the client requested the resource, we also will discuss several protocols used for peer proxy-cache redirection: the Internet Cache Protocol (ICP), the Cache Array Routing Protocol (CARP), and the Hyper Text Caching Protocol (HTCP).
Explicit Browser Configuration
Most browsers can be configured to contact a proxy server for content-there is a pull-down menu where the user can enter the proxy's name or IP address and port number. The browser then contacts the proxy for all requests. Rather than relying on users to correctly configure their browsers to use proxies, some service providers require users to download preconfigured browsers. These browsers know the address of the proxy to contact.
Explicit browser configuration has two main disadvantages:
· Browsers configured to use proxies do not contact the origin server even if the proxy is not responding. If the proxy is down or if the browser is incorrectly configured, the user experiences connectivity problems.
· It is difficult to make changes in network architecture and propagate those changes to all end users. If a service provider wants to add more proxies or take some out of service, browser users have to change their proxy settings.
Proxy Auto-configuration
Explicit configuration of browsers to contact specific proxies can restrict changes in network architecture, because it depends on users to intervene and reconfigure their browsers. An automatic configuration methodology that allows browsers to dynamically configure themselves to contact the correct proxy server solves this problem. Such a methodology exists; it is called the Proxy Auto-configuration (PAC) protocol. PAC was defined by Netscape and is supported by the Netscape Navigator and Microsoft Internet Explorer browsers.
The basic idea behind PAC is to have browsers retrieve a special file, called the PAC file, which specifies the proxy to contact for each URL. The browser must be configured to contact a specific server for the PAC file. The browser then fetches the PAC file every time it is restarted.
The PAC file is a JavaScript file, which must define the function:
function FindProxyForURL(url, host)
Browsers call this function for every requested URL, as follows:
return_value = FindProxyForURL(url_of_request, host_in_url);
where the return value is a string specifying where the browser should request this URL. The return value can be a list of the names of proxies to contact (for example, "PROXY proxy1.domain.com; PROXY proxy2.domain.com") or the string "DIRECT", which means that the browser should go directly to the origin server, bypassing any proxies.
The sequence of operations that illustrate the request for and response to a browser's request for the PAC file are illustrated in Screenshot 20-10. In this example, the server sends back a PAC file with a JavaScript program. The JavaScript program has a function called "FindProxyForURL" that tells the browser to contact the origin server directly if the host in the requested URL is in the "netscape.com" domain, and to go to "proxy1.joes-cache.com" for all other requests. The browser calls this function for each URL it requests and connects according to the results returned by the function.
(Screenshot 20-10.)
The PAC protocol is quite powerful: the JavaScript program can ask the browser to choose a proxy based on any of a number of parameters related to the hostname, such as the DNS address and subnet, and even the day of week or time of day. PAC allows browsers automatically to contact the right proxy with changes in network architecture, as long as the PAC file is updated at the server to reflect changes to the proxy locations. The main drawback with PAC is that the browser must be configured to know which server to fetch the PAC file from, so it is not a completely automatic configuration system. WPAD, discussed in the next section, addresses this problem.
PAC, like preconfigured browsers, is used by some major ISPs today.
Web Proxy Autodiscovery Protocol
The Web Proxy Autodiscovery Protocol (WPAD) aims to provide a way for web browsers to find and use nearby proxies, without requiring the end user to manually configure a proxy setting and without relying on transparent traffic interception. The general problem of defining a web proxy autodiscovery protocol is complicated by the existence of many discovery protocols to choose from and the differences in proxy-use configurations in different browsers.
This section contains an abbreviated and slightly reorganized version of the WPAD Internet draft. The draft currently is being developed as part of the Web Intermediaries Working Group of the IETF.
PAC file autodiscovery
WPAD enables HTTP clients to locate a PAC file and use the PAC file to discover the name of an appropriate proxy server. WPAD does not directly determine the name of the proxy server, because that would circumvent the additional capabilities provided by PAC files (load balancing, request routing to an array of servers, automated failover to backup proxy servers, and so on).
As shown in Screenshot 20-11, the WPAD protocol discovers a PAC file URL, also known as a configuration URL (CURL). The PAC file executes a JavaScript program that returns the address of an appropriate proxy server.
(Screenshot 20-11.)
An HTTP client that implements the WPAD protocol:
· Uses WPAD to find the PAC file CURL
· Fetches the PAC file (a.k.a. configuration file, or CFILE) corresponding to the CURL
· Executes the PAC file to determine the proxy server
· Sends HTTP requests to the proxy server returned by the PAC file
WPAD algorithm
WPAD uses a series of resource-discovery techniques to determine the proper PAC file CURL. Multiple discovery techniques are specified, because not all organizations can use all techniques. WPAD clients attempt each technique, one by one, until they succeed in obtaining a CURL.
The current WPAD specification defines the following techniques, in order:
· DHCP (Dynamic Host Discovery Protocol)
· SLP (Service Location Protocol)
· DNS well-known hostnames
· DNS SRV records
· DNS service URLs in TXT records
Of these five mechanisms, only the DHCP and DNS well-known hostname techniques are required for WPAD clients. We present more details in subsequent sections.
The WPAD client sends a series of resource-discovery requests, using the discovery mechanisms mentioned above, in order. Clients attempt only mechanisms that they support. Whenever a discovery attempt succeeds, the client uses the information obtained to construct a PAC CURL.
If a PAC file is retrieved successfully at that CURL, the process completes. If not, the client resumes where it left off in the predefined series of resource-discovery requests. If, after trying all discovery mechanisms, no PAC file is retrieved, the WPAD protocol fails and the client is configured to use no proxy server.
The client tries DHCP first, followed by SLP. If no PAC file is retrieved, the client moves on to the DNS-based mechanisms.
The client cycles through the DNS SRV, well-known hostnames, and DNS TXT record methods multiple times. Each time, the DNS query QNAME is made less and less specific. In this manner, the client can locate the most specific configuration information possible, but still can fall back on less specific information. Every DNS lookup has the QNAME prefixed with "wpad" to indicate the resource type being requested.
Consider a client with hostname johns-desktop.development.foo.com. This is the sequence of discovery attempts a complete WPAD client would perform:
· DHCP
· SLP
· DNS A lookup on "QNAME=wpad.development.foo.com"
· DNS SRV lookup on "QNAME=wpad.development.foo.com"
· DNS TXT lookup on "QNAME=wpad.development.foo.com"
· DNS A lookup on "QNAME=wpad.foo.com"
· DNS SRV lookup on "QNAME=wpad.foo.com"
· DNS TXT lookup on "QNAME=wpad.foo.com"
Refer to the WPAD specification to get detailed pseudocode that addresses the entire sequence of operations. The following sections discuss the two required mechanisms, DHCP and DNS A lookup. For more details about the reminder of the CURL discovery methods, refer to the WPAD specification.
CURL discovery using DHCP
For this mechanism to work, the CURLs must be stored on DHCP servers that WPAD clients can query. The WPAD client obtains the CURL by sending a DHCP query to a DHCP server. The CURL is contained in DHCP option code 252 (if the DHCP server is configured with this information). All WPAD client implementations are required to support DHCP. The DHCP protocol is detailed in RFC 2131. See RFC 2132 for a list of existing DHCP options.
If the WPAD client already has conducted DHCP queries during its initialization, the DHCP server might already have supplied that value. If the value is not available through a client OS API, the client sends a DHCPINFORM message to query the DHCP server to obtain the value.
The DHCP option code 252 for WPAD is of type STRING and is of arbitrary size. This string contains a URL that points to an appropriate PAC file. For example:
"http://server.domain/proxyconfig.pac"
DNS A record lookup
For this mechanism to work, the IP addresses of suitable proxy servers must be stored on DNS servers that the WPAD clients can query. The WPAD client obtains the CURL by sending an A record lookup to a DNS server. The result of a successful lookup contains an IP address for an appropriate proxy server.
WPAD client implementations are required to support this mechanism. This should be straightforward, as only basic DNS lookup of A records is required. See RFC 2219 for a description of using well-known DNS aliases for resource discovery. For WPAD, the specification uses "well known alias" of "wpad" for web proxy autodiscovery.
The client performs the following DNS lookup:
QNAME=wpad.TGTDOM., QCLASS=IN, QTYPE=A
A successful lookup contains an IP address from which the WPAD client constructs the CURL.
Retrieving the PAC file
Once a candidate CURL is created, the WPAD client usually makes a GET request to the CURL. When making requests, WPAD clients are required to send Accept headers with appropriate CFILE format information that they are capable of handling. For example:
Accept: application/x-ns-proxy-autoconfig
In addition, if the CURL results in a redirect, the clients are required to follow the redirect to its final destination.
When to execute WPAD
The web proxy autodiscovery process is required to occur at least as frequently as one of the following:
· Upon startup of the web client-WPAD is performed only for the start of the first instance. Subsequent instances inherit the settings.
· Whenever there is an indication from the networking stack that the IP address of the client host has changed.
A web client can use either option, depending on what makes sense in its environment. In addition, the client must attempt a discovery cycle upon expiration of a previously downloaded PAC file in accordance with HTTP expiration. It's important that the client obey the timeouts and rerun the WPAD process when the PAC file expires.
Optionally, the client also may implement rerunning the WPAD process on failure of the currently configured proxy if the PAC file does not provide an alternative.
Whenever the client decides to invalidate the current PAC file, it must rerun the entire WPAD protocol to ensure it discovers the currently correct CURL. Specifically, there is no provision in the protocol to do an If-Modified-Since conditional fetch of the PAC file.
A number of network round trips might be required during the WPAD protocol broadcast and/or multicast communications. The WPAD protocol should not be invoked at a more frequent rate than specified above (such as per-URL retrieval).
WPAD spoofing
The IE 5 implementation of WPAD enabled web clients to detect proxy settings automatically, without user intervention. The algorithm used by WPAD prepends the hostname "wpad" to the fully qualified domain name and progressively removes subdomains until it either finds a WPAD server answering the hostname or reaches the third-level domain. For instance, web clients in the domain a.b.microsoft.com would query wpad.a.b.microsoft, wpad.b.microsoft.com, then wpad.microsoft.com.
This exposed a security hole, because in international usage (and certain other configurations), the third-level domain may not be trusted. A malicious user could set up a WPAD server and serve proxy configuration commands of her choice. Subsequent versions of IE (5.01 and later) rectified the problem.
Timeouts
WPAD goes through multiple levels of discovery, and clients must make sure that each phase is time-bound. When possible, limiting each phase to 10 seconds is considered reasonable, but implementors may choose a different value that is more appropriate to their network properties. For example, a device implementation, operating over a wireless network, might use a much larger timeout to account for low bandwidth or high latency.
Administrator considerations
Administrators should configure at least one of the DHCP or DNS A record lookup methods in their environments, as those are the only two that all compatible clients are required to implement. Beyond that, configuring to support mechanisms earlier in the search order will improve client startup time.
One of the major motivations for this protocol structure was to support client location of nearby proxy servers. In many environments, there are several proxy servers (workgroup, corporate gateway, ISP, backbone).
There are a number of possible points at which "nearness" decisions can be made in the WPAD framework:
· DHCP servers for different subnets can return different answers. They also can base decisions on the client cipaddr field or the client identifier option.
· DNS servers can be configured to return different SRV/A/TXT resource records (RRs) for different domain suffixes (for example, QNAMEs wpad.marketing.bigcorp.com and wpad.development.bigcorp.com).
· The web server handling the CURL request can make decisions based on the User-Agent header, Accept header, client IP address/subnet/hostname, topological distribution of nearby proxy servers, etc. This can occur inside a CGI executable created to handle the CURL. As mentioned earlier, it even can be a proxy server handling the CURL requests and making these decisions.
· The PAC file may be expressive enough to select from a set of alternatives at runtime on the client. CARP is based on this premise for an array of caches. It is not inconceivable that the PAC file could compute some network distance or fitness metrics to a set of candidate proxy servers and then select the "closest" or "most responsive" server.