Part 9: How to analyse the HTTP request headers and HTTP response headers

When a visitor requests a page from a server over the internet, in most cases HTTP is used for the communication between the browser and the server.

HTTP stands for HyperText Transfer Protocol (HTTP). The HTTP specification was coordinated by the World Wide Web Consortium (W3C) and Internet Engineering Task Force (IETF). HTTP/1.1 is the most common version today, but some browsers and servers still use HTTP/1.0.

HTTP is a client/server protocol made up of requests and responses. When you requested this page for example, you sent over circa 40 HTTP requests and received a HTTP response for each in return.

A browser sends an HTTP request for a specific URL, and a server hosting that URL sends back an HTTP response. Like many Internet services, the protocol uses a simple, plaintext format. The types of requests are GET, POST, HEAD, PUT, DELETE, OPTIONS, and TRACE. A GET request includes a URL followed by headers. The HTTP response contains a status code, headers and a body.

When a visitor requests a page, a first HTTP request is sent to the server to get the HTML. The HTML of a SharePoint page consists of the page and its controls (menus, language controls, web parts, etc). Based on the contents of that HTML, additional requests are sent to the server to get additional resources like images, videos, JavaScript files and CSS files. If images need to be displayed or if additional Javascript files are necessary for client-side interaction or if styling requires additional css stylesheets, these resources will be downloaded from the server by executing additional HTTP requests.

With each request, the browser sends HTTP request headers and the server responds to each request by sending over an HTTP response with headers.

HTTP headers

Within the HTTP request header, a browser sends along information about itself, like wich type of browser is used, if it accepts zipped resources. If the browser has cookies from the site from previous requests, it will also send the cookies.

The following screenshots are taken from Firebug. You can see the details of the HTTP headers of a specific resource by clicking the + sign next to the resource. This will expand the section with additional information:

The expanded section is divided in tabs. The first tab displays the headers; in other tabs you can inspect the cookies and the cache.

The headers tab contains three sections:

1. Response headers
2. Request headers
3. Response headers from cache

The first screenshot is an example of HTTP request headers.

When a page is loaded for the first time, the HTML of the document and all resources need to be downloaded from the server.  If the resource is requested for the first time, the server will respond with a 200 OK status code and will send over the resource, together with a number of HTTP response headers with additional information about the resource. The following screenshot is an example of HTTP response headers:

The information in the HTTP response headers can differ from resource to resource and also differs from site to site.

Below you can find a list with a number of HTTP headers and a short description. Most of them are explained in more detail in further articles of this series. This list is far from complete, it just contains the most frequent ones.

Key	Description	Possible values(s)
Accept-Encoding	This request header indicates the types of data encodings that are accepted by the browser. The value often indicates whether the browser is able to decode compressed resources or not, and if so, which type of compression.This header is described in more detail in part 16.	gzip, deflate
Connection	This header indicates whether if the connection can stay open for a next request. Opening a connection costs precious time, so it is never a good idea to open and close a connection with each request. Since HTTP/1.1 the default value is Keep-Alive but HTTP/1.1 also comes with a newer value, pipelining.	Keep-Alive, pipelining
Cookie	If cookies are enabled on the browser, this header contains the cookies applicable for the host.
User-Agent	This request header mentions the type of the browser and its version. This can be handy for the server if different rendering need to be applied based on the requesting browser.	Mozilla/5.0 or Internet Explorer or Chrome
Host	This response header contains the URL of the server.
Expires	This header is issued by the server to indicate until when a resource can be stored in the cache. As long as this date is not expired, the resource will be used from cache. As you will see in further articles, the availability of other headers can take precedence over the Expires header forcing the browser to execute a conditional GET.This is described in more detail in part 11 and part 13.	Date/time value
Cache-Control	This response header is issued by the server to indicate how long a resource will remain valid.Cache-Control can have the following values: public: the resource can only be cached by clients private: the resource can only be cached on the server Cache-Control uses the max-age directive to specify how long a component can be cached. It defines the freshness window in seconds. If less than max-age seconds have passed since the component was requested, the browser will use the cached version, thus avoiding an additional HTTP request. You could specify both response headers, Expires and Cache-Control max-age. If both are present, the HTTP specification dictates that the max-age directive will override the Expires header. You can find more details in more detail in part 11.	public, max-age=86400orprivate, max-age=0
Last-Modified	The Last-Modified date is an HTTP response header that indicates when the requested resource was last modified. Read more about this in part 11 and part 12.	Date/time value
If-Modified-Since	This is a request header that is used to ask the server to validate the requested resource. The date in this header will be the Last-Modified date that was received with the previous request.If the resource was modified since this date, the server will respond with a 200 OK status code and send the resource in the body; if the resource was not modified since this date, the server will respond with a 300 Not Modified status code to indicate that the browser can use the resource from its cache.More on this in part 11.	Date/time value
ETag	Entity tags (ETags) are a mechanism that web servers and browsers use to validate cached components. ETags provide another way to evaluate and determine whether the component in the browser’s cache still matches the component or entity on the server.SharePoint has a specific way to calculate ETags. You can read more about this in part 13.	"11713e51210ce1:0"
If-None-Match	This is a request header that is used to ask the server to validate the requested resource. The value in this header will be the ETag that was received with the previous request.If the resource was modified since last request, the server will respond with a 200 OK status code and send the resource in the body; if the resource was not modified since this date, the server will respond with a 300 Not Modified status code to indicate that the browser can use the resource from its cache.More on this in part 11.	"11713e51210ce1:0"
Referer	This request header contains the URL of the page that contained the link the user is currently following to get to the current page.

Connection

This header indicates whether if the connection can stay open for a next request. Opening a connection costs precious time, so it is never a good idea to open and close a connection with each request. Since HTTP/1.1 the default value is Keep-Alive but HTTP/1.1 also comes with a newer value, pipelining.

Pipelining, defined in HTTP/1.1, allows for sending multiple requests over a single socket without waiting for a response. Pipelining has better performance than persistent connections. Unfortunately, pipelining is not supported in Internet Explorer 7 and earlier; and it’s turned off by default in Firefox through version 2. Until pipelining is more widely adopted, Keep-Alive is the way browsers and servers can more efficiently use socket connections for HTTP. This is even more important for HTTPS because establishing new secure socket connections is more time consuming.

SharePoint headers

Additionally, each technology adds its own headers. Most of these headers are pure informational and have no additional value as to ease hackers work.

In the screenshot of the HTTP response headers you can clearly see that the requested page is a SharePoint page.

Key	Description	Possible values(s)
Server	This response header indicates the type of server that is used to host the web application.	Microsoft-IIS/7.5
X-Powered-By	This response header indicates the web technology that is used to develop the web site.	ASP.NET
MicrosoftSharePointTeamServices	This response header indicates the version of SharePoint installed on the servers.	14.0.0.6029
ResourceTag	This response header contains a guid that references to a document. In fact, it is the ETag prefixed with RT:. It is of no use at all.	RT:{F20AB0ED-A8A7-4BEC-B281-22E283EFCD97}
SPRequestGuid	In general, you’ll find this response header on the first request where the page is requested. This GUID is the Log Correlation Id that you can use to go through the SharePoint log files to look for errors with that same GUID.	f525a4a5-cd1a-4f9d-8c35-cba0089e3e60
X-SharePointHealthScore	This response header returns the health score of the server. It is calculated based on a number of performance counters. It is also related with the throthling mechanism in SharePoint 2010. A value higher than zero is an indication for the load balancer to redirect requests to other web front-ends or to queue GET requests. You can read more about this header in a post of Michel Barneveld.	0

As most of these headers are informational, you  can also remove them. This technique is discussed in more detail in part 12. BUT be aware that it is not a good idea to remove these ones:

• SPRequesetGuid: this Guid is the correleation ID used in the SharePoint logs
• MicrosoftSharePointTeamServices: removing this header will prevent search crawls from working correctly (It wouldn’t be able to use sitedata.asmx to efficiently gather changes, and wouldn’t have access to security information)
• X-SharePointHealtScore