How to spy on your browser's HTTP requests and responses
Ask anAgile expert
Are you just getting started with agile? Scaling agile practices in your organization? Leading an agile transformation? Get answers to all your agile questions.
In this article, you will
- Learn how to spy on your Web browser's HTTP requests and responses
- Start to understand why your browser sends HTTP request headers and why web servers send HTTP response headers
What you'll need
- Google Chrome for its developer tools
- Alternatively, you can use other browsers if you can figure out how to use their own development tools
HTTP is no secret at all and it's super easy to spy on your browser
Does it ever seem like cryptic magic what your browser and remote servers do together? Lest you think that HTTP requests and responses are like some secretive handshake that only browser makers and website operators know and can see: they're not. Assuming you are using Google Chrome, do this:
- Open a new window or tab
- Hit the
F12key to pull up the Google Chrome developer tools, and click on the
- Load Google in the new window. You can study the guts of the HTTP request used to GET the Google.com home page.
Notice the request method and request headers section
As you can see, the request method is GET, and below that is a big bag of request headers. Let's highlight just the major headers for now. HTTP request headers will come into play when we start to study how to send data to the Digital.ai Agility, formerly VersionOne, REST API because we'll need to specify a few "custom headers."
HTTP headers provide more information to help the server respond properly
Your browser sends HTTP headers as part of the HTTP request to give the server additional information it needs to know in order to process the request. The most relevant ones, practically speaking, are the accept header and the cookie one. The accept header tells the server what kind of formats the browser wants to get the response in, and the cookie is actually a unique value that identifies me that my browser previously got from Google. My browser sends the cookie back to Google on subsequent requests so that Google can correlate separate requests with me.
Now, scroll the pane down to see the response headers section
Click the response tab
Do you remember in the last article the web's inventor Tim Berners/Lee's statement about the HTTP GET method when he first designed it? He said:
The response from the server was always an HTML page.
Now, let's examine an HTTP POST request
Assuming you still have another browser or tab open and the developer tools handy:
- Navigate to http://www.pastebin.com
- Enter something into the new paste form, scroll down and hit submit
POST request and response highlights
Here are few highlights from the request and response that you can see in your developer tools window:
Request URL:http://pastebin.com/post.php Request Method:POST
The values contained in the payload correspond to HTML form field values. The source code for the HTML form element looks like this:
If, like me, you typed Digital.ai Agility is my favorite agile management tool! into the new paste field (named paste_code in the HTML), then your payload should look similar to this:
//////WebKitFormBoundaryM0FtBA3eFtIeoTma Content/Disposition: form/data; name="post_key" KgvURkjA //////WebKitFormBoundaryM0FtBA3eFtIeoTma Content/Disposition: form/data; name="submit_hidden" submit_hidden //////WebKitFormBoundaryM0FtBA3eFtIeoTma Content/Disposition: form/data; name="paste_code" Digital.ai Agility is my favorite agile management tool!
Again, like we saw for the GET response from Google.com, the content/type header is text/xml, but the new location header specifies where the browser should automatically navigate to to GET the response document, since if you look at the response tab you'll see that it is empty. This is a good design pattern called the Post/Redirect/Get pattern, and we'll learn more about it in later articles.