HTTP (HyperText Transfer Protocol) is the protocol of the world-wide-web. When you visit a webpage with your web browser, the browser is making a series of HTTP requests to web servers somewhere out on the Internet. Those servers will answer with HTTP responses.
The Python Requests library makes it easy to write programs that send and receive HTTP. Instead of having to understand the HTTP protocol in great detail, you can just make very simple HTTP connections using Python objects, and then send and receive messages using the methods of those objects. Let's look at an example:
>>> import requests
>>> response = requests.get('https://www.pythoncircle.com')
That was a basic request for a web page! We used the Requests library to make an HTTP GET request for a specific URL. The URL tells the Requests library the name of the resource (https://www.pythoncircle.com). The result we get is an object of type requests.Response.
Response:
We can check the response text by printing
response.text.
Even with this simple example, the Requests module has done a whole lot of work for us. We didn't have to write any code to find the webserver, make a network connection, construct an HTTP message, wait for a response, or decode the response. Not that HTML can't be messy enough on its own, but let's look at the first bytes of the raw message that we received from the server:
>>> response = requests.get('https://www.google.com', stream=True)
>>> print(response.raw.read()[:100])
b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\xff\xc5Z\xdbz\x9b\xc8\x96\xbe\xcfS`\xf2\xb5-\xc6X\x02$t\xc28\xe3v\xdc\xdd\xee\xce\xa9\xb7\xdd;\xe9\x9d\xce\xf6W@\t\x88\x11`@>D\xd6\x9b\xce\xe5<\xc3\\\xcd\xc5\xfc\xab8\x08\xc9Nz\x1f.&\x8e1U\xb5j\xd5:\xfc\xb5jU\x15\x87;^\xe2\x16\xf7)\x97\x82b\x1e\x1d\x1d\xd2S'
What's all that? The response was compressed with gzip, so it had to be decompressed before we could even read the text of the HTML. One more thing that the Requests library handled for us!
The requests.Response object also contains the exact request that was created for us. We can check out the headers stored in our object to see that the Requests module told the web server that it was okay to compress the content:
>>> response.request.headers['Accept-Encoding']
'gzip, deflate'
>>> response.headers['Content-Encoding']
'gzip'
Checking response status:
How do we know if a request we made got a successful response? You can check out the value of Response.ok, which will be True if the response was good, and False if it wasn't.
>>> response.ok
True
This will only tell you if the webserver says that the response successfully fulfilled the request. The response module can’t determine if that data that you got back is the kind of data that you were expecting. You'll need to do your own checking for that!
you can get the HTTP response code that was returned by looking at Response.status_code:
>>> response.status_code
200
To check your responses to make sure they succeeded before trying to process them further, you could do something like this
response = requests.get(url)
if not response.ok:
raise Exception("GET failed with status code {}".format(response.status_code))
We can also use the Response.raise_for_status() method, which will raise an HTTPError exception only if the response wasn’t successful.
response = requests.get(url)
response.raise_for_status()
HTTP GET Method:
The HTTP GET method, retrieves or gets the resource specified in the URL. By sending a GET request to the web server, you’re asking for the server to GET the resource for you. When you’re browsing the web, most of what you’re doing is using your web browser to issue a whole bunch of GET requests for the text, images, videos, and so forth that your browser will display to you.
A GET request can have parameters. See the URL example below.
https://example.com/path/to/api/cat_pictures?search=grey+kitten&max_results=15The question mark separates the URL resource from the resource's parameters. These parameters are one or more key-value pairs, formatted as a query string. In the example above, the search parameter is set to "grey+kitten", and the max_results parameter is set to 15.
You don't have to write your own code to create an URL like that one. With requests.get(), you can provide a dictionary of parameters, and the Requests module will construct the correct URL for you!
>>> p = {"search": "grey kitten",
... "max_results": 15}
>>> response = requests.get("https://example.com/path/to/api", params=p)
>>> response.request.url
'https://example.com/path/to/api?search=grey+kitten&max_results=15'HTTP POST Method:
Query strings are handy when we want to send small bits of information, but as our data becomes more complex, it can get hard to represent it using query strings.
An alternative, in that case, is using the HTTP POST method. This method sends, or posts, data to a web service. Whenever you fill a web form and press a button to submit, you're using the POST method to send that data back to the webserver. This method tends to be used when there's a bunch of data to transmit.
In our scripts, a POST request looks very similar to a GET request. Instead of setting the params attribute, which gets turned into a query string and appended to the URL, we use the data attribute, which contains the data that will be sent as part of the POST request.
>>> p = {"description": "white kitten",
... "name": "Snowball",
... "age_months": 6}
>>> response = requests.post("https://example.com/path/to/api", data=p)Let's see the generated URL for this request:
>>> response.request.url
'https://example.com/path/to/api'
Where did all of the parameters go? They’re part of the body of the HTTP message. We can see them by checking out the body attribute.
>>> response.request.body
'description=white+kitten&name=Snowball&age_months=6'
If we need to send and receive data from a web service, we can turn our data into dictionaries and then pass that as the data attribute of a POST request. it's common to send and receive data specifically in JSON format, so the Requests module can do the conversion directly for us, using the JSON parameter.
>>> response = requests.post("https://example.com/path/to/api", json=p)
>>> response.request.url
'https://example.com/path/to/api'
>>> response.request.body
b'{"description": "white kitten", "name": "Snowball", "age_months": 6}'
