Working with Requests in Python

Mastering Requests is a fundamental skill for wielding Python’s mighty capabilities for web interfacing and data scraping. This guide provides a comprehensive overview of Requests, oriented toward competent Pythonistas seeking to further hone their craft.

An Introduction to Requests

The Requests library provides an elegant and simple way to issue HTTP requests in Python. Built on top of the urllib3 library, Requests eliminates much of the boilerplate code required when working with URLs and sending network calls. The readability of Requests code — especially with the intuitive verbs offered by the API — makes HTTP scripting uncomplicated and accessible.

Some key advantages of using Requests over the built-in urllib module are:

• Simplified syntax and code that is easier to write and read
• Automatic encoding of parameters and URLs
• Built-in JSON decoding capabilities
• Persistent sessions with connection pooling and cookie persistence
• Timeout support to prevent hanging requests
• Automatic following of redirects
• SSL/TLS verfication

In summary, Requests allows you to focus on interacting with APIs instead of fussing with protocols. For those seeking to scrape web data or leverage web APIs, Requests is likely the go-to choice.

Making Requests in Python

The Requests workflow will be intimately familiar to anyone who has worked with HTTP/URLs before.

To begin, import Requests at the top of your script:


import requests

With Requests imported, you can start issuing requests using the verb methods:


response = requests.get('https://...')

This will make a GET request to the URL, returning a Response object. You can easily access properties of the response:


print(response.status_code) # 200
print(response.headers) # dict of response headers
print(response.content) # Raw response bytes

Additionally, there are equivalent methods for making POST, PUT, DELETE and other HTTP requests. For instance:


requests.post('https://.../users', data={'name':'Johny'})
requests.put('https://.../users/2', data={'name':'Jack'})
requests.delete('https://.../users/3')

The library has many options for customizing requests, handling response data, managing sessions/cookies, configuring timeouts/retries, and more.

Key Features of Requests

Beyond basic request issuing, Requests comes well-stocked with features that make integration smooth:

Automatic JSON Decoding

Requests will automatically decode the response’s JSON for you:


response = requests.get('https://.../get-data')
data = response.json() # Valid JSON objects are turned into Python dicts

Query String Parameters

Requests makes attaching query parameters simple:


payload = {'search_query': 'coffee', 'per_page': 10}
response = requests.get('https://.../search', params=payload)

File Uploads

Sending multipart-encoded files is made easy:


files = {'file': open('report.pdf', 'rb')}
requests.post('https://.../upload', files=files)

Connection Timeouts

Define how long Requests should wait for a response before timing out:


requests.get('https://.../data', timeout=3.0) # seconds

Session Objects

Take advantage of session caching, cookie persistence, and connection pooling by using a session:

session = requests.Session()
session.auth = ('username', 'password') # Enable authentication

response = session.get('https://.../user-profile')

Common Requests Workflows

Now that we’ve covered the basics of Requests, let’s walk through some common usage scenarios:

Scraping a Web Page

Use Requests to fetch content and Beautiful Soup for HTML parsing/scraping:

import requests
from bs4 import BeautifulSoup

response = requests.get("https://...")
soup = BeautifulSoup(response.content, 'html.parser')
h1_tag = soup.find('h1')
print(h1_tag.text) # Print the H1 text

Interacting with a REST API

Leverage Requests to integrate with REST APIs:

import requests

endpoint = "https://.../resources"

Fetch all resources


response = requests.get(endpoint)

Create a new resource


data = {'name': 'New Resource'}
response = requests.post(endpoint, json=data)

Read resource with id 42


response = requests.get(f'{endpoint}/42')

Submitting Web Forms

Post form data to interact with web apps:

import requests

response = requests.post('https://.../signup', data=data)

These snippets reveal how Requests enables you to accomplish most common HTTP workflows with minimal fuss.

Conclusion

In closing, the Requests library makes HTTP scripting in Python a trivial affair. The simple API, developer-friendly design, and powerful feature set have made it a favorite among Python devotees. Regardless of your specific need — whether it’s scraping data, connecting to APIs, or automating web interactions — Requests likely has you covered. Its extensive documentation, community support, and integration with other Python tooling make it a superb choice for meeting your HTTP needs. If you’re not yet leveraging Requests for your Python projects, install it today and see for yourself why it has become a standard library for our language.

joker

Professional data parsing via ZennoPoster, Python, creating browser and keyboard automation scripts. SEO-promotion and website creation: from a business card site to a full-fledged portal.