Python Requests Library: The Complete Guide

requests is the HTTP client most Python developers reach for first, and usually the last one they need. It wraps urllib3 in an API that fits in your head: requests.get(url) returns a response object with .text, .json(), .status_code, and sensible defaults for everything else. This guide covers the whole surface — installation, GET and POST, the json vs data distinction that trips up most API integrations, sessions, timeouts, retries, proxies, redirects, streaming, SSL, and the exception hierarchy — plus the honest limits: requests is in maintenance mode, speaks only HTTP/1.1, and has no async support.

Key Takeaways

• pip install requests, then requests.get(url) — the response object carries .text, .content, .json(), .status_code, and .headers
• json= sets Content-Type: application/json and serializes for you; data= form-encodes. Picking the wrong one is the most common cause of a 400 from a REST API
• requests never times out by default. Always pass timeout= — a (connect, read) tuple is best
• Use a Session for more than one request to the same host: it reuses connections, persists cookies, and carries default headers
• Retries are not built in — mount an HTTPAdapter with a urllib3 Retry on your session
• requests is feature-frozen: no HTTP/2, no async. If you need either, httpx has a near-identical API
• It fetches HTML but does not run JavaScript — a page built client-side comes back as an empty shell no matter what headers you send

Installing requests

pip install requests
python -c "import requests; print(requests.__version__)"

Inside a virtual environment, which is what you want for anything beyond a scratch script:

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install requests

Two optional extras matter in practice:

pip install "requests[socks]"    # SOCKS4/SOCKS5 proxy support (pulls in PySocks)
pip install brotli               # lets requests accept and decode brotli responses

ModuleNotFoundError: No module named 'requests' almost always means you installed into a different interpreter than the one running your script. python -m pip install requests pins the install to the interpreter you just invoked and resolves it nine times out of ten. On Linux, prefer a venv over sudo pip install — installing into the system Python is how you break your package manager.

Quick start: the response object

import requests

resp = requests.get("https://httpbin.org/get", timeout=10)

resp.status_code       # 200
resp.ok                # True for any status below 400
resp.headers["Content-Type"]
resp.text              # body decoded to str
resp.content           # body as raw bytes
resp.json()            # parsed JSON (raises if the body isn't JSON)
resp.url               # final URL, after any redirects
resp.encoding          # the codec used to produce .text

Every HTTP verb has a module-level shortcut, and all of them accept the same keyword arguments:

requests.get(url, params={"q": "python"})
requests.post(url, json={"name": "Jane"})
requests.put(url, json=payload)
requests.patch(url, json={"status": "active"})
requests.delete(url, headers={"Authorization": f"Bearer {token}"})
requests.head(url)                       # headers only, no body
requests.options(url)

DELETE is worth calling out because APIs disagree about it: some want the identifier in the path, some want query parameters, and some accept a JSON body (which requests will happily send, though not every server reads it). All three forms work:

requests.delete(f"https://api.example.com/items/{item_id}", timeout=10)
requests.delete("https://api.example.com/items", params={"id": item_id}, timeout=10)
requests.delete("https://api.example.com/items", json={"ids": [1, 2, 3]}, timeout=10)

.text vs .content vs .json() vs .raw

These four accessors are the single most searched-about part of the library, and the differences are real:

.text and .content are the same bytes; .text just runs them through a codec. Writing .text to a file in binary mode, or .content to a text file, is the usual source of corrupted downloads — always use .content with open(path, "wb").

.raw is the escape hatch, and it has a gotcha: it hands you the undecompressed stream, so a gzipped response reads back as gzip bytes unless you ask for decoding:

resp = requests.get(url, stream=True, timeout=10)
data = resp.raw.read(decode_content=True)     # without decode_content you get gzip bytes

Query parameters

Pass a dict as params and requests handles URL-encoding, including spaces, unicode, and reserved characters:

resp = requests.get("https://httpbin.org/get", params={
    "q": "web scraping",
    "page": 2,
    "safe": True,          # becomes safe=True
}, timeout=10)

print(resp.url)   # https://httpbin.org/get?q=web+scraping&page=2&safe=True

Repeated keys — common in filter APIs — come from a list value or a list of tuples:

requests.get(url, params={"tag": ["python", "http"]})        # ?tag=python&tag=http
requests.get(url, params=[("tag", "python"), ("tag", "http")])

Values of None are dropped entirely, which makes optional parameters easy to express without building the dict conditionally. If you need byte-exact control over encoding or ordering, build the query string yourself with urllib.parse.urlencode and append it — requests will not re-encode a query string that is already in the URL.

POST: json vs data, and why it matters

This is the distinction that generates the most confused bug reports. The two parameters send different content types, and an API that wants one will reject the other:

# Form submission — an HTML <form> posting to a server
requests.post("https://example.com/login",
              data={"username": "jane", "password": "secret"}, timeout=10)

# REST API — JSON body, header set automatically
requests.post("https://api.example.com/users",
              json={"name": "Jane", "roles": ["admin"]}, timeout=10)

Three rules that resolve most of the confusion:

1. Don't set Content-Type yourself when using json=. requests already set it; overriding with your own header dict can leave you sending a mismatched pair.
2. Don't json.dumps() into data=. That sends a JSON string as a form body with the wrong content type. If you genuinely need to control serialization (custom encoder, exact key order), pass the dumped string to data= and set the header explicitly: python requests.post(url, data=json.dumps(payload, default=str), headers={"Content-Type": "application/json"}, timeout=10)
3. Don't pass both. data wins and your JSON is silently ignored.

A dict passed to data= is form-encoded with the same repeated-key rules as params. Bytes and strings pass through untouched — useful for posting XML, NDJSON, or a pre-signed payload.

When a request isn't doing what you expect, prepare it without sending and look at what would go over the wire:

prepared = requests.Request("POST", url, json={"a": 1}).prepare()
print(prepared.headers)   # {'Content-Length': '8', 'Content-Type': 'application/json'}
print(prepared.body)      # b'{"a": 1}'

Headers and the User-Agent

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}
resp = requests.get(url, headers=headers, timeout=10)

Left alone, requests announces itself as python-requests/2.x. Plenty of sites block that string outright, which is why a custom User-Agent is the first fix people try when a page returns 403 in Python but loads fine in a browser. Send a realistic browser string along with the headers a browser would send — Accept, Accept-Language, Referer — because a Chrome User-Agent paired with no Accept-Language is itself a fingerprint.

Header handling details worth knowing: header names are case-insensitive on both request and response (resp.headers["content-type"] works), per-request headers merge with session headers rather than replacing them, and setting a session header to None removes it for that request.

Sessions: connections, cookies, and defaults

A Session is the right default for anything past a single call. It gives you three things at once:

import requests

with requests.Session() as session:
    session.headers.update({"User-Agent": "my-scraper/1.0"})
    session.params = {"api_key": API_KEY}          # sent on every request

    session.post("https://example.com/login",
                 data={"user": "jane", "pass": "secret"}, timeout=10)

    # the login cookie is carried automatically
    profile = session.get("https://example.com/account", timeout=10)

1. Connection reuse. The underlying TCP and TLS connection is kept alive between calls, so the second request to a host skips the handshake — worth hundreds of milliseconds on HTTPS. This is what "persistent connections" and "keep-alive" mean in requests: they're the default inside a session and impossible without one.
2. Cookie persistence. Cookies the server sets are stored in session.cookies and replayed on subsequent requests to matching domains. This is what makes login-then-scrape flows work.
3. Configuration persistence. headers, params, auth, proxies, verify, and cert set on the session apply to every request through it.

Module-level requests.get() creates and discards a session for each call, so it gets none of the above. For a loop of 100 requests to one host, the session version is typically several times faster, and the gap widens with TLS.

The one trap: Session has no timeout attribute. session.timeout = 30 assigns an ignored attribute and every request still hangs forever. Pass timeout= per call, or wrap the session:

class TimeoutSession(requests.Session):
    def request(self, *args, **kwargs):
        kwargs.setdefault("timeout", (5, 30))
        return super().request(*args, **kwargs)

Cookies

# Send cookies with a one-off request
requests.get(url, cookies={"session_id": "abc123"}, timeout=10)

# Read cookies the server set
resp.cookies["session_id"]
resp.cookies.get_dict()

# Set one on a session, scoped to a domain and path
session.cookies.set("consent", "yes", domain="example.com", path="/")

To carry a login across program runs, persist the jar. JSON keeps it readable and avoids the security problems of unpickling data you don't fully control:

import json

with open("cookies.json", "w") as f:
    json.dump(requests.utils.dict_from_cookiejar(session.cookies), f)

with open("cookies.json") as f:
    session.cookies = requests.utils.cookiejar_from_dict(json.load(f))

Timeouts

requests has no default timeout. A server that accepts your connection and then goes silent will hang your script indefinitely — no exception, no traceback, just a process that never finishes. This is the single most important argument in the library:

requests.get(url, timeout=10)              # 10s for connect, 10s for read
requests.get(url, timeout=(3.05, 27))      # (connect, read) — preferred

The two halves bound different failures. Connect covers DNS plus the TCP and TLS handshake, so an unreachable host fails fast. Read bounds the wait between bytes, not the total transfer — a slow but steady 2 GB download never trips a 30-second read timeout, while a server that sends headers and then stalls trips it immediately. There is no built-in total-duration cap; if you need one, enforce it in your own loop or at the scheduler level.

Timeouts raise requests.exceptions.ConnectTimeout and ReadTimeout, both subclasses of Timeout, itself a RequestException.

Retries with backoff

requests deliberately ships with retries disabled. You add them by mounting an HTTPAdapter configured with urllib3's Retry:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=5,
    backoff_factor=0.5,                            # 0.5s, 1s, 2s, 4s, 8s
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD", "OPTIONS"],    # don't blind-retry POSTs
    respect_retry_after_header=True,
    raise_on_status=False,
)

session = requests.Session()
adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)
session.mount("http://", adapter)

resp = session.get("https://api.example.com/data", timeout=(3, 27))

respect_retry_after_header=True doubles as polite rate-limit handling: on a 429 or 503 carrying Retry-After, urllib3 waits exactly as long as the server asked rather than following its own backoff curve. Leave allowed_methods at its default (idempotent methods only) unless you know the endpoint is safe to repeat — retrying a POST can create duplicate orders.

Two version notes: allowed_methods was called method_whitelist in urllib3 1.x, and backoff_factor sleeps backoff_factor * (2 ** (attempt - 1)), so 0.5 is a gentler curve than it looks. When retries are exhausted you get a requests.exceptions.RetryError (or ConnectionError wrapping urllib3's MaxRetryError, depending on what failed).

Retry covers transport-level failures and status codes. It does not help with application-level errors — a 200 response containing {"error": "rate limited"} needs your own loop, ideally with jitter so parallel workers don't retry in lockstep:

import random, time

for attempt in range(5):
    resp = session.get(url, timeout=(3, 27))
    if resp.ok and "error" not in resp.json():
        break
    time.sleep(min(2 ** attempt, 30) + random.uniform(0, 1))

Connection pooling

Under a session, each host gets a urllib3 connection pool. The defaults suit sequential scripts and are too small for threaded work — tune them on the adapter:

adapter = HTTPAdapter(
    pool_connections=20,   # number of distinct host pools cached
    pool_maxsize=20,       # connections kept open per host
    pool_block=False,      # True = wait for a free connection instead of opening extras
    max_retries=retry,
)
session.mount("https://", adapter)

For threaded scraping of a single host, set pool_maxsize to roughly your worker count. Left at the default of 10 with 50 threads, urllib3 opens extra connections and discards them after use, emitting Connection pool is full, discarding connection warnings and quietly throwing away the benefit of pooling. pool_block=True turns the pool into a hard cap, which is what you want when a target explicitly limits concurrent connections.

A Session object is thread-safe for sharing across threads in the common case, but it isn't guaranteed for concurrent mutation — don't reassign session.headers or session.cookies from multiple threads. One session per thread, or one shared read-only session, both work.

Redirects and allow_redirects

requests follows redirects automatically for every method except HEAD:

resp = requests.get("http://github.com", timeout=10)
resp.url          # 'https://github.com/' — the final destination
resp.history      # [<Response [301]>] — every hop, in order
resp.history[0].headers["Location"]

Set allow_redirects=False to get the 3xx response itself instead of the destination:

resp = requests.get(url, allow_redirects=False, timeout=10)
resp.status_code             # 302
resp.headers["Location"]     # where it wanted to send you

Reasons to turn it off: checking whether a shortened URL is malicious before visiting it, auditing a site's redirect chain, capturing a Set-Cookie header that only appears on the redirect response, or reading a Location that contains an OAuth code you need. HEAD is the reverse case — it defaults to not following, so requests.head(url, allow_redirects=True) is how you resolve a final URL without downloading the body.

Following a chain by hand is a short loop:

url, chain = start_url, []
for _ in range(10):
    resp = requests.get(url, allow_redirects=False, timeout=10)
    chain.append((resp.status_code, url))
    if resp.status_code not in (301, 302, 303, 307, 308):
        break
    url = requests.compat.urljoin(url, resp.headers["Location"])   # Location may be relative

Two behaviors that surprise people. Method changes on 301, 302, and 303: a POST becomes a GET and the body is dropped, matching what browsers do. Only 307 and 308 preserve the method and body. Authorization headers are stripped when a redirect crosses to a different host, which is a deliberate security measure — if your credentials vanish mid-chain, that's why. A loop raises TooManyRedirects after 30 hops by default (session.max_redirects).

Streaming, chunked encoding, and large downloads

By default requests downloads the whole body into memory before returning. stream=True returns as soon as headers arrive and leaves the body on the wire:

with requests.get(url, stream=True, timeout=(5, 30)) as resp:
    resp.raise_for_status()
    with open("large-file.zip", "wb") as f:
        for chunk in resp.iter_content(chunk_size=8192):
            f.write(chunk)

Use stream=True for files too large to hold in memory, for responses you may want to abandon after inspecting the headers (checking Content-Length or Content-Type before committing to the download), and for endpoints that stream indefinitely. Avoid it for ordinary API calls — the connection stays checked out of the pool until you consume or close the response, and forgetting is a slow connection leak. Use with or call resp.close(); accessing .text or .content also drains and releases it.

iter_content handles chunked transfer encoding transparently. A server that sends Transfer-Encoding: chunked has no Content-Length, so you cannot know the total size up front — progress bars need a fallback:

total = int(resp.headers.get("Content-Length", 0))   # 0 when chunked

For newline-delimited streams — log tails, NDJSON feeds, server-sent events — iterate lines instead:

with requests.get(url, stream=True, timeout=(5, None)) as resp:
    for line in resp.iter_lines(decode_unicode=True):
        if line:
            event = json.loads(line)

Note timeout=(5, None): a long-lived stream legitimately has gaps between messages, so a read timeout would kill it. Bound the connect phase and leave the read phase open. iter_lines is not reentrant and can split multi-byte characters across chunk boundaries — for anything where exactness matters, use iter_content and buffer yourself.

Downloading images and video is the same pattern, with a content-type check so an error page doesn't get saved as a .jpg:

with requests.get(image_url, stream=True, timeout=(5, 30)) as resp:
    resp.raise_for_status()
    if not resp.headers.get("Content-Type", "").startswith("image/"):
        raise ValueError(f"expected an image, got {resp.headers.get('Content-Type')}")
    with open("photo.jpg", "wb") as f:
        for chunk in resp.iter_content(64 * 1024):
            f.write(chunk)

Uploading files

Pass files= and requests builds the multipart/form-data body, boundary and all:

with open("report.pdf", "rb") as f:
    requests.post(url, files={"document": f}, timeout=30)

Control the filename and MIME type with a tuple, and mix in regular form fields via data= — the two coexist in one multipart body:

with open("report.pdf", "rb") as f:
    requests.post(url,
                  files={"document": ("q3-report.pdf", f, "application/pdf")},
                  data={"category": "finance", "year": "2026"},
                  timeout=30)

Multiple files under the same field name need a list of tuples, since a dict can't hold duplicate keys:

files = [
    ("photos", ("a.jpg", open("a.jpg", "rb"), "image/jpeg")),
    ("photos", ("b.jpg", open("b.jpg", "rb"), "image/jpeg")),
]
requests.post(url, files=files, timeout=60)

In-memory data works too — pass io.BytesIO(blob) or the bytes directly instead of a file handle. Do not set Content-Type yourself here: requests generates a boundary token and puts it in the header, and a hand-written header without the matching boundary produces a body the server cannot parse.

One limit: requests reads each file fully into memory to build the body, so multi-gigabyte uploads need requests-toolbelt's MultipartEncoder, which streams instead.

Authentication

from requests.auth import HTTPBasicAuth, HTTPDigestAuth

requests.get(url, auth=("user", "pass"), timeout=10)              # Basic, shorthand
requests.get(url, auth=HTTPBasicAuth("user", "pass"), timeout=10) # identical, explicit
requests.get(url, auth=HTTPDigestAuth("user", "pass"), timeout=10)

Bearer tokens and API keys are headers, not auth:

session.headers["Authorization"] = f"Bearer {access_token}"
session.headers["X-API-Key"] = api_key

Custom schemes subclass AuthBase, which is how you implement request signing (HMAC, AWS SigV4-style) cleanly:

from requests.auth import AuthBase

class TokenAuth(AuthBase):
    def __init__(self, token):
        self.token = token

    def __call__(self, request):
        request.headers["Authorization"] = f"Token {self.token}"
        return request

requests.get(url, auth=TokenAuth(token), timeout=10)

Read credentials from environment variables or a secret manager, never from a literal in the source. Basic auth in particular is base64, not encryption — it is only safe over HTTPS, and the verify=False shortcut discussed below defeats that protection entirely.

Proxies

proxies = {
    "http":  "http://user:pass@proxy.example.com:8080",
    "https": "http://user:pass@proxy.example.com:8080",
}
requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)
session.proxies.update(proxies)          # or set it once on the session

Both keys take the proxy's own scheme — an HTTP proxy tunneling HTTPS traffic via CONNECT is still http:// in the value; the dict key is the target's scheme. SOCKS needs the extra installed:

pip install "requests[socks]"

proxies = {"https": "socks5h://user:pass@proxy.example.com:1080"}

Use socks5h rather than socks5 for scraping: the h resolves DNS through the proxy, so your local resolver never sees the target hostname and geo-based DNS returns results for the proxy's location.

requests also reads HTTP_PROXY, HTTPS_PROXY, and NO_PROXY from the environment. That's convenient until it isn't — a stale variable in a CI environment routes traffic somewhere unexpected. session.trust_env = False disables environment lookup (along with netrc and REQUESTS_CA_BUNDLE) when you want fully explicit configuration.

Rotating proxies means swapping the dict per request, which requests supports but doesn't manage. Getting a pool worth rotating is the harder half; our proxy provider comparison covers the options, and a scraping API removes the question by rotating server-side.

SSL verification

Certificates are verified by default against the certifi bundle. When you hit SSLError: CERTIFICATE_VERIFY_FAILED, work through the causes in order:

1. Stale CA bundle — pip install -U certifi. This fixes the majority of cases, especially on older machines.
2. Corporate TLS-inspecting proxy or an internal CA — point requests at the CA that actually signed the certificate: python requests.get(url, verify="/path/to/corporate-ca.pem", timeout=10) session.verify = "/path/to/corporate-ca.pem" # or the REQUESTS_CA_BUNDLE env var
3. Genuinely broken remote certificate (expired, hostname mismatch) — decide consciously.

Client certificates for mutual TLS go in cert:

requests.get(url, cert=("client.crt", "client.key"), timeout=10)
requests.get(url, cert="combined.pem", timeout=10)     # cert and key in one file

Disabling verification is a last resort. verify=False makes the connection trivially interceptable — anyone on the path can read and rewrite the traffic, which means it is not acceptable for anything carrying credentials:

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
resp = requests.get(url, verify=False, timeout=10)     # you are now MITM-able

The warning requests prints is doing its job; silencing it without fixing the underlying trust problem just hides a real vulnerability.

Status codes and error handling

requests does not raise on 4xx or 5xx. A 500 is a perfectly valid response object, and resp.json() on an error page is where the confusing traceback usually comes from. Check explicitly:

if resp.status_code == 200:
    ...
if resp.ok:                      # any status below 400
    ...
if resp.status_code == requests.codes.not_found:    # 404, spelled readably
    ...

Or opt into exceptions with raise_for_status(), which raises HTTPError on 4xx and 5xx and returns None otherwise:

resp = requests.get(url, timeout=10)
resp.raise_for_status()
data = resp.json()

The exception hierarchy roots at RequestException, so one except catches everything requests can throw:

import requests
from requests.exceptions import (
    HTTPError, ConnectionError, Timeout, TooManyRedirects,
    SSLError, ProxyError, JSONDecodeError, RequestException,
)

try:
    resp = requests.get(url, timeout=(3.05, 27))
    resp.raise_for_status()
    data = resp.json()
except Timeout:
    log.warning("timed out — safe to retry")
except HTTPError as e:
    log.error("HTTP %s: %s", e.response.status_code, e.response.text[:200])
except JSONDecodeError:
    log.error("expected JSON, got %s", resp.headers.get("Content-Type"))
except ConnectionError:
    log.error("DNS failure, refused connection, or dropped socket")
except RequestException as e:
    log.exception("unexpected requests failure: %s", e)

Order matters: catch the specific classes before RequestException, or the base clause swallows everything. Useful members of the family beyond the above: ConnectTimeout and ReadTimeout (both Timeout), ChunkedEncodingError and ContentDecodingError for malformed response bodies, and MissingSchema / InvalidURL for a bad URL — that last pair catches the classic "I forgot the https://".

HTTPError carries the response on e.response, which is what lets you log the server's actual error message instead of a generic status line.

Compression and encoding

requests advertises Accept-Encoding: gzip, deflate on every request and decompresses transparently, so .content and .text are always the decoded bytes. Add brotli or zstandard to the environment and those encodings are negotiated too. Two consequences catch people out: resp.headers["Content-Encoding"] still reports gzip on a body you're reading as plain text, and len(resp.content) will not match the Content-Length header, which describes the compressed transfer.

Character encoding is a separate layer. .text decodes using resp.encoding, which requests takes from the Content-Type charset. When the header is a bare text/html with no charset, HTTP's rules say ISO-8859-1 — and plenty of pages that are actually UTF-8 send exactly that, producing mojibake like é where é belongs. The fix is to override:

resp.encoding = "utf-8"                   # you know what it is
resp.encoding = resp.apparent_encoding    # let charset detection decide (slower)
text = resp.text

apparent_encoding runs the body through charset_normalizer and is a good fallback for scraping pages of unknown origin, but it costs a scan of the body. For a POST that carries non-ASCII text, encode it yourself and declare the charset so the server doesn't have to guess:

requests.post(url, data="naïve=café".encode("utf-8"),
              headers={"Content-Type": "application/x-www-form-urlencoded; charset=utf-8"},
              timeout=10)

Measuring response time

resp.elapsed is a timedelta requests populates for every response:

resp = requests.get(url, timeout=10)
print(resp.elapsed.total_seconds())

Read it precisely: it covers sending the request through parsing the response headers, and not the body download. For a small JSON response the difference is negligible; for a 200 MB file it's most of the wall-clock time. To measure the full transfer, time it yourself:

import time

start = time.perf_counter()
resp = requests.get(url, timeout=10)
_ = resp.content                       # force the body to be read
total = time.perf_counter() - start

With redirects, resp.elapsed reflects only the final hop — sum [r.elapsed for r in resp.history] + [resp.elapsed] for the whole chain.

Debugging

Response inspection first, since it answers most questions:

resp.request.method, resp.request.url
resp.request.headers          # what was actually sent, after session merging
resp.request.body
resp.status_code, resp.headers, resp.history

For a running log of connections, retries, and redirects, turn up urllib3's logger — requests itself logs almost nothing:

import logging

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG)

Event hooks let you instrument every response through a session without touching call sites:

def log_response(resp, *args, **kwargs):
    print(f"{resp.request.method} {resp.url} -> {resp.status_code} "
          f"in {resp.elapsed.total_seconds():.2f}s")

session.hooks["response"].append(log_response)

For full wire-level dumps including bodies, reach for mitmproxy or Wireshark — requests will not print raw traffic. And when you're comparing behavior against a shell one-liner, our curl guide maps the flags across; curl -v is often the fastest way to establish whether a problem is in your Python or in the server.

Testing HTTP code without hitting the network is its own small ecosystem: responses and requests-mock both intercept at the adapter layer, so your code under test calls requests normally while the library returns canned responses. Both are better than patching requests.get with unittest.mock, which stops testing the request you actually build.

requests vs urllib3 vs httpx

requests is in maintenance mode — it receives security and compatibility fixes, not new protocol features. HTTP/2 and async are not coming; that was a deliberate decision by the maintainers, not a backlog item. This is fine for the overwhelming majority of code. HTTP/1.1 with connection reuse is fast, and threads handle concurrency adequately up to a few dozen workers.

Switch to httpx when you need HTTP/2 multiplexing, async/await, or a sane default timeout; its API is close enough that porting is usually mechanical (requests.get → httpx.get, Session → Client). Drop to urllib3 when you're writing a library that shouldn't pull in requests' dependency tree, or you need pool-level control requests doesn't expose. For everything else, requests being frozen is a feature — the code you write today keeps working.

Using requests for web scraping

For static HTML, requests plus a parser is the whole stack. Beautiful Soup is the usual partner:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

resp = requests.get("https://example.com/articles",
                    headers={"User-Agent": "Mozilla/5.0 ..."}, timeout=10)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "html.parser")
links = [urljoin(resp.url, a["href"]) for a in soup.select("a[href]")]

urljoin against resp.url rather than the URL you requested is the detail that matters — after a redirect, relative links resolve against the final URL, and joining against the original silently produces broken links. Combine this with a session, a realistic User-Agent, timeouts, and a Retry adapter and you have a scraper that survives contact with production. For XML feeds, lxml or ElementTree take the place of Beautiful Soup; the fetch half is identical. Our Python web scraping guide covers the pipeline end to end.

Then there's the wall. requests does not execute JavaScript. It fetches the HTML the server sent and stops there, so a React, Vue, or Angular page comes back as a nearly empty <div id="root"> with the real content nowhere in resp.text. No combination of headers changes this — the data is assembled by code that never runs. The same is true of anti-bot systems that fingerprint the TLS handshake and JavaScript environment: they aren't reading your User-Agent, so a better one won't help.

The options are running a real browser (Playwright, Selenium — heavy, but complete), reverse-engineering the JSON API the page calls (fast when it works, brittle when the site changes), or sending the fetch to a service that renders for you. WebScraping.AI does the last one behind a single request you make with requests, leaving the rest of your code untouched:

import requests

resp = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/spa-products",
    "js": "true",             # rendered in a real browser, proxies included
}, timeout=60)
html = resp.text

Your session, retries, and error handling all still apply — it's an ordinary HTTPS endpoint. The /ai/fields endpoint goes a step further and returns structured JSON from a plain-English description of the data you want, which is what keeps a scheduled job like price monitoring running through site redesigns instead of breaking on every selector change. The API reference lists the other parameters (proxy, country, timeout, device, wait_for).

Note the longer client-side timeout above: rendering a page takes seconds, not milliseconds, so budget for it.

Frequently asked questions

What's the difference between json= and data= in requests? json= serializes your object with json.dumps() and sets Content-Type: application/json. data= form-encodes a dict as application/x-www-form-urlencoded, the format an HTML form posts. Use json= for REST APIs and data= for form submissions. Passing a json.dumps() string to data= sends JSON with the wrong content type, which is the usual cause of an unexplained 400.

Why does my requests script hang forever? Because there is no default timeout. A server that accepts the connection and then stops responding will block indefinitely. Always pass timeout=, ideally as a (connect, read) tuple like (3.05, 27). Note that session.timeout = 30 does nothing — Session has no such attribute; the timeout must be passed per request or injected by subclassing Session.request.

Does requests support HTTP/2? No, and it won't. requests is built on urllib3, which is HTTP/1.1 only, and the project is in maintenance mode. Use httpx with pip install httpx[http2] and httpx.Client(http2=True) if you need it. In practice, HTTP/2 rarely matters for scraping or API work — connection reuse through a Session captures most of the same benefit.

How do I stop requests from following redirects? Pass allow_redirects=False to get the 3xx response with its Location header instead of the final destination. Everything except HEAD follows redirects by default; HEAD is the opposite and needs allow_redirects=True to follow. resp.history holds the full chain when redirects are followed, and resp.url is where you ended up.

When should I use stream=True? When the body is too large to hold in memory, when you want to inspect headers before deciding to download, or when the endpoint streams indefinitely. Iterate with iter_content(chunk_size=8192) for binary or iter_lines() for newline-delimited data, and always use a with block — a streamed response holds its connection out of the pool until closed.

Why is my scraped page missing the content I see in the browser? The page renders that content with JavaScript. requests returns the initial HTML only, so anything a framework builds client-side is absent, and no header changes that. You need a headless browser, the underlying JSON API the page calls, or a rendering API — see the scraping section above.

How do I add retries to requests? Mount an HTTPAdapter(max_retries=Retry(...)) on a Session. urllib3's Retry gives you total, backoff_factor for exponential backoff, status_forcelist for which statuses to retry, and respect_retry_after_header so 429 responses wait exactly as long as the server asks. Keep allowed_methods restricted to idempotent verbs unless you're certain repeating a POST is safe.

Is requests.Session faster than requests.get()? Yes, materially, for repeated calls to the same host. A session reuses the TCP and TLS connection, so every request after the first skips the handshake — typically hundreds of milliseconds on HTTPS. requests.get() builds and discards a session per call and gets none of that, nor cookie or header persistence.