---
title: "Python Requests Library: The Complete Guide"
description: "Complete Python requests guide: GET and POST, json vs data, sessions, headers, timeouts, retries, proxies, redirects, streaming, SSL, and error handling."
url: https://webscraping.ai/blog/python-requests-guide
markdown_index: https://webscraping.ai/llms.txt
---
# Python Requests Library: The Complete Guide

[requests](https://requests.readthedocs.io/) is the HTTP client most Python developers reach for first, and usually the last one they need. It wraps [urllib3](https://webscraping.ai/blog/python-urllib3-guide) in an API that fits in your head: `requests.get(url)` returns a response object with `.text`, `.json()`, `.status_code`, and sensible defaults for everything else. This guide covers the whole surface — installation, GET and POST, the `json` vs `data` distinction that trips up most API integrations, sessions, timeouts, retries, proxies, redirects, streaming, SSL, and the exception hierarchy — plus the honest limits: requests is in maintenance mode, speaks only HTTP/1.1, and has no async support.

## Key Takeaways

- `pip install requests`, then `requests.get(url)` — the response object carries `.text`, `.content`, `.json()`, `.status_code`, and `.headers`
- **`json=` sets `Content-Type: application/json` and serializes for you; `data=` form-encodes.** Picking the wrong one is the most common cause of a 400 from a REST API
- **requests never times out by default.** Always pass `timeout=` — a `(connect, read)` tuple is best
- Use a `Session` for more than one request to the same host: it reuses connections, persists cookies, and carries default headers
- Retries are not built in — mount an `HTTPAdapter` with a urllib3 `Retry` on your session
- requests is **feature-frozen** : no HTTP/2, no async. If you need either, [httpx](https://www.python-httpx.org/) has a near-identical API
- It fetches HTML but does not run JavaScript — a page built client-side comes back as an empty shell no matter what headers you send

## Installing requests

```bash
pip install requests
python -c "import requests; print(requests. __version__ )"
```

Inside a virtual environment, which is what you want for anything beyond a scratch script:

```bash
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install requests
```

Two optional extras matter in practice:

```bash
pip install "requests[socks]" # SOCKS4/SOCKS5 proxy support (pulls in PySocks)
pip install brotli # lets requests accept and decode brotli responses
```

**`ModuleNotFoundError: No module named 'requests'`** almost always means you installed into a different interpreter than the one running your script. `python -m pip install requests` pins the install to the interpreter you just invoked and resolves it nine times out of ten. On Linux, prefer a venv over `sudo pip install` — installing into the system Python is how you break your package manager.

## Quick start: the response object

```python
import requests

resp = requests.get("https://httpbin.org/get", timeout=10)

resp.status_code # 200
resp.ok # True for any status below 400
resp.headers["Content-Type"]
resp.text # body decoded to str
resp.content # body as raw bytes
resp.json() # parsed JSON (raises if the body isn't JSON)
resp.url # final URL, after any redirects
resp.encoding # the codec used to produce .text
```

Every HTTP verb has a module-level shortcut, and all of them accept the same keyword arguments:

```python
requests.get(url, params={"q": "python"})
requests.post(url, json={"name": "Jane"})
requests.put(url, json=payload)
requests.patch(url, json={"status": "active"})
requests.delete(url, headers={"Authorization": f"Bearer {token}"})
requests.head(url) # headers only, no body
requests.options(url)
```

DELETE is worth calling out because APIs disagree about it: some want the identifier in the path, some want query parameters, and some accept a JSON body (which requests will happily send, though not every server reads it). All three forms work:

```python
requests.delete(f"https://api.example.com/items/{item_id}", timeout=10)
requests.delete("https://api.example.com/items", params={"id": item_id}, timeout=10)
requests.delete("https://api.example.com/items", json={"ids": [1, 2, 3]}, timeout=10)
```

### `.text` vs `.content` vs `.json()` vs `.raw`

These four accessors are the single most searched-about part of the library, and the differences are real:

| Accessor | Returns | Use for |
| --- | --- | --- |
| `.text` | `str`, decoded using `.encoding` | HTML, JSON you want to log, plain text |
| `.content` | `bytes`, decompressed but not decoded | Images, PDFs, zip files, anything binary |
| `.json()` | Parsed Python object | JSON API responses |
| `.raw` | The underlying urllib3 stream | Byte-exact proxying (needs `stream=True`) |

`.text` and `.content` are the same bytes; `.text` just runs them through a codec. Writing `.text` to a file in binary mode, or `.content` to a text file, is the usual source of corrupted downloads — **always use `.content` with `open(path, "wb")`.**

`.raw` is the escape hatch, and it has a gotcha: it hands you the _undecompressed_ stream, so a gzipped response reads back as gzip bytes unless you ask for decoding:

```python
resp = requests.get(url, stream=True, timeout=10)
data = resp.raw.read(decode_content=True) # without decode_content you get gzip bytes
```

## Query parameters

Pass a dict as `params` and requests handles URL-encoding, including spaces, unicode, and reserved characters:

```python
resp = requests.get("https://httpbin.org/get", params={
    "q": "web scraping",
    "page": 2,
    "safe": True, # becomes safe=True
}, timeout=10)

print(resp.url) # https://httpbin.org/get?q=web+scraping&page=2&safe=True
```

Repeated keys — common in filter APIs — come from a list value or a list of tuples:

```python
requests.get(url, params={"tag": ["python", "http"]}) # ?tag=python&tag=http
requests.get(url, params=[("tag", "python"), ("tag", "http")])
```

Values of `None` are dropped entirely, which makes optional parameters easy to express without building the dict conditionally. If you need byte-exact control over encoding or ordering, build the query string yourself with `urllib.parse.urlencode` and append it — requests will not re-encode a query string that is already in the URL.

## POST: `json` vs `data`, and why it matters

This is the distinction that generates the most confused bug reports. The two parameters send _different content types_, and an API that wants one will reject the other:

| Parameter | Content-Type sent | Body format |
| --- | --- | --- |
| `data={...}` | `application/x-www-form-urlencoded` | `key=value&key2=value2` |
| `json={...}` | `application/json` | `{"key": "value"}` |
| `data="raw string"` | **none set** | The string, verbatim |
| `files={...}` | `multipart/form-data` | MIME multipart |

```python
# Form submission — an HTML <form> posting to a server
requests.post("https://example.com/login",
              data={"username": "jane", "password": "secret"}, timeout=10)

# REST API — JSON body, header set automatically
requests.post("https://api.example.com/users",
              json={"name": "Jane", "roles": ["admin"]}, timeout=10)
```

Three rules that resolve most of the confusion:

1. **Don't set `Content-Type` yourself when using `json=`.** requests already set it; overriding with your own header dict can leave you sending a mismatched pair.
2. **Don't `json.dumps()` into `data=`.** That sends a JSON string as a form body with the wrong content type. If you genuinely need to control serialization (custom encoder, exact key order), pass the dumped string to `data=` _and_ set the header explicitly: `python
requests.post(url, data=json.dumps(payload, default=str),
             headers={"Content-Type": "application/json"}, timeout=10)
`
3. **Don't pass both.** `data` wins and your JSON is silently ignored.

A dict passed to `data=` is form-encoded with the same repeated-key rules as `params`. Bytes and strings pass through untouched — useful for posting XML, NDJSON, or a pre-signed payload.

When a request isn't doing what you expect, prepare it without sending and look at what would go over the wire:

```python
prepared = requests.Request("POST", url, json={"a": 1}).prepare()
print(prepared.headers) # {'Content-Length': '8', 'Content-Type': 'application/json'}
print(prepared.body) # b'{"a": 1}'
```

## Headers and the User-Agent

```python
headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}
resp = requests.get(url, headers=headers, timeout=10)
```

Left alone, requests announces itself as `python-requests/2.x`. Plenty of sites block that string outright, which is why a custom `User-Agent` is the first fix people try when a page returns 403 in Python but loads fine in a browser. Send a realistic browser string _along with_ the headers a browser would send — `Accept`, `Accept-Language`, `Referer` — because a Chrome User-Agent paired with no `Accept-Language` is itself a fingerprint.

Header handling details worth knowing: header names are case-insensitive on both request and response (`resp.headers["content-type"]` works), per-request headers merge with session headers rather than replacing them, and setting a session header to `None` removes it for that request.

## Sessions: connections, cookies, and defaults

A `Session` is the right default for anything past a single call. It gives you three things at once:

```python
import requests

with requests.Session() as session:
    session.headers.update({"User-Agent": "my-scraper/1.0"})
    session.params = {"api_key": API_KEY} # sent on every request

    session.post("https://example.com/login",
                 data={"user": "jane", "pass": "secret"}, timeout=10)

    # the login cookie is carried automatically
    profile = session.get("https://example.com/account", timeout=10)
```

1. **Connection reuse.** The underlying TCP and TLS connection is kept alive between calls, so the second request to a host skips the handshake — worth hundreds of milliseconds on HTTPS. This is what "persistent connections" and "keep-alive" mean in requests: they're the default _inside_ a session and impossible _without_ one.
2. **Cookie persistence.** Cookies the server sets are stored in `session.cookies` and replayed on subsequent requests to matching domains. This is what makes login-then-scrape flows work.
3. **Configuration persistence.** `headers`, `params`, `auth`, `proxies`, `verify`, and `cert` set on the session apply to every request through it.

Module-level `requests.get()` creates and discards a session for each call, so it gets none of the above. For a loop of 100 requests to one host, the session version is typically several times faster, and the gap widens with TLS.

**The one trap:** `Session` has no `timeout` attribute. `session.timeout = 30` assigns an ignored attribute and every request still hangs forever. Pass `timeout=` per call, or wrap the session:

```python
class TimeoutSession(requests.Session):
    def request(self, *args, **kwargs):
        kwargs.setdefault("timeout", (5, 30))
        return super().request(*args, **kwargs)
```

### Cookies

```python
# Send cookies with a one-off request
requests.get(url, cookies={"session_id": "abc123"}, timeout=10)

# Read cookies the server set
resp.cookies["session_id"]
resp.cookies.get_dict()

# Set one on a session, scoped to a domain and path
session.cookies.set("consent", "yes", domain="example.com", path="/")
```

To carry a login across program runs, persist the jar. JSON keeps it readable and avoids the security problems of unpickling data you don't fully control:

```python
import json

with open("cookies.json", "w") as f:
    json.dump(requests.utils.dict_from_cookiejar(session.cookies), f)

with open("cookies.json") as f:
    session.cookies = requests.utils.cookiejar_from_dict(json.load(f))
```

## Timeouts

**requests has no default timeout.** A server that accepts your connection and then goes silent will hang your script indefinitely — no exception, no traceback, just a process that never finishes. This is the single most important argument in the library:

```python
requests.get(url, timeout=10) # 10s for connect, 10s for read
requests.get(url, timeout=(3.05, 27)) # (connect, read) — preferred
```

The two halves bound different failures. _Connect_ covers DNS plus the TCP and TLS handshake, so an unreachable host fails fast. _Read_ bounds the wait **between bytes** , not the total transfer — a slow but steady 2 GB download never trips a 30-second read timeout, while a server that sends headers and then stalls trips it immediately. There is no built-in total-duration cap; if you need one, enforce it in your own loop or at the scheduler level.

Timeouts raise `requests.exceptions.ConnectTimeout` and `ReadTimeout`, both subclasses of `Timeout`, itself a `RequestException`.

## Retries with backoff

requests deliberately ships with retries disabled. You add them by mounting an `HTTPAdapter` configured with urllib3's `Retry`:

```python
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=5,
    backoff_factor=0.5, # 0.5s, 1s, 2s, 4s, 8s
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD", "OPTIONS"], # don't blind-retry POSTs
    respect_retry_after_header=True,
    raise_on_status=False,
)

session = requests.Session()
adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)
session.mount("http://", adapter)

resp = session.get("https://api.example.com/data", timeout=(3, 27))
```

`respect_retry_after_header=True` doubles as polite rate-limit handling: on a 429 or 503 carrying `Retry-After`, urllib3 waits exactly as long as the server asked rather than following its own backoff curve. Leave `allowed_methods` at its default (idempotent methods only) unless you know the endpoint is safe to repeat — retrying a POST can create duplicate orders.

Two version notes: `allowed_methods` was called `method_whitelist` in urllib3 1.x, and `backoff_factor` sleeps `backoff_factor * (2 ** (attempt - 1))`, so `0.5` is a gentler curve than it looks. When retries are exhausted you get a `requests.exceptions.RetryError` (or `ConnectionError` wrapping urllib3's `MaxRetryError`, depending on what failed).

`Retry` covers transport-level failures and status codes. It does **not** help with application-level errors — a 200 response containing `{"error": "rate limited"}` needs your own loop, ideally with jitter so parallel workers don't retry in lockstep:

```python
import random, time

for attempt in range(5):
    resp = session.get(url, timeout=(3, 27))
    if resp.ok and "error" not in resp.json():
        break
    time.sleep(min(2 ** attempt, 30) + random.uniform(0, 1))
```

## Connection pooling

Under a session, each host gets a urllib3 connection pool. The defaults suit sequential scripts and are too small for threaded work — tune them on the adapter:

```python
adapter = HTTPAdapter(
    pool_connections=20, # number of distinct host pools cached
    pool_maxsize=20, # connections kept open per host
    pool_block=False, # True = wait for a free connection instead of opening extras
    max_retries=retry,
)
session.mount("https://", adapter)
```

For threaded scraping of a single host, set `pool_maxsize` to roughly your worker count. Left at the default of 10 with 50 threads, urllib3 opens extra connections and discards them after use, emitting `Connection pool is full, discarding connection` warnings and quietly throwing away the benefit of pooling. `pool_block=True` turns the pool into a hard cap, which is what you want when a target explicitly limits concurrent connections.

A `Session` object is thread-safe for sharing across threads in the common case, but it isn't guaranteed for concurrent mutation — don't reassign `session.headers` or `session.cookies` from multiple threads. One session per thread, or one shared read-only session, both work.

## Redirects and `allow_redirects`

requests follows redirects automatically for every method except `HEAD`:

```python
resp = requests.get("http://github.com", timeout=10)
resp.url # 'https://github.com/' — the final destination
resp.history # [<Response [301]>] — every hop, in order
resp.history[0].headers["Location"]
```

Set `allow_redirects=False` to get the 3xx response itself instead of the destination:

```python
resp = requests.get(url, allow_redirects=False, timeout=10)
resp.status_code # 302
resp.headers["Location"] # where it wanted to send you
```

Reasons to turn it off: checking whether a shortened URL is malicious before visiting it, auditing a site's redirect chain, capturing a `Set-Cookie` header that only appears on the redirect response, or reading a `Location` that contains an OAuth code you need. `HEAD` is the reverse case — it defaults to _not_ following, so `requests.head(url, allow_redirects=True)` is how you resolve a final URL without downloading the body.

Following a chain by hand is a short loop:

```python
url, chain = start_url, []
for _ in range(10):
    resp = requests.get(url, allow_redirects=False, timeout=10)
    chain.append((resp.status_code, url))
    if resp.status_code not in (301, 302, 303, 307, 308):
        break
    url = requests.compat.urljoin(url, resp.headers["Location"]) # Location may be relative
```

Two behaviors that surprise people. **Method changes on 301, 302, and 303:** a POST becomes a GET and the body is dropped, matching what browsers do. Only 307 and 308 preserve the method and body. **Authorization headers are stripped when a redirect crosses to a different host** , which is a deliberate security measure — if your credentials vanish mid-chain, that's why. A loop raises `TooManyRedirects` after 30 hops by default (`session.max_redirects`).

## Streaming, chunked encoding, and large downloads

By default requests downloads the whole body into memory before returning. `stream=True` returns as soon as headers arrive and leaves the body on the wire:

```python
with requests.get(url, stream=True, timeout=(5, 30)) as resp:
    resp.raise_for_status()
    with open("large-file.zip", "wb") as f:
        for chunk in resp.iter_content(chunk_size=8192):
            f.write(chunk)
```

Use `stream=True` for files too large to hold in memory, for responses you may want to abandon after inspecting the headers (checking `Content-Length` or `Content-Type` before committing to the download), and for endpoints that stream indefinitely. Avoid it for ordinary API calls — the connection stays checked out of the pool until you consume or close the response, and forgetting is a slow connection leak. **Use `with` or call `resp.close()`**; accessing `.text` or `.content` also drains and releases it.

`iter_content` handles chunked transfer encoding transparently. A server that sends `Transfer-Encoding: chunked` has no `Content-Length`, so you cannot know the total size up front — progress bars need a fallback:

```python
total = int(resp.headers.get("Content-Length", 0)) # 0 when chunked
```

For newline-delimited streams — log tails, NDJSON feeds, server-sent events — iterate lines instead:

```python
with requests.get(url, stream=True, timeout=(5, None)) as resp:
    for line in resp.iter_lines(decode_unicode=True):
        if line:
            event = json.loads(line)
```

Note `timeout=(5, None)`: a long-lived stream legitimately has gaps between messages, so a read timeout would kill it. Bound the connect phase and leave the read phase open. `iter_lines` is not reentrant and can split multi-byte characters across chunk boundaries — for anything where exactness matters, use `iter_content` and buffer yourself.

Downloading images and video is the same pattern, with a content-type check so an error page doesn't get saved as a `.jpg`:

```python
with requests.get(image_url, stream=True, timeout=(5, 30)) as resp:
    resp.raise_for_status()
    if not resp.headers.get("Content-Type", "").startswith("image/"):
        raise ValueError(f"expected an image, got {resp.headers.get('Content-Type')}")
    with open("photo.jpg", "wb") as f:
        for chunk in resp.iter_content(64 * 1024):
            f.write(chunk)
```

## Uploading files

Pass `files=` and requests builds the `multipart/form-data` body, boundary and all:

```python
with open("report.pdf", "rb") as f:
    requests.post(url, files={"document": f}, timeout=30)
```

Control the filename and MIME type with a tuple, and mix in regular form fields via `data=` — the two coexist in one multipart body:

```python
with open("report.pdf", "rb") as f:
    requests.post(url,
                  files={"document": ("q3-report.pdf", f, "application/pdf")},
                  data={"category": "finance", "year": "2026"},
                  timeout=30)
```

Multiple files under the same field name need a list of tuples, since a dict can't hold duplicate keys:

```python
files = [
    ("photos", ("a.jpg", open("a.jpg", "rb"), "image/jpeg")),
    ("photos", ("b.jpg", open("b.jpg", "rb"), "image/jpeg")),
]
requests.post(url, files=files, timeout=60)
```

In-memory data works too — pass `io.BytesIO(blob)` or the bytes directly instead of a file handle. Do **not** set `Content-Type` yourself here: requests generates a boundary token and puts it in the header, and a hand-written header without the matching boundary produces a body the server cannot parse.

One limit: requests reads each file fully into memory to build the body, so multi-gigabyte uploads need [requests-toolbelt](https://toolbelt.readthedocs.io/)'s `MultipartEncoder`, which streams instead.

## Authentication

```python
from requests.auth import HTTPBasicAuth, HTTPDigestAuth

requests.get(url, auth=("user", "pass"), timeout=10) # Basic, shorthand
requests.get(url, auth=HTTPBasicAuth("user", "pass"), timeout=10) # identical, explicit
requests.get(url, auth=HTTPDigestAuth("user", "pass"), timeout=10)
```

Bearer tokens and API keys are headers, not `auth`:

```python
session.headers["Authorization"] = f"Bearer {access_token}"
session.headers["X-API-Key"] = api_key
```

Custom schemes subclass `AuthBase`, which is how you implement request signing (HMAC, AWS SigV4-style) cleanly:

```python
from requests.auth import AuthBase

class TokenAuth(AuthBase):
    def __init__ (self, token):
        self.token = token

    def __call__ (self, request):
        request.headers["Authorization"] = f"Token {self.token}"
        return request

requests.get(url, auth=TokenAuth(token), timeout=10)
```

Read credentials from environment variables or a secret manager, never from a literal in the source. Basic auth in particular is base64, not encryption — it is only safe over HTTPS, and the `verify=False` shortcut discussed below defeats that protection entirely.

## Proxies

```python
proxies = {
    "http": "http://user:pass@proxy.example.com:8080",
    "https": "http://user:pass@proxy.example.com:8080",
}
requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)
session.proxies.update(proxies) # or set it once on the session
```

Both keys take the proxy's own scheme — an HTTP proxy tunneling HTTPS traffic via `CONNECT` is still `http://` in the value; the dict key is the _target's_ scheme. SOCKS needs the extra installed:

```bash
pip install "requests[socks]"
```

```python
proxies = {"https": "socks5h://user:pass@proxy.example.com:1080"}
```

Use `socks5h` rather than `socks5` for scraping: the `h` resolves DNS through the proxy, so your local resolver never sees the target hostname and geo-based DNS returns results for the proxy's location.

requests also reads `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` from the environment. That's convenient until it isn't — a stale variable in a CI environment routes traffic somewhere unexpected. `session.trust_env = False` disables environment lookup (along with netrc and `REQUESTS_CA_BUNDLE`) when you want fully explicit configuration.

Rotating proxies means swapping the dict per request, which requests supports but doesn't manage. Getting a pool worth rotating is the harder half; our [proxy provider comparison](https://webscraping.ai/blog/best-proxy-providers-for-web-scraping) covers the options, and a [scraping API](https://webscraping.ai/) removes the question by rotating server-side.

## SSL verification

Certificates are verified by default against the [certifi](https://pypi.org/project/certifi/) bundle. When you hit `SSLError: CERTIFICATE_VERIFY_FAILED`, work through the causes in order:

1. **Stale CA bundle** — `pip install -U certifi`. This fixes the majority of cases, especially on older machines.
2. **Corporate TLS-inspecting proxy or an internal CA** — point requests at the CA that actually signed the certificate: `python
requests.get(url, verify="/path/to/corporate-ca.pem", timeout=10)
session.verify = "/path/to/corporate-ca.pem" # or the REQUESTS_CA_BUNDLE env var
`
3. **Genuinely broken remote certificate** (expired, hostname mismatch) — decide consciously.

Client certificates for mutual TLS go in `cert`:

```python
requests.get(url, cert=("client.crt", "client.key"), timeout=10)
requests.get(url, cert="combined.pem", timeout=10) # cert and key in one file
```

Disabling verification is a last resort. `verify=False` makes the connection trivially interceptable — anyone on the path can read and rewrite the traffic, which means it is _not_ acceptable for anything carrying credentials:

```python
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
resp = requests.get(url, verify=False, timeout=10) # you are now MITM-able
```

The warning requests prints is doing its job; silencing it without fixing the underlying trust problem just hides a real vulnerability.

## Status codes and error handling

**requests does not raise on 4xx or 5xx.** A 500 is a perfectly valid response object, and `resp.json()` on an error page is where the confusing traceback usually comes from. Check explicitly:

```python
if resp.status_code == 200:
    ...
if resp.ok: # any status below 400
    ...
if resp.status_code == requests.codes.not_found: # 404, spelled readably
    ...
```

Or opt into exceptions with `raise_for_status()`, which raises `HTTPError` on 4xx and 5xx and returns `None` otherwise:

```python
resp = requests.get(url, timeout=10)
resp.raise_for_status()
data = resp.json()
```

The exception hierarchy roots at `RequestException`, so one `except` catches everything requests can throw:

```python
import requests
from requests.exceptions import (
    HTTPError, ConnectionError, Timeout, TooManyRedirects,
    SSLError, ProxyError, JSONDecodeError, RequestException,
)

try:
    resp = requests.get(url, timeout=(3.05, 27))
    resp.raise_for_status()
    data = resp.json()
except Timeout:
    log.warning("timed out — safe to retry")
except HTTPError as e:
    log.error("HTTP %s: %s", e.response.status_code, e.response.text[:200])
except JSONDecodeError:
    log.error("expected JSON, got %s", resp.headers.get("Content-Type"))
except ConnectionError:
    log.error("DNS failure, refused connection, or dropped socket")
except RequestException as e:
    log.exception("unexpected requests failure: %s", e)
```

Order matters: catch the specific classes before `RequestException`, or the base clause swallows everything. Useful members of the family beyond the above: `ConnectTimeout` and `ReadTimeout` (both `Timeout`), `ChunkedEncodingError` and `ContentDecodingError` for malformed response bodies, and `MissingSchema` / `InvalidURL` for a bad URL — that last pair catches the classic "I forgot the `https://`".

`HTTPError` carries the response on `e.response`, which is what lets you log the server's actual error message instead of a generic status line.

## Compression and encoding

requests advertises `Accept-Encoding: gzip, deflate` on every request and decompresses transparently, so `.content` and `.text` are always the decoded bytes. Add `brotli` or `zstandard` to the environment and those encodings are negotiated too. Two consequences catch people out: `resp.headers["Content-Encoding"]` still reports `gzip` on a body you're reading as plain text, and `len(resp.content)` will not match the `Content-Length` header, which describes the compressed transfer.

Character encoding is a separate layer. `.text` decodes using `resp.encoding`, which requests takes from the `Content-Type` charset. When the header is a bare `text/html` with no charset, HTTP's rules say ISO-8859-1 — and plenty of pages that are actually UTF-8 send exactly that, producing mojibake like `é` where `é` belongs. The fix is to override:

```python
resp.encoding = "utf-8" # you know what it is
resp.encoding = resp.apparent_encoding # let charset detection decide (slower)
text = resp.text
```

`apparent_encoding` runs the body through `charset_normalizer` and is a good fallback for scraping pages of unknown origin, but it costs a scan of the body. For a POST that carries non-ASCII text, encode it yourself and declare the charset so the server doesn't have to guess:

```python
requests.post(url, data="naïve=café".encode("utf-8"),
              headers={"Content-Type": "application/x-www-form-urlencoded; charset=utf-8"},
              timeout=10)
```

## Measuring response time

`resp.elapsed` is a `timedelta` requests populates for every response:

```python
resp = requests.get(url, timeout=10)
print(resp.elapsed.total_seconds())
```

Read it precisely: it covers sending the request through parsing the response headers, and **not** the body download. For a small JSON response the difference is negligible; for a 200 MB file it's most of the wall-clock time. To measure the full transfer, time it yourself:

```python
import time

start = time.perf_counter()
resp = requests.get(url, timeout=10)
_ = resp.content # force the body to be read
total = time.perf_counter() - start
```

With redirects, `resp.elapsed` reflects only the final hop — sum `[r.elapsed for r in resp.history] + [resp.elapsed]` for the whole chain.

## Debugging

Response inspection first, since it answers most questions:

```python
resp.request.method, resp.request.url
resp.request.headers # what was actually sent, after session merging
resp.request.body
resp.status_code, resp.headers, resp.history
```

For a running log of connections, retries, and redirects, turn up urllib3's logger — requests itself logs almost nothing:

```python
import logging

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG)
```

Event hooks let you instrument every response through a session without touching call sites:

```python
def log_response(resp, *args, **kwargs):
    print(f"{resp.request.method} {resp.url} -> {resp.status_code} "
          f"in {resp.elapsed.total_seconds():.2f}s")

session.hooks["response"].append(log_response)
```

For full wire-level dumps including bodies, reach for [mitmproxy](https://mitmproxy.org/) or Wireshark — requests will not print raw traffic. And when you're comparing behavior against a shell one-liner, our [curl guide](https://webscraping.ai/blog/curl-commands-for-web-scraping) maps the flags across; `curl -v` is often the fastest way to establish whether a problem is in your Python or in the server.

Testing HTTP code without hitting the network is its own small ecosystem: [responses](https://github.com/getsentry/responses) and [requests-mock](https://requests-mock.readthedocs.io/) both intercept at the adapter layer, so your code under test calls `requests` normally while the library returns canned responses. Both are better than patching `requests.get` with `unittest.mock`, which stops testing the request you actually build.

## requests vs urllib3 vs httpx

| | requests | urllib3 | httpx |
| --- | --- | --- | --- |
| API style | High-level, ergonomic | Low-level, explicit | requests-compatible |
| HTTP/2 | No | No | Yes (`pip install httpx[http2]`) |
| Async | No | No | Yes (`AsyncClient`) |
| Connection pooling | Via `Session` | Native (`PoolManager`) | Native |
| Retries | Bolt-on (`HTTPAdapter`) | Built in (`Retry`) | Via transport config |
| Timeout default | **None** | None | 5 seconds |
| Development | Maintenance mode | Active | Active |
| Dependency weight | urllib3, certifi, idna, charset-normalizer | Minimal | httpcore, h11, certifi |

**requests is in maintenance mode** — it receives security and compatibility fixes, not new protocol features. HTTP/2 and async are not coming; that was a deliberate decision by the maintainers, not a backlog item. This is fine for the overwhelming majority of code. HTTP/1.1 with connection reuse is fast, and threads handle concurrency adequately up to a few dozen workers.

Switch to **httpx** when you need HTTP/2 multiplexing, `async`/`await`, or a sane default timeout; its API is close enough that porting is usually mechanical (`requests.get` → `httpx.get`, `Session` → `Client`). Drop to **[urllib3](https://webscraping.ai/blog/python-urllib3-guide)** when you're writing a library that shouldn't pull in requests' dependency tree, or you need pool-level control requests doesn't expose. For everything else, requests being frozen is a feature — the code you write today keeps working.

## Using requests for web scraping

For static HTML, requests plus a parser is the whole stack. [Beautiful Soup](https://webscraping.ai/blog/beautiful-soup-python) is the usual partner:

```python
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

resp = requests.get("https://example.com/articles",
                    headers={"User-Agent": "Mozilla/5.0 ..."}, timeout=10)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "html.parser")
links = [urljoin(resp.url, a["href"]) for a in soup.select("a[href]")]
```

`urljoin` against `resp.url` rather than the URL you requested is the detail that matters — after a redirect, relative links resolve against the _final_ URL, and joining against the original silently produces broken links. Combine this with a session, a realistic User-Agent, timeouts, and a `Retry` adapter and you have a scraper that survives contact with production. For XML feeds, [lxml or ElementTree](https://webscraping.ai/blog/python-xml-parsing) take the place of Beautiful Soup; the fetch half is identical. Our [Python web scraping guide](https://webscraping.ai/blog/web-scraping-with-python) covers the pipeline end to end.

Then there's the wall. **requests does not execute JavaScript.** It fetches the HTML the server sent and stops there, so a React, Vue, or Angular page comes back as a nearly empty `<div id="root">` with the real content nowhere in `resp.text`. No combination of headers changes this — the data is assembled by code that never runs. The same is true of anti-bot systems that fingerprint the TLS handshake and JavaScript environment: they aren't reading your `User-Agent`, so a better one won't help.

The options are running a real browser (Playwright, Selenium — heavy, but complete), reverse-engineering the JSON API the page calls (fast when it works, brittle when the site changes), or sending the fetch to a service that renders for you. [WebScraping.AI](https://webscraping.ai/) does the last one behind a single request you make with requests, leaving the rest of your code untouched:

```python
import requests

resp = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/spa-products",
    "js": "true", # rendered in a real browser, proxies included
}, timeout=60)
html = resp.text
```

Your session, retries, and error handling all still apply — it's an ordinary HTTPS endpoint. The [/ai/fields endpoint](https://webscraping.ai/ai-web-scraping) goes a step further and returns structured JSON from a plain-English description of the data you want, which is what keeps a scheduled job like [price monitoring](https://webscraping.ai/use-cases/price-monitoring) running through site redesigns instead of breaking on every selector change. The [API reference](https://webscraping.ai/docs) lists the other parameters (`proxy`, `country`, `timeout`, `device`, `wait_for`).

Note the longer client-side `timeout` above: rendering a page takes seconds, not milliseconds, so budget for it.

## Frequently asked questions

**What's the difference between `json=` and `data=` in requests?** `json=` serializes your object with `json.dumps()` and sets `Content-Type: application/json`. `data=` form-encodes a dict as `application/x-www-form-urlencoded`, the format an HTML form posts. Use `json=` for REST APIs and `data=` for form submissions. Passing a `json.dumps()` string to `data=` sends JSON with the wrong content type, which is the usual cause of an unexplained 400.

**Why does my requests script hang forever?** Because there is no default timeout. A server that accepts the connection and then stops responding will block indefinitely. Always pass `timeout=`, ideally as a `(connect, read)` tuple like `(3.05, 27)`. Note that `session.timeout = 30` does nothing — `Session` has no such attribute; the timeout must be passed per request or injected by subclassing `Session.request`.

**Does requests support HTTP/2?** No, and it won't. requests is built on urllib3, which is HTTP/1.1 only, and the project is in maintenance mode. Use [httpx](https://www.python-httpx.org/) with `pip install httpx[http2]` and `httpx.Client(http2=True)` if you need it. In practice, HTTP/2 rarely matters for scraping or API work — connection reuse through a `Session` captures most of the same benefit.

**How do I stop requests from following redirects?** Pass `allow_redirects=False` to get the 3xx response with its `Location` header instead of the final destination. Everything except `HEAD` follows redirects by default; `HEAD` is the opposite and needs `allow_redirects=True` to follow. `resp.history` holds the full chain when redirects are followed, and `resp.url` is where you ended up.

**When should I use `stream=True`?** When the body is too large to hold in memory, when you want to inspect headers before deciding to download, or when the endpoint streams indefinitely. Iterate with `iter_content(chunk_size=8192)` for binary or `iter_lines()` for newline-delimited data, and always use a `with` block — a streamed response holds its connection out of the pool until closed.

**Why is my scraped page missing the content I see in the browser?** The page renders that content with JavaScript. requests returns the initial HTML only, so anything a framework builds client-side is absent, and no header changes that. You need a headless browser, the underlying JSON API the page calls, or a rendering API — see the scraping section above.

**How do I add retries to requests?** Mount an `HTTPAdapter(max_retries=Retry(...))` on a `Session`. urllib3's `Retry` gives you `total`, `backoff_factor` for exponential backoff, `status_forcelist` for which statuses to retry, and `respect_retry_after_header` so 429 responses wait exactly as long as the server asks. Keep `allowed_methods` restricted to idempotent verbs unless you're certain repeating a POST is safe.

**Is `requests.Session` faster than `requests.get()`?** Yes, materially, for repeated calls to the same host. A session reuses the TCP and TLS connection, so every request after the first skips the handshake — typically hundreds of milliseconds on HTTPS. `requests.get()` builds and discards a session per call and gets none of that, nor cookie or header persistence.