---
title: "HTTP Headers for Web Scraping: Anatomy of a Real Browser Request"
description: "Every header a real browser sends, how to match them in Python, Node, and curl, plus compression, chunked encoding, auth, TLS certificates and 403 fixes."
url: https://webscraping.ai/blog/http-headers-for-web-scraping
markdown_index: https://webscraping.ai/llms.txt
---
# HTTP Headers for Web Scraping: Anatomy of a Real Browser Request

A browser and a scraper can request the same URL and get different pages back. The difference is almost never the URL — it is the dozen or so HTTP headers the browser attaches to every request and your HTTP client does not. Some of those headers change what the server sends you (compression, language, caching); others are what anti-bot systems read to decide whether you are a person. This guide covers the full request surface: which headers a real Chrome sends in 2026 and in what order, how to reproduce them in Python, Node, and curl, how to decode what comes back, and where header spoofing stops working no matter how carefully you do it.

## Key Takeaways

- A default `python-requests/2.34` request is identifiable in one line; a browser-shaped request needs about eight headers, not one `User-Agent`
- Consistency beats completeness — a Chrome `User-Agent` next to Firefox's `Accept` header is a stronger bot signal than sending no headers at all
- Over HTTP/2 header names are lowercase and `Connection` is illegal; copying an HTTP/1.1 header dump verbatim can itself be the tell
- Declare only the compressions you can actually decode — adding `br, zstd` to `Accept-Encoding` without the decoder installed returns binary garbage
- Header spoofing does not touch the TLS handshake, so a perfect header set still carries a Python JA3/JA4 fingerprint
- `403` means your request looked wrong, `429` means it came too fast — they need opposite fixes

## Why headers decide whether you get blocked

An HTTP request is a method, a path, and a list of headers. The server has nothing else to judge you by before it decides what to return. Bot detection at the request layer asks three questions:

1. **Is anything obviously automated?** Default clients announce themselves: `python-requests/2.34.2`, `curl/8.7.1`, `Go-http-client/2.0`, `axios/1.7.2`. A single string is enough for a WAF rule.
2. **Is the header set complete?** Real browsers send `Accept`, `Accept-Language`, `Accept-Encoding`, `Sec-Fetch-*`, and Client Hints on every navigation. A request with only `User-Agent` is a scraper that read one tutorial.
3. **Is it internally coherent?** This is the one that catches careful scrapers. A `User-Agent` claiming Chrome 150 on Windows, with Firefox's `Accept` value, no `Sec-CH-UA`, and headers in alphabetical order describes a browser that does not exist.

The third check is why "add a user agent" advice stopped working. Detection systems compare the _whole_ request against a profile of what that browser actually sends. Our [user agent guide](https://webscraping.ai/blog/user-agent-rotation-for-web-scraping) covers the `User-Agent` string itself and its rotation in depth — this guide covers everything around it.

## Anatomy of a real browser request

Here is a top-level navigation from Chrome on Windows, in the order Chrome sends it. The order is part of the fingerprint, so it is worth reading as a sequence rather than a set:

```http
GET /products HTTP/1.1
Host: example.com
Connection: keep-alive
sec-ch-ua: "Chromium";v="150", "Google Chrome";v="150", "Not?A_Brand";v="24"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
```

What each one does, and what it costs you to get it wrong:

| Header | What it means | Why a scraper needs it |
| --- | --- | --- |
| `Host` | Target hostname | Set by every client automatically; only matters when you address an IP directly |
| `User-Agent` | Client identity string | The first thing filtered on. Must be a real, current string |
| `Accept` | Content types you'll take, with quality values | Servers content-negotiate on it; a wrong value can return JSON or XML instead of HTML |
| `Accept-Language` | Preferred languages | Controls localization. Mismatching it with your proxy country is a detectable inconsistency |
| `Accept-Encoding` | Compressions you can decode | Omitting it wastes bandwidth and looks unusual; overclaiming it corrupts your data |
| `Referer` | The page you came from | Deep pages reached with no `Referer` look like direct hits nobody makes |
| `Cookie` | Session state | Required after login, consent walls, or a JS challenge |
| `Upgrade-Insecure-Requests` | Will accept an HTTPS upgrade | Sent by browsers on navigations, almost never by scrapers |
| `Sec-Fetch-Site` | Origin relationship: `none`, `same-origin`, `same-site`, `cross-site` | `none` for typed URLs; `same-origin` for in-site links |
| `Sec-Fetch-Mode` | `navigate` for pages, `cors` for XHR/fetch | Must match what you're pretending to be |
| `Sec-Fetch-Dest` | `document`, `empty`, `image`, `script`… | `document` for pages, `empty` for API calls |
| `Sec-Fetch-User` | `?1` — a human triggered this | Only present on user-initiated navigations |
| `sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform` | Low-entropy Client Hints | Chromium sends these unprompted; their version must agree with the `User-Agent` |
| `Priority` | Stream priority (`u=0, i`) | Sent on HTTP/2 and HTTP/3; a minor but real signal |

The `Sec-*` headers are the ones scrapers most often miss. They are _forbidden headers_ in browser JavaScript — a page cannot set them, so a server can trust that they came from the browser itself. That makes them cheap and reliable to check, and it is why a request without them stands out.

The Client Hints (`sec-ch-ua*`) deserve their own note: Chrome's User-Agent Reduction froze most of the detail in the `User-Agent` string and moved it here, so a scraper that only spoofs `User-Agent` is now missing information a real Chrome always volunteers. The [user agent guide](https://webscraping.ai/blog/user-agent-rotation-for-web-scraping) has the current strings and matching hint values.

### Getting the real thing instead of copying a table

Header sets drift with every Chrome release. Rather than trusting any published list — including this one — read them off a live browser:

1. Open DevTools (F12) → **Network**
2. Load the target page
3. Right-click the document request → **Copy** → **Copy as cURL**

You get the exact request your browser made, headers and cookies included, ready to replay in a terminal. Our [curl for web scraping guide](https://webscraping.ai/blog/curl-commands-for-web-scraping) covers turning that command into something reusable, and the [curl converter](https://webscraping.ai/tools/curl-converter) translates it straight into Python, JavaScript, PHP, or Go.

Two habits make this worth repeating rather than doing once: capture from the same _kind_ of navigation you are automating (a link click sends different `Sec-Fetch-*` values than a typed URL), and re-capture when a scraper that worked for months starts returning 403s.

## Sending headers correctly

Every HTTP client takes a dict of headers. The gotchas are in what the client does _around_ your dict.

**Python (requests)** — a `Session` keeps headers, cookies, and the connection pool across requests:

```python
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,"
              "image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate", # only what you can decode
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-User": "?1",
    "Sec-Fetch-Dest": "document",
    "sec-ch-ua": '"Chromium";v="150", "Google Chrome";v="150", "Not?A_Brand";v="24"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
}

session = requests.Session()
session.headers.update(HEADERS)

response = session.get("https://example.com/products", timeout=15)
response.raise_for_status()
```

`Session.headers` is a case-insensitive dict, so assigning `User-Agent` replaces the default rather than adding a second one. Setting a key to `None` removes a default header entirely — the only way to stop requests from sending `Accept-Encoding` at all. The full picture of sessions, retries, and timeouts is in our [Python requests guide](https://webscraping.ai/blog/python-requests-guide).

**Node.js** — built-in `fetch` (undici) speaks HTTP/1.1 by default:

```javascript
const res = await fetch("https://example.com/products", {
  headers: {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
                  "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Dest": "document",
  },
  signal: AbortSignal.timeout(15000),
});
const html = await res.text(); // undici decompresses gzip/br automatically
```

**curl** — `-H` per header, `--compressed` to request and decode compression in one flag:

```bash
curl --compressed \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Dest: document' \
  https://example.com/products
```

Add `-v` to see exactly what went over the wire, including the headers curl added on your behalf.

### Order, casing, and the HTTP/2 trap

Header **order** is stable per browser and is fingerprinted. Python dicts have preserved insertion order since 3.7, and requests sends session headers first, then per-request ones — so define your dict in browser order and it will mostly survive. "Mostly" is the honest word here: clients add `Host`, `Connection`, and `Content-Length` at positions you do not control.

Header **casing** matters more than people expect. HTTP/1.1 header names are case-insensitive by specification, but HTTP/2 and HTTP/3 require them to be **lowercase** on the wire, and Chrome sends `sec-ch-ua` lowercase even over HTTP/1.1 — a detail worth copying exactly.

This creates a trap when replaying a captured request: over HTTP/2 the pseudo-headers (`:method`, `:authority`, `:scheme`, `:path`) come first, `:authority` takes the place of `Host` (browsers don't send `Host` at all), and `Connection`, `Keep-Alive`, `Transfer-Encoding`, and `Upgrade` are **prohibited connection-specific headers**. Sending `Connection: keep-alive` on an HTTP/2 request is a protocol violation that some servers reject outright and any fingerprinter notices instantly, because no browser does it. If your client negotiates HTTP/2 (httpx with `http2=True`, curl by default on HTTPS), drop `Connection` from your header set.

### Where header spoofing stops working

Be clear about what this buys you. Headers are the application layer; detection now runs below it.

- **TLS fingerprinting (JA3/JA4).** Your cipher suite list, extension order, and supported curves are chosen by your TLS library — OpenSSL via Python, BoringSSL in Chrome — and they differ. A perfect Chrome header set arriving on a Python TLS handshake is a contradiction visible before a single header is parsed. Libraries like `curl_cffi` (Python) and `tls-client` (Go) exist specifically to impersonate browser handshakes.
- **HTTP/2 fingerprinting.** SETTINGS frame values, window sizes, and pseudo-header order form an "Akamai fingerprint" that is also library-specific.
- **IP reputation.** A datacenter IP with a flawless browser profile still looks like a datacenter. See our [proxy types guide](https://webscraping.ai/blog/types-of-proxies-for-web-scraping) for what the different pools actually change.
- **Behaviour and JavaScript challenges.** Cloudflare Turnstile, DataDome, and PerimeterX run JS that measures canvas rendering, timing, and event patterns. No header set answers that.

Headers are necessary and not sufficient. They get you past simple WAF rules and header-completeness checks, which is a large share of ordinary sites — and none of the way past a managed anti-bot product.

## Accept-Encoding and compression

`Accept-Encoding` advertises which compressions you can decode; the server picks one and names it in `Content-Encoding`. Getting HTML back compressed is normal — text compresses 70–90%, so this is the single biggest bandwidth lever in scraping.

| Encoding | Notes |
| --- | --- |
| `gzip` | Universal, supported by every stdlib |
| `deflate` | Historically ambiguous (zlib vs raw); still widely accepted |
| `br` (Brotli) | ~15–20% better than gzip on HTML; needs an extra package in Python |
| `zstd` | Chrome 123+ advertises it; fast, but decoder support in HTTP clients is newest |

Most clients decompress transparently: requests handles gzip and deflate out of the box (and Brotli if `brotli`/`brotlicffi` is installed), undici handles gzip and Brotli, curl does it with `--compressed`.

**The failure mode to avoid** : copying `Accept-Encoding: gzip, deflate, br, zstd` out of DevTools into a client that cannot decode `br` or `zstd`. The server takes you at your word, and `response.text` becomes binary noise. Either install the decoders or advertise less:

```python
# Safe: matches what a stock requests install can decode
headers = {"Accept-Encoding": "gzip, deflate"}

# Full browser parity: pip install brotli zstandard (urllib3 2.x uses them)
headers = {"Accept-Encoding": "gzip, deflate, br, zstd"}
```

To decode manually — when you are reading a raw socket, a saved response, or an unusual client:

```python
import gzip, zlib, brotli

# response fetched with stream=True, before .content/.text is touched
raw = response.raw.read(decode_content=False) # undecoded bytes
enc = response.headers.get("Content-Encoding", "").lower()

if enc == "gzip":
    body = gzip.decompress(raw)
elif enc == "deflate":
    try:
        body = zlib.decompress(raw) # zlib-wrapped
    except zlib.error:
        body = zlib.decompress(raw, -zlib.MAX_WBITS) # raw deflate
elif enc == "br":
    body = brotli.decompress(raw)
else:
    body = raw

html = body.decode(response.encoding or "utf-8", errors="replace")
```

Two details that cause silent bugs. First, `Content-Encoding` can list **multiple** encodings applied in order (`br, gzip`), and you must undo them right to left. Second, `Content-Encoding` and `Transfer-Encoding` are different mechanisms: the first compresses the payload end to end, the second describes how the message was framed for this hop. `Content-Length`, when present, describes the _compressed_ size — which is why comparing it to `len(response.text)` never matches.

## Chunked transfer encoding

`Transfer-Encoding: chunked` is how HTTP/1.1 sends a body of unknown length. The server omits `Content-Length` and emits a sequence of size-prefixed chunks terminated by a zero-length one:

```http
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked

17
<html><head><title>Page
16
</title></head></html>
0
```

You will see it on server-rendered pages streamed as they are generated, on long-running APIs, and on log or event endpoints. **Every mainstream HTTP client reassembles chunks for you** — this is a transport detail, not something you normally parse. `response.text` is already the joined body.

It matters in three situations:

**Streaming large responses.** Do not materialize a 500 MB export in memory:

```python
with session.get(url, stream=True, timeout=30) as r:
    r.raise_for_status()
    with open("export.csv", "wb") as f:
        for chunk in r.iter_content(chunk_size=64 * 1024):
            f.write(chunk)
```

`iter_content` yields decompressed bytes in whatever sizes arrive; the chunk boundaries you get are not the HTTP chunk boundaries, and no code should depend on them. Note that `stream=True` holds a connection out of the pool until the body is consumed or the response is closed — the `with` block is what prevents a slow leak.

**Processing lines as they arrive** , for streaming JSON or server-sent events:

```python
with session.get(url, stream=True, timeout=(5, None)) as r:
    for line in r.iter_lines(decode_unicode=True):
        if line:
            handle(json.loads(line))
```

**Truncated responses.** A connection dropped mid-stream leaves you with valid-looking partial HTML and no `Content-Length` to check against — chunked encoding's terminating chunk is the only completeness signal, and clients surface its absence as `ChunkedEncodingError` (requests) or `IncompleteRead` (urllib3). Catch it explicitly and retry rather than parsing half a page.

One protocol note: **HTTP/2 has no chunked encoding**. Framing is built into the protocol, and `Transfer-Encoding` is a prohibited header there. If you are debugging chunk handling on a modern site, check which protocol you actually negotiated first.

In Node, the streaming equivalent is the response body's async iterator:

```javascript
const res = await fetch(url);
for await (const chunk of res.body) {
  process(chunk); // Uint8Array, already de-chunked and decompressed
}
```

## Cookies: a header your client should manage for you

`Cookie` is a request header, but treating it as one is a mistake. Set it manually and you have to parse `Set-Cookie` yourself, honour `Domain`, `Path`, `Expires`, and `Secure`, and merge duplicates — all of which your client already does:

```python
session = requests.Session()
session.get("https://example.com/login") # picks up session cookies
session.post("https://example.com/login", data={"user": "x", "pass": "y"})
session.get("https://example.com/account") # cookies sent automatically
```

Set a cookie by hand only when you are importing state from somewhere else — a browser export, a login performed in Playwright, a token from another service:

```python
session.cookies.set("session_id", "abc123", domain="example.com")
```

Two things that catch people out: a `Cookie` header passed per-request in requests **overrides** the session jar rather than merging with it, and cookies set by JavaScript never appear in `Set-Cookie` at all — those you only get from a real browser. Sites that gate content behind a JS-set cookie cannot be scraped with an HTTP client alone, no matter what headers you send.

## Authentication headers

Three schemes cover almost everything you will meet, all riding the `Authorization` header.

### Basic

Base64 of `username:password`, no hashing, no protection beyond TLS. Never send it over plain HTTP.

```python
from requests.auth import HTTPBasicAuth
session.get(url, auth=HTTPBasicAuth("user", "pass"))
session.get(url, auth=("user", "pass")) # identical shorthand
```

```bash
curl -u user:pass https://example.com/private
```

The wire form is `Authorization: Basic dXNlcjpwYXNz`. Clients send it pre-emptively, so no extra round trip. Base64 is encoding, not encryption — anyone reading the traffic reads the password.

### Digest

A challenge-response scheme (RFC 7616) that never puts the password on the wire. The server answers with `401` and a `WWW-Authenticate: Digest` challenge containing a `realm`, a one-time `nonce`, and a `qop`; the client hashes username, password, realm, nonce, method, and URI together and returns the digest:

```http
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Digest realm="api", qop="auth", algorithm=SHA-256,
                  nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093", opaque="5ccc069c"
```

```python
from requests.auth import HTTPDigestAuth
session.get(url, auth=HTTPDigestAuth("user", "pass"))
```

```bash
curl --digest -u user:pass https://example.com/private
```

The scraping-relevant properties: digest auth **always costs two requests** for the first call, because the client cannot compute a response without the server's nonce. Nonces expire — a long-running scraper will get a `401` with `stale=true` mid-session and must re-handshake, which `HTTPDigestAuth` does automatically but a hand-rolled implementation usually does not. And the `nc` (nonce count) must increment per request against the same nonce; reusing a value gets you rejected.

Where you still find it: routers, IP cameras, printers, enterprise intranets, and older WebDAV endpoints. It is rare on the public web — its practical advantage over Basic disappeared once TLS became universal, and it forces the server to store recoverable password material. For anything new, Basic over TLS or a bearer token is the better answer.

### Bearer tokens

What modern APIs use:

```python
session.headers["Authorization"] = f"Bearer {token}"
```

The behaviour to know is what happens across a redirect: requests **strips** `Authorization` when a redirect crosses to a different host. That is a safety feature — it stops your token being handed to whatever the target redirects to — but it means an authenticated call that bounces to a CDN or a different subdomain silently arrives unauthenticated and returns `401`. When a request with valid credentials fails that way, walk the chain with `allow_redirects=False` and re-authenticate against the final host yourself.

Never hardcode credentials. Read them from the environment or a secret store, and keep them out of the query string, where they land in server logs and referrer headers.

## SSL/TLS certificates

Certificate errors are the most common hard failure in scraping after blocks, and the reflex fix — turn verification off — is the wrong one often enough to be worth unpacking.

By default, Python's requests verifies every certificate against the `certifi` CA bundle, and a mismatch raises:

```text
requests.exceptions.SSLError: HTTPSConnectionPool(host='example.com', port=443):
Max retries exceeded (Caused by SSLError(SSLCertVerificationError(
1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate')))
```

Diagnose before you disable. What the message means:

| Error | Actual cause | Right fix |
| --- | --- | --- |
| `unable to get local issuer certificate` | Missing intermediate in the server's chain, or a CA your bundle lacks | Usually the site's misconfiguration — supply the chain with `verify="chain.pem"` |
| `certificate has expired` | Genuinely expired cert | Nothing to fix client-side; the site is broken |
| `hostname mismatch` / `doesn't match` | Cert issued for a different name; often you're hitting an IP or a shared host | Set the correct `Host`, or use the name on the cert |
| `self-signed certificate` | Internal service, or a corporate MITM proxy intercepting TLS | Add that CA to your trust store — see below |
| `SSLV3_ALERT_HANDSHAKE_FAILURE` | Protocol/cipher mismatch, not a certificate problem at all | Old server needing legacy ciphers, or a custom `SSLContext` |

The first row explains the classic "works in my browser, fails in Python": browsers fetch missing intermediate certificates automatically via the AIA extension and cache CAs they have seen. Python does neither. The site is at fault, but you are the one who has to work around it.

**Trusting an extra CA** — the correct fix for corporate proxies and internal services:

```python
session.verify = "/path/to/corporate-ca.pem" # per session
```

```bash
export REQUESTS_CA_BUNDLE=/path/to/corporate-ca.pem # process-wide
export SSL_CERT_FILE=/path/to/corporate-ca.pem # for urllib/other libs
export NODE_EXTRA_CA_CERTS=/path/to/corporate-ca.pem # Node.js
```

**Client certificates** , when a server requires mutual TLS:

```python
session.cert = ("/path/client.crt", "/path/client.key")
```

**Disabling verification** is a last resort, acceptable for a throwaway request to a site with a genuinely broken chain that serves public data — and unacceptable on anything carrying credentials, because it removes the only defence against a man-in-the-middle:

```python
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
response = session.get(url, verify=False)
```

Scope it to the one host that needs it rather than setting `session.verify = False` globally. In curl the equivalent is `-k`; in Node, `rejectUnauthorized: false` on the agent — never `NODE_TLS_REJECT_UNAUTHORIZED=0`, which disables verification for the whole process including your own API calls.

To inspect a certificate before guessing:

```bash
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates
```

## Conditional requests and caching

If you re-scrape the same pages on a schedule, conditional requests turn most of those fetches into a 304 with no body. Servers advertise validators on the first response:

```http
ETag: "686897696a7c876b7e"
Last-Modified: Wed, 15 Jul 2026 10:31:00 GMT
Cache-Control: max-age=3600
```

Send them back on the next request and the server answers `304 Not Modified` with an empty body if nothing changed:

```python
def fetch_if_changed(session, url, cache):
    entry = cache.get(url, {})
    headers = {}
    if "etag" in entry:
        headers["If-None-Match"] = entry["etag"]
    if "last_modified" in entry:
        headers["If-Modified-Since"] = entry["last_modified"]

    r = session.get(url, headers=headers, timeout=15)
    if r.status_code == 304:
        return None # unchanged, nothing transferred
    r.raise_for_status()
    cache[url] = {
        "etag": r.headers.get("ETag"),
        "last_modified": r.headers.get("Last-Modified"),
    }
    return r.text
```

Note that `raise_for_status()` treats 304 as success, so check for it explicitly first. Caveats worth knowing: many sites serve no validators at all, weak ETags (prefixed `W/`) allow semantically-equivalent-but-different bytes, and some CDNs vary ETags per edge node, which produces false "changed" results. Treat 304 as an optimization, not a change-detection guarantee — if the answer must be right, hash the content you extracted.

## Connections, keep-alive, and timeouts

Opening a TCP connection and completing a TLS handshake costs two to three round trips before any HTTP happens. Reusing connections removes that from every request after the first, and it is a one-line change:

```python
session = requests.Session() # connection pooling, on by default
```

A `Session` keeps a `urllib3` pool per host and reuses sockets automatically. Without one, `requests.get()` in a loop pays a full handshake per URL — commonly a 2–3× throughput difference on HTTPS. To scale the pool for concurrent work:

```python
from requests.adapters import HTTPAdapter

adapter = HTTPAdapter(pool_connections=20, pool_maxsize=50)
session.mount("https://", adapter)
```

`Connection: keep-alive` is the HTTP/1.1 default and you do not need to send it — and as noted above, you must not send it over HTTP/2.

**Always set a timeout.** requests has none by default: a server that accepts your connection and never replies hangs the worker forever. The tuple form separates connect from read, which is what you usually want:

```python
session.get(url, timeout=(5, 30)) # 5s to connect, 30s between bytes
```

The read timeout is per byte received, not for the whole response — a slow server dribbling data can exceed it indefinitely. For a hard ceiling, run the request in a worker with its own deadline. Distinguish the failure modes when you retry: `ConnectTimeout` usually means a dead proxy or blocked IP and should switch endpoints, while `ReadTimeout` often means an overloaded origin and should back off.

## HTTP/1.1 vs HTTP/2 for scraping

W3Techs puts HTTP/2 at about 35% of websites and HTTP/3 at about 41% (September 2026), and browsers use them whenever the server offers them. Most Python scrapers still speak HTTP/1.1, because requests does not support HTTP/2 at all.

| | HTTP/1.1 | HTTP/2 |
| --- | --- | --- |
| Concurrency | One request per connection at a time; 6 parallel connections per host in browsers | Many multiplexed streams on one connection |
| Headers | Sent as text on every request | HPACK-compressed, with a shared table across the connection |
| Header names | Case-insensitive | Must be lowercase |
| Framing | `Content-Length` or chunked | Binary frames; no chunked encoding |
| Connection-specific headers | `Connection`, `Keep-Alive` normal | Prohibited |

What actually matters for scraping:

- **Bandwidth** : HPACK removes most of the repeated header bytes. On many small requests to one host this is a real saving.
- **Concurrency** : multiplexing many requests over one connection cuts socket count and handshake cost.
- **Fingerprinting** : this is the deciding factor for hard targets. A browser negotiates HTTP/2; a client that only speaks HTTP/1.1 is immediately distinguishable from the `User-Agent` it claims. Conversely, an HTTP/2 client with non-browser SETTINGS frames is distinguishable too.

To speak HTTP/2 from Python, use `httpx`:

```python
import httpx

with httpx.Client(http2=True, headers=HEADERS, timeout=15) as client:
    r = client.get("https://example.com/products")
    print(r.http_version) # "HTTP/2"
```

Do not treat the upgrade as a general speedup. For a scraper hitting many different hosts once each, connection reuse never happens and HTTP/2 changes almost nothing; the win is concentrated on many requests to the same origin.

## Status codes and what they actually tell you

The codes you meet in scraping, and the response each one deserves:

| Code | Meaning | What to do |
| --- | --- | --- |
| `200` | Success | Still verify the body — challenge pages and "no results" pages return 200 |
| `301` / `302` | Moved | Follow it, but log where; a redirect to `/login` or `/blocked` is a block in disguise |
| `304` | Not Modified | Your cached copy is current |
| `400` | Bad request | Usually a malformed URL or a header the server rejects |
| `401` | Unauthenticated | Missing or expired credentials; check for a `WWW-Authenticate` challenge |
| `403` | Forbidden | Nearly always bot detection. Fix the _request_, not the rate |
| `404` | Not found | Do not retry. Distinguish genuinely dead URLs from a pattern change |
| `429` | Too many requests | Slow down and honour `Retry-After` |
| `5xx` | Server error | Retry with backoff; `503` often carries `Retry-After` too |

### 403 and 429 need opposite fixes

Conflating them wastes days. **`429` is about rate** — you are recognized and asked to slow down. The server usually tells you how long:

```python
import time

r = session.get(url)
if r.status_code == 429:
    retry_after = r.headers.get("Retry-After")
    if retry_after:
        delay = int(retry_after) if retry_after.isdigit() else 60
    else:
        delay = backoff_delay(attempt) # exponential, with jitter
    time.sleep(delay)
```

`Retry-After` is either seconds or an HTTP date, so parse both. Add jitter to any exponential backoff — synchronized retries from a worker pool recreate the burst that caused the 429.

**`403` is about identity** — retrying the identical request more slowly changes nothing, because the request itself is what was rejected. Work through it in order of cost:

1. Compare your request to a real browser's, header by header, from a fresh "Copy as cURL" capture. Missing `Sec-Fetch-*` or `Accept-Language` is a common cause.
2. Check for a `Referer` requirement on deep pages.
3. Check whether a session cookie is needed — some sites 403 any request without one.
4. Change IP. Datacenter ranges are blocked wholesale by many sites; see the [proxy types guide](https://webscraping.ai/blog/types-of-proxies-for-web-scraping).
5. If the 403 body is a Cloudflare or DataDome interstitial, you are past what headers can solve and need a real browser or a scraping API.

A `403` that returns HTML is worth reading — save the body. It usually names the vendor and tells you which of the five steps applies.

## Letting the API handle the header layer

The header set, the Client Hints, the TLS fingerprint, and the proxy pool all have to agree, and each one drifts on its own schedule. If maintaining that is not the point of your project, [WebScraping.AI](https://webscraping.ai/) handles it server-side: requests go through real Chromium with rotating proxies, so the headers, the handshake, and the IP are consistent by construction.

```python
import requests

html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/products",
    "js": "true",
}).text
```

You can still set specific headers when the target needs them — a `Referer`, a session cookie, a language — with the nested `headers[Name]` syntax:

```python
html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/products",
    "headers[Referer]": "https://example.com/",
    "headers[Accept-Language]": "de-DE,de;q=0.9",
    "headers[Cookie]": "session=abc123",
    "country": "de",
}).text
```

```bash
curl -G https://api.webscraping.ai/html \
  --data-urlencode "api_key=$API_KEY" \
  --data-urlencode "url=https://example.com/products" \
  --data-urlencode "headers[Referer]=https://example.com/" \
  --data-urlencode "js=true"
```

The same `headers` parameter works on [`/text`, `/selected`, `/selected-multiple`, `/ai/question`, and `/ai/fields`](https://webscraping.ai/docs) — so you can skip the parsing step too:

```python
fields = requests.get("https://api.webscraping.ai/ai/fields", params={
    "api_key": API_KEY,
    "url": "https://example.com/product/42",
    "fields[name]": "Product name",
    "fields[price]": "Price with currency",
}).json()["result"] # {"name": "...", "price": "..."}
```

Match the `country` parameter to your `Accept-Language` when you set one — a German language preference arriving from a US exit IP is exactly the kind of inconsistency this section has been about.

## Frequently asked questions

**What headers should I send when web scraping?** At minimum: a current `User-Agent`, `Accept`, `Accept-Language`, and an `Accept-Encoding` listing only what you can decode. Add `Sec-Fetch-Site`, `Sec-Fetch-Mode`, `Sec-Fetch-Dest`, `Sec-Fetch-User`, `Upgrade-Insecure-Requests`, and the `sec-ch-ua*` Client Hints to look like a Chromium navigation, and `Referer` on any page a user would have reached by clicking. Copy the values from a live browser rather than a blog post — including this one.

**Why does my scraper get 403 when the browser loads the page fine?** Something in your request differs from the browser's. In rough order of frequency: a default `User-Agent`, missing `Sec-Fetch-*` headers, no session cookie, no `Referer`, a datacenter IP, or a TLS fingerprint that does not match the browser you claim to be. Capture the browser request with "Copy as cURL", confirm that replaying it works, then remove headers one at a time to find which one mattered.

**Does header order matter for web scraping?** Yes, for sites running serious anti-bot systems. Each browser sends headers in a stable order, and a request whose order does not match any real browser is a signal — alphabetically sorted headers are a common giveaway. Python dicts preserve insertion order, so defining headers in browser order is usually enough; your client will still add `Host` and `Connection` where it wants them.

**What is the difference between Content-Encoding and Transfer-Encoding?** `Content-Encoding` is compression applied to the payload end to end (`gzip`, `br`, `zstd`) — the recipient decompresses it to recover the original bytes. `Transfer-Encoding: chunked` describes how the message was framed for one hop when the length is not known in advance. They are independent and can appear together; HTTP/2 has framing built in and forbids `Transfer-Encoding` entirely.

**Do I need to handle chunked transfer encoding manually?** No. Every mainstream HTTP client reassembles chunks before handing you the body. You only interact with it deliberately — using `stream=True` with `iter_content` or `iter_lines` to process a large or long-lived response incrementally, or when catching the truncation errors (`ChunkedEncodingError`, `IncompleteRead`) that indicate a connection dropped mid-body.

**Should I use `verify=False` to fix SSL certificate errors?** Only as a diagnostic, and never on a request carrying credentials. The usual cause is a missing intermediate certificate in the server's chain, which browsers paper over by fetching it automatically and Python does not — supply the chain with `verify="chain.pem"`. For corporate TLS interception, add the proxy's CA via `REQUESTS_CA_BUNDLE`. If you must disable verification, scope it to the single host that needs it.

**Is HTTP digest authentication still worth using?** For scraping, you use whatever the target requires — and that is still digest on routers, cameras, printers, and older intranet endpoints. For anything you build, no: TLS already protects a Basic credential in transit, digest costs an extra round trip on every fresh nonce, and it requires the server to store recoverable password material. Bearer tokens over TLS are the modern default.

**Can custom headers alone get me past Cloudflare?** No. Cloudflare's managed challenges and bot score run below and above the header layer — TLS and HTTP/2 fingerprints, IP reputation, and JavaScript execution. Correct headers are necessary to avoid being flagged for the easy reasons, but the challenge itself needs a real browser environment or a scraping service that maintains one.