HTTP Headers for Web Scraping: Anatomy of a Real Browser Request

A browser and a scraper can request the same URL and get different pages back. The difference is almost never the URL — it is the dozen or so HTTP headers the browser attaches to every request and your HTTP client does not. Some of those headers change what the server sends you (compression, language, caching); others are what anti-bot systems read to decide whether you are a person. This guide covers the full request surface: which headers a real Chrome sends in 2026 and in what order, how to reproduce them in Python, Node, and curl, how to decode what comes back, and where header spoofing stops working no matter how carefully you do it.

Key Takeaways

• A default python-requests/2.32 request is identifiable in one line; a browser-shaped request needs about eight headers, not one User-Agent
• Consistency beats completeness — a Chrome User-Agent next to Firefox's Accept header is a stronger bot signal than sending no headers at all
• Over HTTP/2 header names are lowercase and Connection is illegal; copying an HTTP/1.1 header dump verbatim can itself be the tell
• Declare only the compressions you can actually decode — adding br, zstd to Accept-Encoding without the decoder installed returns binary garbage
• Header spoofing does not touch the TLS handshake, so a perfect header set still carries a Python JA3/JA4 fingerprint
• 403 means your request looked wrong, 429 means it came too fast — they need opposite fixes

Why headers decide whether you get blocked

An HTTP request is a method, a path, and a list of headers. The server has nothing else to judge you by before it decides what to return. Bot detection at the request layer asks three questions:

1. Is anything obviously automated? Default clients announce themselves: python-requests/2.32.3, curl/8.7.1, Go-http-client/2.0, axios/1.7.2. A single string is enough for a WAF rule.
2. Is the header set complete? Real browsers send Accept, Accept-Language, Accept-Encoding, Sec-Fetch-*, and Client Hints on every navigation. A request with only User-Agent is a scraper that read one tutorial.
3. Is it internally coherent? This is the one that catches careful scrapers. A User-Agent claiming Chrome 150 on Windows, with Firefox's Accept value, no Sec-CH-UA, and headers in alphabetical order describes a browser that does not exist.

The third check is why "add a user agent" advice stopped working. Detection systems compare the whole request against a profile of what that browser actually sends. Our user agent guide covers the User-Agent string itself and its rotation in depth — this guide covers everything around it.

Anatomy of a real browser request

Here is a top-level navigation from Chrome on Windows, in the order Chrome sends it. The order is part of the fingerprint, so it is worth reading as a sequence rather than a set:

GET /products HTTP/1.1
Host: example.com
Connection: keep-alive
sec-ch-ua: "Chromium";v="150", "Google Chrome";v="150", "Not?A_Brand";v="24"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9

What each one does, and what it costs you to get it wrong:

The Sec-* headers are the ones scrapers most often miss. They are forbidden headers in browser JavaScript — a page cannot set them, so a server can trust that they came from the browser itself. That makes them cheap and reliable to check, and it is why a request without them stands out.

The Client Hints (sec-ch-ua*) deserve their own note: Chrome's User-Agent Reduction froze most of the detail in the User-Agent string and moved it here, so a scraper that only spoofs User-Agent is now missing information a real Chrome always volunteers. The user agent guide has the current strings and matching hint values.

Getting the real thing instead of copying a table

Header sets drift with every Chrome release. Rather than trusting any published list — including this one — read them off a live browser:

1. Open DevTools (F12) → Network
2. Load the target page
3. Right-click the document request → Copy → Copy as cURL

You get the exact request your browser made, headers and cookies included, ready to replay in a terminal. Our curl for web scraping guide covers turning that command into something reusable, and the curl converter translates it straight into Python, JavaScript, PHP, or Go.

Two habits make this worth repeating rather than doing once: capture from the same kind of navigation you are automating (a link click sends different Sec-Fetch-* values than a typed URL), and re-capture when a scraper that worked for months starts returning 403s.

Sending headers correctly

Every HTTP client takes a dict of headers. The gotchas are in what the client does around your dict.

Python (requests) — a Session keeps headers, cookies, and the connection pool across requests:

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,"
              "image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate",       # only what you can decode
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-User": "?1",
    "Sec-Fetch-Dest": "document",
    "sec-ch-ua": '"Chromium";v="150", "Google Chrome";v="150", "Not?A_Brand";v="24"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
}

session = requests.Session()
session.headers.update(HEADERS)

response = session.get("https://example.com/products", timeout=15)
response.raise_for_status()

Session.headers is a case-insensitive dict, so assigning User-Agent replaces the default rather than adding a second one. Setting a key to None removes a default header entirely — the only way to stop requests from sending Accept-Encoding at all. The full picture of sessions, retries, and timeouts is in our Python requests guide.

Node.js — built-in fetch (undici) preserves the order you give and speaks HTTP/1.1 by default:

const res = await fetch("https://example.com/products", {
  headers: {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
                  "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Dest": "document",
  },
  signal: AbortSignal.timeout(15000),
});
const html = await res.text();   // undici decompresses gzip/br automatically

curl — -H per header, --compressed to request and decode compression in one flag:

curl --compressed \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Dest: document' \
  https://example.com/products

Add -v to see exactly what went over the wire, including the headers curl added on your behalf.

Order, casing, and the HTTP/2 trap

Header order is stable per browser and is fingerprinted. Python dicts have preserved insertion order since 3.7, and requests sends session headers first, then per-request ones — so define your dict in browser order and it will mostly survive. "Mostly" is the honest word here: clients add Host, Connection, and Content-Length at positions you do not control.

Header casing matters more than people expect. HTTP/1.1 header names are case-insensitive by specification, but HTTP/2 and HTTP/3 require them to be lowercase on the wire, and Chrome sends sec-ch-ua lowercase even over HTTP/1.1 — a detail worth copying exactly.

This creates a trap when replaying a captured request: over HTTP/2 the pseudo-headers (:method, :authority, :scheme, :path) come first, Host does not exist, and Connection, Keep-Alive, Transfer-Encoding, and Upgrade are prohibited connection-specific headers. Sending Connection: keep-alive on an HTTP/2 request is a protocol violation that some servers reject outright and any fingerprinter notices instantly, because no browser does it. If your client negotiates HTTP/2 (httpx with http2=True, curl by default on HTTPS), drop Connection from your header set.

Where header spoofing stops working

Be clear about what this buys you. Headers are the application layer; detection now runs below it.

• TLS fingerprinting (JA3/JA4). Your cipher suite list, extension order, and supported curves are chosen by your TLS library — OpenSSL via Python, BoringSSL in Chrome — and they differ. A perfect Chrome header set arriving on a Python TLS handshake is a contradiction visible before a single header is parsed. Libraries like curl_cffi (Python) and tls-client (Go) exist specifically to impersonate browser handshakes.
• HTTP/2 fingerprinting. SETTINGS frame values, window sizes, and pseudo-header order form an "Akamai fingerprint" that is also library-specific.
• IP reputation. A datacenter IP with a flawless browser profile still looks like a datacenter. See our proxy types guide for what the different pools actually change.
• Behaviour and JavaScript challenges. Cloudflare Turnstile, DataDome, and PerimeterX run JS that measures canvas rendering, timing, and event patterns. No header set answers that.

Headers are necessary and not sufficient. They get you past simple WAF rules and header-completeness checks, which is a large share of ordinary sites — and none of the way past a managed anti-bot product.

Accept-Encoding and compression

Accept-Encoding advertises which compressions you can decode; the server picks one and names it in Content-Encoding. Getting HTML back compressed is normal — text compresses 70–90%, so this is the single biggest bandwidth lever in scraping.

Most clients decompress transparently: requests handles gzip and deflate out of the box (and Brotli if brotli/brotlicffi is installed), undici handles gzip and Brotli, curl does it with --compressed.

The failure mode to avoid: copying Accept-Encoding: gzip, deflate, br, zstd out of DevTools into a client that cannot decode br or zstd. The server takes you at your word, and response.text becomes binary noise. Either install the decoders or advertise less:

# Safe: matches what a stock requests install can decode
headers = {"Accept-Encoding": "gzip, deflate"}

# Full browser parity: pip install brotli zstandard  (urllib3 2.x uses them)
headers = {"Accept-Encoding": "gzip, deflate, br, zstd"}

To decode manually — when you are reading a raw socket, a saved response, or an unusual client:

import gzip, zlib, brotli

raw = response.raw.read()                      # undecoded bytes
enc = response.headers.get("Content-Encoding", "").lower()

if enc == "gzip":
    body = gzip.decompress(raw)
elif enc == "deflate":
    try:
        body = zlib.decompress(raw)            # zlib-wrapped
    except zlib.error:
        body = zlib.decompress(raw, -zlib.MAX_WBITS)   # raw deflate
elif enc == "br":
    body = brotli.decompress(raw)
else:
    body = raw

html = body.decode(response.encoding or "utf-8", errors="replace")

Two details that cause silent bugs. First, Content-Encoding can list multiple encodings applied in order (br, gzip), and you must undo them right to left. Second, Content-Encoding and Transfer-Encoding are different mechanisms: the first compresses the payload end to end, the second describes how the message was framed for this hop. Content-Length, when present, describes the compressed size — which is why comparing it to len(response.text) never matches.

Chunked transfer encoding

Transfer-Encoding: chunked is how HTTP/1.1 sends a body of unknown length. The server omits Content-Length and emits a sequence of size-prefixed chunks terminated by a zero-length one:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked

1a
<html><head><title>Page
14
</title></head></html>
0

You will see it on server-rendered pages streamed as they are generated, on long-running APIs, and on log or event endpoints. Every mainstream HTTP client reassembles chunks for you — this is a transport detail, not something you normally parse. response.text is already the joined body.

It matters in three situations:

Streaming large responses. Do not materialize a 500 MB export in memory:

with session.get(url, stream=True, timeout=30) as r:
    r.raise_for_status()
    with open("export.csv", "wb") as f:
        for chunk in r.iter_content(chunk_size=64 * 1024):
            f.write(chunk)

iter_content yields decompressed bytes in whatever sizes arrive; the chunk boundaries you get are not the HTTP chunk boundaries, and no code should depend on them. Note that stream=True holds a connection out of the pool until the body is consumed or the response is closed — the with block is what prevents a slow leak.

Processing lines as they arrive, for streaming JSON or server-sent events:

with session.get(url, stream=True, timeout=(5, None)) as r:
    for line in r.iter_lines(decode_unicode=True):
        if line:
            handle(json.loads(line))

Truncated responses. A connection dropped mid-stream leaves you with valid-looking partial HTML and no Content-Length to check against — chunked encoding's terminating chunk is the only completeness signal, and clients surface its absence as ChunkedEncodingError (requests) or IncompleteRead (urllib3). Catch it explicitly and retry rather than parsing half a page.

One protocol note: HTTP/2 has no chunked encoding. Framing is built into the protocol, and Transfer-Encoding is a prohibited header there. If you are debugging chunk handling on a modern site, check which protocol you actually negotiated first.

In Node, the streaming equivalent is the response body's async iterator:

const res = await fetch(url);
for await (const chunk of res.body) {
  process(chunk);          // Uint8Array, already de-chunked and decompressed
}

Cookies: a header your client should manage for you

Cookie is a request header, but treating it as one is a mistake. Set it manually and you have to parse Set-Cookie yourself, honour Domain, Path, Expires, and Secure, and merge duplicates — all of which your client already does:

session = requests.Session()
session.get("https://example.com/login")          # picks up session cookies
session.post("https://example.com/login", data={"user": "x", "pass": "y"})
session.get("https://example.com/account")        # cookies sent automatically

Set a cookie by hand only when you are importing state from somewhere else — a browser export, a login performed in Playwright, a token from another service:

session.cookies.set("session_id", "abc123", domain="example.com")

Two things that catch people out: a Cookie header passed per-request in requests overrides the session jar rather than merging with it, and cookies set by JavaScript never appear in Set-Cookie at all — those you only get from a real browser. Sites that gate content behind a JS-set cookie cannot be scraped with an HTTP client alone, no matter what headers you send.

Authentication headers

Three schemes cover almost everything you will meet, all riding the Authorization header.

Basic

Base64 of username:password, no hashing, no protection beyond TLS. Never send it over plain HTTP.

from requests.auth import HTTPBasicAuth
session.get(url, auth=HTTPBasicAuth("user", "pass"))
session.get(url, auth=("user", "pass"))            # identical shorthand

curl -u user:pass https://example.com/private

The wire form is Authorization: Basic dXNlcjpwYXNz. Clients send it pre-emptively, so no extra round trip. Base64 is encoding, not encryption — anyone reading the traffic reads the password.

Digest

A challenge-response scheme (RFC 7616) that never puts the password on the wire. The server answers with 401 and a WWW-Authenticate: Digest challenge containing a realm, a one-time nonce, and a qop; the client hashes username, password, realm, nonce, method, and URI together and returns the digest:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Digest realm="api", qop="auth", algorithm=SHA-256,
                  nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093", opaque="5ccc069c"

from requests.auth import HTTPDigestAuth
session.get(url, auth=HTTPDigestAuth("user", "pass"))

curl --digest -u user:pass https://example.com/private

The scraping-relevant properties: digest auth always costs two requests for the first call, because the client cannot compute a response without the server's nonce. Nonces expire — a long-running scraper will get a 401 with stale=true mid-session and must re-handshake, which HTTPDigestAuth does automatically but a hand-rolled implementation usually does not. And the nc (nonce count) must increment per request against the same nonce; reusing a value gets you rejected.

Where you still find it: routers, IP cameras, printers, enterprise intranets, and older WebDAV endpoints. It is rare on the public web — its practical advantage over Basic disappeared once TLS became universal, and it forces the server to store recoverable password material. For anything new, Basic over TLS or a bearer token is the better answer.

Bearer tokens

What modern APIs use:

session.headers["Authorization"] = f"Bearer {token}"

The behaviour to know is what happens across a redirect: requests strips Authorization when a redirect crosses to a different host. That is a safety feature — it stops your token being handed to whatever the target redirects to — but it means an authenticated call that bounces to a CDN or a different subdomain silently arrives unauthenticated and returns 401. When a request with valid credentials fails that way, walk the chain with allow_redirects=False and re-authenticate against the final host yourself.

Never hardcode credentials. Read them from the environment or a secret store, and keep them out of the query string, where they land in server logs and referrer headers.

SSL/TLS certificates

Certificate errors are the most common hard failure in scraping after blocks, and the reflex fix — turn verification off — is the wrong one often enough to be worth unpacking.

By default, Python's requests verifies every certificate against the certifi CA bundle, and a mismatch raises:

requests.exceptions.SSLError: HTTPSConnectionPool(host='example.com', port=443):
Max retries exceeded (Caused by SSLError(SSLCertVerificationError(
1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate')))

Diagnose before you disable. What the message means:

The first row explains the classic "works in my browser, fails in Python": browsers fetch missing intermediate certificates automatically via the AIA extension and cache CAs they have seen. Python does neither. The site is at fault, but you are the one who has to work around it.

Trusting an extra CA — the correct fix for corporate proxies and internal services:

session.verify = "/path/to/corporate-ca.pem"          # per session

export REQUESTS_CA_BUNDLE=/path/to/corporate-ca.pem   # process-wide
export SSL_CERT_FILE=/path/to/corporate-ca.pem        # for urllib/other libs
export NODE_EXTRA_CA_CERTS=/path/to/corporate-ca.pem  # Node.js

Client certificates, when a server requires mutual TLS:

session.cert = ("/path/client.crt", "/path/client.key")

Disabling verification is a last resort, acceptable for a throwaway request to a site with a genuinely broken chain that serves public data — and unacceptable on anything carrying credentials, because it removes the only defence against a man-in-the-middle:

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
response = session.get(url, verify=False)

Scope it to the one host that needs it rather than setting session.verify = False globally. In curl the equivalent is -k; in Node, rejectUnauthorized: false on the agent — never NODE_TLS_REJECT_UNAUTHORIZED=0, which disables verification for the whole process including your own API calls.

To inspect a certificate before guessing:

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Conditional requests and caching

If you re-scrape the same pages on a schedule, conditional requests turn most of those fetches into a 304 with no body. Servers advertise validators on the first response:

ETag: "686897696a7c876b7e"
Last-Modified: Wed, 15 Jul 2026 10:31:00 GMT
Cache-Control: max-age=3600

Send them back on the next request and the server answers 304 Not Modified with an empty body if nothing changed:

def fetch_if_changed(session, url, cache):
    entry = cache.get(url, {})
    headers = {}
    if "etag" in entry:
        headers["If-None-Match"] = entry["etag"]
    if "last_modified" in entry:
        headers["If-Modified-Since"] = entry["last_modified"]

    r = session.get(url, headers=headers, timeout=15)
    if r.status_code == 304:
        return None                       # unchanged, nothing transferred
    r.raise_for_status()
    cache[url] = {
        "etag": r.headers.get("ETag"),
        "last_modified": r.headers.get("Last-Modified"),
    }
    return r.text

Note that raise_for_status() treats 304 as success, so check for it explicitly first. Caveats worth knowing: many sites serve no validators at all, weak ETags (prefixed W/) allow semantically-equivalent-but-different bytes, and some CDNs vary ETags per edge node, which produces false "changed" results. Treat 304 as an optimization, not a change-detection guarantee — if the answer must be right, hash the content you extracted.

Connections, keep-alive, and timeouts

Opening a TCP connection and completing a TLS handshake costs two to three round trips before any HTTP happens. Reusing connections removes that from every request after the first, and it is a one-line change:

session = requests.Session()          # connection pooling, on by default

A Session keeps a urllib3 pool per host and reuses sockets automatically. Without one, requests.get() in a loop pays a full handshake per URL — commonly a 2–3× throughput difference on HTTPS. To scale the pool for concurrent work:

from requests.adapters import HTTPAdapter

adapter = HTTPAdapter(pool_connections=20, pool_maxsize=50)
session.mount("https://", adapter)

Connection: keep-alive is the HTTP/1.1 default and you do not need to send it — and as noted above, you must not send it over HTTP/2.

Always set a timeout. requests has none by default: a server that accepts your connection and never replies hangs the worker forever. The tuple form separates connect from read, which is what you usually want:

session.get(url, timeout=(5, 30))     # 5s to connect, 30s between bytes

The read timeout is per byte received, not for the whole response — a slow server dribbling data can exceed it indefinitely. For a hard ceiling, run the request in a worker with its own deadline. Distinguish the failure modes when you retry: ConnectTimeout usually means a dead proxy or blocked IP and should switch endpoints, while ReadTimeout often means an overloaded origin and should back off.

HTTP/1.1 vs HTTP/2 for scraping

Roughly 65% of the web serves HTTP/2 or HTTP/3, and browsers use it whenever the server offers it. Most Python scrapers still speak HTTP/1.1, because requests does not support HTTP/2 at all.

What actually matters for scraping:

• Bandwidth: HPACK removes most of the repeated header bytes. On many small requests to one host this is a real saving.
• Concurrency: multiplexing many requests over one connection cuts socket count and handshake cost.
• Fingerprinting: this is the deciding factor for hard targets. A browser negotiates HTTP/2; a client that only speaks HTTP/1.1 is immediately distinguishable from the User-Agent it claims. Conversely, an HTTP/2 client with non-browser SETTINGS frames is distinguishable too.

To speak HTTP/2 from Python, use httpx:

import httpx

with httpx.Client(http2=True, headers=HEADERS, timeout=15) as client:
    r = client.get("https://example.com/products")
    print(r.http_version)      # "HTTP/2"

Do not treat the upgrade as a general speedup. For a scraper hitting many different hosts once each, connection reuse never happens and HTTP/2 changes almost nothing; the win is concentrated on many requests to the same origin.

Status codes and what they actually tell you

The codes you meet in scraping, and the response each one deserves:

403 and 429 need opposite fixes

Conflating them wastes days. 429 is about rate — you are recognized and asked to slow down. The server usually tells you how long:

import time

r = session.get(url)
if r.status_code == 429:
    retry_after = r.headers.get("Retry-After")
    if retry_after:
        delay = int(retry_after) if retry_after.isdigit() else 60
    else:
        delay = backoff_delay(attempt)      # exponential, with jitter
    time.sleep(delay)

Retry-After is either seconds or an HTTP date, so parse both. Add jitter to any exponential backoff — synchronized retries from a worker pool recreate the burst that caused the 429.

403 is about identity — retrying the identical request more slowly changes nothing, because the request itself is what was rejected. Work through it in order of cost:

1. Compare your request to a real browser's, header by header, from a fresh "Copy as cURL" capture. Missing Sec-Fetch-* or Accept-Language is a common cause.
2. Check for a Referer requirement on deep pages.
3. Check whether a session cookie is needed — some sites 403 any request without one.
4. Change IP. Datacenter ranges are blocked wholesale by many sites; see the proxy types guide.
5. If the 403 body is a Cloudflare or DataDome interstitial, you are past what headers can solve and need a real browser or a scraping API.

A 403 that returns HTML is worth reading — save the body. It usually names the vendor and tells you which of the five steps applies.

Letting the API handle the header layer

The header set, the Client Hints, the TLS fingerprint, and the proxy pool all have to agree, and each one drifts on its own schedule. If maintaining that is not the point of your project, WebScraping.AI handles it server-side: requests go through real Chromium with rotating proxies, so the headers, the handshake, and the IP are consistent by construction.

import requests

html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/products",
    "js": "true",
}).text

You can still set specific headers when the target needs them — a Referer, a session cookie, a language — with the nested headers[Name] syntax:

html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/products",
    "headers[Referer]": "https://example.com/",
    "headers[Accept-Language]": "de-DE,de;q=0.9",
    "headers[Cookie]": "session=abc123",
    "country": "de",
}).text

curl -G https://api.webscraping.ai/html \
  --data-urlencode "api_key=$API_KEY" \
  --data-urlencode "url=https://example.com/products" \
  --data-urlencode "headers[Referer]=https://example.com/" \
  --data-urlencode "js=true"

The same headers parameter works on /text, /selected, /selected-multiple, /ai/question, and /ai/fields — so you can skip the parsing step too:

fields = requests.get("https://api.webscraping.ai/ai/fields", params={
    "api_key": API_KEY,
    "url": "https://example.com/product/42",
    "fields[name]": "Product name",
    "fields[price]": "Price with currency",
}).json()

Match the country parameter to your Accept-Language when you set one — a German language preference arriving from a US exit IP is exactly the kind of inconsistency this section has been about.

Frequently asked questions

What headers should I send when web scraping? At minimum: a current User-Agent, Accept, Accept-Language, and an Accept-Encoding listing only what you can decode. Add Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest, Sec-Fetch-User, Upgrade-Insecure-Requests, and the sec-ch-ua* Client Hints to look like a Chromium navigation, and Referer on any page a user would have reached by clicking. Copy the values from a live browser rather than a blog post — including this one.

Why does my scraper get 403 when the browser loads the page fine? Something in your request differs from the browser's. In rough order of frequency: a default User-Agent, missing Sec-Fetch-* headers, no session cookie, no Referer, a datacenter IP, or a TLS fingerprint that does not match the browser you claim to be. Capture the browser request with "Copy as cURL", confirm that replaying it works, then remove headers one at a time to find which one mattered.

Does header order matter for web scraping? Yes, for sites running serious anti-bot systems. Each browser sends headers in a stable order, and a request whose order does not match any real browser is a signal — alphabetically sorted headers are a common giveaway. Python dicts preserve insertion order, so defining headers in browser order is usually enough; your client will still add Host and Connection where it wants them.

What is the difference between Content-Encoding and Transfer-Encoding? Content-Encoding is compression applied to the payload end to end (gzip, br, zstd) — the recipient decompresses it to recover the original bytes. Transfer-Encoding: chunked describes how the message was framed for one hop when the length is not known in advance. They are independent and can appear together; HTTP/2 has framing built in and forbids Transfer-Encoding entirely.

Do I need to handle chunked transfer encoding manually? No. Every mainstream HTTP client reassembles chunks before handing you the body. You only interact with it deliberately — using stream=True with iter_content or iter_lines to process a large or long-lived response incrementally, or when catching the truncation errors (ChunkedEncodingError, IncompleteRead) that indicate a connection dropped mid-body.

Should I use verify=False to fix SSL certificate errors? Only as a diagnostic, and never on a request carrying credentials. The usual cause is a missing intermediate certificate in the server's chain, which browsers paper over by fetching it automatically and Python does not — supply the chain with verify="chain.pem". For corporate TLS interception, add the proxy's CA via REQUESTS_CA_BUNDLE. If you must disable verification, scope it to the single host that needs it.

Is HTTP digest authentication still worth using? For scraping, you use whatever the target requires — and that is still digest on routers, cameras, printers, and older intranet endpoints. For anything you build, no: TLS already protects a Basic credential in transit, digest costs an extra round trip on every fresh nonce, and it requires the server to store recoverable password material. Bearer tokens over TLS are the modern default.

Can custom headers alone get me past Cloudflare? No. Cloudflare's managed challenges and bot score run below and above the header layer — TLS and HTTP/2 fingerprints, IP reputation, and JavaScript execution. Correct headers are necessary to avoid being flagged for the easy reasons, but the challenge itself needs a real browser environment or a scraping service that maintains one.