Python urllib3 Guide: PoolManager, Retries, Timeouts, and SSL

urllib3 is the HTTP client most Python code uses without knowing it — requests, botocore (the AWS SDK), and much of the packaging toolchain are built on top of it. Used directly, it gives you explicit control over connection pooling, retries, and TLS that higher-level libraries hide. This guide covers the confusing naming (urllib vs urllib2 vs urllib3 vs requests), the core PoolManager workflow, and the production concerns: timeouts, retry strategies, SSL verification, proxies, and thread safety.

Key Takeaways

• urllib is the standard library module; urllib3 is a third-party package (pip install urllib3) and, despite the name, not its successor version — they're unrelated codebases
• One PoolManager per application: it reuses TCP/TLS connections across requests, which is where most of the performance comes from
• Set explicit timeouts on every request — the default is to wait forever
• The Retry class gives you exponential backoff, per-status retries, and Retry-After handling in three lines
• urllib3 verifies SSL certificates by default in 2.x; fix certificate problems with certifi instead of disabling verification
• If you're choosing a library for application code, requests (or httpx) is usually more ergonomic — urllib3 shines in libraries, tight loops, and when you need low-level control

urllib vs urllib2 vs urllib3 vs requests

The names are a mess with history behind them:

So "urllib3" is not "urllib version 3" — it started as an independent project (the name refers to it being a third take on HTTP for Python) and never lived in the standard library. urllib.request can fetch a URL without dependencies, but you assemble pooling, retries, and redirects yourself; urllib3 provides all of that, and requests wraps urllib3 in a friendlier API.

When does raw urllib3 beat requests? Library code that shouldn't pull in requests' dependency tree, hot paths where you want zero overhead above the pool, botocore-adjacent code that already ships it, and anywhere you need pool-level knobs requests doesn't expose. For a typical script, use requests and let it drive urllib3 for you. For scraping specifics with requests, see our Python web scraping guide.

Installing urllib3 (and pinning versions)

pip install urllib3
pip show urllib3                 # which version you have

You often already have it as a dependency of requests or boto3 — which is exactly why version pinning comes up so much with this package: botocore pins urllib3 tightly, and blindly upgrading breaks AWS tooling. Install a specific version when you need one:

pip install urllib3==1.26.20         # last 1.x line
pip install "urllib3<2"              # stay on 1.x for legacy compatibility
pip install --upgrade "urllib3>=2,<3"

The 1.x → 2.x migration (2023) renamed method_whitelist to allowed_methods, made cert verification stricter, and dropped some TLS configurations; if a traceback mentions those, you've mixed 2.x code with a 1.x install or vice versa. The v2 migration guide covers the differences; new code should target 2.x.

Quick start: PoolManager and request()

Everything goes through a PoolManager — create one and reuse it for the life of your program:

import urllib3

http = urllib3.PoolManager()

resp = http.request("GET", "https://httpbin.org/get")
print(resp.status)          # 200
print(resp.headers["Content-Type"])
data = resp.json()          # parsed JSON (urllib3 2.x)
text = resp.data.decode("utf-8")   # raw body bytes

(There's also a module-level urllib3.request(...) convenience in 2.x that uses a global pool — fine for one-offs.)

GET with query parameters — pass fields and urllib3 encodes them:

resp = http.request("GET", "https://httpbin.org/get",
                    fields={"q": "web scraping", "page": "2"})

For manual URL building, urllib.parse.urlencode() also works: url + "?" + urlencode(params) — use it when you need repeated keys or precise ordering.

POST — form data, JSON, or raw body:

# Form-encoded
resp = http.request("POST", "https://httpbin.org/post",
                    fields={"user": "vlad"})

# JSON (urllib3 2.x encodes and sets Content-Type for you)
resp = http.request("POST", "https://httpbin.org/post",
                    json={"user": "vlad"})

# Raw body
resp = http.request("POST", "https://httpbin.org/post",
                    body=b"raw bytes", headers={"Content-Type": "application/octet-stream"})

# Multipart file upload: pass a (filename, content[, mime]) tuple in fields
with open("report.pdf", "rb") as f:
    resp = http.request("POST", "https://httpbin.org/post",
                        fields={"file": ("report.pdf", f.read(), "application/pdf")})

Headers go per-request or pool-wide:

http = urllib3.PoolManager(headers={"User-Agent": "my-app/1.0"})
resp = http.request("GET", url, headers={"Accept": "application/json"})

Timeouts: never use the default

By default urllib3 waits indefinitely. Always set a timeout — either per request or on the pool:

resp = http.request("GET", url, timeout=10.0)      # simple: 10s for connect and read

# Separate connect vs read budgets
t = urllib3.Timeout(connect=3.0, read=15.0)
http = urllib3.PoolManager(timeout=t)

Connect timeout bounds the TCP/TLS handshake (a server that's down fails fast); read timeout bounds each wait for data on an established connection (a server that accepted the request but hangs). A slow-but-steady download never trips the read timeout — it's per-read, not total. Timeouts raise urllib3.exceptions.ConnectTimeoutError / ReadTimeoutError, both subclasses of MaxRetryError once retries are exhausted.

Retries with backoff

The Retry class is one of the best reasons to use urllib3 directly:

from urllib3.util import Retry

retry = Retry(
    total=5,
    backoff_factor=0.5,                  # 0.5s, 1s, 2s, 4s, ... between tries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],     # don't blind-retry POSTs
    respect_retry_after_header=True,
)
http = urllib3.PoolManager(retries=retry)

This retries connection errors, read errors, and the listed status codes with exponential backoff, honoring Retry-After on 429/503 responses — which doubles as polite rate-limit handling: the server tells you when to come back and urllib3 waits. When retries run out you get MaxRetryError with the underlying cause attached. Set retries=False to disable retries (and redirect-following) entirely.

For client-side rate limiting (staying under a requests-per-second budget rather than reacting to 429s), urllib3 has no built-in throttle — wrap your call sites with a token bucket or a library like ratelimit, or simply time.sleep() between requests in sequential scrapers.

Connection pooling: what PoolManager actually does

Each unique scheme+host+port gets a ConnectionPool holding open sockets; requests to the same host reuse them, skipping TCP and TLS handshakes. That handshake skip is worth hundreds of milliseconds per request on HTTPS — for scraping one site, pooling alone can double throughput.

Tuning knobs:

http = urllib3.PoolManager(
    num_pools=10,      # how many per-host pools to cache (LRU beyond that)
    maxsize=10,        # connections kept open per host
    block=False,       # if True, cap concurrency at maxsize per host
)

Defaults (num_pools=10, maxsize=1... in practice maxsize defaults small) are fine for casual use; for threaded scraping of one host, set maxsize to roughly your thread count so threads aren't discarding connections. block=True turns the pool into a semaphore — threads wait for a free connection instead of opening extras, useful to guarantee you never hold more than N connections to a host. Connections are returned to the pool automatically when you consume the body (or call release_conn() on streamed responses).

If you're only ever talking to one host, urllib3.connectionpool.HTTPSConnectionPool(host, maxsize=...) skips a dictionary lookup per request, but PoolManager is the right default.

SSL: verification, certifi, and the errors you'll meet

urllib3 2.x verifies certificates against your system trust store by default. When you hit SSLError: CERTIFICATE_VERIFY_FAILED:

1. Outdated CA bundle — pip install -U certifi and point at it explicitly:

import certifi
http = urllib3.PoolManager(ca_certs=certifi.where())

1. Corporate MITM proxy or self-signed cert — add the internal CA: PoolManager(ca_certs="/path/to/company-ca.pem").
2. Genuinely broken remote cert (expired, wrong hostname) — decide consciously. Disabling verification is a last resort for data you'd trust over plain HTTP anyway:

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
http = urllib3.PoolManager(cert_reqs="CERT_NONE")   # you are now MITM-able

Client certificates (mutual TLS) go on the pool too: PoolManager(cert_file="client.crt", key_file="client.key").

Proxies: HTTP and SOCKS

# HTTP/HTTPS proxy for everything this manager does
http = urllib3.ProxyManager("http://proxy.example.com:3128",
                            proxy_headers=urllib3.make_headers(
                                proxy_basic_auth="user:password"))
resp = http.request("GET", "https://httpbin.org/ip")

SOCKS needs an extra: pip install "urllib3[socks]":

from urllib3.contrib.socks import SOCKSProxyManager
http = SOCKSProxyManager("socks5h://user:password@proxy.example.com:1080")

(socks5h resolves DNS through the proxy — usually what you want for scraping.) One manager = one proxy; rotating proxies means creating managers per proxy or letting a web scraping API rotate them server-side. Our proxy provider comparison covers where to get pools worth rotating.

Streaming large responses

resp = http.request("GET", big_file_url, preload_content=False)
with open("dump.bin", "wb") as out:
    for chunk in resp.stream(64 * 1024):
        out.write(chunk)
resp.release_conn()          # hand the socket back to the pool

preload_content=False stops urllib3 from buffering the whole body in memory; resp.stream() (or resp.read(n)) pulls it incrementally. Forgetting release_conn() leaks the connection from the pool — in 2.x, resp.close() or fully draining also releases it.

Errors, logging, and debugging

The exception hierarchy roots at urllib3.exceptions.HTTPError. The ones that matter:

import urllib3
from urllib3.exceptions import (MaxRetryError, NewConnectionError,
                                NameResolutionError, ReadTimeoutError, SSLError)

try:
    resp = http.request("GET", url, timeout=5.0)
    if resp.status >= 400:                 # urllib3 does NOT raise on 4xx/5xx
        handle_http_error(resp.status)
except MaxRetryError as e:                 # wraps the final underlying cause
    print("gave up:", e.reason)
except ReadTimeoutError:
    ...

Unlike requests' raise_for_status(), urllib3 treats a 500 as a perfectly good response — check resp.status yourself (or put the status in status_forcelist so retries handle it).

For visibility, urllib3 logs through the standard logging module:

import logging
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG)   # connection + retry events

That shows connections opened, requests issued, retries scheduled, and warnings. For full wire dumps (headers and bodies), step up to a debugging proxy like mitmproxy — urllib3 doesn't print raw traffic.

Thread safety and concurrency

PoolManager is thread-safe: share one instance across threads, don't create one per thread (that defeats pooling). A standard concurrent-fetch pattern:

from concurrent.futures import ThreadPoolExecutor

http = urllib3.PoolManager(maxsize=20)          # match worker count

def fetch(url):
    return url, http.request("GET", url, timeout=10.0).status

with ThreadPoolExecutor(max_workers=20) as pool:
    for url, status in pool.map(fetch, urls):
        print(status, url)

Note that urllib3 is synchronous — there's no asyncio support in the stable API. If your architecture is async, use aiohttp or httpx instead of forcing urllib3 into an event loop.

urllib3 for web scraping

urllib3 fetches raw HTML fast, and pairs naturally with lxml or Beautiful Soup for parsing. What it can't do: execute JavaScript, solve CAPTCHAs, or look like a residential user. When a site returns empty shells or 403s no matter what headers you send, that's not a urllib3 problem — the fetch itself needs a browser and better IPs. WebScraping.AI does that behind one request — which you can make with urllib3, keeping the rest of your pipeline untouched:

import urllib3

http = urllib3.PoolManager()
resp = http.request("GET", "https://api.webscraping.ai/html", fields={
    "api_key": API_KEY,
    "url": "https://example.com/spa-products",
    "js": "true",                     # rendered in a real browser, proxies included
})
html = resp.data.decode("utf-8")

The /ai/fields endpoint goes further and returns structured JSON described in plain English — no selectors to maintain at all.

Frequently asked questions

Is urllib3 the same as urllib? No. urllib is Python's built-in URL toolkit; urllib3 is an independent third-party package with connection pooling, retries, and modern TLS handling. The "3" is part of the project's name, not a stdlib version number — installing urllib3 doesn't replace or upgrade urllib.

Should I use urllib3 or requests? requests for application code — it's built on urllib3 and adds nicer ergonomics (sessions, raise_for_status(), auth helpers). urllib3 directly when you're writing a library and want minimal dependencies, need pool-level tuning, or want the Retry machinery without an extra layer. Performance differences are small; ergonomics and control are the real axis.

Why am I getting "urllib3 v2 only supports OpenSSL 1.1.1+"? Your Python was compiled against an old OpenSSL (common on older macOS system Pythons and some containers). Options: upgrade Python to a build with modern OpenSSL, or pin urllib3<2 until you can. This is an import-time environment check, not a bug in your code.

How do I send JSON with urllib3? On urllib3 2.x, pass json={"key": "value"} to request() — it serializes and sets Content-Type: application/json. On 1.x, do it manually: body=json.dumps(payload), headers={"Content-Type": "application/json"}. Parse responses with resp.json() (2.x) or json.loads(resp.data).

Does urllib3 follow redirects? Yes, by default (up to 3 in the pool default configuration, and redirects consume retries). Control it per request with redirect=False to get the 3xx response itself, or retries=Retry(redirect=5) for a custom limit. The final URL after redirects is on the response as resp.geturl() / resp.url.

How do I verify which urllib3 version is installed? pip show urllib3 from the shell, or import urllib3; print(urllib3.__version__) in code. If boto3/botocore is involved, check its pin before upgrading — pip install -U urllib3 inside an AWS project frequently creates a dependency conflict that pip check will report.