---
title: "Python urllib3 Guide: PoolManager, Retries, Timeouts, and SSL"
description: "Practical urllib3 tutorial: urllib vs urllib3 vs requests, PoolManager and connection pooling, retry strategies, timeouts, SSL verification, and proxies."
url: https://webscraping.ai/blog/python-urllib3-guide
markdown_index: https://webscraping.ai/llms.txt
---
# Python urllib3 Guide: PoolManager, Retries, Timeouts, and SSL

[urllib3](https://urllib3.readthedocs.io/) is the HTTP client most Python code uses without knowing it — `requests`, `botocore` (the AWS SDK), and much of the packaging toolchain are built on top of it. Used directly, it gives you explicit control over connection pooling, retries, and TLS that higher-level libraries hide. This guide covers the confusing naming (urllib vs urllib2 vs urllib3 vs requests), the core `PoolManager` workflow, and the production concerns: timeouts, retry strategies, SSL verification, proxies, and thread safety.

## Key Takeaways

- `urllib` is the standard library module; **urllib3 is a third-party package** (`pip install urllib3`) and, despite the name, not its successor version — they're unrelated codebases
- One `PoolManager` per application: it reuses TCP/TLS connections across requests, which is where most of the performance comes from
- Set explicit timeouts on every request — the default is to wait forever
- The `Retry` class gives you exponential backoff, per-status retries, and `Retry-After` handling in three lines
- urllib3 verifies SSL certificates by default in 2.x; fix certificate problems with `certifi` instead of disabling verification
- If you're choosing a library for application code, `requests` (or `httpx`) is usually more ergonomic — urllib3 shines in libraries, tight loops, and when you need low-level control

## urllib vs urllib2 vs urllib3 vs requests

The names are a mess with history behind them:

| Name | What it is | Status |
| --- | --- | --- |
| `urllib` | Python 3 stdlib package (`urllib.request`, `.parse`, `.error`) | Built in; verbose for real HTTP work |
| `urllib2` | Python 2's improved stdlib client | Dead with Python 2 — merged into `urllib` in Python 3 |
| `urllib3` | Third-party HTTP client with pooling, retries, TLS | Actively developed; foundation of requests/botocore |
| `requests` | High-level client **built on urllib3** | The ergonomic default for application code |

So "urllib3" is not "urllib version 3" — it started as an independent project (the name refers to it being a third take on HTTP for Python) and never lived in the standard library. `urllib.request` can fetch a URL without dependencies, but you assemble pooling, retries, and redirects yourself; urllib3 provides all of that, and requests wraps urllib3 in a friendlier API.

When does raw urllib3 beat requests? Library code that shouldn't pull in requests' dependency tree, hot paths where you want zero overhead above the pool, botocore-adjacent code that already ships it, and anywhere you need pool-level knobs requests doesn't expose. For a typical script, use requests and let it drive urllib3 for you. For scraping specifics with requests, see our [Python web scraping guide](https://webscraping.ai/blog/web-scraping-with-python).

## Installing urllib3 (and pinning versions)

```bash
pip install urllib3
pip show urllib3 # which version you have
```

You often already have it as a dependency of requests or boto3 — which is exactly why version pinning comes up so much with this package: **botocore pins urllib3 tightly** , and blindly upgrading breaks AWS tooling. Install a specific version when you need one:

```bash
pip install urllib3==1.26.20 # last 1.x line
pip install "urllib3<2" # stay on 1.x for legacy compatibility
pip install --upgrade "urllib3>=2,<3"
```

The 1.x → 2.x migration (2023) renamed `method_whitelist` to `allowed_methods`, made cert verification stricter, and dropped some TLS configurations; if a traceback mentions those, you've mixed 2.x code with a 1.x install or vice versa. The [v2 migration guide](https://urllib3.readthedocs.io/en/stable/v2-migration-guide.html) covers the differences; new code should target 2.x.

## Quick start: PoolManager and request()

Everything goes through a `PoolManager` — create one and reuse it for the life of your program:

```python
import urllib3

http = urllib3.PoolManager()

resp = http.request("GET", "https://httpbin.org/get")
print(resp.status) # 200
print(resp.headers["Content-Type"])
data = resp.json() # parsed JSON (urllib3 2.x)
text = resp.data.decode("utf-8") # raw body bytes
```

(There's also a module-level `urllib3.request(...)` convenience in 2.x that uses a global pool — fine for one-offs.)

GET with query parameters — pass `fields` and urllib3 encodes them:

```python
resp = http.request("GET", "https://httpbin.org/get",
                    fields={"q": "web scraping", "page": "2"})
```

For manual URL building, `urllib.parse.urlencode()` also works: `url + "?" + urlencode(params)` — use it when you need repeated keys or precise ordering.

POST — form data, JSON, or raw body:

```python
# Form-encoded
resp = http.request("POST", "https://httpbin.org/post",
                    fields={"user": "vlad"})

# JSON (urllib3 2.x encodes and sets Content-Type for you)
resp = http.request("POST", "https://httpbin.org/post",
                    json={"user": "vlad"})

# Raw body
resp = http.request("POST", "https://httpbin.org/post",
                    body=b"raw bytes", headers={"Content-Type": "application/octet-stream"})

# Multipart file upload: pass a (filename, content[, mime]) tuple in fields
with open("report.pdf", "rb") as f:
    resp = http.request("POST", "https://httpbin.org/post",
                        fields={"file": ("report.pdf", f.read(), "application/pdf")})
```

Headers go per-request or pool-wide:

```python
http = urllib3.PoolManager(headers={"User-Agent": "my-app/1.0"})
resp = http.request("GET", url, headers={"Accept": "application/json"})
```

## Timeouts: never use the default

By default urllib3 waits indefinitely. Always set a timeout — either per request or on the pool:

```python
resp = http.request("GET", url, timeout=10.0) # simple: 10s for connect and read

# Separate connect vs read budgets
t = urllib3.Timeout(connect=3.0, read=15.0)
http = urllib3.PoolManager(timeout=t)
```

_Connect_ timeout bounds the TCP/TLS handshake (a server that's down fails fast); _read_ timeout bounds each wait for data on an established connection (a server that accepted the request but hangs). A slow-but-steady download never trips the read timeout — it's per-read, not total. Timeouts raise `urllib3.exceptions.ConnectTimeoutError` / `ReadTimeoutError`, both subclasses of `MaxRetryError` once retries are exhausted.

## Retries with backoff

The `Retry` class is one of the best reasons to use urllib3 directly:

```python
from urllib3.util import Retry

retry = Retry(
    total=5,
    backoff_factor=0.5, # 0.5s, 1s, 2s, 4s, ... between tries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"], # don't blind-retry POSTs
    respect_retry_after_header=True,
)
http = urllib3.PoolManager(retries=retry)
```

This retries connection errors, read errors, and the listed status codes with exponential backoff, honoring `Retry-After` on 429/503 responses — which doubles as polite rate-limit handling: the server tells you when to come back and urllib3 waits. When retries run out you get `MaxRetryError` with the underlying cause attached. Set `retries=False` to disable retries (and redirect-following) entirely.

For client-side rate limiting (staying under a requests-per-second budget rather than reacting to 429s), urllib3 has no built-in throttle — wrap your call sites with a token bucket or a library like `ratelimit`, or simply `time.sleep()` between requests in sequential scrapers.

## Connection pooling: what PoolManager actually does

Each unique scheme+host+port gets a `ConnectionPool` holding open sockets; requests to the same host reuse them, skipping TCP and TLS handshakes. That handshake skip is worth hundreds of milliseconds per request on HTTPS — for scraping one site, pooling alone can double throughput.

Tuning knobs:

```python
http = urllib3.PoolManager(
    num_pools=10, # how many per-host pools to cache (LRU beyond that)
    maxsize=10, # connections kept open per host
    block=False, # if True, cap concurrency at maxsize per host
)
```

Defaults (`num_pools=10, maxsize=1`... in practice `maxsize` defaults small) are fine for casual use; for threaded scraping of one host, set `maxsize` to roughly your thread count so threads aren't discarding connections. `block=True` turns the pool into a semaphore — threads wait for a free connection instead of opening extras, useful to guarantee you never hold more than N connections to a host. Connections are returned to the pool automatically when you consume the body (or call `release_conn()` on streamed responses).

If you're only ever talking to one host, `urllib3.connectionpool.HTTPSConnectionPool(host, maxsize=...)` skips a dictionary lookup per request, but `PoolManager` is the right default.

## SSL: verification, certifi, and the errors you'll meet

urllib3 2.x verifies certificates against your system trust store by default. When you hit `SSLError: CERTIFICATE_VERIFY_FAILED`:

1. **Outdated CA bundle** — `pip install -U certifi` and point at it explicitly:

```python
import certifi
http = urllib3.PoolManager(ca_certs=certifi.where())
```

1. **Corporate MITM proxy or self-signed cert** — add the internal CA: `PoolManager(ca_certs="/path/to/company-ca.pem")`.
2. **Genuinely broken remote cert** (expired, wrong hostname) — decide consciously. Disabling verification is a last resort for data you'd trust over plain HTTP anyway:

```python
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
http = urllib3.PoolManager(cert_reqs="CERT_NONE") # you are now MITM-able
```

Client certificates (mutual TLS) go on the pool too: `PoolManager(cert_file="client.crt", key_file="client.key")`.

## Proxies: HTTP and SOCKS

```python
# HTTP/HTTPS proxy for everything this manager does
http = urllib3.ProxyManager("http://proxy.example.com:3128",
                            proxy_headers=urllib3.make_headers(
                                proxy_basic_auth="user:password"))
resp = http.request("GET", "https://httpbin.org/ip")
```

SOCKS needs an extra: `pip install "urllib3[socks]"`:

```python
from urllib3.contrib.socks import SOCKSProxyManager
http = SOCKSProxyManager("socks5h://user:password@proxy.example.com:1080")
```

(`socks5h` resolves DNS through the proxy — usually what you want for scraping.) One manager = one proxy; rotating proxies means creating managers per proxy or letting a [web scraping API](https://webscraping.ai/) rotate them server-side. Our [proxy provider comparison](https://webscraping.ai/blog/best-proxy-providers-for-web-scraping) covers where to get pools worth rotating.

## Streaming large responses

```python
resp = http.request("GET", big_file_url, preload_content=False)
with open("dump.bin", "wb") as out:
    for chunk in resp.stream(64 * 1024):
        out.write(chunk)
resp.release_conn() # hand the socket back to the pool
```

`preload_content=False` stops urllib3 from buffering the whole body in memory; `resp.stream()` (or `resp.read(n)`) pulls it incrementally. Forgetting `release_conn()` leaks the connection from the pool — in 2.x, `resp.close()` or fully draining also releases it.

## Errors, logging, and debugging

The exception hierarchy roots at `urllib3.exceptions.HTTPError`. The ones that matter:

```python
import urllib3
from urllib3.exceptions import (MaxRetryError, NewConnectionError,
                                NameResolutionError, ReadTimeoutError, SSLError)

try:
    resp = http.request("GET", url, timeout=5.0)
    if resp.status >= 400: # urllib3 does NOT raise on 4xx/5xx
        handle_http_error(resp.status)
except MaxRetryError as e: # wraps the final underlying cause
    print("gave up:", e.reason)
except ReadTimeoutError:
    ...
```

Unlike requests' `raise_for_status()`, urllib3 treats a 500 as a perfectly good response — check `resp.status` yourself (or put the status in `status_forcelist` so retries handle it).

For visibility, urllib3 logs through the standard `logging` module:

```python
import logging
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG) # connection + retry events
```

That shows connections opened, requests issued, retries scheduled, and warnings. For full wire dumps (headers and bodies), step up to a debugging proxy like mitmproxy — urllib3 doesn't print raw traffic.

## Thread safety and concurrency

`PoolManager` is thread-safe: share one instance across threads, don't create one per thread (that defeats pooling). A standard concurrent-fetch pattern:

```python
from concurrent.futures import ThreadPoolExecutor

http = urllib3.PoolManager(maxsize=20) # match worker count

def fetch(url):
    return url, http.request("GET", url, timeout=10.0).status

with ThreadPoolExecutor(max_workers=20) as pool:
    for url, status in pool.map(fetch, urls):
        print(status, url)
```

Note that urllib3 is synchronous — there's no asyncio support in the stable API. If your architecture is async, use `aiohttp` or `httpx` instead of forcing urllib3 into an event loop.

## urllib3 for web scraping

urllib3 fetches raw HTML fast, and pairs naturally with [lxml](https://webscraping.ai/blog/python-xml-parsing) or [Beautiful Soup](https://webscraping.ai/blog/beautiful-soup-python) for parsing. What it can't do: execute JavaScript, solve CAPTCHAs, or look like a residential user. When a site returns empty shells or 403s no matter what headers you send, that's not a urllib3 problem — the fetch itself needs a browser and better IPs. [WebScraping.AI](https://webscraping.ai/) does that behind one request — which you can make with urllib3, keeping the rest of your pipeline untouched:

```python
import urllib3

http = urllib3.PoolManager()
resp = http.request("GET", "https://api.webscraping.ai/html", fields={
    "api_key": API_KEY,
    "url": "https://example.com/spa-products",
    "js": "true", # rendered in a real browser, proxies included
})
html = resp.data.decode("utf-8")
```

The [/ai/fields endpoint](https://webscraping.ai/ai-web-scraping) goes further and returns structured JSON described in plain English — no selectors to maintain at all, which is what keeps a scheduled job like [price monitoring](https://webscraping.ai/use-cases/price-monitoring) running through site redesigns. The [API reference](https://webscraping.ai/docs) lists the rest of the parameters (`proxy`, `country`, `wait_for`, `device`), and your `Retry` config keeps working unchanged in front of it.

## Frequently asked questions

**Is urllib3 the same as urllib?** No. `urllib` is Python's built-in URL toolkit; urllib3 is an independent third-party package with connection pooling, retries, and modern TLS handling. The "3" is part of the project's name, not a stdlib version number — installing urllib3 doesn't replace or upgrade `urllib`.

**Should I use urllib3 or requests?** requests for application code — it's built on urllib3 and adds nicer ergonomics (sessions, `raise_for_status()`, auth helpers). urllib3 directly when you're writing a library and want minimal dependencies, need pool-level tuning, or want the `Retry` machinery without an extra layer. Performance differences are small; ergonomics and control are the real axis.

**Why am I getting "urllib3 v2 only supports OpenSSL 1.1.1+"?** Your Python was compiled against an old OpenSSL (common on older macOS system Pythons and some containers). Options: upgrade Python to a build with modern OpenSSL, or pin `urllib3<2` until you can. This is an import-time environment check, not a bug in your code.

**How do I send JSON with urllib3?** On urllib3 2.x, pass `json={"key": "value"}` to `request()` — it serializes and sets `Content-Type: application/json`. On 1.x, do it manually: `body=json.dumps(payload), headers={"Content-Type": "application/json"}`. Parse responses with `resp.json()` (2.x) or `json.loads(resp.data)`.

**Does urllib3 follow redirects?** Yes, by default (up to 3 in the pool default configuration, and redirects consume retries). Control it per request with `redirect=False` to get the 3xx response itself, or `retries=Retry(redirect=5)` for a custom limit. The final URL after redirects is on the response as `resp.geturl()` / `resp.url`.

**How do I verify which urllib3 version is installed?** `pip show urllib3` from the shell, or `import urllib3; print(urllib3. __version__ )` in code. If boto3/botocore is involved, check its pin before upgrading — `pip install -U urllib3` inside an AWS project frequently creates a dependency conflict that `pip check` will report.