What is the difference between urllib and urllib3 in Python?

Python offers different HTTP client libraries, with urllib and urllib3 being two popular options. While they serve similar purposes, they differ significantly in features, performance, and complexity.

urllib (Built-in Standard Library)

urllib is Python's built-in HTTP client package, included with every Python installation. It consists of several modules that handle different URL-related tasks:

urllib Modules

urllib.request - Opens and reads URLs
urllib.error - Exception handling for urllib.request
urllib.parse - URL parsing utilities
urllib.robotparser - Parsing robots.txt files

urllib Features and Limitations

Pros: - No installation required (built into Python) - Simple for basic HTTP requests - Lightweight and minimal dependencies

Cons: - Limited connection management - No connection pooling - Poor performance for multiple requests - Complex API for advanced features - No automatic retry mechanisms - Manual cookie and session handling

urllib Example

import urllib.request
import urllib.parse

# Simple GET request
response = urllib.request.urlopen('https://httpbin.org/get')
data = response.read().decode('utf-8')
print(data)

# POST request with data
post_data = urllib.parse.urlencode({'key': 'value'}).encode('utf-8')
req = urllib.request.Request('https://httpbin.org/post', data=post_data)
response = urllib.request.urlopen(req)
print(response.read().decode('utf-8'))

# Custom headers
req = urllib.request.Request('https://httpbin.org/headers')
req.add_header('User-Agent', 'Custom Agent')
response = urllib.request.urlopen(req)
print(response.read().decode('utf-8'))

urllib3 (Third-party Library)

urllib3 is a powerful, feature-rich HTTP client library that must be installed separately. It's designed for production applications requiring robust HTTP handling.

Installation

pip install urllib3

urllib3 Key Features

Connection pooling - Reuses connections for better performance
Thread safety - Safe for concurrent operations
Automatic retries - Built-in retry logic with backoff
SSL/TLS verification - Comprehensive certificate validation
File uploads - Multipart form data support
Compression - Automatic gzip/deflate handling
Timeout control - Fine-grained timeout configuration
Proxy support - HTTP and HTTPS proxy handling

urllib3 Examples

import urllib3

# Basic usage with connection pooling
http = urllib3.PoolManager()

# Simple GET request
response = http.request('GET', 'https://httpbin.org/get')
print(response.data.decode('utf-8'))
print(f"Status: {response.status}")

# POST request with JSON data
import json
data = {'name': 'John', 'age': 30}
response = http.request(
    'POST', 
    'https://httpbin.org/post',
    body=json.dumps(data),
    headers={'Content-Type': 'application/json'}
)
print(response.data.decode('utf-8'))

# Custom headers and timeout
response = http.request(
    'GET', 
    'https://httpbin.org/delay/2',
    headers={'User-Agent': 'urllib3-client'},
    timeout=urllib3.Timeout(connect=2.0, read=5.0)
)

# File upload
with open('example.txt', 'rb') as f:
    response = http.request(
        'POST',
        'https://httpbin.org/post',
        fields={'file': ('example.txt', f.read(), 'text/plain')}
    )

# Retry configuration
from urllib3.util.retry import Retry
retry_strategy = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504]
)
http = urllib3.PoolManager(retries=retry_strategy)

Performance Comparison

Connection Efficiency

import time

# urllib - creates new connection each time
def urllib_multiple_requests():
    start = time.time()
    for i in range(10):
        response = urllib.request.urlopen('https://httpbin.org/get')
        response.read()
    return time.time() - start

# urllib3 - reuses connections via pooling
def urllib3_multiple_requests():
    http = urllib3.PoolManager()
    start = time.time()
    for i in range(10):
        response = http.request('GET', 'https://httpbin.org/get')
        response.data
    return time.time() - start

# urllib3 is typically 2-3x faster for multiple requests

When to Use Each

Use urllib when:

Simple, one-off HTTP requests
Minimal dependencies required
Basic web scraping tasks
Learning HTTP concepts
Python environments where third-party packages aren't allowed

Use urllib3 when:

Production applications
High-volume web scraping
Need connection pooling
Require advanced features (retries, timeouts, SSL control)
Building robust HTTP clients
Performance is critical

Integration with Other Libraries

Many popular Python libraries use urllib3 internally:

requests - High-level HTTP library built on urllib3
botocore - AWS SDKs use urllib3
httpx - Modern HTTP client (alternative to requests)

# requests uses urllib3 under the hood
import requests
response = requests.get('https://httpbin.org/get')

# Equivalent urllib3 code
import urllib3
http = urllib3.PoolManager()
response = http.request('GET', 'https://httpbin.org/get')

Summary

| Feature | urllib | urllib3 | |---------|--------|---------| | Installation | Built-in | pip install urllib3 | | Connection Pooling | No | Yes | | Thread Safety | Limited | Yes | | Performance | Basic | High | | Retry Logic | Manual | Built-in | | SSL/TLS Control | Basic | Advanced | | API Complexity | Complex for advanced use | Consistent and intuitive | | Memory Usage | Lower | Higher (due to pooling) |

For modern Python applications, especially those involving web scraping or API consumption, urllib3 is generally the better choice due to its superior performance, feature set, and ease of use. However, urllib remains useful for simple scripts where minimal dependencies are preferred.