Can I use proxies with urllib3 and how would I configure them?

Yes, urllib3 provides comprehensive proxy support for HTTP, HTTPS, and SOCKS proxies through its ProxyManager and SOCKSProxyManager classes.

HTTP Proxy Configuration

For basic HTTP proxy configuration, use ProxyManager:

import urllib3

# Basic HTTP proxy
proxy_url = 'http://proxy.example.com:8080'
http = urllib3.ProxyManager(proxy_url)

# Make requests through the proxy
response = http.request('GET', 'http://httpbin.org/ip')
print(response.data.decode('utf-8'))

HTTPS Proxy Configuration

For HTTPS proxies, specify the HTTPS scheme:

import urllib3

# HTTPS proxy
proxy_url = 'https://secure-proxy.example.com:8080'
http = urllib3.ProxyManager(proxy_url)

# Works with both HTTP and HTTPS target URLs
response = http.request('GET', 'https://httpbin.org/ip')
print(response.data.decode('utf-8'))

Proxy Authentication

For authenticated proxies, include credentials in the URL or use headers:

import urllib3

# Method 1: Credentials in URL
proxy_url = 'http://username:password@proxy.example.com:8080'
http = urllib3.ProxyManager(proxy_url)

# Method 2: Using proxy_headers
http = urllib3.ProxyManager(
    'http://proxy.example.com:8080',
    proxy_headers=urllib3.make_headers(proxy_basic_auth='username:password')
)

response = http.request('GET', 'http://httpbin.org/ip')
print(response.data.decode('utf-8'))

SOCKS Proxy Configuration

For SOCKS proxies, install the required dependency and use SOCKSProxyManager:

pip install urllib3[socks]

import urllib3
from urllib3.contrib.socks import SOCKSProxyManager

# SOCKS5 proxy
socks_proxy = 'socks5://proxy.example.com:1080'
http = SOCKSProxyManager(socks_proxy)

# SOCKS5 with authentication
auth_socks_proxy = 'socks5://username:password@proxy.example.com:1080'
http = SOCKSProxyManager(auth_socks_proxy)

# SOCKS4 proxy
socks4_proxy = 'socks4://proxy.example.com:1080'
http = SOCKSProxyManager(socks4_proxy)

response = http.request('GET', 'http://httpbin.org/ip')
print(response.data.decode('utf-8'))

Advanced Proxy Configuration

Configure additional proxy settings like timeouts and retry behavior:

import urllib3

http = urllib3.ProxyManager(
    'http://proxy.example.com:8080',
    timeout=urllib3.Timeout(connect=10, read=30),
    retries=urllib3.Retry(total=3, backoff_factor=0.3),
    maxsize=10,  # Connection pool size
    headers={'User-Agent': 'MyApp/1.0'}
)

response = http.request('GET', 'http://httpbin.org/headers')
print(response.data.decode('utf-8'))

SSL Certificate Handling

When using HTTPS proxies, you may need to handle SSL certificates:

import urllib3

# Disable SSL warnings (not recommended for production)
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

# Option 1: Disable SSL verification (insecure)
http = urllib3.ProxyManager(
    'https://proxy.example.com:8080',
    cert_reqs='CERT_NONE'
)

# Option 2: Use custom CA bundle
http = urllib3.ProxyManager(
    'https://proxy.example.com:8080',
    ca_certs='/path/to/ca-bundle.crt'
)

# Option 3: Use system CA bundle
http = urllib3.ProxyManager(
    'https://proxy.example.com:8080',
    cert_reqs='CERT_REQUIRED',
    ca_certs=urllib3.util.ssl_.DEFAULT_CIPHERS
)

Environment Variable Configuration

Configure proxies using environment variables:

import urllib3
import os

# urllib3 will automatically use these environment variables
os.environ['HTTP_PROXY'] = 'http://proxy.example.com:8080'
os.environ['HTTPS_PROXY'] = 'https://proxy.example.com:8080'
os.environ['NO_PROXY'] = 'localhost,127.0.0.1,.local'

# Standard PoolManager will respect proxy environment variables
http = urllib3.PoolManager()
response = http.request('GET', 'http://httpbin.org/ip')

Error Handling and Best Practices

Implement proper error handling when using proxies:

import urllib3
from urllib3.exceptions import ProxyError, MaxRetryError

try:
    http = urllib3.ProxyManager('http://proxy.example.com:8080')
    response = http.request('GET', 'http://httpbin.org/ip', timeout=10)
    print(f"Response: {response.status}")
    print(response.data.decode('utf-8'))

except ProxyError as e:
    print(f"Proxy connection failed: {e}")
except MaxRetryError as e:
    print(f"Request failed after retries: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")

Security Considerations

• Trust: Only use trusted proxy servers, especially for sensitive data
• SSL Verification: Avoid disabling SSL verification in production
• Credentials: Store proxy credentials securely, not in source code
• Logging: Be aware that proxy servers may log your traffic
• DNS Leaks: SOCKS proxies help prevent DNS leaks compared to HTTP proxies

Choose the appropriate proxy type based on your security and performance requirements.