Yes, you can use urllib3
with a SOCKS proxy by installing the PySocks
library. While urllib3
doesn't provide native SOCKS support, it includes a SOCKSProxyManager
in its contrib module that enables SOCKS proxy functionality.
Installation
Install the required packages:
pip install urllib3[socks]
# Or install separately:
# pip install urllib3 pysocks
Basic SOCKS Proxy Usage
Method 1: Using SOCKSProxyManager (Recommended)
from urllib3.contrib.socks import SOCKSProxyManager
# SOCKS5 proxy
http = SOCKSProxyManager("socks5://localhost:9050/")
# SOCKS4 proxy
http = SOCKSProxyManager("socks4://localhost:1080/")
# Make requests through the proxy
response = http.request('GET', 'http://httpbin.org/ip')
print(response.data.decode())
Method 2: Global Socket Patching
import socks
import socket
from urllib3 import PoolManager
# Configure global SOCKS proxy
socks.set_default_proxy(socks.SOCKS5, "localhost", 9050)
socket.socket = socks.socksocket
# Use regular PoolManager - all connections will use SOCKS
http = PoolManager()
response = http.request('GET', 'http://httpbin.org/ip')
print(response.data.decode())
SOCKS Proxy with Authentication
from urllib3.contrib.socks import SOCKSProxyManager
# SOCKS5 with username/password authentication
http = SOCKSProxyManager("socks5://username:password@proxy.example.com:1080/")
response = http.request('GET', 'http://httpbin.org/ip')
print(response.data.decode())
HTTPS Support
For HTTPS requests through SOCKS proxy:
from urllib3.contrib.socks import SOCKSProxyManager
import urllib3
# Disable SSL warnings if needed (not recommended for production)
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
# SOCKS proxy with SSL certificate verification
http = SOCKSProxyManager(
"socks5://localhost:9050/",
cert_reqs='CERT_REQUIRED',
ca_certs='/path/to/ca-certificates.crt'
)
# For testing, you can disable SSL verification
http = SOCKSProxyManager(
"socks5://localhost:9050/",
cert_reqs='CERT_NONE'
)
response = http.request('GET', 'https://httpbin.org/ip')
print(response.data.decode())
Error Handling
from urllib3.contrib.socks import SOCKSProxyManager
from urllib3.exceptions import ProxyError, HTTPError
import socks
try:
http = SOCKSProxyManager("socks5://localhost:9050/")
response = http.request('GET', 'http://httpbin.org/ip', timeout=10)
print(response.data.decode())
except socks.ProxyConnectionError:
print("Failed to connect to SOCKS proxy")
except ProxyError as e:
print(f"Proxy error: {e}")
except HTTPError as e:
print(f"HTTP error: {e}")
Different SOCKS Versions
from urllib3.contrib.socks import SOCKSProxyManager
# SOCKS4
socks4_proxy = SOCKSProxyManager("socks4://localhost:1080/")
# SOCKS4A (supports domain names)
socks4a_proxy = SOCKSProxyManager("socks4a://localhost:1080/")
# SOCKS5 (most common, supports authentication)
socks5_proxy = SOCKSProxyManager("socks5://localhost:1080/")
Testing Your SOCKS Proxy
from urllib3.contrib.socks import SOCKSProxyManager
import json
def test_socks_proxy(proxy_url):
try:
http = SOCKSProxyManager(proxy_url)
# Test with a service that returns your IP
response = http.request('GET', 'http://httpbin.org/ip')
ip_data = json.loads(response.data.decode())
print(f"Your IP through proxy: {ip_data['origin']}")
# Test HTTPS
response = http.request('GET', 'https://httpbin.org/ip')
ip_data = json.loads(response.data.decode())
print(f"Your HTTPS IP through proxy: {ip_data['origin']}")
return True
except Exception as e:
print(f"Proxy test failed: {e}")
return False
# Test your SOCKS proxy
test_socks_proxy("socks5://localhost:9050/")
Important Notes
SOCKSProxyManager
is part ofurllib3.contrib
, indicating it's a contributed module- SOCKS support requires the
PySocks
library to be installed - SOCKS5 is recommended over SOCKS4 for better security and feature support
- Always handle proxy connection errors gracefully in production code
- Consider connection pooling and timeout settings for better performance
Common Use Cases
SOCKS proxies are commonly used with:
- Tor network: SOCKS5 proxy typically runs on localhost:9050
- SSH tunneling: Create SOCKS proxy with ssh -D 1080 user@server
- VPN services: Many VPN providers offer SOCKS proxy endpoints
- Corporate networks: For accessing external resources through corporate proxies
Always ensure compliance with applicable laws and terms of service when using proxies for web scraping.