Table of contents

Can I use urllib3 with a SOCKS proxy?

Yes, you can use urllib3 with a SOCKS proxy by installing the PySocks library. While urllib3 doesn't provide native SOCKS support, it includes a SOCKSProxyManager in its contrib module that enables SOCKS proxy functionality.

Installation

Install the required packages:

pip install urllib3[socks]
# Or install separately:
# pip install urllib3 pysocks

Basic SOCKS Proxy Usage

Method 1: Using SOCKSProxyManager (Recommended)

from urllib3.contrib.socks import SOCKSProxyManager

# SOCKS5 proxy
http = SOCKSProxyManager("socks5://localhost:9050/")

# SOCKS4 proxy
http = SOCKSProxyManager("socks4://localhost:1080/")

# Make requests through the proxy
response = http.request('GET', 'http://httpbin.org/ip')
print(response.data.decode())

Method 2: Global Socket Patching

import socks
import socket
from urllib3 import PoolManager

# Configure global SOCKS proxy
socks.set_default_proxy(socks.SOCKS5, "localhost", 9050)
socket.socket = socks.socksocket

# Use regular PoolManager - all connections will use SOCKS
http = PoolManager()
response = http.request('GET', 'http://httpbin.org/ip')
print(response.data.decode())

SOCKS Proxy with Authentication

from urllib3.contrib.socks import SOCKSProxyManager

# SOCKS5 with username/password authentication
http = SOCKSProxyManager("socks5://username:password@proxy.example.com:1080/")

response = http.request('GET', 'http://httpbin.org/ip')
print(response.data.decode())

HTTPS Support

For HTTPS requests through SOCKS proxy:

from urllib3.contrib.socks import SOCKSProxyManager
import urllib3

# Disable SSL warnings if needed (not recommended for production)
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

# SOCKS proxy with SSL certificate verification
http = SOCKSProxyManager(
    "socks5://localhost:9050/",
    cert_reqs='CERT_REQUIRED',
    ca_certs='/path/to/ca-certificates.crt'
)

# For testing, you can disable SSL verification
http = SOCKSProxyManager(
    "socks5://localhost:9050/",
    cert_reqs='CERT_NONE'
)

response = http.request('GET', 'https://httpbin.org/ip')
print(response.data.decode())

Error Handling

from urllib3.contrib.socks import SOCKSProxyManager
from urllib3.exceptions import ProxyError, HTTPError
import socks

try:
    http = SOCKSProxyManager("socks5://localhost:9050/")
    response = http.request('GET', 'http://httpbin.org/ip', timeout=10)
    print(response.data.decode())

except socks.ProxyConnectionError:
    print("Failed to connect to SOCKS proxy")
except ProxyError as e:
    print(f"Proxy error: {e}")
except HTTPError as e:
    print(f"HTTP error: {e}")

Different SOCKS Versions

from urllib3.contrib.socks import SOCKSProxyManager

# SOCKS4
socks4_proxy = SOCKSProxyManager("socks4://localhost:1080/")

# SOCKS4A (supports domain names)
socks4a_proxy = SOCKSProxyManager("socks4a://localhost:1080/")

# SOCKS5 (most common, supports authentication)
socks5_proxy = SOCKSProxyManager("socks5://localhost:1080/")

Testing Your SOCKS Proxy

from urllib3.contrib.socks import SOCKSProxyManager
import json

def test_socks_proxy(proxy_url):
    try:
        http = SOCKSProxyManager(proxy_url)

        # Test with a service that returns your IP
        response = http.request('GET', 'http://httpbin.org/ip')
        ip_data = json.loads(response.data.decode())
        print(f"Your IP through proxy: {ip_data['origin']}")

        # Test HTTPS
        response = http.request('GET', 'https://httpbin.org/ip')
        ip_data = json.loads(response.data.decode())
        print(f"Your HTTPS IP through proxy: {ip_data['origin']}")

        return True
    except Exception as e:
        print(f"Proxy test failed: {e}")
        return False

# Test your SOCKS proxy
test_socks_proxy("socks5://localhost:9050/")

Important Notes

  • SOCKSProxyManager is part of urllib3.contrib, indicating it's a contributed module
  • SOCKS support requires the PySocks library to be installed
  • SOCKS5 is recommended over SOCKS4 for better security and feature support
  • Always handle proxy connection errors gracefully in production code
  • Consider connection pooling and timeout settings for better performance

Common Use Cases

SOCKS proxies are commonly used with: - Tor network: SOCKS5 proxy typically runs on localhost:9050 - SSH tunneling: Create SOCKS proxy with ssh -D 1080 user@server - VPN services: Many VPN providers offer SOCKS proxy endpoints - Corporate networks: For accessing external resources through corporate proxies

Always ensure compliance with applicable laws and terms of service when using proxies for web scraping.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon