Table of contents

What should I do if I encounter an SSL error while using urllib3?

SSL errors in urllib3 are common when making HTTPS requests and can stem from various certificate issues, outdated configurations, or server-client mismatches. This guide provides systematic solutions to diagnose and resolve SSL errors effectively.

Common SSL Error Types

Before troubleshooting, identify your specific SSL error:

  • SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] - Certificate verification failed
  • SSLError: [SSL: WRONG_VERSION_NUMBER] - SSL/TLS version mismatch
  • SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] - Deprecated TLS version
  • MaxRetryError with SSL context - Connection SSL errors

Step-by-Step Solutions

1. Verify the Server's SSL Certificate

First, confirm the target server's SSL certificate is valid:

# Check certificate using openssl
openssl s_client -connect example.com:443 -servername example.com

# Or use online tools like SSL Labs SSL Test
# https://www.ssllabs.com/ssltest/

You can also verify in Python:

import ssl
import socket

def check_ssl_cert(hostname, port=443):
    context = ssl.create_default_context()
    with socket.create_connection((hostname, port)) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            print(f"SSL certificate for {hostname}:")
            print(f"Version: {ssock.version()}")
            print(f"Cipher: {ssock.cipher()}")
            cert = ssock.getpeercert()
            print(f"Subject: {cert['subject']}")
            print(f"Issuer: {cert['issuer']}")

check_ssl_cert('httpbin.org')

2. Update Certificate Bundle

Ensure your certificate bundle is current. urllib3 uses the certifi package:

# Update certifi to latest version
pip install --upgrade certifi urllib3

# Check certifi version and location
python -c "import certifi; print(certifi.where())"

3. Basic SSL Configuration

Configure urllib3 with proper SSL settings:

import urllib3
import certifi

# Standard SSL configuration
http = urllib3.PoolManager(
    cert_reqs='CERT_REQUIRED',
    ca_certs=certifi.where()
)

try:
    response = http.request('GET', 'https://httpbin.org/get')
    print(f"Status: {response.status}")
    print(response.data.decode('utf-8'))
except urllib3.exceptions.SSLError as e:
    print(f"SSL Error: {e}")

4. Custom Certificate Bundle

For internal certificates or custom CA bundles:

import urllib3

# Using custom CA bundle
http = urllib3.PoolManager(
    cert_reqs='CERT_REQUIRED',
    ca_certs='/path/to/custom-ca-bundle.pem'
)

# Or combine with system certificates
import certifi
custom_ca_bundle = certifi.where() + ',/path/to/additional-ca.pem'

http = urllib3.PoolManager(
    cert_reqs='CERT_REQUIRED',
    ca_certs=custom_ca_bundle
)

5. Client Certificate Authentication

When servers require client certificates:

import urllib3

# With client certificate
http = urllib3.PoolManager(
    cert_reqs='CERT_REQUIRED',
    cert_file='/path/to/client.pem',
    key_file='/path/to/client.key',
    ca_certs='/path/to/ca-bundle.pem'
)

response = http.request('GET', 'https://secure-api.example.com/')

6. Modern SSL/TLS Configuration

Configure specific TLS versions and ciphers:

import ssl
import urllib3

# Create custom SSL context
ssl_context = ssl.create_default_context()
ssl_context.minimum_version = ssl.TLSVersion.TLSv1_2
ssl_context.maximum_version = ssl.TLSVersion.TLSv1_3

# For servers requiring specific cipher suites
ssl_context.set_ciphers('ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:DHE+CHACHA20:!aNULL:!MD5:!DSS')

http = urllib3.PoolManager(ssl_context=ssl_context)
response = http.request('GET', 'https://tls-v1-2.badssl.com:1012/')

7. Debugging SSL Issues

Enable detailed logging to diagnose problems:

import logging
import urllib3

# Configure logging
logging.basicConfig(level=logging.DEBUG)
urllib3_logger = logging.getLogger("urllib3")
urllib3_logger.setLevel(logging.DEBUG)

# Add SSL-specific debugging
ssl_logger = logging.getLogger("ssl")
ssl_logger.setLevel(logging.DEBUG)

http = urllib3.PoolManager()

try:
    response = http.request('GET', 'https://expired.badssl.com/')
except Exception as e:
    print(f"Error details: {e}")

8. Handling Specific SSL Scenarios

Self-Signed Certificates (Development Only)

import urllib3
import ssl

# Create context that accepts self-signed certificates
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

http = urllib3.PoolManager(ssl_context=ssl_context)
response = http.request('GET', 'https://self-signed.badssl.com/')

Corporate Proxies and Firewalls

import urllib3

# Configure proxy with SSL
proxy_url = 'https://proxy.company.com:8080'
http = urllib3.ProxyManager(
    proxy_url,
    cert_reqs='CERT_REQUIRED',
    ca_certs='/etc/ssl/certs/ca-certificates.crt'
)

response = http.request('GET', 'https://external-api.com/')

9. Production-Ready Error Handling

Implement robust error handling for SSL issues:

import urllib3
import certifi
from urllib3.exceptions import SSLError, MaxRetryError
import time

def make_secure_request(url, max_retries=3):
    http = urllib3.PoolManager(
        cert_reqs='CERT_REQUIRED',
        ca_certs=certifi.where(),
        timeout=urllib3.Timeout(connect=10, read=30)
    )

    for attempt in range(max_retries):
        try:
            response = http.request('GET', url)
            return response
        except SSLError as e:
            print(f"SSL Error on attempt {attempt + 1}: {e}")
            if "CERTIFICATE_VERIFY_FAILED" in str(e):
                print("Certificate verification failed. Check server certificate.")
                break
        except MaxRetryError as e:
            print(f"Max retry error on attempt {attempt + 1}: {e}")
            if attempt < max_retries - 1:
                time.sleep(2 ** attempt)  # Exponential backoff
        except Exception as e:
            print(f"Unexpected error: {e}")
            break

    return None

# Usage
response = make_secure_request('https://httpbin.org/get')
if response:
    print(f"Success: {response.status}")

Security Best Practices

  1. Never disable SSL verification in production unless absolutely necessary
  2. Keep certificates updated - automate certificate bundle updates
  3. Use strong TLS versions - prefer TLS 1.2 or higher
  4. Validate certificate chains - ensure proper CA validation
  5. Monitor certificate expiration - implement alerts for expiring certificates

When to Disable SSL Verification

Only disable SSL verification for:

  • Development/testing environments with self-signed certificates
  • Controlled internal networks where security is managed differently
  • Temporary debugging to isolate SSL issues
import urllib3

# ONLY for development/testing
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
http = urllib3.PoolManager(cert_reqs='CERT_NONE')

# Always document why SSL verification is disabled
# TODO: Remove before production deployment
response = http.request('GET', 'https://internal-dev-server.local/')

Common Troubleshooting Steps

  1. Update packages: pip install --upgrade urllib3 certifi
  2. Check system time: Ensure system clock is accurate
  3. Verify network connectivity: Test basic HTTP connectivity first
  4. Check firewall rules: Ensure HTTPS traffic is allowed
  5. Test with curl: curl -v https://example.com to compare behavior

SSL errors in urllib3 are typically resolvable through proper certificate management and configuration. Always prioritize security while finding the appropriate solution for your specific use case.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon