How can I handle HTTP connection errors and network failures?

Network failures and HTTP connection errors are inevitable challenges in web scraping and API interactions. Implementing robust error handling strategies ensures your applications remain resilient and can recover gracefully from temporary network issues, server downtime, or connection timeouts.

Understanding Common HTTP Connection Errors

Before implementing error handling, it's essential to understand the types of errors you might encounter:

Connection-Level Errors

• Connection Timeout: Server doesn't respond within the specified time
• Connection Refused: Target server actively refuses the connection
• DNS Resolution Failures: Unable to resolve the hostname
• Network Unreachable: Network infrastructure issues

HTTP Status Code Errors

• 4xx Client Errors: 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 429 Too Many Requests
• 5xx Server Errors: 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout

Implementing Retry Logic with Exponential Backoff

Python Implementation with requests

import requests
import time
import random
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

class RobustHTTPClient:
    def __init__(self, max_retries=3, backoff_factor=1, timeout=30):
        self.session = requests.Session()
        self.timeout = timeout

        # Configure retry strategy
        retry_strategy = Retry(
            total=max_retries,
            status_forcelist=[429, 500, 502, 503, 504],
            method_whitelist=["HEAD", "GET", "OPTIONS"],
            backoff_factor=backoff_factor
        )

        adapter = HTTPAdapter(max_retries=retry_strategy)
        self.session.mount("http://", adapter)
        self.session.mount("https://", adapter)

    def get_with_retry(self, url, **kwargs):
        """Make HTTP GET request with custom retry logic"""
        max_attempts = 3
        base_delay = 1

        for attempt in range(max_attempts):
            try:
                response = self.session.get(
                    url, 
                    timeout=self.timeout,
                    **kwargs
                )
                response.raise_for_status()
                return response

            except requests.exceptions.ConnectionError as e:
                if attempt == max_attempts - 1:
                    raise ConnectionError(f"Failed to connect after {max_attempts} attempts: {e}")
                self._wait_with_jitter(base_delay * (2 ** attempt))

            except requests.exceptions.Timeout as e:
                if attempt == max_attempts - 1:
                    raise TimeoutError(f"Request timed out after {max_attempts} attempts: {e}")
                self._wait_with_jitter(base_delay * (2 ** attempt))

            except requests.exceptions.HTTPError as e:
                status_code = e.response.status_code
                if status_code in [429, 500, 502, 503, 504] and attempt < max_attempts - 1:
                    self._wait_with_jitter(base_delay * (2 ** attempt))
                else:
                    raise e

            except requests.exceptions.RequestException as e:
                if attempt == max_attempts - 1:
                    raise e
                self._wait_with_jitter(base_delay * (2 ** attempt))

    def _wait_with_jitter(self, delay):
        """Add random jitter to prevent thundering herd"""
        jitter = random.uniform(0.1, 0.5)
        time.sleep(delay + jitter)

# Usage example
client = RobustHTTPClient(max_retries=5, backoff_factor=2, timeout=30)

try:
    response = client.get_with_retry("https://api.example.com/data")
    print(f"Success: {response.status_code}")
    data = response.json()
except ConnectionError as e:
    print(f"Connection failed: {e}")
except TimeoutError as e:
    print(f"Request timed out: {e}")
except requests.exceptions.HTTPError as e:
    print(f"HTTP error: {e.response.status_code}")

JavaScript Implementation with axios

const axios = require('axios');

class RobustHTTPClient {
    constructor(maxRetries = 3, baseDelay = 1000, timeout = 30000) {
        this.maxRetries = maxRetries;
        this.baseDelay = baseDelay;
        this.timeout = timeout;

        // Configure axios instance
        this.client = axios.create({
            timeout: this.timeout,
            headers: {
                'User-Agent': 'RobustHTTPClient/1.0'
            }
        });

        // Add response interceptor for automatic retries
        this.client.interceptors.response.use(
            response => response,
            error => this.handleError(error)
        );
    }

    async handleError(error) {
        const config = error.config;

        // Initialize retry count
        config.__retryCount = config.__retryCount || 0;

        // Check if we should retry
        if (this.shouldRetry(error) && config.__retryCount < this.maxRetries) {
            config.__retryCount++;

            const delay = this.calculateDelay(config.__retryCount);
            await this.sleep(delay);

            return this.client.request(config);
        }

        return Promise.reject(error);
    }

    shouldRetry(error) {
        // Retry on network errors or specific HTTP status codes
        if (error.code === 'ECONNABORTED' || error.code === 'ENOTFOUND' || 
            error.code === 'ECONNREFUSED' || error.code === 'ETIMEDOUT') {
            return true;
        }

        if (error.response) {
            const status = error.response.status;
            return [429, 500, 502, 503, 504].includes(status);
        }

        return false;
    }

    calculateDelay(attempt) {
        // Exponential backoff with jitter
        const exponentialDelay = this.baseDelay * Math.pow(2, attempt - 1);
        const jitter = Math.random() * 1000;
        return exponentialDelay + jitter;
    }

    sleep(ms) {
        return new Promise(resolve => setTimeout(resolve, ms));
    }

    async get(url, config = {}) {
        try {
            const response = await this.client.get(url, config);
            return response.data;
        } catch (error) {
            throw this.formatError(error);
        }
    }

    formatError(error) {
        if (error.code === 'ECONNABORTED') {
            return new Error(`Request timeout: ${error.message}`);
        } else if (error.code === 'ENOTFOUND') {
            return new Error(`DNS resolution failed: ${error.message}`);
        } else if (error.code === 'ECONNREFUSED') {
            return new Error(`Connection refused: ${error.message}`);
        } else if (error.response) {
            return new Error(`HTTP ${error.response.status}: ${error.response.statusText}`);
        } else {
            return new Error(`Network error: ${error.message}`);
        }
    }
}

// Usage example
const client = new RobustHTTPClient(5, 1000, 30000);

async function fetchData() {
    try {
        const data = await client.get('https://api.example.com/data');
        console.log('Success:', data);
    } catch (error) {
        console.error('Failed to fetch data:', error.message);
    }
}

fetchData();

Circuit Breaker Pattern

The circuit breaker pattern prevents cascading failures by temporarily stopping requests to a failing service:

import time
from enum import Enum

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=60, expected_exception=Exception):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.expected_exception = expected_exception

        self.failure_count = 0
        self.last_failure_time = None
        self.state = CircuitState.CLOSED

    def call(self, func, *args, **kwargs):
        if self.state == CircuitState.OPEN:
            if self._should_attempt_reset():
                self.state = CircuitState.HALF_OPEN
            else:
                raise Exception("Circuit breaker is OPEN")

        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except self.expected_exception as e:
            self._on_failure()
            raise e

    def _should_attempt_reset(self):
        return (time.time() - self.last_failure_time) >= self.recovery_timeout

    def _on_success(self):
        self.failure_count = 0
        self.state = CircuitState.CLOSED

    def _on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()

        if self.failure_count >= self.failure_threshold:
            self.state = CircuitState.OPEN

# Usage with HTTP requests
def make_request(url):
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    return response.json()

circuit_breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=30)

try:
    data = circuit_breaker.call(make_request, "https://api.example.com/data")
    print("Data retrieved successfully")
except Exception as e:
    print(f"Request failed: {e}")

Advanced Error Handling Strategies

Connection Pooling and Keep-Alive

import requests
from requests.adapters import HTTPAdapter

class OptimizedHTTPClient:
    def __init__(self, pool_connections=10, pool_maxsize=10):
        self.session = requests.Session()

        # Configure connection pooling
        adapter = HTTPAdapter(
            pool_connections=pool_connections,
            pool_maxsize=pool_maxsize,
            max_retries=0  # Handle retries manually
        )

        self.session.mount('http://', adapter)
        self.session.mount('https://', adapter)

        # Set keep-alive headers
        self.session.headers.update({
            'Connection': 'keep-alive',
            'Keep-Alive': 'timeout=30, max=100'
        })

    def get(self, url, **kwargs):
        return self.session.get(url, **kwargs)

Fallback Mechanisms

class MultiEndpointClient:
    def __init__(self, endpoints, timeout=30):
        self.endpoints = endpoints
        self.timeout = timeout
        self.current_endpoint = 0

    def get_data(self, path):
        """Try multiple endpoints until one succeeds"""
        last_error = None

        for i in range(len(self.endpoints)):
            endpoint_index = (self.current_endpoint + i) % len(self.endpoints)
            endpoint = self.endpoints[endpoint_index]

            try:
                url = f"{endpoint}{path}"
                response = requests.get(url, timeout=self.timeout)
                response.raise_for_status()

                # Update successful endpoint for next request
                self.current_endpoint = endpoint_index
                return response.json()

            except requests.exceptions.RequestException as e:
                last_error = e
                print(f"Endpoint {endpoint} failed: {e}")
                continue

        raise Exception(f"All endpoints failed. Last error: {last_error}")

# Usage
endpoints = [
    "https://primary-api.example.com",
    "https://backup-api.example.com",
    "https://fallback-api.example.com"
]

client = MultiEndpointClient(endpoints)
try:
    data = client.get_data("/api/v1/data")
except Exception as e:
    print(f"All endpoints failed: {e}")

Integration with Web Scraping Tools

When working with browser automation tools, error handling becomes even more critical. For comprehensive error handling in browser-based scraping, you might want to explore how to handle errors in Puppeteer for browser-specific error scenarios.

Timeout Management

# Configure curl with comprehensive timeout settings
curl --connect-timeout 10 \
     --max-time 30 \
     --retry 3 \
     --retry-delay 5 \
     --retry-max-time 120 \
     "https://api.example.com/data"

# Monitor network connectivity
ping -c 4 google.com
traceroute google.com
nslookup api.example.com

Monitoring and Logging

Implement comprehensive logging to track connection issues:

import logging
import requests
from datetime import datetime

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('http_errors.log'),
        logging.StreamHandler()
    ]
)

class MonitoredHTTPClient:
    def __init__(self):
        self.session = requests.Session()
        self.metrics = {
            'total_requests': 0,
            'successful_requests': 0,
            'failed_requests': 0,
            'connection_errors': 0,
            'timeout_errors': 0
        }

    def get(self, url, **kwargs):
        self.metrics['total_requests'] += 1
        start_time = datetime.now()

        try:
            response = self.session.get(url, **kwargs)
            duration = (datetime.now() - start_time).total_seconds()

            logging.info(f"HTTP {response.status_code} - {url} - {duration:.2f}s")

            if response.status_code >= 400:
                self.metrics['failed_requests'] += 1
                logging.warning(f"HTTP error {response.status_code} for {url}")
            else:
                self.metrics['successful_requests'] += 1

            return response

        except requests.exceptions.ConnectionError as e:
            self.metrics['connection_errors'] += 1
            logging.error(f"Connection error for {url}: {e}")
            raise

        except requests.exceptions.Timeout as e:
            self.metrics['timeout_errors'] += 1
            logging.error(f"Timeout error for {url}: {e}")
            raise

    def get_metrics(self):
        return self.metrics.copy()

Best Practices Summary

1. Implement exponential backoff with random jitter to avoid thundering herd problems
2. Set appropriate timeouts for both connection and read operations
3. Use connection pooling to improve performance and reduce overhead
4. Implement circuit breakers to prevent cascading failures
5. Log errors comprehensively for debugging and monitoring
6. Plan fallback strategies including alternative endpoints or cached data
7. Monitor network metrics to identify patterns and improve reliability

For additional context on handling network-related issues in browser automation scenarios, consider reviewing how to handle timeouts in Puppeteer which covers timeout management in browser-controlled environments.

By implementing these robust error handling strategies, your web scraping and API integration applications will be much more resilient to network failures and connection issues, ensuring better reliability and user experience.