Table of contents

How do I debug HTTP requests and responses in Requests?

Debugging HTTP requests and responses is a crucial skill when working with the Python Requests library. Whether you're building web scrapers, API clients, or automated testing tools, understanding how to effectively debug network interactions can save you hours of troubleshooting. This guide covers comprehensive debugging techniques, from basic response inspection to advanced logging and monitoring approaches.

Basic Response Inspection

The simplest way to debug requests is by examining the response object properties. The Requests library provides extensive information about both the request and response.

import requests

# Make a request
response = requests.get('https://httpbin.org/get')

# Basic debugging information
print(f"Status Code: {response.status_code}")
print(f"URL: {response.url}")
print(f"Headers: {response.headers}")
print(f"Content Type: {response.headers.get('content-type')}")
print(f"Response Time: {response.elapsed.total_seconds()}s")

# Response content
print(f"Response Text: {response.text[:200]}...")  # First 200 chars
print(f"Response JSON: {response.json()}")  # If JSON response

Request Object Inspection

Access the original request details through the response object:

import requests

response = requests.post('https://httpbin.org/post', 
                        json={'key': 'value'},
                        headers={'User-Agent': 'My App 1.0'})

# Inspect the request that was sent
request = response.request
print(f"Request Method: {request.method}")
print(f"Request URL: {request.url}")
print(f"Request Headers: {dict(request.headers)}")
print(f"Request Body: {request.body}")

Advanced Logging with urllib3

Enable detailed HTTP logging to see the raw HTTP traffic:

import logging
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

# Enable urllib3 debug logging
logging.basicConfig(level=logging.DEBUG)
urllib3_logger = logging.getLogger('urllib3')
urllib3_logger.setLevel(logging.DEBUG)

# Make request with detailed logging
response = requests.get('https://httpbin.org/get')

This will output detailed information including: - Connection pool creation - HTTP request headers - HTTP response headers - SSL handshake details - Connection reuse information

Custom Debug Session

Create a custom session with built-in debugging capabilities:

import requests
import json
from datetime import datetime

class DebugSession(requests.Session):
    def request(self, method, url, **kwargs):
        # Log request details
        print(f"\n[{datetime.now()}] {method} {url}")
        print(f"Headers: {json.dumps(dict(kwargs.get('headers', {})), indent=2)}")

        if 'data' in kwargs:
            print(f"Data: {kwargs['data']}")
        if 'json' in kwargs:
            print(f"JSON: {json.dumps(kwargs['json'], indent=2)}")

        # Make the request
        start_time = datetime.now()
        response = super().request(method, url, **kwargs)
        end_time = datetime.now()

        # Log response details
        print(f"Status: {response.status_code}")
        print(f"Response Time: {(end_time - start_time).total_seconds()}s")
        print(f"Response Headers: {json.dumps(dict(response.headers), indent=2)}")

        return response

# Usage
session = DebugSession()
response = session.get('https://httpbin.org/get')

HTTP Event Hooks

Use request hooks to debug specific events during the request lifecycle:

import requests

def log_request(request, *args, **kwargs):
    print(f"Sending request to {request.url}")
    print(f"Request headers: {dict(request.headers)}")

def log_response(response, *args, **kwargs):
    print(f"Received response: {response.status_code}")
    print(f"Response headers: {dict(response.headers)}")
    print(f"Response time: {response.elapsed.total_seconds()}s")

# Create session with hooks
session = requests.Session()
session.hooks['response'].append(log_response)

# Make request
response = session.get('https://httpbin.org/get', hooks={'response': log_response})

Debugging Connection Issues

Handle and debug common connection problems:

import requests
from requests.exceptions import (
    ConnectionError, 
    Timeout, 
    TooManyRedirects, 
    RequestException
)

def debug_request(url, **kwargs):
    try:
        response = requests.get(url, timeout=10, **kwargs)
        response.raise_for_status()
        return response
    except ConnectionError as e:
        print(f"Connection Error: {e}")
        print(f"Failed to connect to {url}")
    except Timeout as e:
        print(f"Timeout Error: {e}")
        print(f"Request to {url} timed out")
    except TooManyRedirects as e:
        print(f"Redirect Error: {e}")
        print(f"Too many redirects for {url}")
    except requests.HTTPError as e:
        print(f"HTTP Error: {e}")
        print(f"Status Code: {e.response.status_code}")
        print(f"Response Text: {e.response.text}")
    except RequestException as e:
        print(f"Request Error: {e}")

    return None

# Usage
response = debug_request('https://httpbin.org/status/404')

Debugging SSL and Certificate Issues

Handle SSL-related debugging:

import requests
import ssl
from requests.adapters import HTTPAdapter
from urllib3.util.ssl_ import create_urllib3_context

# Custom SSL context for debugging
def create_debug_ssl_context():
    context = create_urllib3_context()
    context.check_hostname = False
    context.verify_mode = ssl.CERT_NONE
    return context

# Debug SSL issues
def debug_ssl_request(url):
    try:
        # First try with SSL verification
        response = requests.get(url, verify=True, timeout=10)
        print("SSL verification successful")
        return response
    except requests.exceptions.SSLError as e:
        print(f"SSL Error: {e}")

        # Try without SSL verification (for debugging only)
        try:
            response = requests.get(url, verify=False, timeout=10)
            print("Request successful without SSL verification")
            print("WARNING: SSL verification disabled - use only for debugging")
            return response
        except Exception as inner_e:
            print(f"Request failed even without SSL verification: {inner_e}")

    return None

Request/Response Middleware

Create middleware to automatically log all requests and responses:

import requests
import json
import time
from functools import wraps

def debug_requests(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print(f"\n=== DEBUG: {func.__name__} ===")
        print(f"Args: {args}")
        print(f"Kwargs: {kwargs}")

        start_time = time.time()
        try:
            result = func(*args, **kwargs)
            end_time = time.time()

            if hasattr(result, 'status_code'):
                print(f"Status Code: {result.status_code}")
                print(f"URL: {result.url}")
                print(f"Request Time: {end_time - start_time:.3f}s")
                print(f"Response Size: {len(result.content)} bytes")

            return result
        except Exception as e:
            end_time = time.time()
            print(f"Error after {end_time - start_time:.3f}s: {e}")
            raise

    return wrapper

# Apply debugging to requests functions
requests.get = debug_requests(requests.get)
requests.post = debug_requests(requests.post)

# Now all requests will be automatically debugged
response = requests.get('https://httpbin.org/get')

Environment-Based Debug Configuration

Set up conditional debugging based on environment variables:

import os
import requests
import logging

# Configure debug level based on environment
DEBUG_LEVEL = os.getenv('HTTP_DEBUG_LEVEL', 'INFO').upper()

if DEBUG_LEVEL == 'DEBUG':
    # Enable all HTTP debugging
    logging.basicConfig(level=logging.DEBUG)
    logging.getLogger('urllib3').setLevel(logging.DEBUG)

    # Enable request/response body logging
    import http.client as http_client
    http_client.HTTPConnection.debuglevel = 1

def make_request(method, url, **kwargs):
    """Enhanced request function with configurable debugging"""

    if DEBUG_LEVEL in ['DEBUG', 'VERBOSE']:
        print(f"\n[REQUEST] {method.upper()} {url}")
        if kwargs.get('headers'):
            print(f"Headers: {kwargs['headers']}")
        if kwargs.get('data'):
            print(f"Data: {kwargs['data']}")
        if kwargs.get('json'):
            print(f"JSON: {kwargs['json']}")

    try:
        response = requests.request(method, url, **kwargs)

        if DEBUG_LEVEL in ['DEBUG', 'VERBOSE']:
            print(f"[RESPONSE] Status: {response.status_code}")
            print(f"Response Headers: {dict(response.headers)}")

        if DEBUG_LEVEL == 'DEBUG':
            print(f"Response Body: {response.text[:500]}...")

        return response

    except Exception as e:
        if DEBUG_LEVEL in ['DEBUG', 'VERBOSE', 'ERROR']:
            print(f"[ERROR] {type(e).__name__}: {e}")
        raise

# Usage
response = make_request('GET', 'https://httpbin.org/get')

Debugging with Network Monitoring Tools

Integrate with external tools for advanced debugging. When working with complex web scraping scenarios that require JavaScript execution, you might want to compare your Requests behavior with browser-based tools like monitoring network requests in Puppeteer to understand differences in request patterns.

import requests
import mitmproxy
from requests.packages.urllib3.exceptions import InsecureRequestWarning

# Disable SSL warnings when using proxy
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Configure proxy for traffic inspection
proxies = {
    'http': 'http://localhost:8080',
    'https': 'http://localhost:8080'
}

# Make requests through proxy for inspection
response = requests.get('https://httpbin.org/get', 
                       proxies=proxies, 
                       verify=False)

Performance Debugging

Monitor and debug performance issues:

import requests
import time
from contextlib import contextmanager

@contextmanager
def time_request(description="Request"):
    start = time.time()
    try:
        yield
    finally:
        end = time.time()
        print(f"{description} took {end - start:.3f} seconds")

def debug_performance():
    session = requests.Session()

    # Test connection pooling
    with time_request("First request"):
        response1 = session.get('https://httpbin.org/get')

    with time_request("Second request (should be faster due to connection reuse)"):
        response2 = session.get('https://httpbin.org/get')

    # Test with new session (new connection)
    with time_request("New session request"):
        response3 = requests.get('https://httpbin.org/get')

    print(f"Connection info:")
    print(f"Response 1 - Connection: {response1.headers.get('connection', 'unknown')}")
    print(f"Response 2 - Connection: {response2.headers.get('connection', 'unknown')}")

debug_performance()

Using Command Line Tools for HTTP Debugging

For quick debugging, you can also use command-line tools alongside your Python scripts:

# Use curl to test the same endpoint
curl -v https://httpbin.org/get

# Monitor network traffic with tcpdump (Linux/macOS)
sudo tcpdump -i any -A -s 0 'tcp port 80 or tcp port 443'

# Use netstat to check connection status
netstat -an | grep :80

Best Practices for Production Debugging

  1. Use appropriate log levels: Don't enable DEBUG logging in production
  2. Sanitize sensitive data: Remove authentication tokens and personal data from logs
  3. Implement structured logging: Use JSON logging for better parsing
  4. Monitor response times: Track performance metrics
  5. Handle errors gracefully: Provide meaningful error messages
import logging
import json
from datetime import datetime

# Production-ready debug configuration
class ProductionDebugger:
    def __init__(self, log_level=logging.INFO):
        self.logger = logging.getLogger(__name__)
        self.logger.setLevel(log_level)

        handler = logging.StreamHandler()
        formatter = logging.Formatter(
            '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
        )
        handler.setFormatter(formatter)
        self.logger.addHandler(handler)

    def sanitize_headers(self, headers):
        """Remove sensitive information from headers"""
        sensitive_keys = ['authorization', 'cookie', 'x-api-key']
        return {k: v if k.lower() not in sensitive_keys else '[REDACTED]' 
                for k, v in headers.items()}

    def log_request(self, method, url, **kwargs):
        log_data = {
            'type': 'request',
            'method': method,
            'url': url,
            'timestamp': datetime.now().isoformat()
        }

        if 'headers' in kwargs:
            log_data['headers'] = self.sanitize_headers(kwargs['headers'])

        self.logger.info(json.dumps(log_data))

    def log_response(self, response):
        log_data = {
            'type': 'response',
            'status_code': response.status_code,
            'url': response.url,
            'response_time': response.elapsed.total_seconds(),
            'content_length': len(response.content),
            'timestamp': datetime.now().isoformat()
        }

        self.logger.info(json.dumps(log_data))

        if response.status_code >= 400:
            self.logger.error(f"HTTP Error {response.status_code}: {response.text[:200]}")

# Usage
debugger = ProductionDebugger()

Advanced Debugging with Pytest Fixtures

When writing tests, create reusable debugging fixtures:

import pytest
import requests
from unittest.mock import patch

@pytest.fixture
def debug_session():
    """Fixture that provides a session with debug logging enabled"""
    session = requests.Session()
    session.hooks['response'].append(lambda r, *args, **kwargs: print(f"Response: {r.status_code}"))
    return session

@pytest.fixture
def mock_response():
    """Fixture for mocking HTTP responses during testing"""
    with patch('requests.get') as mock_get:
        mock_response = mock_get.return_value
        mock_response.status_code = 200
        mock_response.json.return_value = {'test': 'data'}
        yield mock_response

def test_api_call(debug_session):
    response = debug_session.get('https://httpbin.org/get')
    assert response.status_code == 200

Common Debugging Scenarios

Debugging Authentication Issues

import requests
from requests.auth import HTTPBasicAuth

def debug_auth(url, username, password):
    """Debug authentication issues"""

    # Test without auth first
    try:
        response = requests.get(url)
        print(f"No auth - Status: {response.status_code}")
    except Exception as e:
        print(f"No auth failed: {e}")

    # Test with basic auth
    try:
        response = requests.get(url, auth=HTTPBasicAuth(username, password))
        print(f"Basic auth - Status: {response.status_code}")
        print(f"Auth header sent: {response.request.headers.get('Authorization', 'None')}")
    except Exception as e:
        print(f"Basic auth failed: {e}")

# Usage
debug_auth('https://httpbin.org/basic-auth/user/pass', 'user', 'pass')

Debugging Encoding Issues

import requests
import chardet

def debug_encoding(url):
    """Debug text encoding issues"""
    response = requests.get(url)

    print(f"Response encoding (detected): {response.encoding}")
    print(f"Response encoding (apparent): {response.apparent_encoding}")

    # Detect encoding using chardet
    detected = chardet.detect(response.content)
    print(f"Chardet detection: {detected}")

    # Try different encodings
    for encoding in ['utf-8', 'latin-1', 'cp1252']:
        try:
            text = response.content.decode(encoding)
            print(f"Encoding {encoding}: Success (first 100 chars: {text[:100]})")
        except UnicodeDecodeError as e:
            print(f"Encoding {encoding}: Failed - {e}")

# Usage
debug_encoding('https://httpbin.org/encoding/utf8')

Conclusion

Effective debugging of HTTP requests and responses in the Requests library requires a combination of built-in tools, custom logging, and systematic approaches. Start with basic response inspection for simple issues, then progress to advanced logging and monitoring for complex scenarios. When dealing with JavaScript-heavy applications that require browser automation, you may need to complement Requests debugging with tools for handling browser sessions in Puppeteer to get a complete picture of the application's network behavior.

Remember to balance debugging detail with performance and security considerations, especially in production environments. The techniques covered in this guide will help you quickly identify and resolve HTTP-related issues in your Python applications, from simple API calls to complex web scraping scenarios.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I handle chunked transfer encoding with Requests?
How do I set connection and read timeouts separately?
How do I handle SSL/TLS certificate verification errors?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon