Table of contents

How to Handle Response Headers with Curl

HTTP response headers contain crucial metadata about web responses, including content type, server information, caching directives, and security headers. Understanding how to capture and analyze these headers with curl is essential for effective web scraping, API testing, and debugging web applications.

Understanding HTTP Response Headers

Response headers provide valuable information about the server's response, including: - Content-Type: The media type of the response body - Content-Length: Size of the response body in bytes - Set-Cookie: Cookie information for session management - Location: Redirect destination for 3xx status codes - Cache-Control: Caching directives - Server: Information about the web server - Date: Timestamp when the response was generated

Basic Header Display with curl

Display Headers Only

Use the -I or --head flag to retrieve only the response headers:

curl -I https://httpbin.org/get

This sends a HEAD request and displays only the headers:

HTTP/2 200 
date: Wed, 19 Jul 2025 10:30:45 GMT
content-type: application/json
content-length: 314
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

Include Headers with Response Body

Use the -i or --include flag to display headers along with the response body:

curl -i https://httpbin.org/get

This shows both headers and content in a single output.

Advanced Header Handling Techniques

Save Headers to a File

Use the -D or --dump-header option to save headers to a file:

curl -D headers.txt https://httpbin.org/get

The headers will be saved to headers.txt while the response body is displayed normally.

Output Headers to Separate Files

# Save headers and body to different files
curl -D response_headers.txt -o response_body.html https://example.com

Extract Specific Headers

Use grep to filter specific headers:

curl -I https://httpbin.org/get | grep -i content-type

Output: content-type: application/json

Parsing Headers in Scripts

Bash Script Example

#!/bin/bash

# Function to extract header value
get_header() {
    local url=$1
    local header_name=$2
    curl -s -I "$url" | grep -i "^$header_name:" | cut -d' ' -f2- | tr -d '\r'
}

# Usage examples
url="https://httpbin.org/get"
content_type=$(get_header "$url" "content-type")
server=$(get_header "$url" "server")

echo "Content-Type: $content_type"
echo "Server: $server"

Python Script for Header Processing

import subprocess
import re

def get_response_headers(url):
    """Extract all response headers using curl"""
    result = subprocess.run(
        ['curl', '-s', '-I', url],
        capture_output=True,
        text=True
    )

    headers = {}
    for line in result.stdout.split('\n'):
        if ':' in line:
            key, value = line.split(':', 1)
            headers[key.strip().lower()] = value.strip()

    return headers

# Usage
url = "https://httpbin.org/get"
headers = get_response_headers(url)

print(f"Content-Type: {headers.get('content-type', 'Not found')}")
print(f"Server: {headers.get('server', 'Not found')}")
print(f"Date: {headers.get('date', 'Not found')}")

JavaScript/Node.js Implementation

const { exec } = require('child_process');

function getResponseHeaders(url) {
    return new Promise((resolve, reject) => {
        exec(`curl -s -I "${url}"`, (error, stdout, stderr) => {
            if (error) {
                reject(error);
                return;
            }

            const headers = {};
            const lines = stdout.split('\n');

            lines.forEach(line => {
                if (line.includes(':')) {
                    const [key, ...valueParts] = line.split(':');
                    headers[key.trim().toLowerCase()] = valueParts.join(':').trim();
                }
            });

            resolve(headers);
        });
    });
}

// Usage
(async () => {
    try {
        const headers = await getResponseHeaders('https://httpbin.org/get');
        console.log('Content-Type:', headers['content-type']);
        console.log('Server:', headers['server']);
        console.log('Date:', headers['date']);
    } catch (error) {
        console.error('Error:', error);
    }
})();

Handling Specific Header Scenarios

Following Redirects and Capturing Headers

# Follow redirects and show headers for each step
curl -I -L https://httpbin.org/redirect/3

The -L flag follows redirects, and you'll see headers for each redirect step.

Handling Authentication Headers

# Check authentication requirements
curl -I https://httpbin.org/basic-auth/user/pass

# With authentication
curl -I -u user:pass https://httpbin.org/basic-auth/user/pass

Checking Cache Headers

# Extract cache-related headers
curl -I https://httpbin.org/cache/60 | grep -E "(cache-control|expires|etag|last-modified)"

Advanced Header Analysis

Check for Security Headers

#!/bin/bash

check_security_headers() {
    local url=$1
    echo "Security Headers Analysis for: $url"
    echo "========================================"

    headers=$(curl -s -I "$url")

    # Check for important security headers
    echo "Strict-Transport-Security: $(echo "$headers" | grep -i "strict-transport-security" || echo "Missing")"
    echo "Content-Security-Policy: $(echo "$headers" | grep -i "content-security-policy" || echo "Missing")"
    echo "X-Frame-Options: $(echo "$headers" | grep -i "x-frame-options" || echo "Missing")"
    echo "X-Content-Type-Options: $(echo "$headers" | grep -i "x-content-type-options" || echo "Missing")"
}

check_security_headers "https://example.com"

Monitor Header Changes

#!/bin/bash

# Monitor headers over time
monitor_headers() {
    local url=$1
    local interval=${2:-60}

    while true; do
        timestamp=$(date '+%Y-%m-%d %H:%M:%S')
        echo "[$timestamp] Checking headers for $url"
        curl -s -I "$url" | grep -E "(server|date|content-length)" >> header_log.txt
        echo "---" >> header_log.txt
        sleep $interval
    done
}

monitor_headers "https://api.example.com/status" 300

Troubleshooting Common Issues

Handle Different HTTP Versions

# Force HTTP/1.1
curl -I --http1.1 https://example.com

# Force HTTP/2
curl -I --http2 https://example.com

Deal with Malformed Headers

# More lenient parsing
curl -I --ignore-content-length https://problematic-site.com

Debug Header Issues

# Verbose output for debugging
curl -v -I https://example.com 2>&1 | grep -E "(> |< )"

Best Practices for Header Handling

  1. Always Check Status Codes: Parse the first line to understand response status
  2. Handle Case Sensitivity: HTTP headers are case-insensitive; normalize when processing
  3. Account for Multiple Values: Some headers can appear multiple times
  4. Set Timeouts: Use --max-time to prevent hanging requests
  5. Follow Redirects Carefully: Use -L but be aware of redirect loops

Production-Ready Header Parser

#!/bin/bash

parse_response_headers() {
    local url=$1
    local timeout=${2:-30}
    local max_redirects=${3:-5}

    # Create temporary file for headers
    local header_file=$(mktemp)

    # Make request with error handling
    if curl -s -I -L \
        --max-time "$timeout" \
        --max-redirs "$max_redirects" \
        -D "$header_file" \
        "$url" > /dev/null; then

        # Parse status code
        local status=$(head -n1 "$header_file" | cut -d' ' -f2)
        echo "Status: $status"

        # Extract common headers
        echo "Content-Type: $(grep -i '^content-type:' "$header_file" | cut -d' ' -f2- | tr -d '\r')"
        echo "Content-Length: $(grep -i '^content-length:' "$header_file" | cut -d' ' -f2- | tr -d '\r')"
        echo "Server: $(grep -i '^server:' "$header_file" | cut -d' ' -f2- | tr -d '\r')"

    else
        echo "Error: Failed to retrieve headers from $url"
        rm -f "$header_file"
        return 1
    fi

    # Cleanup
    rm -f "$header_file"
}

# Usage
parse_response_headers "https://httpbin.org/get"

Integration with Web Scraping Workflows

When building web scraping applications, response headers provide critical information for:

  • Rate Limiting: Check X-RateLimit-* headers
  • Content Validation: Verify Content-Type before parsing
  • Caching Strategy: Use Cache-Control and ETag headers
  • Authentication Status: Monitor authentication-related headers

While curl excels at command-line header analysis, modern web scraping tools like Puppeteer for handling browser sessions or monitoring network requests in Puppeteer provide more sophisticated header handling capabilities for JavaScript-heavy applications.

Conclusion

Mastering curl's header handling capabilities is essential for web developers, system administrators, and security professionals. Whether you're debugging API responses, monitoring web services, or building automated testing scripts, the techniques covered in this guide provide a solid foundation for working with HTTP response headers effectively.

Remember to combine header analysis with proper error handling, timeout management, and security considerations to build robust applications that can handle real-world web interactions reliably.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

What does the --resolve option do in Curl?
How do I use Curl to send requests with query parameters?
What is the difference between --head and --request HEAD in Curl?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon