What is the Purpose of the --verbose Flag in Curl?

The --verbose flag (or -v for short) in Curl is one of the most powerful debugging tools available for HTTP requests and web scraping operations. It provides detailed information about the entire request-response cycle, making it essential for troubleshooting connectivity issues, analyzing server responses, and understanding how your HTTP requests are being processed.

Understanding Curl's Verbose Output

When you use the --verbose flag, Curl displays comprehensive information about:

Connection establishment - DNS lookup, TCP connection, and SSL handshake details
Request headers - All headers sent to the server
Response headers - All headers received from the server
Protocol information - HTTP version, status codes, and redirects
SSL/TLS details - Certificate information and encryption protocols
Timing information - Connection and transfer speeds

Basic Verbose Usage

Here's a simple example of using the verbose flag:

curl --verbose https://httpbin.org/get

This command will output something like:

* Trying 54.175.219.8:443...
* Connected to httpbin.org (54.175.219.8) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=httpbin.org
*  start date: Dec 28 2023 23:30:00 GMT
*  expire date: Mar 27 2024 23:30:00 GMT
*  subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b8c5c0a2e0)
> GET /get HTTP/2
> Host: httpbin.org
> user-agent: curl/7.81.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Thu, 18 Jul 2024 10:30:00 GMT
< content-type: application/json
< content-length: 258
< server: gunicorn/19.9.0
< access-control-allow-origin: *
< access-control-allow-credentials: true
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.81.0"
  },
  "origin": "203.0.113.1",
  "url": "https://httpbin.org/get"
}
* Connection #0 to host httpbin.org left open

Interpreting Verbose Output

Connection Information

Lines starting with * show connection details: - DNS resolution and IP address - Port numbers and connection IDs - SSL/TLS handshake process - Certificate validation

Request Headers

Lines starting with > show the request being sent: ```

GET /get HTTP/2 Host: httpbin.org user-agent: curl/7.81.0 accept: / ```

Response Headers

Lines starting with < show the server's response headers: < HTTP/2 200 < date: Thu, 18 Jul 2024 10:30:00 GMT < content-type: application/json < content-length: 258

Practical Examples for Web Scraping

Debugging Failed Requests

When a web scraping request fails, verbose output helps identify the issue:

curl --verbose https://example.com/api/data

This might reveal: - SSL certificate problems - Redirect loops - Authentication requirements - Rate limiting responses

Analyzing Response Headers

Understanding server responses is crucial for web scraping:

curl --verbose --head https://example.com

This shows only headers, useful for checking: - Content-Type for parsing strategy - Cache-Control headers - Rate limiting headers (X-RateLimit-*) - Server software information

Following Redirects with Details

curl --verbose --location https://bit.ly/example

The verbose output shows each redirect step, helping you understand redirect chains that might affect your scraping logic.

Advanced Verbose Options

Combining with Other Flags

# Verbose output with custom headers
curl --verbose --header "Authorization: Bearer token123" https://api.example.com

# Verbose POST request with data
curl --verbose --data '{"key":"value"}' --header "Content-Type: application/json" https://api.example.com/submit

# Verbose output with proxy
curl --verbose --proxy http://proxy.example.com:8080 https://target.com

Saving Verbose Output

# Save verbose output to file
curl --verbose https://example.com 2> debug.log

# Include timing information
curl --verbose --write-out "Total time: %{time_total}s\n" https://example.com

Web Scraping Applications

API Development and Testing

When developing web scraping solutions that interact with APIs, verbose output helps verify:

# Check API authentication
curl --verbose --header "X-API-Key: your-key" https://api.example.com/data

# Verify rate limiting headers
curl --verbose https://api.example.com/endpoint

SSL/TLS Troubleshooting

For HTTPS websites, verbose output reveals SSL issues:

# Check certificate details
curl --verbose https://secure-site.com

# Disable certificate verification for testing
curl --verbose --insecure https://self-signed-site.com

Performance Analysis

Verbose output combined with timing options provides performance insights:

curl --verbose --write-out "DNS: %{time_namelookup}s, Connect: %{time_connect}s, Total: %{time_total}s\n" https://example.com

Scripting with Verbose Output

When automating web scraping tasks, you can capture and parse verbose output:

Python Example

import subprocess
import re

def get_response_code(url):
    result = subprocess.run(['curl', '--verbose', '--silent', url], 
                          capture_output=True, text=True)

    # Parse HTTP status from stderr (verbose output)
    status_match = re.search(r'< HTTP/[\d.]+ (\d+)', result.stderr)
    if status_match:
        return int(status_match.group(1))
    return None

# Usage
status = get_response_code('https://httpbin.org/get')
print(f"Response status: {status}")

JavaScript/Node.js Example

const { spawn } = require('child_process');

function curlVerbose(url) {
    return new Promise((resolve, reject) => {
        const curl = spawn('curl', ['--verbose', '--silent', url]);
        let stderr = '';
        let stdout = '';

        curl.stderr.on('data', (data) => {
            stderr += data.toString();
        });

        curl.stdout.on('data', (data) => {
            stdout += data.toString();
        });

        curl.on('close', (code) => {
            const headers = parseHeaders(stderr);
            resolve({ 
                body: stdout, 
                headers: headers,
                verboseOutput: stderr 
            });
        });
    });
}

function parseHeaders(verboseOutput) {
    const responseHeaders = {};
    const lines = verboseOutput.split('\n');

    lines.forEach(line => {
        if (line.startsWith('< ') && line.includes(': ')) {
            const [key, value] = line.substring(2).split(': ', 2);
            responseHeaders[key.toLowerCase()] = value;
        }
    });

    return responseHeaders;
}

// Usage
curlVerbose('https://httpbin.org/get')
    .then(result => {
        console.log('Response headers:', result.headers);
        console.log('Content-Type:', result.headers['content-type']);
    });

Best Practices for Using Verbose Output

1. Security Considerations

Be careful when logging verbose output in production, as it may contain sensitive information
Remove authentication headers from logs
Use --trace-ascii for even more detailed debugging when needed

2. Performance Impact

Verbose output adds overhead to requests
Disable verbose mode in production scraping scripts
Use selectively for debugging specific issues

3. Log Management

Redirect verbose output to files for analysis: curl -v url 2> debug.log
Parse logs programmatically for automated monitoring
Rotate logs to prevent disk space issues

Common Troubleshooting Scenarios

Connection Timeouts

curl --verbose --connect-timeout 10 --max-time 30 https://slow-site.com

Proxy Issues

curl --verbose --proxy http://proxy:8080 --proxy-user username:password https://target.com

Cookie Handling

curl --verbose --cookie-jar cookies.txt --cookie cookies.txt https://site-with-sessions.com

Conclusion

The --verbose flag in Curl is an indispensable tool for web scraping and HTTP debugging. It provides complete visibility into the request-response cycle, enabling developers to troubleshoot issues, optimize performance, and understand server behavior. Whether you're debugging a failed API call, analyzing response headers for scraping logic, or investigating SSL certificate issues, verbose output gives you the detailed information needed to solve complex web scraping challenges.

By mastering the verbose flag and understanding how to interpret its output, you'll be better equipped to build robust web scraping solutions and diagnose issues quickly when they arise. Remember to use it judiciously in production environments and always consider the security implications of logging detailed request information.

Table of contents