Table of contents

What is the Purpose of the --include Flag in Curl?

The --include flag (or its short form -i) in curl is a fundamental option that instructs curl to include HTTP response headers in the output alongside the response body. This flag is essential for debugging HTTP requests, analyzing API responses, and understanding server behavior during web scraping and API development.

Understanding the --include Flag

By default, curl only displays the response body when making HTTP requests. However, HTTP headers contain crucial information about the response, including status codes, content types, cookies, caching directives, and custom headers. The --include flag makes this metadata visible, providing a complete picture of the HTTP transaction.

Basic Syntax

curl --include [URL]
curl -i [URL]

Practical Examples

Basic Usage

Let's start with a simple example to see the difference:

Without --include:

curl https://httpbin.org/json

Output:

{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      }
    ],
    "title": "Sample Slide Show"
  }
}

With --include:

curl --include https://httpbin.org/json

Output: ``` HTTP/2 200 date: Mon, 15 Jan 2024 10:30:45 GMT content-type: application/json content-length: 429 server: gunicorn/19.9.0 access-control-allow-origin: * access-control-allow-credentials: true

{ "slideshow": { "author": "Yours Truly", "date": "date of publication", "slides": [ { "title": "Wake up to WonderWidgets!", "type": "all" } ], "title": "Sample Slide Show" } } ```

Analyzing API Responses

When working with APIs, headers provide essential information for proper integration:

curl -i -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{"name": "John Doe", "email": "john@example.com"}'

The response might include: ``` HTTP/1.1 201 Created Location: https://api.example.com/users/12345 Content-Type: application/json X-RateLimit-Remaining: 99 X-RateLimit-Reset: 1641988245

{ "id": 12345, "name": "John Doe", "email": "john@example.com", "created_at": "2024-01-15T10:30:45Z" } ```

Debugging Authentication Issues

Headers are crucial for understanding authentication flows:

curl -i -H "Authorization: Bearer your-token-here" \
  https://api.example.com/protected-resource

This helps identify authentication problems through status codes and error headers: ``` HTTP/1.1 401 Unauthorized WWW-Authenticate: Bearer realm="api" Content-Type: application/json

{ "error": "Invalid or expired token" } ```

Key Use Cases for --include

1. Status Code Analysis

Headers reveal the exact HTTP status code and reason phrase:

curl -i https://httpbin.org/status/404

Output: HTTP/2 404 content-length: 0 server: gunicorn/19.9.0

2. Content Type Detection

Understanding response format through Content-Type headers:

curl -i https://api.github.com/users/octocat

Shows: HTTP/2 200 content-type: application/json; charset=utf-8

3. Cookie Management

Tracking session cookies and authentication tokens:

curl -i -c cookies.txt https://example.com/login

Headers might include: Set-Cookie: sessionid=abc123; Path=/; HttpOnly Set-Cookie: csrftoken=xyz789; Path=/

4. Rate Limiting Information

Many APIs include rate limiting headers:

curl -i https://api.github.com/user

Response headers: X-RateLimit-Limit: 60 X-RateLimit-Remaining: 59 X-RateLimit-Reset: 1641988245

5. Redirect Analysis

Understanding redirect chains and locations:

curl -i https://short.link/abc123

Shows redirect information: HTTP/1.1 301 Moved Permanently Location: https://example.com/full-url

Advanced Usage Patterns

Combining with Other Flags

The --include flag works well with other curl options:

# Follow redirects and show all headers
curl -i -L https://bit.ly/example

# Include headers with verbose output
curl -i -v https://api.example.com/data

# Save headers and body to separate files
curl -i -o response.txt https://api.example.com/data

Processing Headers in Scripts

When automating with scripts, you can extract specific header information:

Bash script example:

#!/bin/bash
response=$(curl -i -s https://api.example.com/data)
status_code=$(echo "$response" | head -n1 | cut -d' ' -f2)
content_type=$(echo "$response" | grep -i "content-type" | cut -d' ' -f2-)

echo "Status: $status_code"
echo "Content-Type: $content_type"

Python integration:

import subprocess
import re

def get_response_with_headers(url):
    result = subprocess.run(['curl', '-i', '-s', url], 
                          capture_output=True, text=True)
    response = result.stdout

    # Split headers and body
    parts = response.split('\r\n\r\n', 1)
    headers = parts[0]
    body = parts[1] if len(parts) > 1 else ''

    return headers, body

headers, body = get_response_with_headers('https://api.example.com/data')
print("Headers:", headers)
print("Body:", body)

JavaScript/Node.js example:

const { exec } = require('child_process');

function getCurlResponse(url) {
    return new Promise((resolve, reject) => {
        exec(`curl -i -s "${url}"`, (error, stdout, stderr) => {
            if (error) {
                reject(error);
                return;
            }

            const parts = stdout.split('\r\n\r\n');
            const headers = parts[0];
            const body = parts.slice(1).join('\r\n\r\n');

            resolve({ headers, body });
        });
    });
}

// Usage
getCurlResponse('https://api.example.com/data')
    .then(({ headers, body }) => {
        console.log('Headers:', headers);
        console.log('Body:', body);
    })
    .catch(console.error);

Comparison with Related Options

--include vs --head

  • --include (-i): Shows headers + body
  • --head (-I): Shows only headers (HEAD request)
# Get headers and body
curl -i https://example.com

# Get only headers
curl -I https://example.com

--include vs --dump-header

  • --include (-i): Headers included in stdout with body
  • --dump-header (-D): Headers saved to separate file
# Headers mixed with body
curl -i https://example.com

# Headers saved to file, body to stdout
curl -D headers.txt https://example.com

--include vs --verbose

  • --include (-i): Shows response headers only
  • --verbose (-v): Shows both request and response headers plus connection details
# Response headers only
curl -i https://example.com

# Complete transaction details
curl -v https://example.com

Best Practices

1. Use for Development and Debugging

Always use --include during API development to understand server responses:

curl -i -X POST https://api.example.com/webhooks \
  -H "Content-Type: application/json" \
  -d '{"url": "https://myapp.com/webhook", "events": ["push"]}'

2. Combine with Verbose Mode for Complete Analysis

For thorough debugging, combine with verbose output:

curl -i -v https://api.example.com/data

3. Save Complete Responses for Analysis

Preserve both headers and body for later analysis:

curl -i -o complete_response.txt https://api.example.com/data

4. Monitor API Changes

Use in monitoring scripts to track API behavior changes:

#!/bin/bash
curl -i -s https://api.example.com/status | grep -E "HTTP/|Server:|X-Version:"

5. Extract Specific Headers

Parse headers for specific information:

# Extract rate limit information
curl -i -s https://api.github.com/user | grep -i "x-ratelimit"

# Check content type
curl -i -s https://api.example.com/data | grep -i "content-type"

Common Troubleshooting Scenarios

SSL Certificate Issues

curl -i -k https://self-signed.example.com

Headers reveal SSL-related information and potential security warnings.

Content Encoding Problems

curl -i -H "Accept-Encoding: gzip" https://example.com

Shows compression-related headers: Content-Encoding: gzip Vary: Accept-Encoding

CORS Issues

curl -i -X OPTIONS -H "Origin: https://myapp.com" https://api.example.com

Reveals CORS headers: Access-Control-Allow-Origin: https://myapp.com Access-Control-Allow-Methods: GET, POST, PUT, DELETE Access-Control-Allow-Headers: Content-Type, Authorization

Authentication Token Validation

curl -i -H "Authorization: Bearer token123" https://api.example.com/me

Check authentication status and token validity through response headers and status codes.

Cache Behavior Analysis

curl -i -H "If-None-Match: W/\"etag-value\"" https://example.com/api/data

Understand caching behavior through ETag and cache-control headers: HTTP/1.1 304 Not Modified ETag: W/"etag-value" Cache-Control: max-age=3600

Integration with Modern Development Workflows

API Testing and Documentation

The --include flag is invaluable when monitoring network requests in Puppeteer or validating API responses during automated testing. It provides the same level of detail that browser developer tools show for HTTP transactions.

Debugging Web Scraping Issues

When handling authentication in Puppeteer or other scraping tools, curl with --include helps verify authentication headers and session management before implementing complex automation logic.

Performance Analysis

Use --include to analyze response times and server performance indicators:

curl -i -w "Time: %{time_total}s\nSize: %{size_download} bytes\n" \
  https://api.example.com/large-dataset

Real-World Use Cases

API Integration Testing

Before integrating with a third-party API, use --include to understand the complete response structure:

# Test OAuth token endpoint
curl -i -X POST https://api.service.com/oauth/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials&client_id=your_id&client_secret=your_secret"

Webhook Development

Validate webhook endpoints and understand response expectations:

curl -i -X POST https://yourapp.com/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: push" \
  -d '{"repository": {"name": "test-repo"}}'

CDN and Caching Analysis

Analyze content delivery and caching behavior:

curl -i https://cdn.example.com/static/app.js

This reveals important caching headers like: Cache-Control: public, max-age=31536000 ETag: "abc123" Last-Modified: Wed, 21 Oct 2023 07:28:00 GMT

Error Handling and Status Codes

Understanding HTTP Status Codes

The --include flag makes HTTP status codes immediately visible:

# Success responses (2xx)
curl -i https://api.example.com/users/123  # 200 OK

# Redirection responses (3xx)
curl -i https://old-api.example.com/users  # 301 Moved Permanently

# Client error responses (4xx)
curl -i https://api.example.com/users/999  # 404 Not Found

# Server error responses (5xx)
curl -i https://broken-api.example.com/data  # 500 Internal Server Error

Custom Error Handling

Process different status codes in scripts:

#!/bin/bash
response=$(curl -i -s https://api.example.com/data)
status=$(echo "$response" | head -n1 | awk '{print $2}')

case $status in
    200) echo "Success: Data retrieved" ;;
    404) echo "Error: Resource not found" ;;
    500) echo "Error: Server error" ;;
    *) echo "Unexpected status: $status" ;;
esac

Conclusion

The --include flag is an essential tool for any developer working with HTTP APIs, web scraping, or debugging network requests. It provides complete visibility into HTTP transactions, enabling better understanding of server behavior, proper error handling, and effective debugging.

Whether you're developing against REST APIs, troubleshooting authentication flows, analyzing response patterns, or integrating with third-party services, the --include flag transforms curl from a simple content fetcher into a powerful HTTP analysis tool.

Key benefits of using --include:

  • Complete HTTP transaction visibility: See both headers and body in one command
  • Debugging capabilities: Identify issues through status codes and error headers
  • API integration support: Understand authentication, rate limiting, and caching
  • Performance analysis: Monitor response times and content delivery
  • Development workflow enhancement: Validate endpoints before complex implementation

By incorporating --include into your development workflow, you gain insights that are crucial for building robust applications and understanding the full context of HTTP communications. This flag bridges the gap between simple command-line requests and comprehensive HTTP debugging, making it indispensable for modern web development and API integration tasks.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I use Curl to handle response headers?
What does the --resolve option do in Curl?
How do I use Curl to send requests with query parameters?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon