How do I set custom headers with Curl?

Setting custom HTTP headers with Curl is essential for web scraping, API interactions, and mimicking browser behavior. The -H (or --header) flag allows you to specify custom headers in your HTTP requests, enabling you to authenticate, set content types, modify user agents, and more.

Basic Syntax for Custom Headers

The fundamental syntax for setting custom headers with Curl is:

curl -H "Header-Name: Header-Value" https://example.com

You can specify multiple headers by using the -H flag multiple times:

curl -H "Header-1: Value-1" -H "Header-2: Value-2" https://example.com

Common Use Cases for Custom Headers

1. Setting User-Agent Headers

One of the most common scenarios is setting a custom User-Agent to mimic different browsers:

# Chrome User-Agent
curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" https://example.com

# Mobile Safari User-Agent
curl -H "User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1" https://example.com

2. Authentication Headers

Bearer Token Authentication

curl -H "Authorization: Bearer YOUR_ACCESS_TOKEN" https://api.example.com/data

Basic Authentication

curl -H "Authorization: Basic $(echo -n 'username:password' | base64)" https://api.example.com/data

API Key Authentication

curl -H "X-API-Key: your-api-key-here" https://api.example.com/data

3. Content-Type Headers for POST Requests

When sending data with POST requests, setting the correct Content-Type is crucial:

# JSON data
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"name": "John", "email": "john@example.com"}' \
  https://api.example.com/users

# Form data
curl -X POST \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "name=John&email=john@example.com" \
  https://api.example.com/users

# XML data
curl -X POST \
  -H "Content-Type: application/xml" \
  -d '<user><name>John</name><email>john@example.com</email></user>' \
  https://api.example.com/users

4. Accept Headers

Specify the response format you prefer:

# Request JSON response
curl -H "Accept: application/json" https://api.example.com/data

# Request XML response
curl -H "Accept: application/xml" https://api.example.com/data

# Request HTML response
curl -H "Accept: text/html" https://example.com

Advanced Header Examples

Setting Multiple Headers for Web Scraping

When scraping websites that have anti-bot measures, you might need to set multiple headers to appear more like a real browser:

curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
     -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" \
     -H "Accept-Language: en-US,en;q=0.5" \
     -H "Accept-Encoding: gzip, deflate" \
     -H "Connection: keep-alive" \
     -H "Upgrade-Insecure-Requests: 1" \
     https://example.com

Custom Headers for APIs

Many APIs require specific headers for proper functionality:

# GitHub API example
curl -H "Accept: application/vnd.github.v3+json" \
     -H "Authorization: token YOUR_GITHUB_TOKEN" \
     https://api.github.com/user/repos

# Stripe API example
curl -H "Authorization: Bearer sk_test_..." \
     -H "Content-Type: application/x-www-form-urlencoded" \
     -X POST \
     -d "amount=2000&currency=usd" \
     https://api.stripe.com/v1/charges

Referer Headers

Set the referer header to indicate where the request is coming from:

curl -H "Referer: https://google.com" https://example.com/page

Custom Headers for CORS

When dealing with Cross-Origin Resource Sharing:

curl -H "Origin: https://example.com" \
     -H "Access-Control-Request-Method: POST" \
     -H "Access-Control-Request-Headers: Content-Type" \
     -X OPTIONS \
     https://api.example.com/data

Working with Header Files

For complex requests with many headers, you can store them in a file:

Create a headers file (headers.txt): User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Accept: application/json Authorization: Bearer YOUR_TOKEN Content-Type: application/json X-Custom-Header: custom-value

Then use it with Curl:

curl -H @headers.txt https://api.example.com/data

Header Manipulation and Removal

Removing Default Headers

To remove headers that Curl sets by default, set them to an empty value:

# Remove User-Agent header
curl -H "User-Agent:" https://example.com

# Remove Accept header
curl -H "Accept:" https://example.com

Overriding Default Headers

You can override Curl's default headers by simply setting them:

# Override default Accept header
curl -H "Accept: application/json" https://example.com

Integration with Web Scraping Workflows

When using Curl for web scraping, custom headers are often essential. For more complex scenarios involving JavaScript-rendered content, you might need to combine Curl with other tools. For instance, while Curl handles static content efficiently, handling authentication in browser automation tools becomes necessary for dynamic content.

Debugging and Monitoring Headers

Viewing Request Headers

Use the -v (verbose) flag to see the headers being sent:

curl -v -H "Custom-Header: value" https://example.com

Viewing Response Headers Only

To see only response headers without the body:

curl -I -H "Custom-Header: value" https://example.com

Including Response Headers in Output

To include response headers in the output:

curl -i -H "Custom-Header: value" https://example.com

Best Practices for Custom Headers

Use Realistic User-Agents: When web scraping, use realistic user-agent strings from actual browsers to avoid detection.
Set Appropriate Accept Headers: Always set Accept headers that match the content type you expect to receive.
Include Standard Browser Headers: For web scraping, include headers like Accept-Language, Accept-Encoding, and Connection to mimic real browser behavior.
Respect Rate Limits: When using API keys in headers, ensure you respect the API's rate limiting policies.
Secure Sensitive Headers: Never expose API keys or tokens in logs or version control. Use environment variables:

curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/data

Error Handling and Troubleshooting

Common Header-Related Errors

401 Unauthorized: Check your authentication headers
415 Unsupported Media Type: Verify your Content-Type header
403 Forbidden: Your User-Agent might be blocked

Testing Headers

Test your headers with a service that echoes them back:

curl -H "Test-Header: value" https://httpbin.org/headers

Performance Considerations

When setting custom headers, be aware that:

Some servers may reject requests with too many custom headers
Large header values can impact request performance
Some CDNs and proxies may strip certain custom headers

For scenarios requiring more sophisticated header management and session handling, consider using browser automation tools that can monitor network requests and responses for comprehensive web scraping workflows.

Conclusion

Setting custom headers with Curl is straightforward using the -H flag. Whether you're authenticating with APIs, mimicking browser behavior for web scraping, or setting content types for data submission, mastering custom headers is essential for effective HTTP client usage. Remember to always respect website terms of service and rate limits when using these techniques for web scraping or API interactions.

Table of contents