What Does the --write-out Option Do in cURL?
The --write-out (or -w) option in cURL is a powerful feature that allows you to extract detailed information about HTTP requests and responses. This option enables you to output various metrics, timing data, and response details that are essential for debugging, performance monitoring, and automation scripts. Unlike the response body, which is written to stdout by default, the --write-out output provides metadata about the request execution.
Basic Syntax and Usage
The basic syntax for the --write-out option is:
curl --write-out "format_string" [URL]
# or
curl -w "format_string" [URL]
The format string can contain literal text mixed with variable names that cURL will replace with actual values.
Simple Example
curl -w "Status: %{http_code}\n" https://httpbin.org/status/200
This command will output the HTTP status code after the response body.
Available Variables
cURL provides numerous variables that you can use with --write-out. Here are the most commonly used ones:
HTTP Response Variables
%{http_code}- HTTP status code (200, 404, 500, etc.)%{http_version}- HTTP version used (1.0, 1.1, 2.0)%{response_code}- Numerical response code%{redirect_url}- URL that would be followed on redirect
Timing Variables
%{time_total}- Total time for the complete operation%{time_namelookup}- Time for DNS lookup%{time_connect}- Time to establish connection%{time_appconnect}- Time for SSL/TLS handshake%{time_pretransfer}- Time from start until file transfer begins%{time_starttransfer}- Time until first byte is received%{time_redirect}- Time for all redirects
Size and Transfer Variables
%{size_download}- Total bytes downloaded%{size_upload}- Total bytes uploaded%{size_header}- Total bytes of headers received%{size_request}- Total bytes sent in request%{speed_download}- Average download speed (bytes/sec)%{speed_upload}- Average upload speed (bytes/sec)
Connection Variables
%{num_connects}- Number of new connections made%{num_redirects}- Number of redirects followed%{ssl_verify_result}- SSL certificate verification result
Practical Examples
Performance Monitoring
Create a comprehensive timing report:
curl -w "DNS lookup: %{time_namelookup}s\nConnect: %{time_connect}s\nSSL handshake: %{time_appconnect}s\nPre-transfer: %{time_pretransfer}s\nStart transfer: %{time_starttransfer}s\nTotal: %{time_total}s\nHTTP code: %{http_code}\nSize: %{size_download} bytes\nSpeed: %{speed_download} bytes/sec\n" \
https://example.com
Health Check Script
#!/bin/bash
response=$(curl -s -w "%{http_code},%{time_total},%{size_download}" https://api.example.com/health)
status_code=$(echo $response | cut -d',' -f1)
response_time=$(echo $response | cut -d',' -f2)
response_size=$(echo $response | cut -d',' -f3)
echo "Status: $status_code"
echo "Response time: ${response_time}s"
echo "Response size: $response_size bytes"
if [ "$status_code" -eq 200 ]; then
echo "Service is healthy"
else
echo "Service check failed"
exit 1
fi
JSON Output Format
For easier parsing in scripts, you can format the output as JSON:
curl -w '{"status_code":%{http_code},"total_time":%{time_total},"dns_time":%{time_namelookup},"connect_time":%{time_connect},"size_download":%{size_download},"speed_download":%{speed_download}}' \
-s -o /dev/null \
https://httpbin.org/json
Using Format Files
For complex formats, you can store the format string in a file:
Create a file named curl-format.txt:
time_namelookup: %{time_namelookup} seconds\n
time_connect: %{time_connect} seconds\n
time_appconnect: %{time_appconnect} seconds\n
time_pretransfer: %{time_pretransfer} seconds\n
time_redirect: %{time_redirect} seconds\n
time_starttransfer: %{time_starttransfer} seconds\n
----------\n
time_total: %{time_total} seconds\n
status_code: %{http_code}\n
Then use it with:
curl -w "@curl-format.txt" https://example.com
Integration with Web Scraping
When building web scraping applications, the --write-out option becomes invaluable for monitoring request performance and debugging issues. You can combine it with other tools for comprehensive monitoring:
Python Integration
import subprocess
import json
def measure_request_performance(url):
format_string = '{"status_code":%{http_code},"total_time":%{time_total},"dns_time":%{time_namelookup},"connect_time":%{time_connect},"size_download":%{size_download}}'
result = subprocess.run([
'curl', '-s', '-w', format_string, '-o', '/dev/null', url
], capture_output=True, text=True)
if result.returncode == 0:
metrics = json.loads(result.stdout)
return metrics
else:
raise Exception(f"cURL failed: {result.stderr}")
# Usage
try:
metrics = measure_request_performance("https://httpbin.org/delay/2")
print(f"Status: {metrics['status_code']}")
print(f"Total time: {metrics['total_time']} seconds")
print(f"Download size: {metrics['size_download']} bytes")
except Exception as e:
print(f"Error: {e}")
JavaScript/Node.js Integration
const { spawn } = require('child_process');
function measureRequestPerformance(url) {
return new Promise((resolve, reject) => {
const formatString = '{"status_code":%{http_code},"total_time":%{time_total},"size_download":%{size_download}}';
const curl = spawn('curl', ['-s', '-w', formatString, '-o', '/dev/null', url]);
let output = '';
let error = '';
curl.stdout.on('data', (data) => {
output += data.toString();
});
curl.stderr.on('data', (data) => {
error += data.toString();
});
curl.on('close', (code) => {
if (code === 0) {
try {
const metrics = JSON.parse(output);
resolve(metrics);
} catch (e) {
reject(new Error('Failed to parse JSON output'));
}
} else {
reject(new Error(`cURL failed with code ${code}: ${error}`));
}
});
});
}
// Usage
measureRequestPerformance('https://httpbin.org/delay/1')
.then(metrics => {
console.log(`Status: ${metrics.status_code}`);
console.log(`Total time: ${metrics.total_time} seconds`);
console.log(`Download size: ${metrics.size_download} bytes`);
})
.catch(error => {
console.error('Error:', error.message);
});
Advanced Use Cases
API Rate Limiting Monitoring
Monitor API rate limits and response times:
#!/bin/bash
api_url="https://api.example.com/data"
format='{"status":%{http_code},"time":%{time_total},"size":%{size_download},"speed":%{speed_download}}\n'
for i in {1..10}; do
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
metrics=$(curl -s -w "$format" -o /dev/null "$api_url")
echo "$timestamp - Request $i: $metrics"
sleep 1
done
SSL Certificate Validation
Check SSL certificate details:
curl -w "SSL verify result: %{ssl_verify_result}\nHTTP version: %{http_version}\nStatus: %{http_code}\n" \
https://expired.badssl.com/
Load Testing with Timing Analysis
#!/bin/bash
url="https://httpbin.org/delay/1"
requests=50
total_time=0
echo "Running $requests requests to $url"
for i in $(seq 1 $requests); do
time=$(curl -w "%{time_total}" -s -o /dev/null "$url")
total_time=$(echo "$total_time + $time" | bc -l)
echo "Request $i: ${time}s"
done
average=$(echo "scale=3; $total_time / $requests" | bc -l)
echo "Average response time: ${average}s"
Best Practices
1. Suppress Response Body for Metrics-Only Requests
When you only need metrics, redirect the response body to /dev/null:
curl -w "%{http_code}" -s -o /dev/null https://example.com
2. Use Silent Mode
Include -s to suppress progress information:
curl -s -w "Time: %{time_total}s\n" https://example.com
3. Format for Machine Parsing
Use JSON or CSV format for easy parsing in scripts:
# CSV format
curl -w "%{http_code},%{time_total},%{size_download}\n" -s -o /dev/null https://example.com
# JSON format
curl -w '{"code":%{http_code},"time":%{time_total}}\n' -s -o /dev/null https://example.com
Common Pitfalls and Troubleshooting
Variable Case Sensitivity
Variable names are case-sensitive. %{HTTP_CODE} will not work; use %{http_code}.
Escaping in Shell Scripts
When using --write-out in shell scripts, be careful with quote escaping:
# Correct
curl -w "Status: %{http_code}\n" https://example.com
# Also correct with single quotes
curl -w 'Status: %{http_code}\n' https://example.com
JSON Format Validation
Always validate JSON output when parsing:
import json
try:
metrics = json.loads(curl_output)
except json.JSONDecodeError:
print("Invalid JSON output from cURL")
Conclusion
The --write-out option in cURL is an essential tool for developers who need to monitor, debug, and analyze HTTP requests. Whether you're building web scrapers, monitoring APIs, or conducting performance tests, this feature provides valuable insights into request execution. By combining timing data, status codes, and transfer metrics, you can create robust monitoring solutions and identify performance bottlenecks in your applications.
For more advanced web scraping scenarios where you need to handle JavaScript-heavy sites, consider exploring tools like monitoring network requests in Puppeteer or learn about handling authentication in Puppeteer for more complex scenarios.
