What does the --retry option do in Curl?

The --retry option in curl is a powerful feature that enables automatic retrying of failed HTTP requests. When a request fails due to network issues, server errors, or other transient problems, curl will automatically attempt the request again up to the specified number of times before giving up.

Understanding the --retry Option

The --retry option tells curl to retry the request if it receives certain types of errors. By default, curl will retry on:

Timeout errors
Connection failures
DNS lookup failures
Transfer errors
HTTP 500, 502, 503, and 504 status codes

Basic Syntax

curl --retry <number> [URL]

Where <number> is the maximum number of retry attempts (0-2147483647).

Simple Examples

Basic Retry Example

# Retry up to 3 times on failure
curl --retry 3 https://api.example.com/data

Retry with Verbose Output

# See retry attempts in action
curl --retry 5 --verbose https://unreliable-api.com/endpoint

Advanced Retry Configuration

Retry Delay Options

You can control the delay between retry attempts using additional options:

# Wait 2 seconds between retries
curl --retry 3 --retry-delay 2 https://api.example.com/data

# Use exponential backoff (delay doubles each time)
curl --retry 5 --retry-delay 1 --retry-max-time 60 https://api.example.com/data

Retry All Errors

By default, curl only retries on specific error conditions. To retry on all HTTP error codes:

# Retry on any HTTP error (4xx, 5xx)
curl --retry 3 --retry-all-errors https://api.example.com/data

Setting Maximum Retry Time

Limit the total time spent on retries:

# Stop retrying after 30 seconds total
curl --retry 10 --retry-max-time 30 https://api.example.com/data

Practical Use Cases

Web Scraping with Retry Logic

When scraping websites, temporary server issues are common. Here's how to handle them:

# Scrape with robust retry logic
curl --retry 5 \
     --retry-delay 2 \
     --retry-max-time 60 \
     --retry-all-errors \
     --user-agent "Mozilla/5.0 (compatible; WebScraper)" \
     https://example.com/data

API Integration

For API calls in production environments:

# Robust API call with authentication
curl --retry 3 \
     --retry-delay 1 \
     --retry-connrefused \
     --header "Authorization: Bearer $TOKEN" \
     --header "Content-Type: application/json" \
     --data '{"query": "example"}' \
     https://api.service.com/v1/search

Health Check Scripts

Perfect for monitoring scripts:

#!/bin/bash
# Health check with retry
curl --retry 2 \
     --retry-delay 5 \
     --max-time 10 \
     --fail \
     --silent \
     https://myservice.com/health || echo "Service is down!"

Retry Options Reference

| Option | Description | Example | |--------|-------------|---------| | --retry <num> | Number of retry attempts | --retry 5 | | --retry-delay <seconds> | Delay between retries | --retry-delay 3 | | --retry-max-time <seconds> | Maximum total retry time | --retry-max-time 60 | | --retry-all-errors | Retry on all HTTP errors | --retry-all-errors | | --retry-connrefused | Retry on connection refused | --retry-connrefused |

Programming Language Integration

Python with subprocess

import subprocess
import json

def fetch_with_retry(url, retries=3):
    """Fetch URL with curl retry logic"""
    cmd = [
        'curl',
        '--retry', str(retries),
        '--retry-delay', '2',
        '--retry-all-errors',
        '--silent',
        '--show-error',
        url
    ]

    try:
        result = subprocess.run(cmd, capture_output=True, text=True, check=True)
        return result.stdout
    except subprocess.CalledProcessError as e:
        print(f"Failed after {retries} retries: {e.stderr}")
        return None

# Usage
data = fetch_with_retry('https://api.example.com/data.json')
if data:
    parsed = json.loads(data)
    print(parsed)

Node.js with child_process

const { execSync } = require('child_process');

function fetchWithRetry(url, retries = 3) {
    const command = `curl --retry ${retries} --retry-delay 2 --retry-all-errors --silent "${url}"`;

    try {
        const output = execSync(command, { encoding: 'utf8' });
        return output;
    } catch (error) {
        console.error(`Failed after ${retries} retries:`, error.message);
        return null;
    }
}

// Usage
const data = fetchWithRetry('https://api.example.com/data.json');
if (data) {
    const parsed = JSON.parse(data);
    console.log(parsed);
}

Best Practices

1. Choose Appropriate Retry Counts

# For critical operations
curl --retry 5 --retry-delay 2 https://critical-api.com/data

# For non-critical operations
curl --retry 2 --retry-delay 1 https://optional-service.com/data

2. Implement Exponential Backoff

# Exponential backoff helps avoid overwhelming servers
curl --retry 4 \
     --retry-delay 1 \
     --retry-max-time 30 \
     https://api.example.com/data

3. Combine with Timeouts

# Prevent hanging requests
curl --retry 3 \
     --retry-delay 2 \
     --connect-timeout 10 \
     --max-time 30 \
     https://slow-api.com/data

4. Log Retry Attempts

# Enable verbose logging for debugging
curl --retry 3 \
     --retry-delay 1 \
     --verbose \
     --stderr retry.log \
     https://api.example.com/data

Common Error Scenarios

Network Connectivity Issues

# Handle intermittent network problems
curl --retry 5 \
     --retry-delay 3 \
     --retry-connrefused \
     https://remote-server.com/api

Server Overload

# Handle 503 Service Unavailable errors
curl --retry 3 \
     --retry-delay 5 \
     --retry-all-errors \
     https://busy-server.com/endpoint

DNS Resolution Problems

# Retry on DNS failures
curl --retry 2 \
     --retry-delay 2 \
     --resolve "example.com:443:1.2.3.4" \
     https://example.com/api

Integration with Web Scraping Workflows

When building robust web scraping systems, combining curl's retry functionality with other reliability features creates a more resilient solution. For complex scenarios involving JavaScript-heavy sites, you might need to complement curl with tools that can handle dynamic content and browser automation.

For monitoring and debugging retry behavior, consider implementing comprehensive error handling strategies similar to those used in browser automation tools.

Monitoring Retry Performance

Script for Retry Analysis

#!/bin/bash
# Monitor retry success rates
LOG_FILE="retry_stats.log"

for i in {1..10}; do
    echo "Attempt $i:" >> $LOG_FILE
    curl --retry 3 \
         --retry-delay 1 \
         --write-out "HTTP: %{http_code}, Time: %{time_total}s, Retries: %{num_redirects}\n" \
         --silent \
         --output /dev/null \
         https://api.example.com/test >> $LOG_FILE
    sleep 2
done

Conclusion

The --retry option in curl is an essential feature for building reliable HTTP clients and web scraping tools. By automatically handling transient failures, it significantly improves the robustness of your applications. Remember to:

Use appropriate retry counts based on your use case
Implement delays between retries to avoid overwhelming servers
Combine with timeouts to prevent hanging requests
Monitor and log retry behavior for optimization

When used correctly, the retry mechanism can dramatically improve the success rate of your HTTP requests while maintaining good citizenship on the web.

Table of contents