Table of contents

What is the purpose of the --fail flag in Curl?

The --fail flag in Curl is a crucial option that changes how Curl handles HTTP error status codes (4xx and 5xx responses). By default, Curl considers any HTTP response a "success" from a network perspective, regardless of the status code. The --fail flag modifies this behavior to treat HTTP error responses as actual failures, causing Curl to exit with a non-zero status code.

Default Curl Behavior vs --fail Flag

Without --fail (Default Behavior)

By default, Curl will successfully complete a request and return exit code 0 even when encountering HTTP error status codes:

# This command will exit with code 0 even if the page returns 404
curl https://httpbin.org/status/404
echo "Exit code: $?"
# Output: Exit code: 0

# Default behavior with 500 error
curl https://httpbin.org/status/500
echo "Exit code: $?"
# Output: Exit code: 0

With --fail Flag

When using the --fail flag, Curl will exit with a non-zero status code for HTTP errors:

# This command will exit with code 22 for 404 error
curl --fail https://httpbin.org/status/404
echo "Exit code: $?"
# Output: Exit code: 22

# This will also exit with code 22 for 500 error
curl --fail https://httpbin.org/status/500
echo "Exit code: $?"
# Output: Exit code: 22

HTTP Status Codes Affected by --fail

The --fail flag specifically targets these HTTP status code ranges:

  • 4xx Client Errors: 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, etc.
  • 5xx Server Errors: 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, etc.

Status codes 1xx, 2xx, and 3xx are still considered successful and won't trigger the fail behavior.

Practical Use Cases

1. Script Error Handling

The --fail flag is essential for robust shell scripts and automation:

#!/bin/bash

# Download a file and handle errors properly
if curl --fail -o "data.json" "https://api.example.com/data"; then
    echo "Download successful"
    # Process the file
    cat data.json | jq '.'
else
    echo "Download failed with exit code: $?"
    exit 1
fi

2. API Health Checks

Use --fail for monitoring API endpoints:

# Simple health check script
curl --fail --silent --max-time 10 https://api.example.com/health
if [ $? -eq 0 ]; then
    echo "API is healthy"
else
    echo "API health check failed"
    # Send alert or take corrective action
fi

3. CI/CD Pipeline Integration

In continuous integration environments, --fail ensures build failures on HTTP errors:

# In a CI/CD pipeline
curl --fail --silent --show-error \
     --header "Authorization: Bearer $API_TOKEN" \
     --data '{"deployment": "production"}' \
     https://api.example.com/deploy

# If the API returns an error, the pipeline will fail appropriately

Combining --fail with Other Curl Options

Silent Mode with Error Display

# Hide progress bar but show errors
curl --fail --silent --show-error https://api.example.com/endpoint

Retry Logic with --fail

# Retry failed requests up to 3 times
curl --fail --retry 3 --retry-delay 2 https://api.example.com/data

Custom Headers and Authentication

# API request with authentication and error handling
curl --fail \
     --header "Content-Type: application/json" \
     --header "Authorization: Bearer $TOKEN" \
     --data '{"query": "SELECT * FROM users"}' \
     https://api.example.com/query

Programming Language Integration

Python with subprocess

import subprocess
import sys

def make_curl_request(url):
    try:
        result = subprocess.run([
            'curl', '--fail', '--silent', '--show-error', url
        ], capture_output=True, text=True, check=True)

        return result.stdout
    except subprocess.CalledProcessError as e:
        print(f"Curl request failed with exit code {e.returncode}")
        print(f"Error output: {e.stderr}")
        return None

# Usage
data = make_curl_request("https://api.example.com/data")
if data:
    print("Request successful:", data)
else:
    print("Request failed")

Node.js with child_process

const { exec } = require('child_process');

function makeCurlRequest(url) {
    return new Promise((resolve, reject) => {
        const command = `curl --fail --silent --show-error "${url}"`;

        exec(command, (error, stdout, stderr) => {
            if (error) {
                reject({
                    exitCode: error.code,
                    stderr: stderr,
                    message: `Curl failed with exit code ${error.code}`
                });
            } else {
                resolve(stdout);
            }
        });
    });
}

// Usage
makeCurlRequest('https://api.example.com/data')
    .then(data => console.log('Success:', data))
    .catch(error => console.error('Request failed:', error.message));

Advanced Error Handling Patterns

Custom Exit Code Mapping

#!/bin/bash

make_request() {
    local url=$1
    local output_file=$2

    curl --fail --silent --show-error --output "$output_file" "$url"
    local exit_code=$?

    case $exit_code in
        0)
            echo "Success: Data saved to $output_file"
            ;;
        22)
            echo "HTTP error: Server returned an error status"
            ;;
        6)
            echo "Network error: Couldn't resolve host"
            ;;
        7)
            echo "Network error: Failed to connect to host"
            ;;
        28)
            echo "Timeout error: Operation timed out"
            ;;
        *)
            echo "Unknown error: Exit code $exit_code"
            ;;
    esac

    return $exit_code
}

# Usage
make_request "https://api.example.com/data" "response.json"

Logging and Monitoring

#!/bin/bash

LOG_FILE="/var/log/api_monitor.log"

monitor_endpoint() {
    local endpoint=$1
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    if curl --fail --silent --max-time 30 "$endpoint" >/dev/null; then
        echo "[$timestamp] SUCCESS: $endpoint" >> "$LOG_FILE"
        return 0
    else
        local exit_code=$?
        echo "[$timestamp] FAILURE: $endpoint (exit code: $exit_code)" >> "$LOG_FILE"

        # Send alert for critical endpoints
        if [[ "$endpoint" == *"critical"* ]]; then
            echo "Critical endpoint failure detected" | mail -s "API Alert" admin@example.com
        fi

        return $exit_code
    fi
}

# Monitor multiple endpoints
endpoints=(
    "https://api.example.com/health"
    "https://api.example.com/critical/service"
    "https://api.example.com/user/status"
)

for endpoint in "${endpoints[@]}"; do
    monitor_endpoint "$endpoint"
done

When Not to Use --fail

There are scenarios where you might want to handle HTTP errors manually rather than using --fail:

Custom Error Processing

# Get the HTTP status code for custom handling
http_code=$(curl --silent --output response.txt --write-out "%{http_code}" https://api.example.com/data)

case $http_code in
    200)
        echo "Success: Processing data"
        cat response.txt | jq '.'
        ;;
    401)
        echo "Authentication required: Refreshing token"
        # Custom token refresh logic
        ;;
    429)
        echo "Rate limited: Waiting before retry"
        sleep 60
        # Retry logic
        ;;
    *)
        echo "Unexpected status code: $http_code"
        cat response.txt
        ;;
esac

Error Recovery Strategies

Exponential Backoff with --fail

#!/bin/bash

exponential_backoff_request() {
    local url=$1
    local max_attempts=5
    local delay=1

    for ((attempt=1; attempt<=max_attempts; attempt++)); do
        echo "Attempt $attempt of $max_attempts"

        if curl --fail --silent --show-error --max-time 30 "$url"; then
            echo "Request successful on attempt $attempt"
            return 0
        fi

        if [ $attempt -lt $max_attempts ]; then
            echo "Request failed, waiting ${delay}s before retry..."
            sleep $delay
            delay=$((delay * 2))  # Exponential backoff
        fi
    done

    echo "All attempts failed"
    return 1
}

# Usage
exponential_backoff_request "https://api.example.com/data"

Fallback Endpoints

#!/bin/bash

request_with_fallback() {
    local primary_url=$1
    local fallback_url=$2
    local output_file=$3

    echo "Trying primary endpoint: $primary_url"
    if curl --fail --silent --show-error --output "$output_file" "$primary_url"; then
        echo "Primary endpoint successful"
        return 0
    fi

    echo "Primary failed, trying fallback: $fallback_url"
    if curl --fail --silent --show-error --output "$output_file" "$fallback_url"; then
        echo "Fallback endpoint successful"
        return 0
    fi

    echo "Both endpoints failed"
    return 1
}

# Usage
request_with_fallback \
    "https://api.example.com/data" \
    "https://backup-api.example.com/data" \
    "response.json"

Integration with Modern Tools

Using --fail in Docker Health Checks

FROM alpine:latest
RUN apk add --no-cache curl

COPY check-health.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/check-health.sh

HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD curl --fail --silent http://localhost:8080/health || exit 1

CMD ["your-app"]

Kubernetes Liveness Probes

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: your-app:latest
    livenessProbe:
      exec:
        command:
        - /bin/sh
        - -c
        - curl --fail --silent --max-time 5 http://localhost:8080/health
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 3

Best Practices Summary

  1. Always use --fail in scripts where HTTP errors should cause script failure
  2. Combine with --silent --show-error to suppress progress but show error messages
  3. Use appropriate timeouts with --max-time to prevent hanging
  4. Implement retry logic for transient failures
  5. Log both successes and failures for monitoring and debugging
  6. Consider fallback mechanisms for critical operations
  7. Use exponential backoff for retry strategies
  8. Monitor and alert on repeated failures

The --fail flag is an essential tool for creating robust, production-ready scripts and automation workflows that properly handle HTTP error conditions. When building applications that need to monitor network requests in Puppeteer or handle errors in Puppeteer, understanding proper error handling patterns like those demonstrated with Curl's --fail flag becomes invaluable for creating reliable web scraping and API interaction workflows.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I use Curl to send multipart form data?
What does the --max-time option do in Curl?
How do I use Curl with OAuth authentication?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon