Making Asynchronous Requests with Curl

Curl is inherently synchronous - it blocks execution until a request completes. However, there are several approaches to achieve asynchronous behavior with curl.

Understanding Curl's Synchronous Nature

By default, curl waits for each request to complete before proceeding:

curl https://api.example.com/data1
curl https://api.example.com/data2  # Only runs after first request completes

Method 1: Background Processes

Run curl commands in the background using &:

# Start multiple requests simultaneously
curl https://api.example.com/data1 > response1.json &
curl https://api.example.com/data2 > response2.json &
curl https://api.example.com/data3 > response3.json &

# Wait for all background jobs to complete
wait
echo "All requests completed"

Method 2: Parallel Execution with GNU Parallel

Install GNU Parallel for more control over concurrent requests:

# Install on Ubuntu/Debian
sudo apt-get install parallel

# Run multiple URLs in parallel
parallel -j5 curl -o "output_{#}.json" ::: \
  https://api.example.com/data1 \
  https://api.example.com/data2 \
  https://api.example.com/data3

Method 3: Curl with xargs

Process multiple URLs concurrently:

# Create a file with URLs
echo -e "https://api.example.com/data1\nhttps://api.example.com/data2" > urls.txt

# Execute with limited parallelism
cat urls.txt | xargs -n 1 -P 5 curl -O

Method 4: Shell Script with Process Management

#!/bin/bash

# Function to make curl request
make_request() {
    local url=$1
    local output=$2
    curl -s "$url" > "$output"
    echo "Completed: $url"
}

# Start multiple requests
make_request "https://api.example.com/data1" "output1.json" &
make_request "https://api.example.com/data2" "output2.json" &
make_request "https://api.example.com/data3" "output3.json" &

# Wait for all to complete
wait
echo "All requests finished"

Method 5: Using Programming Languages

Python with asyncio and subprocess

import asyncio
import subprocess
import json

async def async_curl(url, output_file):
    """Execute curl asynchronously using subprocess"""
    process = await asyncio.create_subprocess_exec(
        'curl', '-s', url, '-o', output_file,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE
    )
    await process.communicate()
    return f"Completed: {url}"

async def main():
    tasks = [
        async_curl('https://api.example.com/data1', 'output1.json'),
        async_curl('https://api.example.com/data2', 'output2.json'),
        async_curl('https://api.example.com/data3', 'output3.json')
    ]

    results = await asyncio.gather(*tasks)
    for result in results:
        print(result)

# Run the async function
asyncio.run(main())

Node.js with child_process

const { spawn } = require('child_process');
const util = require('util');

function asyncCurl(url, outputFile) {
    return new Promise((resolve, reject) => {
        const curl = spawn('curl', ['-s', url, '-o', outputFile]);

        curl.on('close', (code) => {
            if (code === 0) {
                resolve(`Completed: ${url}`);
            } else {
                reject(`curl failed with code ${code}`);
            }
        });
    });
}

async function main() {
    try {
        const requests = [
            asyncCurl('https://api.example.com/data1', 'output1.json'),
            asyncCurl('https://api.example.com/data2', 'output2.json'),
            asyncCurl('https://api.example.com/data3', 'output3.json')
        ];

        const results = await Promise.all(requests);
        results.forEach(result => console.log(result));
    } catch (error) {
        console.error('Error:', error);
    }
}

main();

Method 6: Advanced Parallel Processing

Using curl's built-in parallel feature with config files

# Create config files for each request
echo "url = https://api.example.com/data1" > config1.txt
echo "output = output1.json" >> config1.txt

echo "url = https://api.example.com/data2" > config2.txt
echo "output = output2.json" >> config2.txt

# Run in parallel
curl --parallel --parallel-max 5 --config config1.txt --config config2.txt

Best Practices

1. Limit Concurrency: Don't overwhelm servers with too many simultaneous requests
2. Error Handling: Always check exit codes and handle failures
3. Rate Limiting: Respect API rate limits even with parallel requests
4. Resource Management: Monitor system resources when running many concurrent requests

Example: Real-world Async Curl Script

#!/bin/bash

URLS=(
    "https://api.example.com/users"
    "https://api.example.com/posts"
    "https://api.example.com/comments"
)

MAX_PARALLEL=3
TIMEOUT=30

# Function to make request with error handling
fetch_url() {
    local url=$1
    local filename=$(basename "$url").json

    if curl -s --max-time $TIMEOUT "$url" > "$filename"; then
        echo "✓ Success: $url -> $filename"
    else
        echo "✗ Failed: $url"
        return 1
    fi
}

# Export function for parallel execution
export -f fetch_url
export TIMEOUT

# Execute with GNU Parallel
printf '%s\n' "${URLS[@]}" | parallel -j$MAX_PARALLEL fetch_url

echo "All async requests completed"

This comprehensive approach allows you to effectively achieve asynchronous behavior with curl while maintaining control over concurrency and error handling.