What does the --connect-timeout option do in Curl?

The --connect-timeout option in cURL sets the maximum time in seconds that cURL will wait for a connection to be established with the target server. This option is crucial for controlling how long your application waits before giving up on unresponsive servers, making it an essential tool for robust web scraping and API interactions.

Understanding Connection Timeout vs Total Timeout

Before diving into --connect-timeout, it's important to distinguish between connection timeout and total timeout:

Connection timeout (--connect-timeout): Time limit for establishing the initial TCP connection
Maximum time (--max-time or -m): Total time limit for the entire operation, including connection, data transfer, and processing

The connection timeout specifically controls the handshake phase when your client attempts to establish a TCP connection with the server.

Basic Syntax and Usage

The basic syntax for the --connect-timeout option is:

curl --connect-timeout <seconds> <URL>

Simple Examples

# Set connection timeout to 10 seconds
curl --connect-timeout 10 https://example.com

# Short form using -t flag (not available for connect-timeout)
# Note: There is no short form for --connect-timeout

# Combine with other timeout options
curl --connect-timeout 15 --max-time 60 https://api.example.com/data

Practical Use Cases

1. Handling Slow or Unresponsive Servers

When scraping websites or calling APIs, some servers may be slow to respond or temporarily unavailable:

# Fail fast if server doesn't respond within 5 seconds
curl --connect-timeout 5 https://slow-server.com/api/endpoint

# More conservative approach for reliable servers
curl --connect-timeout 30 https://reliable-api.com/data

2. Batch Processing and Automation

For scripts that process multiple URLs, connection timeouts prevent hanging on unresponsive endpoints:

#!/bin/bash
urls=(
    "https://site1.com"
    "https://site2.com"
    "https://site3.com"
)

for url in "${urls[@]}"; do
    echo "Checking $url..."
    if curl --connect-timeout 10 --max-time 30 -s -o /dev/null "$url"; then
        echo "✓ $url is accessible"
    else
        echo "✗ $url failed or timed out"
    fi
done

3. Load Testing and Health Checks

When monitoring service availability, quick connection timeouts help identify connectivity issues:

# Health check with strict timing
curl --connect-timeout 3 --max-time 10 \
     --fail --silent --show-error \
     https://api.service.com/health

Advanced Configuration Examples

Combining with Retry Logic

# Retry up to 3 times with 2-second delays, 5-second connection timeout
curl --connect-timeout 5 \
     --retry 3 \
     --retry-delay 2 \
     --retry-connrefused \
     https://unreliable-server.com/data

Using with Different Protocols

The --connect-timeout option works with various protocols:

# HTTP/HTTPS
curl --connect-timeout 10 https://example.com

# FTP
curl --connect-timeout 15 ftp://files.example.com/document.pdf

# SFTP
curl --connect-timeout 20 sftp://secure.example.com/data/file.json

Complex Web Scraping Scenario

# Comprehensive web scraping command
curl --connect-timeout 10 \
     --max-time 60 \
     --user-agent "Mozilla/5.0 (compatible; WebScraper/1.0)" \
     --header "Accept: text/html,application/xhtml+xml" \
     --compressed \
     --location \
     --fail \
     --silent \
     --show-error \
     --output scraped_content.html \
     https://target-website.com/page

Programming Language Integration

Python with subprocess

import subprocess
import json

def fetch_with_timeout(url, connect_timeout=10, max_timeout=60):
    """
    Fetch URL using cURL with connection timeout
    """
    cmd = [
        'curl',
        '--connect-timeout', str(connect_timeout),
        '--max-time', str(max_timeout),
        '--fail',
        '--silent',
        '--show-error',
        url
    ]

    try:
        result = subprocess.run(cmd, capture_output=True, text=True, check=True)
        return result.stdout
    except subprocess.CalledProcessError as e:
        print(f"cURL failed: {e.stderr}")
        return None

# Usage
content = fetch_with_timeout('https://api.example.com/data', connect_timeout=5)
if content:
    print("Successfully fetched content")

Node.js using child_process

const { exec } = require('child_process');
const util = require('util');
const execPromise = util.promisify(exec);

async function fetchWithCurl(url, connectTimeout = 10) {
    const command = `curl --connect-timeout ${connectTimeout} --fail --silent "${url}"`;

    try {
        const { stdout, stderr } = await execPromise(command);
        if (stderr) {
            console.error('cURL error:', stderr);
            return null;
        }
        return stdout;
    } catch (error) {
        console.error('Connection failed:', error.message);
        return null;
    }
}

// Usage
fetchWithCurl('https://api.example.com/data', 5)
    .then(data => {
        if (data) {
            console.log('Data received:', data.length, 'characters');
        }
    });

Error Handling and Troubleshooting

Common Error Scenarios

Connection timeout exceeded:

curl: (28) Connection timeout after 10000 milliseconds

Server unreachable:

curl: (7) Failed to connect to example.com port 443: Connection refused

DNS resolution timeout:

curl: (6) Could not resolve host: nonexistent-domain.com

Debugging Connection Issues

# Verbose output to diagnose connection problems
curl --connect-timeout 10 \
     --verbose \
     --trace-time \
     https://problematic-server.com

# Test with different timeout values
for timeout in 5 10 15 30; do
    echo "Testing with ${timeout}s timeout..."
    time curl --connect-timeout $timeout https://example.com > /dev/null 2>&1
    echo "Exit code: $?"
done

Best Practices and Recommendations

1. Choose Appropriate Timeout Values

Fast networks/local services: 3-5 seconds
Public APIs/websites: 10-15 seconds
Slow or distant servers: 20-30 seconds
File downloads: 30-60 seconds

2. Combine with Other Timeout Options

# Recommended combination for web scraping
curl --connect-timeout 15 \
     --max-time 120 \
     --speed-limit 1000 \
     --speed-time 30 \
     https://large-file-server.com/download

3. Environment-Specific Configuration

Create configuration files for different environments:

# ~/.curlrc for development
connect-timeout = 5
max-time = 30
user-agent = "Development/1.0"

# Production script with explicit timeouts
curl --connect-timeout 20 --max-time 300 https://production-api.com

Integration with Modern Web Scraping

While cURL's --connect-timeout option provides excellent control over connection timing, modern web scraping often requires more sophisticated timeout handling for JavaScript-heavy sites. For dynamic content that loads after the initial connection, you might need tools that can handle timeouts in more complex scenarios or manage authentication flows where connection timing is just one part of the overall process.

Monitoring and Logging

Implementing Connection Timeout Monitoring

#!/bin/bash
LOG_FILE="connection_monitoring.log"

monitor_endpoint() {
    local url=$1
    local timeout=${2:-10}
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    start_time=$(date +%s.%N)
    if curl --connect-timeout $timeout --max-time 30 --fail --silent "$url" > /dev/null; then
        end_time=$(date +%s.%N)
        duration=$(echo "$end_time - $start_time" | bc)
        echo "$timestamp SUCCESS $url ${duration}s" >> "$LOG_FILE"
        return 0
    else
        echo "$timestamp FAILED $url timeout=${timeout}s" >> "$LOG_FILE"
        return 1
    fi
}

# Monitor multiple endpoints
monitor_endpoint "https://api1.example.com/health" 5
monitor_endpoint "https://api2.example.com/status" 10
monitor_endpoint "https://slow-service.com/ping" 30

Performance Considerations

The --connect-timeout option directly impacts your application's performance characteristics:

Too short: May cause false failures on slow but functional networks
Too long: Can cause your application to hang on truly unresponsive servers
Optimal range: Usually between 10-30 seconds for most web scraping scenarios

For high-performance web scraping operations, consider implementing parallel requests with individual timeout controls to maximize throughput while maintaining reliability.

Conclusion

The --connect-timeout option in cURL is a fundamental tool for building robust, reliable web scraping and API interaction systems. By setting appropriate connection timeouts, you can ensure your applications fail fast on unresponsive servers while allowing sufficient time for legitimate slow connections.

Remember to combine --connect-timeout with other timeout options like --max-time for comprehensive timeout management, and always test your timeout values under realistic network conditions to find the optimal balance between reliability and performance for your specific use case.

Table of contents