Table of contents

How do I implement retry logic for failed requests in PHP?

Implementing retry logic for failed HTTP requests is crucial for building robust PHP applications that can handle network failures, temporary server errors, and rate limiting. This guide covers various approaches to implement retry mechanisms with exponential backoff, custom error handling, and best practices.

Why Implement Retry Logic?

Retry logic helps your PHP applications handle: - Network timeouts and connection failures - Temporary server errors (5xx status codes) - Rate limiting (429 status codes) - DNS resolution failures - Service unavailability

Basic Retry Implementation with cURL

Here's a simple retry mechanism using PHP's cURL extension:

<?php
function makeRequestWithRetry($url, $maxRetries = 3, $delay = 1) {
    $retryCount = 0;

    while ($retryCount <= $maxRetries) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_TIMEOUT, 30);
        curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($ch, CURLOPT_USERAGENT, 'PHP Retry Client/1.0');

        $response = curl_exec($ch);
        $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        $error = curl_error($ch);
        curl_close($ch);

        // Success condition
        if ($response !== false && $httpCode >= 200 && $httpCode < 300) {
            return [
                'success' => true,
                'data' => $response,
                'http_code' => $httpCode,
                'retries' => $retryCount
            ];
        }

        // Check if we should retry
        if ($retryCount < $maxRetries && shouldRetry($httpCode, $error)) {
            $retryCount++;
            sleep($delay);
            $delay *= 2; // Exponential backoff
            continue;
        }

        // Max retries reached or non-retryable error
        return [
            'success' => false,
            'error' => $error ?: "HTTP $httpCode",
            'http_code' => $httpCode,
            'retries' => $retryCount
        ];
    }
}

function shouldRetry($httpCode, $error) {
    // Retry on network errors
    if (!empty($error)) {
        return true;
    }

    // Retry on specific HTTP status codes
    $retryableCodes = [408, 429, 500, 502, 503, 504];
    return in_array($httpCode, $retryableCodes);
}

// Usage example
$result = makeRequestWithRetry('https://api.example.com/data');
if ($result['success']) {
    echo "Request successful after {$result['retries']} retries\n";
    echo "Response: " . $result['data'] . "\n";
} else {
    echo "Request failed: {$result['error']}\n";
}
?>

Advanced Retry Class with Exponential Backoff

For more sophisticated retry logic, create a dedicated retry class:

<?php
class HttpRetryClient {
    private $maxRetries;
    private $baseDelay;
    private $maxDelay;
    private $backoffMultiplier;
    private $jitter;

    public function __construct(
        int $maxRetries = 3,
        float $baseDelay = 1.0,
        float $maxDelay = 60.0,
        float $backoffMultiplier = 2.0,
        bool $jitter = true
    ) {
        $this->maxRetries = $maxRetries;
        $this->baseDelay = $baseDelay;
        $this->maxDelay = $maxDelay;
        $this->backoffMultiplier = $backoffMultiplier;
        $this->jitter = $jitter;
    }

    public function request($url, $options = []) {
        $retryCount = 0;
        $lastError = null;

        while ($retryCount <= $this->maxRetries) {
            try {
                $response = $this->makeRequest($url, $options);

                if ($this->isSuccessful($response)) {
                    return [
                        'success' => true,
                        'response' => $response,
                        'retries' => $retryCount
                    ];
                }

                if (!$this->shouldRetry($response)) {
                    break;
                }

            } catch (Exception $e) {
                $lastError = $e;

                if (!$this->isRetryableException($e)) {
                    break;
                }
            }

            if ($retryCount < $this->maxRetries) {
                $this->sleep($retryCount);
                $retryCount++;
            } else {
                break;
            }
        }

        return [
            'success' => false,
            'error' => $lastError ? $lastError->getMessage() : 'Max retries exceeded',
            'retries' => $retryCount
        ];
    }

    private function makeRequest($url, $options) {
        $ch = curl_init();

        $defaultOptions = [
            CURLOPT_URL => $url,
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT => 30,
            CURLOPT_CONNECTTIMEOUT => 10,
            CURLOPT_FOLLOWLOCATION => true,
            CURLOPT_USERAGENT => 'PHP Retry Client/2.0',
            CURLOPT_HEADER => true,
        ];

        curl_setopt_array($ch, array_merge($defaultOptions, $options));

        $response = curl_exec($ch);

        if ($response === false) {
            $error = curl_error($ch);
            curl_close($ch);
            throw new Exception("cURL error: $error");
        }

        $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        $headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
        curl_close($ch);

        return [
            'http_code' => $httpCode,
            'headers' => substr($response, 0, $headerSize),
            'body' => substr($response, $headerSize)
        ];
    }

    private function isSuccessful($response) {
        return $response['http_code'] >= 200 && $response['http_code'] < 300;
    }

    private function shouldRetry($response) {
        $retryableCodes = [408, 429, 500, 502, 503, 504];
        return in_array($response['http_code'], $retryableCodes);
    }

    private function isRetryableException($exception) {
        $message = strtolower($exception->getMessage());
        $retryableErrors = [
            'timeout',
            'connection',
            'network',
            'dns',
            'resolve'
        ];

        foreach ($retryableErrors as $error) {
            if (strpos($message, $error) !== false) {
                return true;
            }
        }

        return false;
    }

    private function sleep($retryCount) {
        $delay = $this->baseDelay * pow($this->backoffMultiplier, $retryCount);
        $delay = min($delay, $this->maxDelay);

        if ($this->jitter) {
            $delay = $delay * (0.5 + mt_rand() / mt_getrandmax() * 0.5);
        }

        usleep($delay * 1000000); // Convert to microseconds
    }
}

// Usage example
$client = new HttpRetryClient(
    maxRetries: 5,
    baseDelay: 0.5,
    maxDelay: 30.0,
    backoffMultiplier: 2.0,
    jitter: true
);

$result = $client->request('https://api.example.com/data', [
    CURLOPT_HTTPHEADER => ['Content-Type: application/json'],
    CURLOPT_POST => true,
    CURLOPT_POSTFIELDS => json_encode(['key' => 'value'])
]);

if ($result['success']) {
    echo "Success after {$result['retries']} retries\n";
    $response = $result['response'];
    echo "HTTP Code: {$response['http_code']}\n";
    echo "Response Body: {$response['body']}\n";
} else {
    echo "Failed: {$result['error']}\n";
}
?>

Using Guzzle HTTP Client with Retry Middleware

For more advanced HTTP handling, use Guzzle with retry middleware:

<?php
require_once 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Exception\RequestException;
use GuzzleHttp\Handler\CurlHandler;
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Middleware;
use GuzzleHttp\Psr7\Request;
use GuzzleHttp\Psr7\Response;

class GuzzleRetryClient {
    private $client;

    public function __construct() {
        $stack = HandlerStack::create(new CurlHandler());

        // Add retry middleware
        $stack->push(Middleware::retry(
            $this->retryDecider(),
            $this->retryDelay()
        ));

        $this->client = new Client([
            'handler' => $stack,
            'timeout' => 30,
            'connect_timeout' => 10,
        ]);
    }

    private function retryDecider() {
        return function (
            $retries,
            Request $request,
            Response $response = null,
            RequestException $exception = null
        ) {
            // Limit the number of retries
            if ($retries >= 5) {
                return false;
            }

            // Retry connection exceptions
            if ($exception instanceof RequestException) {
                return true;
            }

            // Retry on server errors
            if ($response && $response->getStatusCode() >= 500) {
                return true;
            }

            // Retry on rate limiting
            if ($response && $response->getStatusCode() === 429) {
                return true;
            }

            return false;
        };
    }

    private function retryDelay() {
        return function ($numberOfRetries) {
            return 1000 * pow(2, $numberOfRetries); // Exponential backoff in milliseconds
        };
    }

    public function get($url, $options = []) {
        try {
            $response = $this->client->get($url, $options);
            return [
                'success' => true,
                'status_code' => $response->getStatusCode(),
                'headers' => $response->getHeaders(),
                'body' => $response->getBody()->getContents()
            ];
        } catch (RequestException $e) {
            return [
                'success' => false,
                'error' => $e->getMessage(),
                'status_code' => $e->getResponse() ? $e->getResponse()->getStatusCode() : null
            ];
        }
    }

    public function post($url, $data, $options = []) {
        try {
            $options['json'] = $data;
            $response = $this->client->post($url, $options);
            return [
                'success' => true,
                'status_code' => $response->getStatusCode(),
                'headers' => $response->getHeaders(),
                'body' => $response->getBody()->getContents()
            ];
        } catch (RequestException $e) {
            return [
                'success' => false,
                'error' => $e->getMessage(),
                'status_code' => $e->getResponse() ? $e->getResponse()->getStatusCode() : null
            ];
        }
    }
}

// Usage example
$client = new GuzzleRetryClient();
$result = $client->get('https://api.example.com/data');

if ($result['success']) {
    echo "Request successful\n";
    echo "Status: {$result['status_code']}\n";
    echo "Body: {$result['body']}\n";
} else {
    echo "Request failed: {$result['error']}\n";
}
?>

Circuit Breaker Pattern Implementation

For high-traffic applications, implement a circuit breaker pattern to prevent cascading failures:

<?php
class CircuitBreaker {
    private $failureThreshold;
    private $recoveryTimeout;
    private $failureCount = 0;
    private $lastFailureTime;
    private $state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN

    public function __construct($failureThreshold = 5, $recoveryTimeout = 60) {
        $this->failureThreshold = $failureThreshold;
        $this->recoveryTimeout = $recoveryTimeout;
    }

    public function call(callable $operation) {
        if ($this->state === 'OPEN') {
            if (time() - $this->lastFailureTime >= $this->recoveryTimeout) {
                $this->state = 'HALF_OPEN';
            } else {
                throw new Exception('Circuit breaker is OPEN');
            }
        }

        try {
            $result = $operation();
            $this->onSuccess();
            return $result;
        } catch (Exception $e) {
            $this->onFailure();
            throw $e;
        }
    }

    private function onSuccess() {
        $this->failureCount = 0;
        $this->state = 'CLOSED';
    }

    private function onFailure() {
        $this->failureCount++;
        $this->lastFailureTime = time();

        if ($this->failureCount >= $this->failureThreshold) {
            $this->state = 'OPEN';
        }
    }

    public function getState() {
        return $this->state;
    }
}

// Combined with retry logic
class ResilientHttpClient {
    private $retryClient;
    private $circuitBreaker;

    public function __construct() {
        $this->retryClient = new HttpRetryClient();
        $this->circuitBreaker = new CircuitBreaker();
    }

    public function request($url, $options = []) {
        try {
            return $this->circuitBreaker->call(function() use ($url, $options) {
                return $this->retryClient->request($url, $options);
            });
        } catch (Exception $e) {
            return [
                'success' => false,
                'error' => $e->getMessage(),
                'circuit_breaker_state' => $this->circuitBreaker->getState()
            ];
        }
    }
}
?>

Best Practices for Retry Logic

1. Use Exponential Backoff with Jitter

Prevent thundering herd problems by adding randomness to delay calculations.

2. Set Reasonable Limits

// Good retry configuration
$config = [
    'max_retries' => 3,
    'base_delay' => 1.0,
    'max_delay' => 60.0,
    'timeout' => 30
];

3. Handle Rate Limiting Properly

private function handleRateLimit($response) {
    $headers = $response['headers'];
    if (isset($headers['Retry-After'])) {
        $retryAfter = (int) $headers['Retry-After'];
        sleep($retryAfter);
        return true;
    }
    return false;
}

4. Log Retry Attempts

private function logRetry($url, $attempt, $error) {
    error_log("Retry attempt $attempt for $url: $error");
}

5. Monitor and Alert

Track retry patterns to identify systemic issues:

class RetryMetrics {
    public static function recordRetry($service, $attempts, $success) {
        // Send metrics to monitoring system
        // Example: StatsD, Prometheus, etc.
    }
}

Integration with Web Scraping

When implementing retry logic for web scraping projects, similar to how you might handle errors in Puppeteer, consider these additional factors:

function scrapeWithRetry($url, $selector) {
    $client = new HttpRetryClient();

    $result = $client->request($url, [
        CURLOPT_USERAGENT => 'Mozilla/5.0 (compatible; WebScraper/1.0)',
        CURLOPT_HTTPHEADER => [
            'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Accept-Language: en-US,en;q=0.5',
            'Accept-Encoding: gzip, deflate'
        ]
    ]);

    if ($result['success']) {
        $dom = new DOMDocument();
        @$dom->loadHTML($result['response']['body']);
        $xpath = new DOMXPath($dom);
        $elements = $xpath->query($selector);

        return array_map(function($element) {
            return $element->textContent;
        }, iterator_to_array($elements));
    }

    return [];
}

Command Line Testing

Test your retry logic with command line tools:

# Test with curl to simulate timeouts
curl --max-time 1 --retry 3 --retry-delay 2 https://httpbin.org/delay/5

# Test with PHP script
php -r "
include 'retry_client.php';
\$client = new HttpRetryClient();
\$result = \$client->request('https://httpbin.org/status/500');
var_dump(\$result);
"

Monitoring and Debugging

Add comprehensive logging and monitoring to your retry implementation:

class RetryLogger {
    public static function logAttempt($url, $attempt, $httpCode, $error = null) {
        $timestamp = date('Y-m-d H:i:s');
        $message = "[$timestamp] Retry $attempt for $url - HTTP $httpCode";
        if ($error) {
            $message .= " - Error: $error";
        }
        error_log($message);
    }

    public static function logSuccess($url, $totalAttempts, $duration) {
        $timestamp = date('Y-m-d H:i:s');
        $message = "[$timestamp] Success for $url after $totalAttempts attempts in {$duration}ms";
        error_log($message);
    }

    public static function logFailure($url, $totalAttempts, $lastError) {
        $timestamp = date('Y-m-d H:i:s');
        $message = "[$timestamp] Final failure for $url after $totalAttempts attempts - $lastError";
        error_log($message);
    }
}

Conclusion

Implementing robust retry logic in PHP requires careful consideration of failure modes, backoff strategies, and monitoring. Whether you're building a simple web scraper or a complex API client, proper retry mechanisms ensure your applications remain resilient in the face of network failures and temporary service outages.

The examples provided here offer various approaches from basic retry loops to sophisticated circuit breaker patterns. Choose the implementation that best fits your application's requirements and traffic patterns. Remember to always monitor your retry behavior and adjust parameters based on real-world performance data.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

What are the best practices for organizing PHP web scraping code?
How can I scrape data from single-page applications (SPAs) using PHP?
How do I handle session management during web scraping with PHP?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon