Table of contents

How do I implement concurrent requests in Guzzle for faster scraping?

When scraping multiple web pages or API endpoints, making requests sequentially can be extremely slow. Guzzle, the popular PHP HTTP client library, provides powerful concurrency features that allow you to send multiple HTTP requests simultaneously, dramatically improving your scraping performance. This guide covers everything you need to know about implementing concurrent requests in Guzzle.

Understanding Guzzle Concurrency

Guzzle supports asynchronous HTTP requests through promises and request pools. Instead of waiting for each request to complete before starting the next one, concurrent requests allow multiple HTTP operations to run in parallel, significantly reducing the total execution time for large-scale scraping operations.

Key Benefits of Concurrent Requests

  • Faster execution: Multiple requests run simultaneously instead of sequentially
  • Better resource utilization: Takes advantage of I/O wait times
  • Scalable scraping: Handle hundreds or thousands of URLs efficiently
  • Improved user experience: Reduced waiting times for data collection

Basic Concurrent Requests with Promises

The simplest way to implement concurrent requests in Guzzle is using promises. Here's a basic example:

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Promise;

$client = new Client();

// Create an array of promise objects
$promises = [
    'page1' => $client->getAsync('https://example.com/page1'),
    'page2' => $client->getAsync('https://example.com/page2'),
    'page3' => $client->getAsync('https://example.com/page3'),
];

// Wait for all promises to complete
$responses = Promise\settle($promises)->wait();

// Process responses
foreach ($responses as $key => $response) {
    if ($response['state'] === 'fulfilled') {
        echo "Success for {$key}: " . $response['value']->getStatusCode() . "\n";
        $body = $response['value']->getBody()->getContents();
        // Process the response body here
    } else {
        echo "Failed for {$key}: " . $response['reason']->getMessage() . "\n";
    }
}
?>

Using Guzzle Pools for Large-Scale Scraping

For scraping large numbers of URLs, Guzzle's Pool class provides better memory management and control over concurrency levels:

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;
use Psr\Http\Message\ResponseInterface;
use GuzzleHttp\Exception\RequestException;

$client = new Client();

// Array of URLs to scrape
$urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3',
    // Add more URLs as needed
];

// Generator function to create requests
$requests = function ($urls) {
    foreach ($urls as $url) {
        yield new Request('GET', $url);
    }
};

// Create a pool with concurrent limit
$pool = new Pool($client, $requests($urls), [
    'concurrency' => 5, // Limit to 5 concurrent requests
    'fulfilled' => function (ResponseInterface $response, $index) {
        // Handle successful response
        echo "Request {$index} completed successfully\n";
        $body = $response->getBody()->getContents();

        // Process scraped content here
        // Example: extract data, save to database, etc.
        processScrapedData($body, $index);
    },
    'rejected' => function (RequestException $reason, $index) {
        // Handle failed request
        echo "Request {$index} failed: " . $reason->getMessage() . "\n";
    },
]);

// Execute the pool
$promise = $pool->promise();
$promise->wait();

function processScrapedData($html, $index) {
    // Your data processing logic here
    // Parse HTML, extract specific elements, etc.
    echo "Processing data from request {$index}\n";
}
?>

Advanced Concurrent Scraping with Custom Options

For more sophisticated scraping scenarios, you can customize request options and implement retry logic:

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;
use Psr\Http\Message\ResponseInterface;
use GuzzleHttp\Exception\RequestException;

class ConcurrentScraper
{
    private $client;
    private $results = [];

    public function __construct()
    {
        $this->client = new Client([
            'timeout' => 30,
            'headers' => [
                'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
            ]
        ]);
    }

    public function scrapeUrls($urls, $concurrency = 10)
    {
        $requests = $this->createRequests($urls);

        $pool = new Pool($this->client, $requests, [
            'concurrency' => $concurrency,
            'fulfilled' => [$this, 'onFulfilled'],
            'rejected' => [$this, 'onRejected'],
        ]);

        $pool->promise()->wait();
        return $this->results;
    }

    private function createRequests($urls)
    {
        foreach ($urls as $index => $url) {
            yield $index => new Request('GET', $url, [
                'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                'Accept-Language' => 'en-US,en;q=0.5',
                'Accept-Encoding' => 'gzip, deflate',
                'Connection' => 'keep-alive',
                'Upgrade-Insecure-Requests' => '1',
            ]);
        }
    }

    public function onFulfilled(ResponseInterface $response, $index)
    {
        $statusCode = $response->getStatusCode();
        $body = $response->getBody()->getContents();

        $this->results[$index] = [
            'status' => 'success',
            'status_code' => $statusCode,
            'content' => $body,
            'content_length' => strlen($body)
        ];

        echo "✓ Request {$index} completed ({$statusCode})\n";
    }

    public function onRejected(RequestException $reason, $index)
    {
        $this->results[$index] = [
            'status' => 'failed',
            'error' => $reason->getMessage(),
            'code' => $reason->getCode()
        ];

        echo "✗ Request {$index} failed: {$reason->getMessage()}\n";
    }
}

// Usage example
$scraper = new ConcurrentScraper();
$urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3',
    // Add more URLs
];

$results = $scraper->scrapeUrls($urls, 5);

// Process results
foreach ($results as $index => $result) {
    if ($result['status'] === 'success') {
        // Parse HTML content, extract data, etc.
        echo "Processing content from URL {$index}\n";
    }
}
?>

Rate Limiting and Respectful Scraping

When implementing concurrent requests, it's crucial to be respectful to target servers. Here's how to add rate limiting:

<?php
use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;

class RateLimitedScraper
{
    private $client;
    private $lastRequestTime = 0;
    private $minDelay = 1; // Minimum delay between requests in seconds

    public function __construct($delay = 1)
    {
        $this->minDelay = $delay;
        $this->client = new Client(['timeout' => 30]);
    }

    public function scrapeWithRateLimit($urls)
    {
        $requests = $this->createDelayedRequests($urls);

        $pool = new Pool($this->client, $requests, [
            'concurrency' => 3, // Lower concurrency for rate limiting
            'fulfilled' => function ($response, $index) {
                echo "Request {$index} completed\n";
                // Process response
            },
            'rejected' => function ($reason, $index) {
                echo "Request {$index} failed\n";
            },
        ]);

        $pool->promise()->wait();
    }

    private function createDelayedRequests($urls)
    {
        foreach ($urls as $index => $url) {
            // Add delay between requests
            if ($this->lastRequestTime > 0) {
                $elapsed = microtime(true) - $this->lastRequestTime;
                if ($elapsed < $this->minDelay) {
                    usleep(($this->minDelay - $elapsed) * 1000000);
                }
            }

            $this->lastRequestTime = microtime(true);
            yield $index => new Request('GET', $url);
        }
    }
}
?>

Performance Optimization Tips

1. Optimal Concurrency Levels

The ideal number of concurrent requests depends on several factors:

// For most web servers, 5-10 concurrent requests work well
$concurrency = min(10, count($urls));

// For APIs with rate limits, use lower concurrency
$concurrency = 3;

// For your own servers or APIs, you can go higher
$concurrency = 20;

2. Memory Management

For large-scale scraping, manage memory efficiently:

// Use streaming for large responses
$client = new Client([
    'stream' => true, // Stream large responses
    'timeout' => 30,
]);

// Process responses immediately to free memory
'fulfilled' => function (ResponseInterface $response, $index) {
    $content = $response->getBody()->getContents();

    // Process and save data immediately
    processAndSave($content, $index);

    // Clear response from memory
    unset($content);
},

3. Error Handling and Retries

Implement robust error handling for production scraping:

class RetryableScraper
{
    private $maxRetries = 3;
    private $retryDelay = 1;

    public function scrapeWithRetry($urls)
    {
        $failedUrls = [];

        for ($attempt = 1; $attempt <= $this->maxRetries; $attempt++) {
            echo "Attempt {$attempt}\n";

            $results = $this->scrapeUrls($attempt === 1 ? $urls : $failedUrls);
            $failedUrls = $this->getFailedUrls($results);

            if (empty($failedUrls)) {
                break; // All requests succeeded
            }

            if ($attempt < $this->maxRetries) {
                sleep($this->retryDelay * $attempt); // Exponential backoff
            }
        }

        return $results;
    }
}

JavaScript Execution Considerations

While Guzzle excels at concurrent HTTP requests for static content, it cannot execute JavaScript. For websites that rely heavily on JavaScript for content rendering, consider using browser automation tools like Puppeteer for handling JavaScript-heavy websites or implementing proper session management for more complex scraping scenarios.

Concurrent Requests with Authentication

When scraping authenticated endpoints concurrently, you'll need to handle sessions properly:

<?php
use GuzzleHttp\Client;
use GuzzleHttp\Cookie\CookieJar;

class AuthenticatedConcurrentScraper
{
    private $client;
    private $cookieJar;

    public function __construct()
    {
        $this->cookieJar = new CookieJar();
        $this->client = new Client([
            'cookies' => $this->cookieJar,
            'timeout' => 30
        ]);
    }

    public function login($loginUrl, $username, $password)
    {
        // Perform login to establish session
        $response = $this->client->post($loginUrl, [
            'form_params' => [
                'username' => $username,
                'password' => $password
            ]
        ]);

        return $response->getStatusCode() === 200;
    }

    public function scrapeProtectedUrls($urls)
    {
        $promises = [];
        foreach ($urls as $key => $url) {
            $promises[$key] = $this->client->getAsync($url);
        }

        return Promise\settle($promises)->wait();
    }
}
?>

Best Practices Summary

  1. Start with low concurrency (3-5 requests) and gradually increase based on server response
  2. Implement proper error handling and retry logic for failed requests
  3. Use rate limiting to be respectful to target servers
  4. Monitor memory usage when scraping large numbers of pages
  5. Set appropriate timeouts to avoid hanging requests
  6. Use proper User-Agent headers to identify your scraper
  7. Respect robots.txt and website terms of service
  8. Handle cookies and sessions properly for authenticated scraping
  9. Process responses immediately to manage memory efficiently
  10. Monitor response status codes and adjust concurrency accordingly

Troubleshooting Common Issues

Connection Pool Exhaustion

// Limit concurrent connections per host
$client = new Client([
    'curl' => [
        CURLOPT_MAXCONNECTS => 10
    ]
]);

SSL Certificate Issues

// For development only - disable SSL verification
$client = new Client([
    'verify' => false // Don't use in production
]);

Timeout Handling

// Different timeouts for different scenarios
$client = new Client([
    'connect_timeout' => 5,  // Connection timeout
    'timeout' => 30,         // Total request timeout
    'read_timeout' => 10     // Read timeout
]);

Conclusion

Implementing concurrent requests in Guzzle can dramatically improve your web scraping performance. By using promises for small-scale operations or pools for large-scale scraping, you can reduce execution time from hours to minutes. Remember to implement proper rate limiting, error handling, and monitoring to ensure your scraping operations are both efficient and respectful to target websites.

The key is finding the right balance between speed and server respect – start conservatively and optimize based on your specific use case and the target website's capacity. For JavaScript-heavy sites, consider combining Guzzle with browser automation tools for the best of both worlds.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How can I use Guzzle with proxy servers for web scraping?
What are Guzzle middleware and how do they work?
How do I handle multipart form uploads with Guzzle?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon