How do I use Guzzle to follow meta refresh redirects when scraping?

Guzzle is a PHP HTTP client that simplifies sending HTTP requests and integrates with web services. By default, Guzzle will follow standard 3xx HTTP redirects, but it does not natively handle meta refresh redirects, which are client-side directives typically found within HTML content to instruct the browser to load a different URL after a certain time interval.

To follow meta refresh redirects when scraping with Guzzle, you'll need to manually parse the HTML response and look for the http-equiv="refresh" meta tag. If such a tag is found, you can extract the target URL and the delay, and then issue a new request to the target URL after the specified delay.

Here's an example of how you might implement this in PHP using Guzzle:

<?php

require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Exception\RequestException;
use Symfony\Component\DomCrawler\Crawler;

$client = new Client();

// Function to handle meta refresh
function followMetaRefresh($url) {
    global $client;

    try {
        $response = $client->get($url);
        $html = (string) $response->getBody();
        $crawler = new Crawler($html);

        // Look for meta refresh tag
        $metaRefresh = $crawler->filter('meta[http-equiv="refresh"]');
        if ($metaRefresh->count() > 0) {
            $content = $metaRefresh->attr('content');
            $parts = explode(';', $content);
            foreach ($parts as $part) {
                if (preg_match('/url=(.+)/i', $part, $matches)) {
                    $refreshUrl = trim($matches[1]);
                    // Use the refresh URL for the next request
                    if (filter_var($refreshUrl, FILTER_VALIDATE_URL)) {
                        echo "Following meta refresh to: $refreshUrl\n";
                        return followMetaRefresh($refreshUrl);
                    }
                }
            }
        }

        // Return the final HTML if no meta refresh is found
        return $html;
    } catch (RequestException $e) {
        echo "An error occurred: " . $e->getMessage() . "\n";
        return null;
    }
}

// Start by scraping a given URL
$startUrl = 'http://example.com';
$html = followMetaRefresh($startUrl);

// Do something with the final HTML...

In this example, we're using the Symfony DomCrawler component to make it easier to parse the HTML and find the meta refresh tag. You'll need to include the symfony/dom-crawler package in your composer.json to use it:

composer require symfony/dom-crawler

Please note the following:

• This code will follow the meta refresh redirects recursively until no more meta refresh tags are found in the HTML.
• Make sure to respect the site's robots.txt and terms of service when scraping content.
• The delay specified in the meta refresh is ignored in this example. In a real-world scenario, you might want to implement a sleep/delay mechanism to wait for the specified time before making the next HTTP request.
• The filter_var function is used to validate the URL extracted from the meta refresh tag. You might need to adjust the URL extraction logic if the URL in the content attribute is relative or malformed.
• Always handle exceptions and errors gracefully to ensure your scraper doesn't crash when it encounters unexpected conditions.