How do I handle different character encodings when scraping with Goutte?

Goutte is a PHP library that provides a simple API to crawl websites and extract data from the HTML/XML responses. When scraping websites with Goutte, you might encounter pages with various character encodings, which can lead to garbled text if not handled correctly.

Here's how to handle different character encodings when scraping with Goutte:

1. Detect Character Encoding: The first step is to determine the character encoding of the page you are scraping. Websites usually specify their character encoding in the HTTP headers or within the <meta> tags in the HTML.

2. Use mb_convert_encoding: If the character encoding is not UTF-8, you can use the PHP function mb_convert_encoding to convert the extracted content to UTF-8 or any other desired encoding.

Here's an example of how to handle character encoding with Goutte:

require 'vendor/autoload.php';

use Goutte\Client;

$client = new Client();

// Go to the website you want to scrape
$crawler = $client->request('GET', 'http://example.com');

// Get the Content-Type header to find out the character encoding
$response = $client->getResponse();
$contentType = $response->getHeader('Content-Type');

// Use a regular expression to parse the charset from the content type
preg_match('/charset=([^;]+)/', $contentType, $matches);
$charset = $matches[1] ?? 'UTF-8'; // Default to UTF-8 if charset is not found

// Extract the content
$htmlContent = $crawler->html();

// Convert the content to UTF-8 if necessary
if (strtoupper($charset) !== 'UTF-8') {
    $htmlContent = mb_convert_encoding($htmlContent, 'UTF-8', $charset);
}

// Continue with your scraping...

Note: In the example above, we're assuming that the character encoding is specified in the Content-Type header. If it's not there, you might need to look for a <meta> tag in the HTML that specifies the charset:

// If the charset isn't in the HTTP header, check the HTML
if (!isset($charset)) {
    $crawler->filter('meta[http-equiv="Content-Type"]')->each(function ($node) use (&$charset) {
        $metaContentType = $node->attr('content');
        preg_match('/charset=([^;]+)/', $metaContentType, $matches);
        $charset = $matches[1] ?? null;
    });
}

if ($charset === null) {
    // Default to UTF-8 or use other methods to detect the encoding
    $charset = 'UTF-8';
}

When the charset is determined, you can convert the encoding as shown earlier.

It's important to note that Goutte itself does not handle encoding conversion. It's up to you to detect and convert encodings as needed based on the content you're scraping.

Additionally, if you scrape a lot of pages with various encodings, you might want to consider using a library like iconv or mbstring to detect and handle encodings more robustly.