How do I handle character encoding issues when scraping with PHP?

Character encoding issues can cause headaches when scraping content from the web with PHP, as you might end up with garbled text if encoding isn't handled properly. Here's how you can avoid or fix encoding issues:

1. Set the Default Encoding

Ensure that the default character encoding is set to UTF-8 in your PHP environment. This can be done by setting the default_charset in your php.ini file:

default_charset = "UTF-8"

Or you can set it at runtime using the ini_set function:

ini_set('default_charset', 'UTF-8');

2. Use mb_* Functions

Use mb_* (multibyte string) functions for string manipulation, which are encoding-aware. Before using these functions, make sure the mbstring extension is enabled in your PHP setup.

3. Specify the Encoding in file_get_contents

When using file_get_contents to fetch the content, make sure to specify the encoding in the HTTP context options if the source encoding is known:

$context = stream_context_create(array(
    'http' => array(
        'header' => "Content-Type: text/html; charset=UTF-8"
    )
));

$html = file_get_contents('http://example.com', false, $context);

4. Convert Encoding When Necessary

If the source page uses a different encoding, convert it to UTF-8 using mb_convert_encoding:

$sourceEncoding = 'ISO-8859-1'; // Replace with the actual source encoding
$html = mb_convert_encoding($html, 'UTF-8', $sourceEncoding);

5. Use DOMDocument with Proper Encoding

When parsing HTML, use DOMDocument and make sure to handle encoding properly:

$dom = new DOMDocument();

// Use @ to suppress warnings, then handle errors properly
@$dom->loadHTML(mb_convert_encoding($html, 'HTML-ENTITIES', 'UTF-8'));

6. Check HTTP Headers

Check the Content-Type header of the HTTP response to detect the encoding:

$headers = get_headers('http://example.com', 1);
$contentType = $headers['Content-Type'] ?? '';
preg_match('/charset=(.*)/', $contentType, $matches);
$charset = $matches[1] ?? 'UTF-8';

$html = file_get_contents('http://example.com');
$html = mb_convert_encoding($html, 'UTF-8', $charset);

7. Use cURL with Encoding Options

If you're using cURL, set the encoding options accordingly:

$ch = curl_init('http://example.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_ENCODING, 'UTF-8');
$html = curl_exec($ch);
curl_close($ch);

8. Handle Meta Tags

Some pages may specify their encoding in a meta tag. You can parse this tag to find out the encoding:

preg_match('/<meta.*?charset=["\']?([^"\'\s]+)/i', $html, $matches);
$charset = $matches[1] ?? 'UTF-8';
$html = mb_convert_encoding($html, 'UTF-8', $charset);

9. Database Encoding

If you're storing scraped data in a database, ensure the database and the connection are both set to use UTF-8. For MySQL, for instance, you can set the connection charset:

$pdo = new PDO('mysql:host=localhost;dbname=your_db', 'username', 'password');
$pdo->exec("SET NAMES 'utf8'");

10. Look Out for BOM

The Byte Order Mark (BOM) can cause issues when parsing files. You can check for and remove the BOM:

$bom = pack('H*','EFBBBF');
$html = preg_replace("/^$bom/", '', $html);

By following these steps, you should be able to handle most character encoding issues when scraping web pages using PHP. Always test to ensure that the text is being displayed correctly after scraping and conversion.