How do you handle encoding issues with Cheerio?

Cheerio is a fast, flexible, and lean implementation of core jQuery designed specifically for the server to parse, manipulate, and render HTML. When dealing with web scraping using Cheerio in Node.js, you may encounter encoding issues, especially if the source document doesn't use UTF-8 encoding. Here are some ways to handle encoding issues with Cheerio:

1. Ensure Correct Encoding in HTTP Requests

When you make HTTP requests to fetch the HTML content, you should ensure that the response is properly encoded. You can use the request module (or axios, got, or any other HTTP request library) with the iconv-lite library to handle different encodings.

const request = require('request');
const iconv = require('iconv-lite');
const cheerio = require('cheerio');

request({
    url: 'http://example.com',
    encoding: null, // Prevents automatic string decoding
}, function (error, response, body) {
    if (!error && response.statusCode == 200) {
        const contentType = response.headers['content-type'];
        let encoding = 'utf8';

        if (contentType.includes('charset=ISO-8859-1')) {
            encoding = 'ISO-8859-1';
        }
        // Convert from an encoded buffer to js string.
        const html = iconv.decode(body, encoding);

        const $ = cheerio.load(html);
        // Proceed with your scraping...
    }
});

2. Convert Encodings with iconv-lite

If you have already downloaded the content and you have it as a Buffer, you can use iconv-lite to convert it to the correct encoding before loading it with Cheerio:

const fs = require('fs');
const iconv = require('iconv-lite');
const cheerio = require('cheerio');

const buffer = fs.readFileSync('page.html');
const html = iconv.decode(buffer, 'ISO-8859-1');

const $ = cheerio.load(html);
// Proceed with your scraping...

3. Setting the Correct Encoding in Cheerio

When you load the content into Cheerio, you can specify the encoding if it's known:

const cheerio = require('cheerio');

const html = fs.readFileSync('page.html', 'latin1'); // Assuming the HTML is in latin1

const $ = cheerio.load(html, {
    decodeEntities: true // Set to false if you don't want to decode HTML entities
});
// Proceed with your scraping...

4. Autodetect Encoding

For a more automated approach, you might consider using a library that can autodetect the encoding of a text. One such library is jschardet.

const fs = require('fs');
const jschardet = require('jschardet');
const iconv = require('iconv-lite');
const cheerio = require('cheerio');

const buffer = fs.readFileSync('page.html');
const detectedEncoding = jschardet.detect(buffer).encoding;
const html = iconv.decode(buffer, detectedEncoding);

const $ = cheerio.load(html);
// Proceed with your scraping...

When handling encoding issues, always check the HTTP Content-Type header if available, as it may contain the charset information. If the content type is not available or not reliable, you may need to use encoding detection libraries as shown above.

Keep in mind that web scraping might be against the terms of service of some websites, and you should always respect the robots.txt file and other usage policies.