What are the common errors encountered when using Headless Chromium and how can I solve them?

Headless Chromium is powerful for web scraping and automated testing, but developers often encounter specific errors. Here's a comprehensive guide to the most common issues and their proven solutions:

Browser Launch Failures

1. Chromium Won't Start

Common Error Messages: - Failed to launch the browser process! - No usable sandbox! - ENOENT: no such file or directory, stat '/usr/bin/google-chrome'

Solutions:

# Install dependencies on Ubuntu/Debian
sudo apt-get install -y gconf-service libasound2 libatk1.0-0 libcairo-gobject2 \
libgtk-3-0 libgdk-pixbuf2.0-0 libxss1 fonts-liberation libappindicator1 \
libnss3 lsb-release xdg-utils wget

# For Docker environments
RUN apt-get update && apt-get install -y \
    wget \
    gnupg \
    ca-certificates \
    && wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list \
    && apt-get update \
    && apt-get install -y google-chrome-stable

Use --no-sandbox for restricted environments (Docker, CI/CD):

browser = await puppeteer.launch({
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
    headless: true
});

Network and Navigation Issues

2. Timeout Errors

Common Error Messages: - TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded - TimeoutError: waiting for selector timed out - TimeoutError: Page didn't respond within timeout

Solutions:

// Increase navigation timeout
await page.setDefaultNavigationTimeout(60000);
await page.setDefaultTimeout(60000);

// Use appropriate wait conditions
await page.goto('https://example.com', { 
    waitUntil: 'networkidle0', // Wait until no network requests for 500ms
    timeout: 60000 
});

// Wait for specific elements
await page.waitForSelector('#content', { timeout: 30000 });

3. Network Connection Errors

Common Error Messages: - net::ERR_CONNECTION_REFUSED - net::ERR_NAME_NOT_RESOLVED - net::ERR_INTERNET_DISCONNECTED

Solutions:

// Add retry logic for network failures
async function navigateWithRetry(page, url, maxRetries = 3) {
    for (let i = 0; i < maxRetries; i++) {
        try {
            await page.goto(url, { waitUntil: 'networkidle2' });
            return;
        } catch (error) {
            if (i === maxRetries - 1) throw error;
            console.log(`Retry ${i + 1} for ${url}`);
            await new Promise(resolve => setTimeout(resolve, 2000));
        }
    }
}

Element Interaction Problems

4. Element Not Found

Common Error Messages: - Error: No node found for selector: #elementId - Error: Node is detached from document - Error: Element is not attached to the DOM

Solutions:

// Wait for element before interacting
await page.waitForSelector('#button', { visible: true });
await page.click('#button');

// Use more robust selectors
await page.waitForXPath('//button[contains(text(), "Submit")]');

// Handle dynamic content
await page.waitForFunction(
    () => document.querySelectorAll('.item').length > 0,
    { timeout: 30000 }
);

Security and Certificate Issues

5. SSL/TLS Certificate Errors

Common Error Messages: - net::ERR_CERT_AUTHORITY_INVALID - net::ERR_CERT_COMMON_NAME_INVALID - Your connection is not private

Solutions:

// Bypass SSL errors (use cautiously)
browser = await puppeteer.launch({
    args: [
        '--ignore-certificate-errors',
        '--ignore-ssl-errors',
        '--allow-running-insecure-content',
        '--disable-web-security'
    ]
});

// Or handle certificate errors in page context
await page.setBypassCSP(true);

System Resource Issues

6. Memory and Resource Errors

Common Error Messages: - out of memory - Cannot create browser context - Target closed

Solutions:

// Limit resource usage
browser = await puppeteer.launch({
    args: [
        '--max-old-space-size=4096',
        '--disable-dev-shm-usage',
        '--disable-extensions',
        '--disable-plugins',
        '--disable-images' // For faster loading
    ]
});

// Close pages after use
const page = await browser.newPage();
// ... use page
await page.close();

7. Permission Denied Errors

Common Error Messages: - Error: EACCES: permission denied - Error: Could not find browser revision

Solutions:

# Fix file permissions
chmod +x /usr/bin/google-chrome
chown -R user:user /home/user/.cache/puppeteer

# Set proper environment variables
export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
export PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome

Hardware-Related Issues

8. GPU and Display Errors

Common Error Messages: - libGL error: No matching fbConfigs or visuals found - Xvfb failed to start

Solutions:

// Disable GPU acceleration
browser = await puppeteer.launch({
    args: [
        '--disable-gpu',
        '--disable-software-rasterizer',
        '--disable-background-timer-throttling',
        '--disable-backgrounding-occluded-windows',
        '--disable-renderer-backgrounding'
    ]
});

Complete Examples

Robust Puppeteer Implementation

const puppeteer = require('puppeteer');

async function createRobustBrowser() {
    return await puppeteer.launch({
        args: [
            '--no-sandbox',
            '--disable-setuid-sandbox',
            '--disable-dev-shm-usage',
            '--disable-gpu',
            '--disable-web-security',
            '--disable-features=site-per-process',
            '--ignore-certificate-errors',
            '--allow-running-insecure-content'
        ],
        headless: true,
        timeout: 60000
    });
}

async function scrapeWithErrorHandling(url) {
    let browser;
    let page;

    try {
        browser = await createRobustBrowser();
        page = await browser.newPage();

        // Set timeouts and user agent
        await page.setDefaultNavigationTimeout(60000);
        await page.setDefaultTimeout(30000);
        await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36');

        // Navigate with retry logic
        await navigateWithRetry(page, url);

        // Wait for content to load
        await page.waitForSelector('body', { timeout: 30000 });

        // Extract data
        const data = await page.evaluate(() => {
            return {
                title: document.title,
                url: window.location.href,
                content: document.body.innerText.substring(0, 1000)
            };
        });

        return data;

    } catch (error) {
        console.error(`Scraping failed for ${url}:`, error.message);

        // Handle specific errors
        if (error.message.includes('timeout')) {
            console.log('Timeout error - consider increasing timeout values');
        } else if (error.message.includes('net::ERR_')) {
            console.log('Network error - check connectivity and URL');
        } else if (error.message.includes('No node found')) {
            console.log('Element not found - check selectors');
        }

        throw error;

    } finally {
        if (page) await page.close();
        if (browser) await browser.close();
    }
}

async function navigateWithRetry(page, url, maxRetries = 3) {
    for (let attempt = 1; attempt <= maxRetries; attempt++) {
        try {
            await page.goto(url, { 
                waitUntil: 'networkidle2',
                timeout: 60000 
            });
            return;
        } catch (error) {
            console.log(`Navigation attempt ${attempt} failed: ${error.message}`);

            if (attempt === maxRetries) {
                throw error;
            }

            // Wait before retry
            await new Promise(resolve => setTimeout(resolve, 2000 * attempt));
        }
    }
}

// Usage
scrapeWithErrorHandling('https://example.com')
    .then(data => console.log('Success:', data))
    .catch(error => console.error('Failed:', error.message));

Selenium Python Implementation

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, WebDriverException
import time

class RobustChromeDriver:
    def __init__(self):
        self.driver = None
        self.setup_driver()

    def setup_driver(self):
        options = Options()
        options.add_argument('--headless')
        options.add_argument('--no-sandbox')
        options.add_argument('--disable-dev-shm-usage')
        options.add_argument('--disable-gpu')
        options.add_argument('--disable-web-security')
        options.add_argument('--ignore-certificate-errors')
        options.add_argument('--allow-running-insecure-content')
        options.add_argument('--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36')

        # Set page load strategy
        options.page_load_strategy = 'eager'

        try:
            self.driver = webdriver.Chrome(options=options)
            self.driver.set_page_load_timeout(60)
            self.driver.implicitly_wait(10)
        except Exception as e:
            print(f"Failed to initialize Chrome driver: {e}")
            raise

    def navigate_with_retry(self, url, max_retries=3):
        for attempt in range(1, max_retries + 1):
            try:
                self.driver.get(url)
                # Wait for page to be ready
                WebDriverWait(self.driver, 30).until(
                    EC.presence_of_element_located((By.TAG_NAME, "body"))
                )
                return True

            except TimeoutException:
                print(f"Timeout on attempt {attempt} for {url}")
                if attempt == max_retries:
                    raise
                time.sleep(2 * attempt)

            except WebDriverException as e:
                print(f"WebDriver error on attempt {attempt}: {e}")
                if attempt == max_retries:
                    raise
                time.sleep(2 * attempt)

        return False

    def safe_find_element(self, by, value, timeout=10):
        try:
            element = WebDriverWait(self.driver, timeout).until(
                EC.presence_of_element_located((by, value))
            )
            return element
        except TimeoutException:
            print(f"Element not found: {by}={value}")
            return None

    def close(self):
        if self.driver:
            self.driver.quit()

# Usage example
def scrape_with_error_handling(url):
    driver = None
    try:
        driver = RobustChromeDriver()

        if driver.navigate_with_retry(url):
            # Extract data
            title = driver.driver.title

            # Wait for specific elements
            content_element = driver.safe_find_element(By.TAG_NAME, "body")
            content = content_element.text[:1000] if content_element else "No content found"

            return {
                'title': title,
                'url': driver.driver.current_url,
                'content': content
            }
        else:
            return None

    except Exception as e:
        print(f"Scraping failed: {e}")
        return None
    finally:
        if driver:
            driver.close()

# Usage
result = scrape_with_error_handling('https://example.com')
if result:
    print("Success:", result)
else:
    print("Scraping failed")

Best Practices for Error Prevention

1. Always use proper error handling with try-catch blocks
2. Set appropriate timeouts for your use case
3. Implement retry logic for network-related operations
4. Use robust selectors that are less likely to break
5. Monitor resource usage and clean up properly
6. Test in your target environment (Docker, CI/CD, etc.)
7. Keep Chrome/Chromium updated to avoid known issues
8. Respect robots.txt and website terms of service

These solutions cover the majority of Headless Chromium errors you'll encounter. Remember to always test your implementation in your specific environment and adjust configurations based on your needs.