Table of contents

Can I use extensions with Headless Chromium?

Yes, you can use extensions with Headless Chromium, but it requires specific configuration and has important limitations. While Chrome extensions are designed for GUI-based browsers, many background extensions can still function in headless mode for automation and testing purposes.

Key Limitations

Before implementing extensions in headless mode, understand these constraints:

  • UI-dependent extensions won't work (popup windows, browser action buttons)
  • Content scripts and background scripts generally work fine
  • Extension installation requires unpacked extension directories
  • Debugging is more complex without browser DevTools UI
  • Some automation tools only support extensions in headful mode

Method 1: Selenium with Python

Installation and Setup

First, install the required dependencies:

pip install selenium webdriver-manager

Basic Extension Loading

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# Path to your unpacked extension directory
extension_path = '/path/to/your/extension/'

# Configure Chrome options
chrome_options = Options()
chrome_options.add_argument('--headless=new')  # Use new headless mode
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument(f'--load-extension={extension_path}')

# Optional: Disable extension security for testing
chrome_options.add_argument('--disable-extensions-file-access-check')
chrome_options.add_argument('--disable-web-security')

# Auto-download and setup ChromeDriver
service = Service(ChromeDriverManager().install())

# Initialize driver
driver = webdriver.Chrome(service=service, options=chrome_options)

try:
    # Your automation code here
    driver.get('https://example.com')

    # Wait for extension to load and execute
    driver.implicitly_wait(5)

    # Check if extension is working (example)
    extension_elements = driver.find_elements('css selector', '[data-extension-id]')
    print(f"Extension elements found: {len(extension_elements)}")

finally:
    driver.quit()

Loading Multiple Extensions

# Load multiple extensions
extension_paths = [
    '/path/to/extension1/',
    '/path/to/extension2/',
    '/path/to/extension3/'
]

chrome_options = Options()
chrome_options.add_argument('--headless=new')
chrome_options.add_argument('--disable-gpu')

# Load all extensions
for path in extension_paths:
    chrome_options.add_argument(f'--load-extension={path}')

# Alternative: Load extensions as comma-separated list
# chrome_options.add_argument(f'--load-extension={",".join(extension_paths)}')

driver = webdriver.Chrome(service=service, options=chrome_options)

Method 2: Puppeteer with JavaScript

Installation

npm install puppeteer

Headful Mode (Recommended for Extensions)

const puppeteer = require('puppeteer');
const path = require('path');

(async () => {
  const extensionPath = path.resolve('/path/to/your/extension/');

  const browser = await puppeteer.launch({
    headless: false, // Extensions require headful mode in Puppeteer
    devtools: false, // Set to true for debugging
    args: [
      `--disable-extensions-except=${extensionPath}`,
      `--load-extension=${extensionPath}`,
      '--no-first-run',
      '--disable-default-apps',
      '--disable-popup-blocking',
      '--disable-translate',
      '--disable-background-timer-throttling',
      '--disable-renderer-backgrounding',
      '--disable-backgrounding-occluded-windows'
    ]
  });

  const page = await browser.newPage();

  try {
    await page.goto('https://example.com', { waitUntil: 'networkidle2' });

    // Wait for extension to initialize
    await page.waitForTimeout(2000);

    // Check if extension injected content
    const extensionContent = await page.evaluate(() => {
      return document.querySelector('[data-extension]') !== null;
    });

    console.log('Extension active:', extensionContent);

  } catch (error) {
    console.error('Error:', error);
  } finally {
    await browser.close();
  }
})();

Experimental Headless Mode

// Note: This is experimental and may not work with all extensions
const browser = await puppeteer.launch({
  headless: 'new', // Use new headless mode
  args: [
    `--load-extension=${extensionPath}`,
    '--disable-extensions-file-access-check',
    '--disable-web-security',
    '--allow-running-insecure-content'
  ]
});

Method 3: Playwright with Extensions

const { chromium } = require('playwright');

(async () => {
  const extensionPath = '/path/to/your/extension/';

  const context = await chromium.launchPersistentContext('', {
    headless: false, // Extensions require headful mode
    args: [
      `--disable-extensions-except=${extensionPath}`,
      `--load-extension=${extensionPath}`
    ]
  });

  const page = await context.newPage();
  await page.goto('https://example.com');

  // Your automation code here

  await context.close();
})();

Extension Preparation

Creating an Unpacked Extension

If you have a .crx file, you need to extract it:

# Create directory for extension
mkdir my-extension

# Extract CRX file (if you have one)
unzip extension.crx -d my-extension/

# Or download from Chrome Web Store using browser developer tools
# Navigate to chrome://extensions/, enable Developer mode,
# and use "Pack extension" to create unpacked version

Manifest Requirements

Ensure your extension's manifest.json includes proper permissions:

{
  "manifest_version": 3,
  "name": "My Extension",
  "version": "1.0",
  "permissions": [
    "activeTab",
    "storage"
  ],
  "content_scripts": [{
    "matches": ["<all_urls>"],
    "js": ["content.js"]
  }],
  "background": {
    "service_worker": "background.js"
  }
}

Debugging and Troubleshooting

Common Issues and Solutions

  1. Extension not loading: Check file permissions and path correctness
  2. Scripts not executing: Verify content script matches and permissions
  3. Background scripts failing: Check for console errors in extension context

Debugging Code

# Enable Chrome logging for debugging
chrome_options.add_argument('--enable-logging')
chrome_options.add_argument('--v=1')
chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_experimental_option('excludeSwitches', ['enable-automation'])

# Check loaded extensions
driver.get('chrome://extensions/')

Best Practices

  1. Test in headful mode first to ensure extension works correctly
  2. Use unpacked extensions for easier debugging and modification
  3. Handle extension load timing with appropriate waits
  4. Monitor browser console for extension-related errors
  5. Keep extensions minimal to reduce complexity in headless environments

Alternative Approaches

If extensions don't work in your headless setup, consider:

  • Direct API integration instead of browser extensions
  • Userscripts injected via --user-data-dir
  • Custom browser modifications for specific functionality
  • Hybrid approaches using both headful and headless modes

Extensions in headless Chromium are powerful for automation but require careful setup and testing to ensure reliable functionality.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon