How can I make Headless Chromium use less CPU and memory?

Running headless Chromium can be resource-intensive, which might become an issue, especially when running multiple instances or on machines with limited resources. Here are some strategies to help reduce CPU and memory usage when using headless Chromium:

1. Disable Features and Extensions

Headless Chromium can be configured to disable unnecessary features and extensions that consume additional resources:

Python (with Selenium):

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--disable-gpu')  # GPU hardware acceleration isn't needed for headless
options.add_argument('--no-sandbox')  # Disable the sandbox for all software features
options.add_argument('--disable-dev-shm-usage')  # Overcome limited resource problems
options.add_argument('--disable-extensions')  # Disabling extensions can save resources
options.add_argument('--disable-plugins')  # Disable plugins
# Add any other arguments you think might reduce resource usage

driver = webdriver.Chrome(options=options)

2. Use a Lightweight User-Agent

A lightweight user-agent can sometimes help in reducing the page load time and resources by requesting a simpler version of the webpage, which is less resource-intensive to render.

Python (with Selenium):

options.add_argument('--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36')

3. Limit Tab/Window Count

Each new tab or window in Chromium can consume additional CPU and memory. Limit the number of open tabs/windows to what is necessary.

4. Block Unnecessary Content

You can block images, stylesheets, or JavaScript which are often not needed when scraping websites, to save bandwidth and reduce CPU usage.

Python (with Selenium):

prefs = {
    'profile.managed_default_content_settings.images': 2,
    'profile.managed_default_content_settings.stylesheets': 2,
    'profile.managed_default_content_settings.javascript': 2,
options.add_experimental_option('prefs', prefs)

5. Use Headless Browsers Optimized for Low Resource Usage

Consider using tools like puppeteer-core with pyppeteer in Python or puppeteer in JavaScript, which are optimized for headless browsing and may offer better performance than Selenium with headless Chrome.

Python (with Pyppeteer):

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch(headless=True, args=[
    page = await browser.newPage()
    await page.goto('')
    await browser.close()


6. Optimize Page Load Strategy

For scraping, it might be enough to wait for the DOM to be loaded without waiting for all resources (like images) to be loaded.

Python (with Selenium):

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

caps = DesiredCapabilities.CHROME
caps['pageLoadStrategy'] = 'none'  # Do not wait for full page load

driver = webdriver.Chrome(desired_capabilities=caps, options=options)

7. Reduce CPU Priority

You can reduce the CPU priority of the headless Chromium process if you are running on a Unix-like system.

Bash (console command):

nice -n 10 chromium-browser --headless --disable-gpu ...

8. Memory and CPU Profiling

Profile your headless Chromium to understand where it consumes the most resources, and optimize or eliminate those tasks.

9. Use a Lighter Alternative

For simple scraping tasks, consider using lighter alternatives like requests-html in Python or cheerio with axios in JavaScript, which are much less resource-intensive than a full-blown browser.

Python (with requests-html):

from requests_html import HTMLSession

session = HTMLSession()
response = session.get('')

10. Server-Side Rendering (SSR) / Pre-rendering

If you control the website being scraped, implement SSR to serve pre-rendered HTML which can be easily scraped without the need for a JavaScript engine.

Remember to always comply with the terms of service of the website you are scraping and ensure that your scraping activities do not negatively impact the website's performance.

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping