What Is a Headless Browser? Headless Chrome for Scraping and Testing

A headless browser is a real web browser running without a visible window — it loads pages, executes JavaScript, and renders layouts exactly like the Chrome on your desktop, but it's driven by code instead of clicks. That makes it the standard tool for scraping JavaScript-heavy sites, automated testing, screenshots, and PDF generation. This guide covers how headless Chrome/Chromium actually works: launching it from the command line, driving it from Puppeteer, Playwright, and Selenium, the flags that matter, and the operational problems — memory, debugging, detection — that appear the moment you run one in production.

Key Takeaways

• A headless browser is a full browser engine (DOM, JavaScript, network stack) minus the GUI — not a lightweight HTML fetcher
• chrome --headless=new --dump-dom <url> gives you rendered HTML from a terminal with zero code
• Since Chrome 112, "new headless" is the same browser binary as regular Chrome, which removed a whole class of rendering differences
• Each headless Chrome instance eats 100–500 MB of RAM; flags and page-level resource blocking cut that substantially
• Headless browsers are detectable — the HeadlessChrome user agent, missing plugins, and CDP fingerprints give them away without countermeasures
• Use one when the page needs JavaScript to produce your data; for static HTML, a plain HTTP client is 10–50× cheaper

What is a headless browser?

"Headless" means no graphical interface: no window, no rendered pixels on a screen, no human input. Everything else is intact — the browser parses HTML, applies CSS, runs JavaScript, manages cookies and storage, and produces a DOM identical to what a visible browser would build. Programs control it through an automation protocol (Chrome's DevTools Protocol, Firefox's remote protocol, or WebDriver) and read results back as HTML, screenshots, PDFs, or extracted data.

The practical options in 2026:

• Headless Chrome / Chromium — the de facto standard; what Puppeteer and most scraping infrastructure runs
• Headless Firefox — firefox --headless, driven via Selenium or Playwright
• WebKit via Playwright — closest stand-in for Safari
• Purpose-built engines (Lightpanda, and historically PhantomJS) — smaller and faster, but with partial web compatibility; test carefully before trusting them with real sites

Chromium is the open-source browser Chrome is built from — for headless work they behave the same, and "headless Chromium" usually just means the Chromium build Puppeteer or Playwright downloaded for you.

Why use one?

• Web scraping: sites built on React/Vue/Angular return near-empty HTML; the data appears only after JavaScript runs. A headless browser executes that JavaScript. (If the data is in the raw HTML, skip the browser — parse it directly.)
• End-to-end testing: CI servers have no display; headless browsers run the same test suites 2–15% faster and parallelize cheaply
• Screenshots and PDFs: pixel-accurate page captures and print-quality PDFs on a server
• Performance monitoring: Lighthouse audits and synthetic checks run on headless Chrome
• AI agents: tools that browse the web programmatically are headless browser fleets under the hood

Running headless Chrome from the command line

No libraries needed — Chrome itself takes a --headless flag:

# Rendered DOM after JavaScript execution
chrome --headless=new --dump-dom https://example.com/

# Screenshot and PDF
chrome --headless=new --screenshot=page.png --window-size=1280,800 https://example.com/
chrome --headless=new --print-to-pdf=page.pdf https://example.com/

(On macOS the binary is /Applications/Google Chrome.app/Contents/MacOS/Google Chrome; on Linux it's google-chrome or chromium-browser.)

--headless=new selects the current headless implementation, unified with regular Chrome since version 112 — same binary, same rendering, same feature set. The pre-112 "old headless" was a separate stripped-down browser that rendered some pages differently; if you see advice mentioning --headless=old, it's outdated (the old mode was removed from the Chrome binary entirely in 2024, surviving only as the separate chrome-headless-shell build for legacy screenshot workloads).

Command-line rendering is great for quick checks and cron jobs, but real scraping needs interaction — clicking, waiting, extracting structured data. That's what the automation libraries are for.

Driving headless Chrome from code

Puppeteer (Node.js, Chrome-focused):

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();          // headless by default
  const page = await browser.newPage();
  await page.goto('https://example.com/products', { waitUntil: 'networkidle2' });
  const titles = await page.$$eval('.product h2', els => els.map(e => e.innerText));
  console.log(titles);
  await browser.close();
})();

Playwright (Python/Node/Java/.NET, three browser engines — see our Playwright scraping guide):

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()                    # headless by default
    page = browser.new_page()
    page.goto("https://example.com/products")
    titles = page.locator(".product h2").all_inner_texts()
    browser.close()

Selenium (any language, any browser — see the Selenium FAQ):

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
driver.get("https://example.com/products")
titles = [e.text for e in driver.find_elements("css selector", ".product h2")]
driver.quit()

All three run the same headless Chrome underneath; they differ in API ergonomics, language support, and waiting models. Puppeteer and Playwright install their own browser builds; Selenium uses the Chrome on your system.

The command-line flags that matter

Headless Chrome in production is largely flag tuning. The ones that earn their place:

A production launch typically looks like:

chrome --headless=new --no-sandbox --disable-dev-shm-usage \
       --window-size=1920,1080 --user-data-dir=/tmp/profile \
       --dump-dom https://example.com/

Keeping CPU and memory under control

The cost model: every headless Chrome instance is a real browser — expect 100–300 MB baseline plus 50–150 MB per open page, more on media-heavy sites. A scraping fleet dies from memory before anything else. What actually helps:

• Reuse the browser, recycle pages. Launching Chrome costs ~1s and hundreds of MB; opening a page in an existing browser is nearly free. Keep one browser, iterate pages — but restart the browser every few hundred pages, because leaks accumulate.
• Block what you don't need. Images, fonts, video, and analytics scripts are most of the bandwidth and much of the CPU. Both Puppeteer and Playwright can abort requests by type:

await page.setRequestInterception(true);
page.on('request', req =>
  ['image', 'font', 'media'].includes(req.resourceType()) ? req.abort() : req.continue()
);

• Cap concurrency by memory, not optimism. Concurrent pages ≈ (available RAM − 1 GB) / 300 MB is a sane starting point.
• Useful flags: --js-flags="--max-old-space-size=512" bounds V8's heap per process; --renderer-process-limit=4 limits process explosion on multi-tab work.
• Set timeouts everywhere. A hung networkidle wait on a page with a chatty websocket keeps a renderer alive forever. Every navigation needs a timeout and a finally that closes the page.

Cookies, sessions, and local storage

Headless Chrome handles cookies and localStorage exactly like regular Chrome — per profile, in memory unless you persist them. Two persistence strategies:

• Profile directory (--user-data-dir, or launch_persistent_context in Playwright): everything survives restarts — cookies, localStorage, cache. Heavyweight but complete.
• Export/import state: log in once, save the session, reuse it across fresh browser instances. Playwright's context.storage_state(path="auth.json") captures cookies and localStorage in one JSON file; Puppeteer's browserContext.cookies()/setCookie() covers the cookie half.

The second approach is the scraping workhorse: authenticate once (even manually in a headful browser), then stamp the saved state onto every worker. Watch for sites that bind sessions to IP or fingerprint — a session exported from your office IP may be invalidated when replayed through a datacenter proxy.

Proxies

Route headless Chrome through a proxy with a launch flag — --proxy-server=http://proxy:8080 — or per-context in Playwright, which is the practical way to give each parallel session its own exit IP:

context = browser.new_context(proxy={
    "server": "http://proxy.example.com:8000",
    "username": "user", "password": "pass",
})

Chrome's flag itself has no syntax for credentials; Puppeteer handles auth via page.authenticate(). One browser = one proxy with the flag approach; Playwright contexts or multiple browser instances are the way around it. Residential rotation, health checks, and geo-targeting are their own project — our proxy providers comparison covers that landscape.

Screenshots and PDFs

The one-liners from the CLI section scale up in code, with control over viewport, full-page capture, and encoding:

await page.setViewport({ width: 1280, height: 800, deviceScaleFactor: 2 });
await page.screenshot({ path: 'page.png', fullPage: true });
await page.pdf({ path: 'page.pdf', format: 'A4', printBackground: true });

Screenshot gotchas that cost people afternoons: lazy-loaded images need a scroll-through before fullPage capture; web fonts need a document.fonts.ready wait to avoid fallback-font renders; and animated elements want prefers-reduced-motion emulation for stable output. PDFs use the print stylesheet, so a page can screenshot beautifully and PDF terribly.

Debugging: logs and watching the invisible

When a headless run misbehaves, visibility is the problem. In rough order of usefulness:

1. Run headful. headless: false (plus slowMo: 250 in Puppeteer/Playwright) shows you exactly what the browser sees. Most "headless bugs" reproduce instantly.
2. Capture the browser console. Page errors are invisible unless you forward them: page.on('console', msg => console.log(msg.text())) and page.on('pageerror', ...).
3. Screenshot on failure. A try/catch that saves error.png + page.content() turns mystery timeouts into diagnosable pages (usually: a cookie banner, a CAPTCHA, or a login wall).
4. Chrome's own logs. --enable-logging=stderr --v=1 prints browser-side events; --dump-dom piped to a file shows what actually rendered from a bare CLI run.
5. Remote DevTools. Launch with --remote-debugging-port=9222, open http://localhost:9222 in a normal browser, and attach full DevTools to the headless instance — live inspector against an invisible browser.

Headless browser detection (and what stealth can and can't do)

Sites detect headless browsers, and vanilla headless Chrome announces itself:

• The default user agent literally contains HeadlessChrome
• navigator.webdriver is true under automation
• Plugin lists, WebGL renderer strings, and font sets differ from desktop Chrome
• Anti-bot vendors (Cloudflare, DataDome, Akamai) fingerprint at the TLS and CDP-behavior level, plus IP reputation and mouse/scroll biometrics

Countermeasures exist in layers: overriding the UA and navigator.webdriver is trivial; puppeteer-extra-plugin-stealth and playwright-stealth patch dozens of JS-visible leaks; residential proxies fix IP reputation. That defeats casual detection. It does not reliably beat the major anti-bot vendors, who watch signals below the JavaScript layer and update weekly — a patched browser on a clean IP still fails a managed challenge that decides to be suspicious. Treat stealth as harm reduction, not a solution; when a target runs serious bot protection, the economical answer is usually a scraping API that fights that war full-time (below).

Running headless Chrome in Docker

Containers are the natural home for headless fleets, with three standard traps — sandbox, shared memory, and zombie processes:

FROM mcr.microsoft.com/playwright/python:v1.54.0-noble   # browsers + deps preinstalled
COPY scraper.py .
CMD ["python", "scraper.py"]

docker run --init --ipc=host --shm-size=1gb my-scraper

• --init reaps zombie renderer processes Chrome leaves behind
• --ipc=host or --shm-size=1gb fixes the 64 MB /dev/shm crash (Failed to load resource: net::ERR_INSUFFICIENT_RESOURCES, random tab crashes)
• Official images (mcr.microsoft.com/playwright, ghcr.io/puppeteer/puppeteer) ship every system library Chrome needs — hand-rolling apt-get dependency lists is how Dockerfiles rot

Run as a non-root user so you don't need --no-sandbox; if you must run as root, the flag is unavoidable and the container boundary becomes your sandbox.

Common errors, decoded

Do you need to run this yourself?

A headless browser fleet is real infrastructure: memory management, proxy rotation, stealth patching, session pools, and an anti-bot arms race that never pauses. If the goal is just rendered pages as data, WebScraping.AI runs that fleet for you — headless Chrome with rotating residential proxies and anti-detection handling, behind one HTTP call:

import requests

# Rendered HTML — JavaScript executed, proxies and anti-bot handled
html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/spa-products",
    "js": "true",
}).text

# Or extract structured fields without writing selectors at all
fields = requests.get("https://api.webscraping.ai/ai/fields", params={
    "api_key": API_KEY,
    "url": "https://example.com/product/42",
    "fields[name]": "Product name",
    "fields[price]": "Price with currency",
}).json()

The usual division of labor: prototype locally with Puppeteer or Playwright where you can watch the browser, then hand the production fetch to the /html endpoint and keep your parsing code. Your scraper stays a hundred lines of extraction logic instead of a browser-farm operations manual.

Frequently asked questions

What's the difference between headless Chrome and headless Chromium? Chrome is Google's branded build of the open-source Chromium project — for headless automation they render identically. Puppeteer and Playwright download their own Chromium-lineage builds ("Chrome for Testing"), so what most people call "headless Chrome" is technically Chromium anyway. It only matters for the few features Chrome adds (licensed codecs like H.264, some DRM) — if you scrape video-heavy sites, use a real Chrome binary via the channel option.

Is a headless browser faster than a normal browser? Somewhat — skipping GPU compositing and paint saves work, and benchmarks put headless runs a few percent to ~15% faster, plus far better parallelism on servers. The dramatic speedups come from what headless enables: blocking images/fonts, no window management, dozens of concurrent instances. It is still orders of magnitude heavier than a plain HTTP request — don't render pages that curl could fetch.

Can websites detect headless browsers? Yes. Default fingerprints (the HeadlessChrome UA, navigator.webdriver, missing plugins) are trivially detectable, and anti-bot vendors detect much subtler signals. Stealth plugins and residential proxies beat basic checks; sophisticated protection usually still wins. See the detection section above for the honest breakdown.

Which headless browser is best for scraping? Headless Chromium via Playwright is the current default recommendation: best waiting model, per-context proxies, three engines behind one API. Puppeteer is equally capable if you're Node-only; Selenium remains the choice for maximum language/browser breadth or existing test infrastructure. PhantomJS is dead (unmaintained since 2018) — don't start anything new on it.

How much RAM does headless Chrome need? Rule of thumb: 300–500 MB per browser instance with one active page, growing with page complexity and tab count. A 4 GB server comfortably runs ~8–10 concurrent pages with resource blocking enabled. Restart browsers periodically — long-lived instances leak.

Do I always need a headless browser for scraping? No — it's the expensive tool for a specific problem: content that only exists after JavaScript runs. Check the raw response first (curl the URL or view source): if your data is there, an HTTP client plus an HTML parser is faster and 10–50× cheaper. Many "JavaScript sites" also expose the underlying JSON API in the network tab, which beats rendering entirely.