Puppeteer Web Scraping: The Complete Node.js Guide

Puppeteer is the browser automation library that made headless Chrome scraping mainstream: a Node.js API that drives a real browser, executes JavaScript exactly like a user's Chrome does, and exposes everything — DOM, network, cookies, screenshots — to your script. This guide covers the full scraping workflow: installation, selectors, waiting, clicking and forms, AJAX and single-page apps, authentication and sessions, PDFs, proxies and stealth, Docker, performance, plus an honest Puppeteer vs Playwright comparison and what to do when a real browser alone isn't enough.

Key Takeaways

• npm i puppeteer installs the library plus a matching Chrome for Testing build — no manual browser setup
• Prefer page.waitForSelector() (or the newer page.locator() API) over fixed sleeps; most flaky scrapers are just waiting bugs
• The fastest way to scrape a JavaScript-heavy site is often not parsing its DOM but capturing its own JSON API calls with page.on('response')
• Reuse one browser with many pages, close pages when done, and restart the browser every few hundred pages to keep memory flat
• puppeteer-extra-plugin-stealth hides the obvious automation signals, but modern anti-bot systems (Cloudflare, DataDome) still detect it — plan for proxies or a scraping API on protected targets
• For PDFs use page.pdf() (headless only); for screenshots page.screenshot({ fullPage: true })

What is Puppeteer?

Puppeteer is an open-source Node.js library from Google's Chrome team, released in 2017. It controls Chrome or Chromium over the DevTools Protocol (and, since v23, Firefox over WebDriver BiDi), giving your script the same view of a page that a real user has: JavaScript executed, XHR responses applied, DOM fully rendered. If your stack is .NET rather than Node, the same API exists in C# as PuppeteerSharp.

That's exactly what plain HTTP clients can't do. fetch() + Cheerio sees only the initial HTML; if the prices, listings, or reviews you need are rendered client-side by React or Vue, they simply aren't there. Puppeteer runs the page for real, so anything a browser can show, your scraper can read. The trade-off is cost: every page means a full browser rendering pipeline, so Puppeteer is the tool for JavaScript-heavy targets, not a default for every fetch — see our headless browser guide for when a real browser is and isn't worth it.

Prerequisites and installation

You need Node.js 18+ and one command:

npm i puppeteer

This downloads both the library and a pinned Chrome for Testing build (~150 MB) that's guaranteed compatible — you never wrestle with browser/driver version mismatches. If you want to use an existing Chrome install (e.g. in a constrained Docker image), npm i puppeteer-core skips the download and you pass executablePath yourself.

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch();        // headless by default
const page = await browser.newPage();
await page.goto('https://example.com', { waitUntil: 'networkidle2' });
console.log(await page.title());
await browser.close();

While developing, launch with { headless: false, slowMo: 50 } to watch the browser do what your code says — the single best debugging trick Puppeteer has.

Selectors and extracting data

Puppeteer queries the live DOM with CSS selectors (see our CSS selectors FAQ for the selector syntax itself):

// One element / all elements (ElementHandles)
const el = await page.$('h1');
const rows = await page.$$('table tr');

// Evaluate in the page and return plain data — the workhorses
const title = await page.$eval('h1', el => el.textContent.trim());
const products = await page.$$eval('.product', els =>
  els.map(el => ({
    name: el.querySelector('.name')?.textContent.trim(),
    price: el.querySelector('.price')?.textContent.trim(),
    url: el.querySelector('a')?.href,
  }))
);

$eval/$$eval run your callback inside the browser, so only serializable data comes back — no DOM handles to leak. For anything beyond a single selector, page.evaluate() lets you run arbitrary JavaScript in the page context and return a JSON-shaped result. XPath is available via the ::-p-xpath() selector prefix, but CSS covers nearly every scraping case.

Waiting: the part everyone gets wrong

Most "Puppeteer is flaky" complaints are timing bugs: the script queried the DOM before the site's JavaScript finished building it. The old page.waitFor() and page.waitForTimeout() are gone from current Puppeteer — and fixed sleeps were always the wrong tool. Wait for conditions instead:

// Wait until the element exists (default timeout 30s)
await page.waitForSelector('.results .product', { timeout: 15000 });

// Wait for an arbitrary predicate in the page
await page.waitForFunction(() => document.querySelectorAll('.product').length > 20);

// Wait for navigation triggered by a click — combine, don't sequence
await Promise.all([
  page.waitForNavigation({ waitUntil: 'networkidle2' }),
  page.click('a.next-page'),
]);

Modern Puppeteer also has a locator API with built-in waiting, closer to Playwright's model:

await page.locator('button.load-more').click();   // waits for visible + enabled, then clicks
const text = await page.locator('.price').map(el => el.textContent).wait();

On page.goto(), pick waitUntil deliberately: 'domcontentloaded' is fastest, 'networkidle2' (≤2 in-flight requests for 500 ms) is the pragmatic choice for SPAs, 'networkidle0' the strict one. If you truly need a pause, new Promise(r => setTimeout(r, ms)) works — but reach for it last.

Clicking, typing, and forms

Simulating user interaction is half of scraping dynamic sites — dismissing cookie banners, submitting search forms, expanding "load more" sections:

await page.click('#accept-cookies');                  // single click
await page.click('.item', { clickCount: 2 });         // double-click
await page.type('#search', 'mechanical keyboard', { delay: 30 });  // types like a human
await page.keyboard.press('Enter');
await page.select('select#sort', 'price-asc');        // <select> dropdowns
await page.hover('.menu-trigger');                    // reveal hover menus

If a click "does nothing", the usual causes: the element is covered by an overlay (dismiss the modal first), it's outside the viewport (Puppeteer scrolls automatically, but sticky headers can intercept the click), or the site listens for trusted events on a child element — page.click() dispatches real input events at coordinates, so click the visible child, not the wrapper. Pop-ups that open new tabs land in browser.pages(); native dialogs (alert/confirm) are handled with page.on('dialog', d => d.dismiss()).

Scraping AJAX, infinite scroll, and single-page apps

For content that loads after the initial render, you have two strategies — and the second is usually better.

Strategy 1: wait for the rendered DOM. Scroll, wait for selectors, extract:

// Infinite scroll: keep scrolling until the item count stops growing
let previous = 0;
while (true) {
  const count = await page.$$eval('.item', els => els.length);
  if (count === previous) break;
  previous = count;
  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await page.waitForNetworkIdle({ idleTime: 500 }).catch(() => {});
}

Strategy 2: capture the site's own API. SPAs fetch their data as JSON; intercept it and skip HTML parsing entirely:

page.on('response', async (response) => {
  if (response.url().includes('/api/products') && response.ok()) {
    const data = await response.json();     // clean, structured, no selectors
    results.push(...data.items);
  }
});
await page.goto('https://spa-shop.example.com/catalog');

Request interception also cuts rendering cost dramatically — block what you don't need:

await page.setRequestInterception(true);
page.on('request', (req) =>
  ['image', 'font', 'media', 'stylesheet'].includes(req.resourceType())
    ? req.abort()
    : req.continue()
);

For crawling an SPA, remember client-side route changes don't fire full navigations — wait on page.waitForFunction(() => location.pathname === '/target') or on the API response you expect, not on waitForNavigation.

Authentication, cookies, and sessions

Three layers, three tools:

HTTP basic auth is one call: await page.authenticate({ username, password }) (also used for authenticating proxies).

Form login is just clicking and typing — the interesting part is not doing it on every run. Persist the session:

// Log in once, save cookies
await page.type('#email', process.env.SCRAPE_USER);
await page.type('#password', process.env.SCRAPE_PASS);
await Promise.all([page.waitForNavigation(), page.click('button[type=submit]')]);
const cookies = await browser.cookies();
fs.writeFileSync('cookies.json', JSON.stringify(cookies));

// Later runs: restore and skip the login form
await browser.setCookie(...JSON.parse(fs.readFileSync('cookies.json')));

(On older Puppeteer versions the same pair is page.cookies() / page.setCookie().) The heavier alternative is puppeteer.launch({ userDataDir: './profile' }), which persists the whole browser profile — cookies, localStorage, cache — across runs, exactly like a real Chrome profile. Use isolated contexts (browser.createBrowserContext()) when you need several accounts in one browser without shared cookies.

Sites protecting logins with 2FA, device checks, or CAPTCHA are telling you automated login isn't welcome — check the terms and our legality guide before scripting around them.

Screenshots and PDFs

await page.screenshot({ path: 'page.png', fullPage: true });
await page.screenshot({ path: 'el.png', clip: await (await page.$('.chart')).boundingBox() });

await page.pdf({ path: 'report.pdf', format: 'A4', printBackground: true });

page.pdf() works only in headless mode and renders the print stylesheet — set printBackground: true or pages come out white. Puppeteer is genuinely excellent at HTML-to-PDF (invoices, reports); many teams run it for that alone.

Iframes, multiple tabs, and downloads

Iframes have their own DOM — page.$ can't see into them. Get the frame first:

const frame = page.frames().find(f => f.url().includes('checkout'));
const total = await frame.$eval('.total', el => el.textContent);

Multiple tabs: browser.pages() lists open tabs; catch tabs opened by clicks with browser.waitForTarget() or the targetcreated event. Tabs are cheap — parallelize with several pages in one browser rather than several browsers.

Downloads go through a CDP session:

const client = await browser.target().createCDPSession();
await client.send('Browser.setDownloadBehavior', {
  behavior: 'allow', downloadPath: '/tmp/scrape-downloads',
});
await page.click('a.export-csv');

If the "download" is really a URL, skip the browser: grab href and fetch it directly with the session cookies.

Performance, parallelism, and memory

Chrome is the cost center — a rendering page holds 100–400 MB of RAM. The rules that keep large crawls flat:

• One browser, many pages. puppeteer.launch() is expensive; browser.newPage() is cheap. Run 5–10 concurrent pages per browser process.
• Always close pages in a finally block. Leaked pages are the classic Puppeteer memory leak.
• Block images/fonts/CSS via request interception (above) — typically 2–5× faster page loads.
• Recycle the browser every few hundred pages; long-lived Chrome processes accumulate memory regardless of your code.
• For a managed worker pool with retries and concurrency limits, puppeteer-cluster does the bookkeeping for you.

const results = await Promise.all(urls.map(async (url) => {
  const page = await browser.newPage();
  try {
    await page.goto(url, { waitUntil: 'domcontentloaded' });
    return await page.$eval('h1', el => el.textContent);
  } finally {
    await page.close();
  }
}));

Errors worth planning for: TimeoutError (retry with backoff — transient slowness is normal), net::ERR_* navigation failures, and crashed pages (page.on('error')). Wrap per-URL work so one bad page doesn't kill the crawl.

Proxies, stealth, and getting blocked

Puppeteer automates a real browser but doesn't hide the automation. Headless Chrome leaks signals — navigator.webdriver, missing plugins, headless UA hints — and sites cross-check IP reputation, TLS fingerprints, and behavior.

Proxies are set per browser launch:

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://proxy.example.com:8000'],
});
await page.authenticate({ username: 'user', password: 'pass' });  // proxy credentials

One launch = one proxy; rotating per-request means multiple browsers or a local forwarder like proxy-chain.

Stealth: puppeteer-extra-plugin-stealth patches the well-known giveaways and gets you past basic checks. Be honest with yourself about its limits: it's a cat-and-mouse patchset, and current commercial anti-bot systems (Cloudflare Bot Management, DataDome, PerimeterX) fingerprint deeper than it patches. On protected targets, expect to add rotating residential proxies, human-like pacing, and CAPTCHA handling — and expect maintenance forever. That maintenance cost is the actual argument for scraping APIs, not any single technical wall.

Running Puppeteer in Docker

Chrome needs system libraries a slim Node image doesn't have. The path of least pain is the official image, which ships all dependencies:

FROM ghcr.io/puppeteer/puppeteer:latest
COPY --chown=pptruser . /app
WORKDIR /app
RUN npm ci
CMD ["node", "scrape.js"]

Two flags matter in containers:

const browser = await puppeteer.launch({
  args: ['--no-sandbox', '--disable-dev-shm-usage'],
});

--disable-dev-shm-usage stops crashes from Docker's tiny default /dev/shm; --no-sandbox is required when running as root (better: run as a non-root user like the official image's pptruser and keep the sandbox). Pin your Puppeteer version so the bundled Chrome doesn't shift under you between builds.

Puppeteer vs Playwright

The comparison everyone asks about — the short answer is they're siblings (Microsoft's Playwright team originally built Puppeteer at Google), and for scraping the differences are real but not dramatic:

Choose Puppeteer when you're all-in on Node and Chrome — it's lighter, mature, and the Chrome DevTools integration is unmatched. Choose Playwright for Python/Java/.NET teams, WebKit/Safari coverage, per-context proxies, or its stricter auto-waiting model, which produces fewer flaky scrapes with less code — our Playwright web scraping guide covers it end to end. Migrating between them is straightforward; don't agonize over the choice.

When Puppeteer isn't enough

Puppeteer solves rendering. It does not solve IP reputation, anti-bot walls, CAPTCHAs, or the ops burden of running a Chrome fleet at scale. When those become the problem, the pragmatic move is to let an API do the browser part — WebScraping.AI renders pages in real Chrome with rotating datacenter or residential proxies and returns the result over plain HTTP:

// Rendered HTML through a managed browser + residential proxies
const html = await fetch(
  'https://api.webscraping.ai/html?' + new URLSearchParams({
    api_key: process.env.WSAI_KEY,
    url: 'https://example.com/product/42',
    js: 'true',
    proxy: 'residential',
  })
).then(r => r.text());

// Or skip parsing entirely — structured fields from any page
const fields = await fetch(
  'https://api.webscraping.ai/ai/fields?' + new URLSearchParams({
    api_key: process.env.WSAI_KEY,
    url: 'https://example.com/product/42',
    fields: JSON.stringify({ name: 'Product name', price: 'Price with currency' }),
  })
).then(r => r.json());

A common hybrid: prototype locally with Puppeteer, then swap page.goto() + page.content() for the /html endpoint in production and keep your Cheerio/$$eval parsing logic unchanged. See the AI web scraping overview for the extraction endpoints.

Frequently asked questions

Is Puppeteer good for web scraping? Yes — for JavaScript-rendered sites it's one of the best options in the Node ecosystem: real Chrome rendering, network interception, and a huge community. For static pages, plain fetch + Cheerio is 10–50× cheaper; don't launch Chrome for HTML that's already in the response. Our JavaScript scraping libraries roundup maps the whole toolbox.

Puppeteer or Playwright for scraping? Functionally close. Puppeteer if you're Node-only and Chrome-focused; Playwright if you want Python/Java/.NET, WebKit coverage, or default auto-waiting. Both are free and actively maintained, and neither beats the other at avoiding bot detection — that battle is won by proxies and infrastructure, not the automation library.

Why did page.waitForTimeout stop working? It was removed in Puppeteer v22 along with the long-deprecated page.waitFor(). Replace fixed pauses with waitForSelector/waitForFunction/waitForNetworkIdle, or new Promise(r => setTimeout(r, ms)) if you genuinely need a sleep.

Can Puppeteer bypass Cloudflare or CAPTCHAs? Not reliably. The stealth plugin passes basic checks, but managed challenges and CAPTCHAs from modern anti-bot vendors still stop unattended browsers, and each Chrome release resets the arms race. For protected targets, a scraping API that owns the challenge layer — which is exactly what WebScraping.AI does — is usually cheaper than maintaining your own evasion stack.

How much does Puppeteer cost to run? The library is free (Apache 2.0); the cost is Chrome: roughly 100–400 MB RAM per concurrent page plus significant CPU during rendering. Size your infrastructure by peak concurrent pages, and expect a fleet of small workers rather than one big process.

Is scraping with Puppeteer legal? Automating a browser is legal in itself; what matters is what you scrape and how — public vs. gated data, terms of service, rate pressure, and privacy law. Our web scraping legality guide covers the current case law.