---
title: "Puppeteer Web Scraping: The Complete Node.js Guide"
description: "How to scrape websites with Puppeteer: setup, selectors, waiting, clicks, AJAX, cookies, PDFs, proxies, stealth, Docker, and Puppeteer vs Playwright."
url: https://webscraping.ai/blog/puppeteer-web-scraping
markdown_index: https://webscraping.ai/llms.txt
---
# Puppeteer Web Scraping: The Complete Node.js Guide

Puppeteer is the browser automation library that made headless Chrome scraping mainstream: a Node.js API that drives a real browser, executes JavaScript exactly like a user's Chrome does, and exposes everything — DOM, network, cookies, screenshots — to your script. This guide covers the full scraping workflow: installation, selectors, waiting, clicking and forms, AJAX and single-page apps, authentication and sessions, PDFs, proxies and stealth, Docker, performance, plus an honest Puppeteer vs Playwright comparison and what to do when a real browser alone isn't enough.

## Key Takeaways

- `npm i puppeteer` installs the library plus a matching Chrome for Testing build — no manual browser setup
- Prefer `page.waitForSelector()` (or the newer `page.locator()` API) over fixed sleeps; most flaky scrapers are just waiting bugs
- The fastest way to scrape a JavaScript-heavy site is often not parsing its DOM but capturing its own JSON API calls with `page.on('response')`
- Reuse one browser with many pages, close pages when done, and restart the browser every few hundred pages to keep memory flat
- `puppeteer-extra-plugin-stealth` hides the obvious automation signals, but modern anti-bot systems (Cloudflare, DataDome) still detect it — plan for proxies or a scraping API on protected targets
- For PDFs use `page.pdf()`; for screenshots `page.screenshot({ fullPage: true })`

## What is Puppeteer?

Puppeteer is an open-source Node.js library from Google's Chrome team, released in 2017. It controls Chrome or Chromium over the DevTools Protocol (and, since v23, Firefox over WebDriver BiDi), giving your script the same view of a page that a real user has: JavaScript executed, XHR responses applied, DOM fully rendered. If your stack is .NET rather than Node, the same API exists in C# as [PuppeteerSharp](https://webscraping.ai/blog/puppeteersharp-web-scraping).

That's exactly what plain HTTP clients can't do. `fetch()` + [Cheerio](https://webscraping.ai/blog/cheerio-web-scraping) sees only the initial HTML; if the prices, listings, or reviews you need are rendered client-side by React or Vue, they simply aren't there. Puppeteer runs the page for real, so anything a browser can show, your scraper can read. The trade-off is cost: every page means a full browser rendering pipeline, so Puppeteer is the tool for JavaScript-heavy targets, not a default for every fetch — see our [headless browser guide](https://webscraping.ai/blog/headless-browser-guide) for when a real browser is and isn't worth it.

## Prerequisites and installation

You need Node.js 22.12+ (Puppeteer 25 dropped older Node versions and is ESM-only) and one command:

```bash
npm i puppeteer
```

This downloads the library plus a pinned **Chrome for Testing** build (roughly 170–280 MB depending on OS) and the lighter `chrome-headless-shell`, both guaranteed compatible — you never wrestle with browser/driver version mismatches. If you want to use an existing Chrome install (e.g. in a constrained Docker image), `npm i puppeteer-core` skips the download and you pass `executablePath` yourself.

```javascript
import puppeteer from 'puppeteer'; // Puppeteer 25+ is ESM-only: use .mjs or "type": "module"

const browser = await puppeteer.launch(); // headless by default
const page = await browser.newPage();
await page.goto('https://example.com', { waitUntil: 'networkidle2' });
console.log(await page.title());
await browser.close();
```

While developing, launch with `{ headless: false, slowMo: 50 }` to watch the browser do what your code says — the single best debugging trick Puppeteer has.

## Selectors and extracting data

Puppeteer queries the live DOM with CSS selectors (see our [CSS selectors cheat sheet](https://webscraping.ai/blog/css-selectors-cheat-sheet) for the selector syntax itself):

```javascript
// One element / all elements (ElementHandles)
const el = await page.$('h1');
const rows = await page.$$('table tr');

// Evaluate in the page and return plain data — the workhorses
const title = await page.$eval('h1', el => el.textContent.trim());
const products = await page.$$eval('.product', els =>
  els.map(el => ({
    name: el.querySelector('.name')?.textContent.trim(),
    price: el.querySelector('.price')?.textContent.trim(),
    url: el.querySelector('a')?.href,
  }))
);
```

`$eval`/`$$eval` run your callback inside the browser, so only serializable data comes back — no DOM handles to leak. For anything beyond a single selector, `page.evaluate()` lets you run arbitrary JavaScript in the page context and return a JSON-shaped result. XPath is available via the `::-p-xpath()` selector prefix, but CSS covers nearly every scraping case.

## Waiting: the part everyone gets wrong

Most "Puppeteer is flaky" complaints are timing bugs: the script queried the DOM before the site's JavaScript finished building it. The old `page.waitFor()` and `page.waitForTimeout()` are gone from current Puppeteer — and fixed sleeps were always the wrong tool. Wait for conditions instead:

```javascript
// Wait until the element exists (default timeout 30s)
await page.waitForSelector('.results .product', { timeout: 15000 });

// Wait for an arbitrary predicate in the page
await page.waitForFunction(() => document.querySelectorAll('.product').length > 20);

// Wait for navigation triggered by a click — combine, don't sequence
await Promise.all([
  page.waitForNavigation({ waitUntil: 'networkidle2' }),
  page.click('a.next-page'),
]);
```

Modern Puppeteer also has a locator API with built-in waiting, closer to Playwright's model:

```javascript
await page.locator('button.load-more').click(); // waits for visible + enabled, then clicks
const text = await page.locator('.price').map(el => el.textContent).wait();
```

On `page.goto()`, pick `waitUntil` deliberately: `'domcontentloaded'` is fastest, `'networkidle2'` (≤2 in-flight requests for 500 ms) is the pragmatic choice for SPAs, `'networkidle0'` the strict one. If you truly need a pause, `new Promise(r => setTimeout(r, ms))` works — but reach for it last.

## Clicking, typing, and forms

Simulating user interaction is half of scraping dynamic sites — dismissing cookie banners, submitting search forms, expanding "load more" sections:

```javascript
await page.click('#accept-cookies'); // single click
await page.click('.item', { clickCount: 2 }); // double-click
await page.type('#search', 'mechanical keyboard', { delay: 30 }); // types like a human
await page.keyboard.press('Enter');
await page.select('select#sort', 'price-asc'); // <select> dropdowns
await page.hover('.menu-trigger'); // reveal hover menus
```

If a click "does nothing", the usual causes: the element is covered by an overlay (dismiss the modal first), it's outside the viewport (Puppeteer scrolls automatically, but sticky headers can intercept the click), or the site listens for trusted events on a child element — `page.click()` dispatches real input events at coordinates, so click the visible child, not the wrapper. Pop-ups that open new tabs land in `browser.pages()`; native dialogs (`alert`/`confirm`) are handled with `page.on('dialog', d => d.dismiss())`.

## Scraping AJAX, infinite scroll, and single-page apps

For content that loads after the initial render, you have two strategies — and the second is usually better.

**Strategy 1: wait for the rendered DOM.** Scroll, wait for selectors, extract:

```javascript
// Infinite scroll: keep scrolling until the item count stops growing
let previous = 0;
while (true) {
  const count = await page.$$eval('.item', els => els.length);
  if (count === previous) break;
  previous = count;
  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await page.waitForNetworkIdle({ idleTime: 500 }).catch(() => {});
}
```

**Strategy 2: capture the site's own API.** SPAs fetch their data as JSON; intercept it and skip HTML parsing entirely:

```javascript
page.on('response', async (response) => {
  if (response.url().includes('/api/products') && response.ok()) {
    const data = await response.json(); // clean, structured, no selectors
    results.push(...data.items);
  }
});
await page.goto('https://spa-shop.example.com/catalog');
```

Request interception also cuts rendering cost dramatically — block what you don't need:

```javascript
await page.setRequestInterception(true);
page.on('request', (req) =>
  ['image', 'font', 'media', 'stylesheet'].includes(req.resourceType())
    ? req.abort()
    : req.continue()
);
```

Capturing the API rather than the DOM is what makes long-running crawls maintainable — [price monitoring](https://webscraping.ai/use-cases/price-monitoring) and [product data aggregation](https://webscraping.ai/use-cases/product-data-aggregation) jobs survive redesigns when they read the JSON the storefront reads.

For crawling an SPA, remember client-side route changes don't fire full navigations — wait on `page.waitForFunction(() => location.pathname === '/target')` or on the API response you expect, not on `waitForNavigation`.

## Authentication, cookies, and sessions

Three layers, three tools:

**HTTP basic auth** is one call: `await page.authenticate({ username, password })` (also used for authenticating proxies).

**Form login** is just clicking and typing — the interesting part is not doing it on every run. Persist the session:

```javascript
// Log in once, save cookies
await page.type('#email', process.env.SCRAPE_USER);
await page.type('#password', process.env.SCRAPE_PASS);
await Promise.all([page.waitForNavigation(), page.click('button[type=submit]')]);
const cookies = await browser.cookies();
fs.writeFileSync('cookies.json', JSON.stringify(cookies));

// Later runs: restore and skip the login form
await browser.setCookie(...JSON.parse(fs.readFileSync('cookies.json')));
```

(On older Puppeteer versions the same pair is `page.cookies()` / `page.setCookie()`.) The heavier alternative is `puppeteer.launch({ userDataDir: './profile' })`, which persists the whole browser profile — cookies, localStorage, cache — across runs, exactly like a real Chrome profile. Use isolated contexts (`browser.createBrowserContext()`) when you need several accounts in one browser without shared cookies.

Sites protecting logins with 2FA, device checks, or CAPTCHA are telling you automated login isn't welcome — check the terms and our [legality guide](https://webscraping.ai/blog/is-web-scraping-legal) before scripting around them.

## Screenshots and PDFs

```javascript
await page.screenshot({ path: 'page.png', fullPage: true });
await page.screenshot({ path: 'el.png', clip: await (await page.$('.chart')).boundingBox() });

await page.pdf({ path: 'report.pdf', format: 'A4', printBackground: true });
```

`page.pdf()` renders the print stylesheet — set `printBackground: true` or pages come out white. Puppeteer is genuinely excellent at HTML-to-PDF (invoices, reports); many teams run it for that alone.

## Iframes, multiple tabs, and downloads

**Iframes** have their own DOM — `page.$` can't see into them. Get the frame first:

```javascript
const frame = page.frames().find(f => f.url().includes('checkout'));
const total = await frame.$eval('.total', el => el.textContent);
```

**Multiple tabs** : `browser.pages()` lists open tabs; catch tabs opened by clicks with `browser.waitForTarget()` or the `targetcreated` event. Tabs are cheap — parallelize with several pages in one browser rather than several browsers.

**Downloads** go through a CDP session:

```javascript
const client = await browser.target().createCDPSession();
await client.send('Browser.setDownloadBehavior', {
  behavior: 'allow', downloadPath: '/tmp/scrape-downloads',
});
await page.click('a.export-csv');
```

If the "download" is really a URL, skip the browser: grab `href` and fetch it directly with the session cookies.

## Performance, parallelism, and memory

Chrome is the cost center — a rendering page holds 100–400 MB of RAM. The rules that keep large crawls flat:

- **One browser, many pages.** `puppeteer.launch()` is expensive; `browser.newPage()` is cheap. Run 5–10 concurrent pages per browser process.
- **Always close pages** in a `finally` block. Leaked pages are the classic Puppeteer memory leak.
- **Block images/fonts/CSS** via request interception (above) — typically 2–5× faster page loads.
- **Recycle the browser** every few hundred pages; long-lived Chrome processes accumulate memory regardless of your code.
- For a managed worker pool with retries and concurrency limits, [`puppeteer-cluster`](https://github.com/thomasdondorf/puppeteer-cluster) does the bookkeeping for you.

```javascript
const results = await Promise.all(urls.map(async (url) => {
  const page = await browser.newPage();
  try {
    await page.goto(url, { waitUntil: 'domcontentloaded' });
    return await page.$eval('h1', el => el.textContent);
  } finally {
    await page.close();
  }
}));
```

Errors worth planning for: `TimeoutError` (retry with backoff — transient slowness is normal), `net::ERR_*` navigation failures, and crashed pages (`page.on('error')`). Wrap per-URL work so one bad page doesn't kill the crawl.

## Proxies, stealth, and getting blocked

Puppeteer automates a real browser but doesn't hide the automation. Headless Chrome leaks signals — `navigator.webdriver`, missing plugins, headless UA hints — and sites cross-check IP reputation, TLS fingerprints, and behavior.

**Proxies** are set per browser launch, or per browser context:

```javascript
const browser = await puppeteer.launch({
  args: ['--proxy-server=http://proxy.example.com:8000'],
});
await page.authenticate({ username: 'user', password: 'pass' }); // proxy credentials

// Or one proxy per isolated context inside the same browser
const context = await browser.createBrowserContext({ proxyServer: 'http://proxy2.example.com:8000' });
```

Rotating per _request_ (rather than per context) still needs a local forwarder like `proxy-chain` or a rotating proxy endpoint.

**Stealth** : [`puppeteer-extra-plugin-stealth`](https://github.com/berstend/puppeteer-extra) patches the well-known giveaways and gets you past basic checks. Be honest with yourself about its limits: it's a cat-and-mouse patchset, and current commercial anti-bot systems (Cloudflare Bot Management, DataDome, PerimeterX) fingerprint deeper than it patches. On protected targets, expect to add rotating residential proxies, human-like pacing, and CAPTCHA handling — and expect maintenance forever. That maintenance cost is the actual argument for scraping APIs, not any single technical wall.

## Running Puppeteer in Docker

Chrome needs system libraries a slim Node image doesn't have. The path of least pain is the official image, which ships all dependencies:

```dockerfile
FROM ghcr.io/puppeteer/puppeteer:latest
COPY --chown=pptruser . /app
WORKDIR /app
RUN npm ci
CMD ["node", "scrape.js"]
```

Two flags matter in containers:

```javascript
const browser = await puppeteer.launch({
  args: ['--no-sandbox', '--disable-dev-shm-usage'],
});
```

`--disable-dev-shm-usage` stops crashes from Docker's tiny default `/dev/shm`; `--no-sandbox` is required when running as root (better: run as a non-root user like the official image's `pptruser` and keep the sandbox). Pin your Puppeteer version so the bundled Chrome doesn't shift under you between builds.

## Puppeteer vs Playwright

The comparison everyone asks about — the short answer is they're siblings (Microsoft's Playwright team originally built Puppeteer at Google), and for scraping the differences are real but not dramatic:

| | Puppeteer | Playwright |
| --- | --- | --- |
| Languages | JavaScript/TypeScript | JS/TS, Python, Java, .NET |
| Browsers | Chrome/Chromium, Firefox (BiDi) | Chromium, Firefox, WebKit |
| Auto-waiting | Locator API (newer, opt-in) | Built into every locator (default) |
| Isolated sessions | Browser contexts (per-context proxy) | Browser contexts (per-context proxy) |
| Maintainer | Google Chrome team | Microsoft |

Choose **Puppeteer** when you're all-in on Node and Chrome — it's lighter, mature, and the Chrome DevTools integration is unmatched. Choose **Playwright** for Python/Java/.NET teams, WebKit/Safari coverage, or its stricter auto-waiting model, which produces fewer flaky scrapes with less code — our [Playwright web scraping guide](https://webscraping.ai/blog/playwright-web-scraping) covers it end to end. Migrating between them is straightforward; don't agonize over the choice.

## When Puppeteer isn't enough

Puppeteer solves rendering. It does not solve IP reputation, anti-bot walls, CAPTCHAs, or the ops burden of running a Chrome fleet at scale. When those become the problem, the pragmatic move is to let an API do the browser part — [WebScraping.AI](https://webscraping.ai/) renders pages in real Chrome with rotating datacenter or residential proxies and returns the result over plain HTTP:

```javascript
// Rendered HTML through a managed browser + residential proxies
const html = await fetch(
  'https://api.webscraping.ai/html?' + new URLSearchParams({
    api_key: process.env.WSAI_KEY,
    url: 'https://example.com/product/42',
    js: 'true',
    proxy: 'residential',
  })
).then(r => r.text());

// Or skip parsing entirely — structured fields from any page
const fields = await fetch(
  'https://api.webscraping.ai/ai/fields?' + new URLSearchParams({
    api_key: process.env.WSAI_KEY,
    url: 'https://example.com/product/42',
    'fields[name]': 'Product name',
    'fields[price]': 'Price with currency',
  })
).then(r => r.json());
```

A common hybrid: prototype locally with Puppeteer, then swap `page.goto()` + `page.content()` for the [/html endpoint](https://webscraping.ai/docs) in production and keep your Cheerio/`$$eval` parsing logic unchanged. See the [AI web scraping overview](https://webscraping.ai/ai-web-scraping) for the extraction endpoints, or the [n8n node](https://webscraping.ai/integrations/n8n) and [MCP server](https://webscraping.ai/integrations/mcp-server) when the scrape belongs in a workflow or an agent instead of a Node script.

## Frequently asked questions

**Is Puppeteer good for web scraping?** Yes — for JavaScript-rendered sites it's one of the best options in the Node ecosystem: real Chrome rendering, network interception, and a huge community. For static pages, plain `fetch` + [Cheerio](https://webscraping.ai/blog/cheerio-web-scraping) is 10–50× cheaper; don't launch Chrome for HTML that's already in the response. Our [JavaScript scraping libraries roundup](https://webscraping.ai/blog/javascript-web-scraping-libraries) maps the whole toolbox.

**Puppeteer or Playwright for scraping?** Functionally close. Puppeteer if you're Node-only and Chrome-focused; Playwright if you want Python/Java/.NET, WebKit coverage, or default auto-waiting. Both are free and actively maintained, and neither beats the other at avoiding bot detection — that battle is won by proxies and infrastructure, not the automation library.

**Why did page.waitForTimeout stop working?** It was removed in Puppeteer v22 along with the long-deprecated `page.waitFor()`. Replace fixed pauses with `waitForSelector`/`waitForFunction`/`waitForNetworkIdle`, or `new Promise(r => setTimeout(r, ms))` if you genuinely need a sleep.

**Can Puppeteer bypass Cloudflare or CAPTCHAs?** Not reliably. The stealth plugin passes basic checks, but managed challenges and CAPTCHAs from modern anti-bot vendors still stop unattended browsers, and each Chrome release resets the arms race. For protected targets, a scraping API that owns the challenge layer — which is exactly what [WebScraping.AI](https://webscraping.ai/) does — is usually cheaper than maintaining your own evasion stack.

**How much does Puppeteer cost to run?** The library is free (Apache 2.0); the cost is Chrome: roughly 100–400 MB RAM per concurrent page plus significant CPU during rendering. Size your infrastructure by peak concurrent pages, and expect a fleet of small workers rather than one big process.

**Is scraping with Puppeteer legal?** Automating a browser is legal in itself; what matters is what you scrape and how — public vs. gated data, terms of service, rate pressure, and privacy law. Our [web scraping legality guide](https://webscraping.ai/blog/is-web-scraping-legal) covers the current case law.