---
title: "Playwright Web Scraping: The Complete Guide (Python & Node.js)"
description: "How to scrape websites with Playwright: install, locators, waits, headless mode, proxies, screenshots, Docker, and how it compares to Selenium and Puppeteer."
url: https://webscraping.ai/blog/playwright-web-scraping
markdown_index: https://webscraping.ai/llms.txt
---
# Playwright Web Scraping: The Complete Guide (Python & Node.js)

Playwright is the browser automation library that made JavaScript-heavy sites scrapeable without pain: one API drives Chromium, Firefox, and WebKit, waits for elements automatically, and works from Python, Node.js, Java, or C#. This guide covers the full web scraping workflow — installation, locators, waiting, headless mode, proxies, network interception, sessions, screenshots, Docker — plus an honest comparison with Selenium and Puppeteer and what to do when Playwright alone gets blocked.

## Key Takeaways

- `pip install playwright && playwright install chromium` (or `npm init playwright@latest`) gets you scraping in two commands
- Locators auto-wait for elements — most explicit `sleep`/wait code you'd write in Selenium simply disappears
- Browsers run headless by default; flip `headless=False` while developing to watch what your scraper does
- Proxies, user agents, cookies, and geolocation are set per browser context, so one browser can run many isolated sessions
- Route interception can block images/fonts (faster crawls) or capture the site's own JSON API responses — often better than parsing HTML
- Playwright automates a real browser but doesn't hide it — for anti-bot walls, pair it with rotating proxies or a scraping API

## What is Playwright?

Playwright is an open-source browser automation framework from Microsoft, released in 2020 by the team that originally built Puppeteer at Google. It controls real browsers — Chromium, Firefox, and WebKit — over a fast bidirectional protocol, with official bindings for JavaScript/TypeScript, Python, Java, and .NET.

It was designed for end-to-end testing, but the same properties make it a first-class scraping tool: it executes JavaScript like a real user's browser, its auto-waiting model eliminates most timing bugs, and its context model makes parallel, isolated sessions cheap. If the data you need only appears after scripts run — React storefronts, infinite scroll feeds, dashboards — a plain HTTP client can't see it, and Playwright can.

## Installing Playwright

Playwright ships in two parts: the library and the browser binaries (`playwright install` downloads patched builds; your system Chrome is not used by default).

**Python:**

```bash
pip install playwright
playwright install chromium # or: playwright install (all three engines)
```

**Node.js:**

```bash
npm init playwright@latest # scaffolded, or:
npm i -D playwright && npx playwright install chromium
```

On Linux servers, `playwright install --with-deps chromium` also apt-installs the system libraries browsers need. Install only the engines you scrape with — each browser is a 100–300 MB download.

## Your first scraper

Python offers sync and async APIs; the sync API is the comfortable default for scripts:

```python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch() # headless by default
    page = browser.new_page()
    page.goto("https://example.com/products")

    for item in page.locator(".product-card").all():
        name = item.locator("h2").inner_text()
        price = item.locator(".price").inner_text()
        print(name, price)

    browser.close()
```

The same scraper in Node.js:

```javascript
const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com/products');

  for (const item of await page.locator('.product-card').all()) {
    const name = await item.locator('h2').innerText();
    const price = await item.locator('.price').innerText();
    console.log(name, price);
  }

  await browser.close();
})();
```

For high-concurrency crawlers in Python, the async API (`async_playwright`) lets one process drive many pages; the method names are identical with `await` in front.

## Locators and selectors

`page.locator()` is the core of modern Playwright. A locator is a lazy _description_ of an element — it's resolved fresh every time you act on it, and every action auto-waits for the element to be ready. That's the difference from the older `query_selector`, which grabs a one-time element handle that can go stale.

```python
# CSS and XPath
page.locator("div.product-card")
page.locator("//div[@class='product-card']") # XPath inferred from // prefix

# User-facing locators — more resilient to markup changes
page.get_by_role("button", name="Next page")
page.get_by_text("Add to cart")
page.get_by_label("Search")

# Chaining and filtering
page.locator(".product-card").filter(has_text="SSD").locator(".price")

# Multiple matches
page.locator(".product-card").count()
page.locator(".product-card").nth(2)
page.locator(".product-card").all()
```

Extracting data from located elements:

```python
name = page.locator("h1").inner_text() # rendered text
raw = page.locator("h1").text_content() # includes hidden text
href = page.locator("a.next").get_attribute("href")
html = page.locator("#reviews").inner_html()
full = page.content() # entire rendered page HTML
```

`page.content()` matters for scraping: it returns the _current_ DOM as HTML, so you can feed a fully rendered page into BeautifulSoup, lxml, or Cheerio and keep your existing parsing code.

## Waiting: how Playwright handles timing

Playwright's answer to the `time.sleep(5)` you see in old Selenium scripts is **auto-waiting** : every locator action waits (up to the action timeout, 30 s by default) for the element to be attached, visible, stable, and enabled before proceeding. Most scrapers need no explicit waits at all.

When you do need to wait explicitly, wait for a _condition_, not a duration:

```python
# Navigation waits for the 'load' event by default; DOM-ready is often enough
page.goto(url, wait_until="domcontentloaded")

# Wait for a specific element to appear
page.locator(".results").wait_for(state="visible")

# Wait for the site's own API call to finish
with page.expect_response(lambda r: "/api/search" in r.url) as resp:
    page.get_by_role("button", name="Search").click()
data = resp.value.json()

# Wait until network goes quiet — use sparingly; sites with analytics
# pings or long-polling may never reach networkidle
page.wait_for_load_state("networkidle")
```

A hard `page.wait_for_timeout(3000)` exists but is a last resort — it makes scrapers both slower _and_ flakier than condition-based waits.

For infinite scroll, combine scrolling with a growth check:

```python
previous = 0
while True:
    page.mouse.wheel(0, 20000)
    page.wait_for_timeout(800) # brief settle between scrolls
    count = page.locator(".product-card").count()
    if count == previous:
        break
    previous = count
```

## Headless mode

Playwright runs browsers **headless by default** — no window, less CPU/RAM, works on servers. During development, run headed to watch the scraper drive the page:

```python
browser = p.chromium.launch(headless=False, slow_mo=300) # slow_mo: ms between actions
```

Two useful facts: by default Playwright's headless Chromium is the lighter `chromium-headless-shell` build, not the full browser — pass `channel="chromium"` to `launch()` to opt into Chrome's "new" headless mode, which is the real browser and behaves closer to headed — and headless is still _detectable_ — sites can spot `HeadlessChrome` in the default user agent and subtler environment signals, which is where the [stealth section](https://webscraping.ai/blog/playwright-web-scraping#user-agents-fingerprints-and-getting-blocked) comes in.

## Browser contexts: isolated parallel sessions

A `BrowserContext` is an incognito-style profile inside one browser process: separate cookies, storage, cache, and settings. Contexts are cheap (milliseconds to create) — the standard pattern for parallel scraping is one browser, many contexts:

```python
browser = p.chromium.launch()

us_ctx = browser.new_context(
    locale="en-US",
    timezone_id="America/New_York",
    viewport={"width": 1920, "height": 1080},
)
de_ctx = browser.new_context(locale="de-DE", timezone_id="Europe/Berlin")

page_us = us_ctx.new_page()
page_de = de_ctx.new_page() # completely isolated from page_us
```

Contexts are also where sessions live. Log in once, save the storage state, and reuse it across runs — no more re-running login flows:

```python
context = browser.new_context()
page = context.new_page()
page.goto("https://example.com/login")
page.get_by_label("Email").fill("demo@example.com")
page.get_by_label("Password").fill("secret")
page.get_by_role("button", name="Log in").click()
context.storage_state(path="session.json") # cookies + localStorage

# Later runs: start already logged in
context = browser.new_context(storage_state="session.json")
```

## Network interception

`page.route()` intercepts every request the page makes. The two scraping superpowers this unlocks:

**1. Block heavy resources for faster, cheaper crawls:**

```python
page.route("**/*", lambda route:
    route.abort() if route.request.resource_type in ("image", "font", "media")
    else route.continue_())
```

**2. Capture the site's internal API instead of parsing HTML.** Most dynamic sites fetch JSON and render it client-side; reading that JSON directly is faster and more robust than scraping the DOM:

```python
def handle_response(response):
    if "/api/products" in response.url and response.ok:
        for product in response.json()["items"]:
            print(product["name"], product["price"])

page.on("response", handle_response)
page.goto("https://example.com/catalog")
```

You can also modify requests in flight — add headers, rewrite URLs, or return mocked responses with `route.fulfill()`. Reading the site's own JSON is the single biggest robustness win on feed-shaped targets — [job boards](https://webscraping.ai/use-cases/job-listing-aggregation), [property listings](https://webscraping.ai/use-cases/property-listing-aggregation), [retail catalogs](https://webscraping.ai/use-cases/price-monitoring) — because the JSON schema changes far less often than the markup around it.

## Using proxies

Proxies are configured at launch (for the whole browser) or per context (different IPs per session — the pattern rotating-proxy scraping needs):

```python
# Whole browser
browser = p.chromium.launch(proxy={
    "server": "http://proxy.example.com:8080",
    "username": "proxyuser",
    "password": "proxypass",
})

# Per context — one browser, many exit IPs
context = browser.new_context(proxy={"server": "http://us-proxy.example.com:8080"})
```

SOCKS5 works via `"server": "socks5://..."` (no auth support for SOCKS in Chromium). To verify the proxy is live, hit an IP-echo endpoint before scraping. One static datacenter IP rarely survives serious scraping — residential rotation is what actually keeps large crawls alive; see our breakdown of [proxy types](https://webscraping.ai/blog/types-of-proxies-for-web-scraping) and [residential proxy providers](https://webscraping.ai/blog/best-residential-proxy-providers-for-web-scraping).

## User agents, fingerprints, and getting blocked

Playwright's defaults advertise automation: the user agent says `HeadlessChrome`, and `navigator.webdriver` is `true`. The basics are context options:

```python
context = browser.new_context(
    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
               "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    viewport={"width": 1920, "height": 1080},
    locale="en-US",
)
```

Rotate user agents across contexts rather than hardcoding one forever (our [user-agent rotation guide](https://webscraping.ai/blog/user-agent-rotation-for-web-scraping) covers sane strategies). Beyond headers, anti-bot vendors fingerprint TLS, canvas, WebGL, fonts, and timing; the `playwright-stealth`/`playwright-extra` plugins patch the obvious JS-level tells, and headed mode with a real display fools more checks than headless. But understand the ceiling: against Cloudflare, DataDome, or PerimeterX, patched fingerprints are an arms race you're on the losing side of — that's a build-vs-buy decision, not a code snippet.

## Screenshots and PDFs

Playwright screenshots are pixel-accurate renders of the live page — useful for archiving evidence alongside scraped data and for debugging headless runs:

```python
page.screenshot(path="page.png") # viewport
page.screenshot(path="full.png", full_page=True) # entire scrollable page
page.locator(".pricing-table").screenshot(path="el.png") # single element
page.pdf(path="page.pdf") # Chromium only
```

On failures, dump both a screenshot and `page.content()` — the pair answers "what did the bot actually see?" faster than any log line.

## Timeouts and error handling

Playwright's defaults: 30 seconds for actions and navigation. Slow proxies need more; fast local crawls want less, so failures surface quickly:

```python
context.set_default_timeout(15_000) # all actions in this context
page.goto(url, timeout=60_000) # per call
page.locator(".price").inner_text(timeout=5_000)
```

Timeouts throw `TimeoutError` — catch it per page so one dead URL doesn't kill a crawl:

```python
from playwright.sync_api import TimeoutError as PlaywrightTimeout

for url in urls:
    try:
        page.goto(url, wait_until="domcontentloaded")
        scrape(page)
    except PlaywrightTimeout:
        failed.append(url)
```

Two hygiene rules for long-running scrapers: always close pages/contexts you're done with (each leaks memory otherwise), and restart the browser process every few hundred pages — Chromium's memory only grows.

## Running Playwright in Docker

The official image ships browsers and every system dependency preinstalled — matching your library version to the image tag is the one thing to get right:

```dockerfile
FROM mcr.microsoft.com/playwright/python:v1.63.0-noble
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt # playwright==1.63.0 to match the image
COPY . .
CMD ["python", "scraper.py"]
```

(For Node.js: `mcr.microsoft.com/playwright:v1.63.0-noble`.) Run with `--init` to reap zombie Chromium processes, and `--ipc=host` to avoid shared-memory crashes on Chromium. Headless-only workloads need no display server; if a site behaves differently headed, `xvfb-run python scraper.py` inside the container gets you a virtual display.

## Playwright vs Selenium vs Puppeteer

| | Playwright | Selenium | Puppeteer |
| --- | --- | --- | --- |
| Browsers | Chromium, Firefox, WebKit | All majors incl. Safari | Chrome/Chromium, Firefox (via WebDriver BiDi) |
| Languages | JS/TS, Python, Java, .NET | Most languages | JS/TS only |
| Auto-waiting | Built into every action | Manual explicit waits | Partial |
| Protocol | Own bidirectional protocol | WebDriver (HTTP) + BiDi | CDP (Chrome), WebDriver BiDi (Firefox) |
| Network interception | First-class | Limited (BiDi improving) | First-class |
| Speed | Fast | Slowest of the three | Fast |
| Ecosystem age | 2020+ | 2004+, huge legacy base | 2017+ |

The practical read: **Selenium** still wins when you must test real Safari, drive an exotic browser/grid, or work in a language Playwright doesn't support — and it powers a lot of existing scraping code (see our [Python Selenium guide](https://webscraping.ai/blog/python-selenium)). **Puppeteer** is fine when you're all-in on Node + Chrome. For new scraping projects in 2026, Playwright is the default: same speed class as Puppeteer, more languages, more browsers, and the best waiting/locator model of the three, which directly translates into fewer flaky scrapes.

## When Playwright isn't enough

Playwright solves JavaScript rendering. It does not solve the operational side of scraping at scale:

- **Anti-bot walls.** Cloudflare, DataDome, and CAPTCHAs detect automation fingerprints regardless of how the browser is driven. Stealth plugins help at the margins; they don't win the arms race.
- **Proxy fleets.** Serious crawls need residential rotation, geo-targeting, retry logic, and IP health tracking — infrastructure, not scraping code.
- **Browser fleet cost.** Every concurrent page is a real Chromium eating ~100+ MB of RAM. A thousand-page-per-minute crawl is a cluster engineering project.

If you'd rather keep writing extraction logic and skip that infrastructure, the same job can run through an API: [WebScraping.AI](https://webscraping.ai/) renders pages in headless Chrome with rotating residential proxies and anti-bot handling behind one HTTP call:

```python
import requests

# Fully rendered HTML — JavaScript executed, proxies handled
html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/spa-products",
    "js": "true",
}).text

# Or skip HTML parsing entirely — ask questions about the page
answer = requests.get("https://api.webscraping.ai/ai/question", params={
    "api_key": API_KEY,
    "url": "https://example.com/product/42",
    "question": "What is the price and is it in stock?",
}).text
```

A common hybrid: prototype the crawl in Playwright locally, then swap `page.goto()` + `page.content()` for the [/html endpoint](https://webscraping.ai/docs) in production and keep your locator-free parsing code unchanged. See the [AI web scraping overview](https://webscraping.ai/ai-web-scraping) for the extraction endpoints, or the [n8n node](https://webscraping.ai/integrations/n8n) if the crawl belongs in a workflow rather than a script.

## Frequently asked questions

**Is Playwright good for web scraping?** Yes — for JavaScript-heavy sites it's arguably the best open-source option: real browser rendering, auto-waiting locators, network interception, and multi-language support. For static HTML pages, a plain HTTP client with a parser is 10–50× cheaper; don't launch Chromium to fetch pages `requests` could.

**Playwright or BeautifulSoup — which should I use?** They're different layers, not competitors. BeautifulSoup parses HTML you already have; Playwright obtains HTML that requires a browser to produce. A common pipeline is Playwright for fetching (`page.content()`) and BeautifulSoup for parsing — see our [Python web scraping guide](https://webscraping.ai/blog/web-scraping-with-python).

**How do I scrape iframes and shadow DOM?** Iframes: `page.frame_locator("#checkout-frame").locator("button")` targets elements inside the frame. Shadow DOM: Playwright locators pierce open shadow roots automatically — a plain `page.locator("my-widget .price")` works where raw CSS `querySelector` fails. Closed shadow roots remain inaccessible to all tools.

**Can Playwright bypass Cloudflare or CAPTCHAs?** Not reliably on its own. Stealth patches and residential proxies raise the success rate; managed challenges and CAPTCHAs still stop unattended browsers. When a target sits behind serious anti-bot protection, a scraping API that handles the challenge layer is usually cheaper than fighting it — that's exactly the case [WebScraping.AI](https://webscraping.ai/) exists for.

**How much does Playwright cost to run?** The library is free (Apache 2.0). The real cost is compute: each concurrent Chromium page uses roughly 100–400 MB of RAM plus meaningful CPU during rendering. Budget capacity by peak concurrent pages, not total page count.

**Is scraping with Playwright legal?** Automating a browser is legal; what matters is what you scrape and how you use it — public vs. gated data, terms of service, rate pressure, and privacy law. Our [web scraping legality guide](https://webscraping.ai/blog/is-web-scraping-legal) covers the current state of the case law.