Playwright Web Scraping: The Complete Guide (Python & Node.js)

Playwright is the browser automation library that made JavaScript-heavy sites scrapeable without pain: one API drives Chromium, Firefox, and WebKit, waits for elements automatically, and works from Python, Node.js, Java, or C#. This guide covers the full web scraping workflow — installation, locators, waiting, headless mode, proxies, network interception, sessions, screenshots, Docker — plus an honest comparison with Selenium and Puppeteer and what to do when Playwright alone gets blocked.

Key Takeaways

• pip install playwright && playwright install chromium (or npm init playwright@latest) gets you scraping in two commands
• Locators auto-wait for elements — most explicit sleep/wait code you'd write in Selenium simply disappears
• Browsers run headless by default; flip headless=False while developing to watch what your scraper does
• Proxies, user agents, cookies, and geolocation are set per browser context, so one browser can run many isolated sessions
• Route interception can block images/fonts (faster crawls) or capture the site's own JSON API responses — often better than parsing HTML
• Playwright automates a real browser but doesn't hide it — for anti-bot walls, pair it with rotating proxies or a scraping API

What is Playwright?

Playwright is an open-source browser automation framework from Microsoft, released in 2020 by the team that originally built Puppeteer at Google. It controls real browsers — Chromium, Firefox, and WebKit — over a fast bidirectional protocol, with official bindings for JavaScript/TypeScript, Python, Java, and .NET.

It was designed for end-to-end testing, but the same properties make it a first-class scraping tool: it executes JavaScript like a real user's browser, its auto-waiting model eliminates most timing bugs, and its context model makes parallel, isolated sessions cheap. If the data you need only appears after scripts run — React storefronts, infinite scroll feeds, dashboards — a plain HTTP client can't see it, and Playwright can.

Installing Playwright

Playwright ships in two parts: the library and the browser binaries (playwright install downloads patched builds; your system Chrome is not used by default).

Python:

pip install playwright
playwright install chromium   # or: playwright install  (all three engines)

Node.js:

npm init playwright@latest    # scaffolded, or:
npm i -D playwright && npx playwright install chromium

On Linux servers, playwright install --with-deps chromium also apt-installs the system libraries browsers need. Install only the engines you scrape with — each browser is a 100–300 MB download.

Your first scraper

Python offers sync and async APIs; the sync API is the comfortable default for scripts:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()          # headless by default
    page = browser.new_page()
    page.goto("https://example.com/products")

    for item in page.locator(".product-card").all():
        name = item.locator("h2").inner_text()
        price = item.locator(".price").inner_text()
        print(name, price)

    browser.close()

The same scraper in Node.js:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com/products');

  for (const item of await page.locator('.product-card').all()) {
    const name = await item.locator('h2').innerText();
    const price = await item.locator('.price').innerText();
    console.log(name, price);
  }

  await browser.close();
})();

For high-concurrency crawlers in Python, the async API (async_playwright) lets one process drive many pages; the method names are identical with await in front.

Locators and selectors

page.locator() is the core of modern Playwright. A locator is a lazy description of an element — it's resolved fresh every time you act on it, and every action auto-waits for the element to be ready. That's the difference from the older query_selector, which grabs a one-time element handle that can go stale.

# CSS and XPath
page.locator("div.product-card")
page.locator("//div[@class='product-card']")   # XPath inferred from // prefix

# User-facing locators — more resilient to markup changes
page.get_by_role("button", name="Next page")
page.get_by_text("Add to cart")
page.get_by_label("Search")

# Chaining and filtering
page.locator(".product-card").filter(has_text="SSD").locator(".price")

# Multiple matches
page.locator(".product-card").count()
page.locator(".product-card").nth(2)
page.locator(".product-card").all()

Extracting data from located elements:

name  = page.locator("h1").inner_text()          # rendered text
raw   = page.locator("h1").text_content()        # includes hidden text
href  = page.locator("a.next").get_attribute("href")
html  = page.locator("#reviews").inner_html()
full  = page.content()                            # entire rendered page HTML

page.content() matters for scraping: it returns the current DOM as HTML, so you can feed a fully rendered page into BeautifulSoup, lxml, or Cheerio and keep your existing parsing code.

Waiting: how Playwright handles timing

Playwright's answer to the time.sleep(5) you see in old Selenium scripts is auto-waiting: every locator action waits (up to the action timeout, 30 s by default) for the element to be attached, visible, stable, and enabled before proceeding. Most scrapers need no explicit waits at all.

When you do need to wait explicitly, wait for a condition, not a duration:

# Navigation waits for the 'load' event by default; DOM-ready is often enough
page.goto(url, wait_until="domcontentloaded")

# Wait for a specific element to appear
page.locator(".results").wait_for(state="visible")

# Wait for the site's own API call to finish
with page.expect_response(lambda r: "/api/search" in r.url) as resp:
    page.get_by_role("button", name="Search").click()
data = resp.value.json()

# Wait until network goes quiet — use sparingly; sites with analytics
# pings or long-polling may never reach networkidle
page.wait_for_load_state("networkidle")

A hard page.wait_for_timeout(3000) exists but is a last resort — it makes scrapers both slower and flakier than condition-based waits.

For infinite scroll, combine scrolling with a growth check:

previous = 0
while True:
    page.mouse.wheel(0, 20000)
    page.wait_for_timeout(800)          # brief settle between scrolls
    count = page.locator(".product-card").count()
    if count == previous:
        break
    previous = count

Headless mode

Playwright runs browsers headless by default — no window, less CPU/RAM, works on servers. During development, run headed to watch the scraper drive the page:

browser = p.chromium.launch(headless=False, slow_mo=300)  # slow_mo: ms between actions

Two useful facts: modern Chromium's headless mode is the same browser engine as headed (the old stripped-down "headless shell" is gone, so rendering differences are rare), and headless is still detectable — sites can spot HeadlessChrome in the default user agent and subtler environment signals, which is where the stealth section comes in.

Browser contexts: isolated parallel sessions

A BrowserContext is an incognito-style profile inside one browser process: separate cookies, storage, cache, and settings. Contexts are cheap (milliseconds to create) — the standard pattern for parallel scraping is one browser, many contexts:

browser = p.chromium.launch()

us_ctx = browser.new_context(
    locale="en-US",
    timezone_id="America/New_York",
    viewport={"width": 1920, "height": 1080},
)
de_ctx = browser.new_context(locale="de-DE", timezone_id="Europe/Berlin")

page_us = us_ctx.new_page()
page_de = de_ctx.new_page()   # completely isolated from page_us

Contexts are also where sessions live. Log in once, save the storage state, and reuse it across runs — no more re-running login flows:

context = browser.new_context()
page = context.new_page()
page.goto("https://example.com/login")
page.get_by_label("Email").fill("demo@example.com")
page.get_by_label("Password").fill("secret")
page.get_by_role("button", name="Log in").click()
context.storage_state(path="session.json")     # cookies + localStorage

# Later runs: start already logged in
context = browser.new_context(storage_state="session.json")

Network interception

page.route() intercepts every request the page makes. The two scraping superpowers this unlocks:

1. Block heavy resources for faster, cheaper crawls:

page.route("**/*", lambda route:
    route.abort() if route.request.resource_type in ("image", "font", "media")
    else route.continue_())

2. Capture the site's internal API instead of parsing HTML. Most dynamic sites fetch JSON and render it client-side; reading that JSON directly is faster and more robust than scraping the DOM:

def handle_response(response):
    if "/api/products" in response.url and response.ok:
        for product in response.json()["items"]:
            print(product["name"], product["price"])

page.on("response", handle_response)
page.goto("https://example.com/catalog")

You can also modify requests in flight — add headers, rewrite URLs, or return mocked responses with route.fulfill().

Using proxies

Proxies are configured at launch (for the whole browser) or per context (different IPs per session — the pattern rotating-proxy scraping needs):

# Whole browser
browser = p.chromium.launch(proxy={
    "server": "http://proxy.example.com:8080",
    "username": "proxyuser",
    "password": "proxypass",
})

# Per context — one browser, many exit IPs
context = browser.new_context(proxy={"server": "http://us-proxy.example.com:8080"})

SOCKS5 works via "server": "socks5://..." (no auth support for SOCKS in Chromium). To verify the proxy is live, hit an IP-echo endpoint before scraping. One static datacenter IP rarely survives serious scraping — residential rotation is what actually keeps large crawls alive; see our breakdown of proxy types and residential proxy providers.

User agents, fingerprints, and getting blocked

Playwright's defaults advertise automation: the user agent says HeadlessChrome, and navigator.webdriver is true. The basics are context options:

context = browser.new_context(
    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
               "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    viewport={"width": 1920, "height": 1080},
    locale="en-US",
)

Rotate user agents across contexts rather than hardcoding one forever (our user-agent rotation guide covers sane strategies). Beyond headers, anti-bot vendors fingerprint TLS, canvas, WebGL, fonts, and timing; the playwright-stealth/playwright-extra plugins patch the obvious JS-level tells, and headed mode with a real display fools more checks than headless. But understand the ceiling: against Cloudflare, DataDome, or PerimeterX, patched fingerprints are an arms race you're on the losing side of — that's a build-vs-buy decision, not a code snippet.

Screenshots and PDFs

Playwright screenshots are pixel-accurate renders of the live page — useful for archiving evidence alongside scraped data and for debugging headless runs:

page.screenshot(path="page.png")                        # viewport
page.screenshot(path="full.png", full_page=True)        # entire scrollable page
page.locator(".pricing-table").screenshot(path="el.png")  # single element
page.pdf(path="page.pdf")                               # Chromium, headless only

On failures, dump both a screenshot and page.content() — the pair answers "what did the bot actually see?" faster than any log line.

Timeouts and error handling

Playwright's defaults: 30 seconds for actions and navigation. Slow proxies need more; fast local crawls want less, so failures surface quickly:

context.set_default_timeout(15_000)             # all actions in this context
page.goto(url, timeout=60_000)                  # per call
page.locator(".price").inner_text(timeout=5_000)

Timeouts throw TimeoutError — catch it per page so one dead URL doesn't kill a crawl:

from playwright.sync_api import TimeoutError as PlaywrightTimeout

for url in urls:
    try:
        page.goto(url, wait_until="domcontentloaded")
        scrape(page)
    except PlaywrightTimeout:
        failed.append(url)

Two hygiene rules for long-running scrapers: always close pages/contexts you're done with (each leaks memory otherwise), and restart the browser process every few hundred pages — Chromium's memory only grows.

Running Playwright in Docker

The official image ships browsers and every system dependency preinstalled — matching your library version to the image tag is the one thing to get right:

FROM mcr.microsoft.com/playwright/python:v1.46.0-jammy
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt   # playwright==1.46.0 to match the image
COPY . .
CMD ["python", "scraper.py"]

(For Node.js: mcr.microsoft.com/playwright:v1.46.0-jammy.) Run with --init to reap zombie Chromium processes, and --ipc=host to avoid shared-memory crashes on Chromium. Headless-only workloads need no display server; if a site behaves differently headed, xvfb-run python scraper.py inside the container gets you a virtual display.

Playwright vs Selenium vs Puppeteer

The practical read: Selenium still wins when you must test real Safari, drive an exotic browser/grid, or work in a language Playwright doesn't support — and it powers a lot of existing scraping code (see our Python Selenium guide). Puppeteer is fine when you're all-in on Node + Chrome. For new scraping projects in 2026, Playwright is the default: same speed class as Puppeteer, more languages, more browsers, and the best waiting/locator model of the three, which directly translates into fewer flaky scrapes.

When Playwright isn't enough

Playwright solves JavaScript rendering. It does not solve the operational side of scraping at scale:

• Anti-bot walls. Cloudflare, DataDome, and CAPTCHAs detect automation fingerprints regardless of how the browser is driven. Stealth plugins help at the margins; they don't win the arms race.
• Proxy fleets. Serious crawls need residential rotation, geo-targeting, retry logic, and IP health tracking — infrastructure, not scraping code.
• Browser fleet cost. Every concurrent page is a real Chromium eating ~100+ MB of RAM. A thousand-page-per-minute crawl is a cluster engineering project.

If you'd rather keep writing extraction logic and skip that infrastructure, the same job can run through an API: WebScraping.AI renders pages in headless Chrome with rotating residential proxies and anti-bot handling behind one HTTP call:

import requests

# Fully rendered HTML — JavaScript executed, proxies handled
html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/spa-products",
    "js": "true",
}).text

# Or skip HTML parsing entirely — ask questions about the page
answer = requests.get("https://api.webscraping.ai/ai/question", params={
    "api_key": API_KEY,
    "url": "https://example.com/product/42",
    "question": "What is the price and is it in stock?",
}).text

A common hybrid: prototype the crawl in Playwright locally, then swap page.goto() + page.content() for the /html endpoint in production and keep your locator-free parsing code unchanged. See the AI web scraping overview for the extraction endpoints.

Frequently asked questions

Is Playwright good for web scraping? Yes — for JavaScript-heavy sites it's arguably the best open-source option: real browser rendering, auto-waiting locators, network interception, and multi-language support. For static HTML pages, a plain HTTP client with a parser is 10–50× cheaper; don't launch Chromium to fetch pages requests could.

Playwright or BeautifulSoup — which should I use? They're different layers, not competitors. BeautifulSoup parses HTML you already have; Playwright obtains HTML that requires a browser to produce. A common pipeline is Playwright for fetching (page.content()) and BeautifulSoup for parsing — see our Python web scraping guide.

How do I scrape iframes and shadow DOM? Iframes: page.frame_locator("#checkout-frame").locator("button") targets elements inside the frame. Shadow DOM: Playwright locators pierce open shadow roots automatically — a plain page.locator("my-widget .price") works where raw CSS querySelector fails. Closed shadow roots remain inaccessible to all tools.

Can Playwright bypass Cloudflare or CAPTCHAs? Not reliably on its own. Stealth patches and residential proxies raise the success rate; managed challenges and CAPTCHAs still stop unattended browsers. When a target sits behind serious anti-bot protection, a scraping API that handles the challenge layer is usually cheaper than fighting it — that's exactly the case WebScraping.AI exists for.

How much does Playwright cost to run? The library is free (Apache 2.0). The real cost is compute: each concurrent Chromium page uses roughly 100–400 MB of RAM plus meaningful CPU during rendering. Budget capacity by peak concurrent pages, not total page count.

Is scraping with Playwright legal? Automating a browser is legal; what matters is what you scrape and how you use it — public vs. gated data, terms of service, rate pressure, and privacy law. Our web scraping legality guide covers the current state of the case law.