Web Scraping with JavaScript and Node.js

Almost every JavaScript scraping job is one of two shapes. If the data is already in the HTML the server sends, fetch plus Cheerio extracts it in about fifteen lines and one HTTP round trip. If the data only appears after the browser runs the page's own JavaScript, you need Playwright or Puppeteer — a real browser, a slower run, and a browser binary on disk. Everything below is about telling those two cases apart in ten seconds, writing the scraper for whichever one you got, and handling the specific things that break real scrapers: redirects, shadow DOM, iframes, logins, and rate limits.

Every code sample here runs against a public sandbox site, so you can paste and execute it as-is.

Key Takeaways

• Test before you choose a tool: curl -s URL | grep 'a value you can see on the page'. If the string is there, you never need a browser.
• Node 22+ ships fetch globally. A modern Node scraper needs no HTTP client dependency at all — axios and node-fetch are optional now, not required.
• Look for the JSON before you launch Chromium. Many "JavaScript-rendered" pages call a plain JSON endpoint you can hit directly, which is faster and more stable than parsing rendered HTML.
• Browser-side JavaScript cannot scrape other sites. CORS blocks your script from reading a cross-origin response, which is why every JS scraping tutorial is really a Node.js tutorial.
• Read the status code to diagnose a block: a 403 on request #1 is a headers problem; a 403 or 429 after a burst is a rate/IP problem. Fix them in that order — headers, then delay, then proxies.
• Waiting beats sleeping. waitForSelector on the element you actually want is the difference between a scraper that works and one that intermittently returns empty arrays.
• Randomised delays and fake mouse jitter do very little against commercial anti-bot systems. Rate discipline, a coherent fingerprint, and IP quality are what actually move the needle.

Does this page need a browser?

This is the only question that matters at the start, and it takes one command to answer. Pick a value you can see on the rendered page and look for it in the raw HTML:

curl -s https://books.toscrape.com/ | grep -c "A Light in the Attic"   # 2 -> in the HTML
curl -s https://quotes.toscrape.com/js/ | grep -c '<div class="quote">' # 0 -> rendered by JS

The first page is server-rendered: the book titles are in the bytes the server sends, so Cheerio can read them. The second builds its <div class="quote"> elements in the browser, so the raw HTML has none. Same test in Node, if you prefer to stay in one language:

const html = await fetch('https://quotes.toscrape.com/js/').then(r => r.text());
console.log(html.includes('Albert Einstein')); // true — but not inside a .quote element

Note the trap in that second result. The quote data is in the source, sitting inside a <script> tag as a JavaScript array — it just isn't in the DOM yet. That's a very common pattern and it means you can often skip the browser anyway. More on that below.

The browser DevTools version of the same check: open the Network tab, reload, click the document request, and read the Response tab (not Elements — Elements shows the DOM after JavaScript has run, which will mislead you every time).

Setting up a Node.js scraping project

You need Node 22 or 24 (both LTS lines in 2026). Node 18 is end-of-life; avoid it.

node -v            # v22.x or v24.x
mkdir book-scraper && cd book-scraper
npm init -y
npm pkg set type=module
npm install cheerio

npm pkg set type=module enables import syntax and top-level await, which every example here uses. That's the whole setup for static scraping — no HTTP client needed, because fetch is built into the runtime.

Scraping static HTML with fetch and Cheerio

Cheerio parses an HTML string and gives you a jQuery-style API over it. It doesn't run scripts, doesn't render layout, and doesn't download images — it's a parser, which is exactly why it's fast.

import * as cheerio from 'cheerio';

const res = await fetch('https://books.toscrape.com/', {
  headers: {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' +
      'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
    'Accept-Language': 'en-US,en;q=0.9',
  },
});

// fetch does NOT throw on 4xx/5xx — you have to check
if (!res.ok) throw new Error(`HTTP ${res.status} for ${res.url}`);

const $ = cheerio.load(await res.text());

const books = $('article.product_pod').map((_, el) => ({
  title: $(el).find('h3 a').attr('title'),
  price: $(el).find('.price_color').text().trim(),
  inStock: $(el).find('.instock.availability').text().trim() === 'In stock',
  url: new URL($(el).find('h3 a').attr('href'), res.url).href,
})).get();

console.log(books.length, books[0]);
// 20 {
//   title: 'A Light in the Attic',
//   price: '£51.77',
//   inStock: true,
//   url: 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html'
// }

Three details in there that break most first-attempt scrapers:

• res.ok. Unlike axios, fetch resolves successfully on a 404 or 403. Without that check you'll parse an error page and get a silent empty array.
• .map(...).get(). Cheerio's .map() returns a Cheerio object, not an array. .get() unwraps it. Forgetting .get() is the single most common Cheerio bug.
• new URL(href, res.url). Scraped links are usually relative. Resolving against res.url (not the URL you requested) also survives redirects — see the next section for why that distinction matters.

Following pagination

async function scrapeAllPages(startUrl) {
  const all = [];
  let url = startUrl;

  while (url) {
    const res = await fetch(url);
    if (!res.ok) throw new Error(`HTTP ${res.status} for ${url}`);
    const $ = cheerio.load(await res.text());

    $('article.product_pod').each((_, el) => {
      all.push({
        title: $(el).find('h3 a').attr('title'),
        price: $(el).find('.price_color').text().trim(),
      });
    });

    const next = $('li.next a').attr('href');
    url = next ? new URL(next, res.url).href : null;

    await new Promise(r => setTimeout(r, 500)); // stay polite
  }

  return all;
}

const books = await scrapeAllPages('https://books.toscrape.com/');
console.log(books.length); // 1000

Sequential with a delay is the right default. Parallelise only after the scraper works, and cap concurrency when you do — see the concurrency section below.

Handling redirects

Redirects are where scrapers silently start collecting the wrong data — you asked for a product page, got bounced to a country selector, and your selectors quietly return nothing. fetch follows redirects by default and tells you what happened:

const res = await fetch('https://httpbin.org/redirect/2');

console.log(res.redirected); // true — at least one hop happened
console.log(res.url);        // final URL after all hops
console.log(res.status);     // 200 — the status of the FINAL response

res.url is the value you want for resolving relative links and for deciding whether you ended up somewhere useful. The original URL you passed to fetch is gone from the response object, so keep it yourself if you need to log the pair.

Inspecting the chain hop by hop

Node's fetch behaves differently from a browser's here, and it's the difference that makes redirect debugging possible at all. In a browser, redirect: 'manual' gives you an opaque response with no status and no headers. In Node, it hands you the real 3xx response with a readable Location header:

async function traceRedirects(startUrl, maxHops = 10) {
  const chain = [];
  let url = startUrl;

  for (let i = 0; i < maxHops; i++) {
    const res = await fetch(url, { redirect: 'manual' });

    if (res.status < 300 || res.status > 399) {
      return { chain, finalUrl: url, status: res.status, res };
    }

    const location = res.headers.get('location');
    if (!location) throw new Error(`${res.status} with no Location header at ${url}`);

    const next = new URL(location, url).href; // Location may be relative
    if (chain.some(hop => hop.to === next)) throw new Error('Redirect loop');

    chain.push({ from: url, to: next, status: res.status });
    url = next;
  }

  throw new Error(`More than ${maxHops} redirects`);
}

const { chain, finalUrl } = await traceRedirects('https://httpbin.org/redirect/3');
console.log(chain.map(h => `${h.status} ${h.from} -> ${h.to}`).join('\n'));
console.log('Landed on:', finalUrl);

Four things worth knowing before you rely on automatic following:

• 307 and 308 preserve the method and body; 301 and 302 historically don't. A redirected POST may arrive as a GET with your body dropped. If you're scraping behind a form submission, check which code you got.
• Node strips Authorization and Cookie headers on a cross-origin redirect. This follows the fetch spec and it is a security feature, not a bug — but it means a scraper that authenticates to example.com and gets bounced to cdn.example.net arrives unauthenticated. Handle those hops manually if you need the credentials to survive.
• redirect: 'error' rejects the promise on any 3xx, which is the right setting when a redirect means your assumptions are wrong and you'd rather fail loudly.
• Meta refresh and JavaScript redirects are not HTTP redirects. <meta http-equiv="refresh"> and location.href = ... produce a 200 with a nearly empty body, and fetch will happily hand you that empty page. If res.redirected is false but the content is missing, grep the HTML for http-equiv="refresh" before blaming your selectors.

Browser automation follows all of these transparently, including the JavaScript ones, which is sometimes reason enough to use it.

The data isn't in the HTML — find the JSON API first

Before installing a browser, spend two minutes in the Network tab filtering by Fetch/XHR. Pages that render client-side almost always fetch their data from an endpoint that returns clean JSON, and calling it directly is faster, more stable, and immune to CSS changes.

The sandbox has a live example. https://quotes.toscrape.com/scroll is an infinite-scroll page; the endpoint behind it is:

curl -s 'https://quotes.toscrape.com/api/quotes?page=1'

const res = await fetch('https://quotes.toscrape.com/api/quotes?page=1');
const { quotes, has_next } = await res.json();

console.log(quotes.length, has_next);          // 10 true
console.log(quotes[0].text, '—', quotes[0].author.name);
// "The world as we have created it is a process of our thinking..." — Albert Einstein

No browser, no selectors, structured data with the author already split out. When you find one of these, stop looking for a scraping library and just write a loop over page.

The second shortcut is the inline-script case shown earlier: quotes.toscrape.com/js embeds its data as var data = [...] inside a <script> tag. You can pull that out with Cheerio and a regex rather than booting Chromium. Look for __NEXT_DATA__, __NUXT__, or window.__INITIAL_STATE__ — those cover a large share of modern React, Next.js, and Nuxt sites.

Scraping JavaScript-rendered pages with Playwright

When the data genuinely only exists after scripts run, you need a real browser. Playwright is the better default in 2026: it bundles Chromium, Firefox, and WebKit, has auto-waiting built into its locator API, and installs its browsers with one command.

npm install playwright
npx playwright install chromium

import { chromium } from 'playwright';

const browser = await chromium.launch();          // headless by default
const page = await browser.newPage();

await page.goto('https://quotes.toscrape.com/js/', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('div.quote');          // wait for the DATA, not a timer

const quotes = await page.$$eval('div.quote', els =>
  els.map(el => ({
    text: el.querySelector('span.text').textContent,
    author: el.querySelector('small.author').textContent,
  }))
);

console.log(quotes.length, quotes[0]);
// 10 { text: '"The world as we have created it..."', author: 'Albert Einstein' }

await browser.close();

The pattern that matters: waitUntil: 'domcontentloaded' followed by waitForSelector on the element you actually want. Waiting for networkidle is slow and unreliable on pages with polling or analytics beacons, and a fixed setTimeout is a race condition you'll rediscover in production at 3am.

Puppeteer is nearly identical if you already use it — import puppeteer from 'puppeteer', puppeteer.launch(), same goto/waitForSelector/$$eval calls. It drives Chrome only. Since v22 headless: true means modern headless Chrome, so the old headless: 'new' value is no longer needed. For the operational side of running either one — memory, flags, crash loops — see the headless browser guide.

For crawls spanning thousands of URLs, don't hand-roll the queue. Crawlee wraps Cheerio, Playwright, and Puppeteer behind one interface with a request queue, retries, and concurrency limits already built.

Scraping shadow DOM content

Shadow DOM lets a component keep its own isolated DOM tree. The practical consequence for scraping is that document.querySelector('.price') returns null even though you can see the price on screen, because the element lives inside a shadow root attached to some custom element. In DevTools you'll see a #shadow-root node above the content you want.

Check the served HTML first. Declarative shadow DOM ships the content in the response body inside a <template shadowrootmode="open"> element, and Cheerio can read that without a browser:

const $ = cheerio.load(html);
// Declarative shadow roots are ordinary <template> elements in the source
$('template[shadowrootmode]').each((_, tpl) => {
  console.log($(tpl).find('.price').text());
});

If the shadow root is built at runtime by JavaScript, you need a browser. Playwright pierces open shadow roots automatically — its CSS engine crosses shadow boundaries, so you write the selector as if the encapsulation weren't there:

// No special syntax needed: this crosses into the shadow root
const price = await page.locator('product-card .price').textContent();
await page.locator('product-card button.add-to-cart').click();

The >> piercing syntax you'll find in older tutorials is legacy and no longer required. Note that Playwright's CSS piercing does not cross into <iframe> elements — those are a separate document and need frameLocator, covered next.

Puppeteer needs the explicit pierce/ query handler, or a manual traversal:

// Puppeteer's piercing query handler
const text = await page.$eval('pierce/product-card .price', el => el.textContent);

// Or traverse yourself — also works for finding what's in there at all
const found = await page.evaluate(() => {
  function search(root, selector) {
    const hit = root.querySelector(selector);
    if (hit) return hit.textContent;
    for (const el of root.querySelectorAll('*')) {
      if (el.shadowRoot) {
        const nested = search(el.shadowRoot, selector);
        if (nested) return nested;
      }
    }
    return null;
  }
  return search(document, '.price');
});

Closed shadow roots are a genuine dead end. When a component calls attachShadow({ mode: 'closed' }), element.shadowRoot is null for everyone — your code, Playwright, and Puppeteer alike. There is no supported selector that reaches inside. The productive move is to stop fighting the DOM and go find the data somewhere else: the XHR response that populated the component, a data-* attribute on the host element, or a JSON blob in a script tag. That is usually cleaner than the shadow DOM approach would have been anyway.

Extracting data from iframes

An <iframe> is a whole separate document with its own URL. The parent page's HTML contains only the <iframe src="..."> tag — none of the content — which is why Cheerio returns nothing and why page.locator() on the parent finds nothing either.

The cheapest fix is usually to skip the parent entirely. Read the src, request that URL directly, and parse it like any other page:

const $ = cheerio.load(parentHtml);
const frameUrl = new URL($('iframe#content').attr('src'), pageUrl).href;
const frameHtml = await fetch(frameUrl).then(r => r.text());
const $frame = cheerio.load(frameHtml);

This works far more often than people expect, and it turns a browser job back into an HTTP job. It fails when the iframe requires the parent's cookies or a Referer, in which case pass those headers along.

When you do need the browser, Playwright's frameLocator is the modern API:

const frame = page.frameLocator('iframe#checkout');
await frame.locator('#card-number').fill('4242424242424242');
const total = await frame.locator('.order-total').textContent();

Puppeteer goes through the element handle, or matches on the frame's URL:

const handle = await page.$('iframe#checkout');
const frame = await handle.contentFrame();
const total = await frame.$eval('.order-total', el => el.textContent);

// Or find it among all frames
const byUrl = page.frames().find(f => f.url().includes('/checkout'));

Two notes. First, the same-origin policy that blocks in-page JavaScript from reading a cross-origin iframe does not apply to Playwright or Puppeteer — they drive the browser from outside, so cross-origin frames are readable like any other. The postMessage handshakes described in older guides solve a different problem (communicating between two pages you control) and aren't relevant to scraping. Second, ad and consent iframes are frequently nested several levels deep and load late, so wait on a selector inside the frame rather than on the <iframe> tag itself.

Sessions, cookies, and logging in

Log in once, save the session, and reuse it. Re-authenticating on every run is slow, and it's the fastest way to get an account flagged.

Playwright's storageState captures cookies and localStorage, which matters because most modern apps keep their auth token in local storage rather than a cookie:

import { chromium } from 'playwright';

// First run: authenticate and persist the session
const browser = await chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();

await page.goto('https://example.com/login');
await page.fill('#email', process.env.SCRAPE_USER);
await page.fill('#password', process.env.SCRAPE_PASS);
await page.click('button[type=submit]');
await page.waitForURL('**/dashboard');

await context.storageState({ path: 'session.json' });
await browser.close();

// Later runs: start already logged in
const context = await browser.newContext({ storageState: 'session.json' });

Keep credentials in environment variables or a secrets manager, never in the source. And check that the session is still valid at the start of each run rather than assuming — a scrape that silently collects the logged-out version of every page is worse than one that crashes.

Handing the session off to plain HTTP

Once you have cookies, you rarely need the browser any more. Node's fetch has no cookie jar — it will not store or resend Set-Cookie for you — so build the header yourself and enjoy requests that are an order of magnitude cheaper than a browser page load:

const cookies = await context.cookies();
const cookieHeader = cookies.map(c => `${c.name}=${c.value}`).join('; ');

const res = await fetch('https://example.com/api/orders?page=1', {
  headers: { Cookie: cookieHeader },
});
const orders = await res.json();

This browser-for-auth, HTTP-for-volume split is the single biggest performance win available on authenticated scrapes.

Two-factor authentication, honestly

There is no technique that produces a valid second factor you don't have. Anything framed as "bypassing 2FA" is either account compromise or wishful thinking. What you actually have are three legitimate options, in the order you should try them:

1. Use the supported credential. Check whether the service offers an API with a personal access token, an app-specific password, or an OAuth client. This exists far more often than people check, it's stable across UI redesigns, and it's the only option that doesn't break the moment the login flow changes. Do this first.

2. Hand off the second factor to a human, once. Run headed, let a person complete the challenge, then save the session and run headless from there. This is the realistic answer for most one-off and low-frequency scrapes:

const browser = await chromium.launch({ headless: false });
const context = await browser.newContext();
const page = await context.newPage();

await page.goto('https://example.com/login');
await page.fill('#email', process.env.SCRAPE_USER);
await page.fill('#password', process.env.SCRAPE_PASS);
await page.click('button[type=submit]');

// Human completes the 2FA challenge in the visible browser.
// Wait on the post-login state, generously.
console.log('Complete the 2FA prompt in the browser window...');
await page.waitForURL('**/dashboard', { timeout: 5 * 60 * 1000 });

await context.storageState({ path: 'session.json' });

Tick a "remember this device" box if the site offers one — that's the site telling you how long the handoff will last. Schedule the re-authentication rather than discovering it through a failed run.

3. Generate TOTP codes yourself — only for accounts you own and control. If you enrolled the authenticator and hold the base32 seed, you can compute the same six digits your phone would:

import { authenticator } from 'otplib';

const code = authenticator.generate(process.env.TOTP_SECRET);
await page.fill('#totp-code', code);
await page.click('#verify');

Be clear-eyed about the tradeoff: storing the TOTP seed next to the password puts both factors in one place, which is most of the security 2FA was giving you. Do it only for a service account created for this purpose, keep the seed in a secrets manager rather than an .env file that gets committed, and don't do it at all for an account that also controls anything else you care about.

SMS codes and push approvals have no legitimate automation path — intercepting them means compromising the phone or the carrier account. If a target only supports those, option 2 is your ceiling.

All of this applies to accounts you are authorized to automate. Scraping behind a login is where terms-of-service and legal exposure concentrate, and "I had the password" is not the same as "I was permitted to automate this" — see our guide to web scraping legality.

Mimicking human behavior — what actually works

There's a large genre of tutorial promising that Bezier-curve mouse paths, simulated typos, and randomised scroll patterns will get you past bot detection. Set expectations honestly: against commercial anti-bot systems, they mostly don't.

Cloudflare, DataDome, and PerimeterX score you on TLS fingerprint, canvas and WebGL rendering, HTTP/2 frame ordering, IP reputation, and account history — signals your mouse path never touches. Headless Chrome loses several of those before your script executes a single line. Patching navigator.webdriver and shipping curved mouse movements is an arms race against companies with more engineers pointed at it than you have.

What genuinely reduces friction, roughly in order of impact:

1. Send a coherent fingerprint. Node's fetch sends the literal User-Agent: node, which is the most common reason a first request returns 403. Set a real browser User-Agent and Accept-Language — and keep them consistent with each other. A Windows Chrome user agent paired with a Linux TLS signature and a de-DE language header is more suspicious than the honest default. See user agent rotation for doing this at volume.
2. Slow down. Request rate and burst shape are the signals most under your control and the ones most sites actually act on. A scraper that takes six hours instead of six minutes is often the entire fix.
3. Keep sessions coherent. Reuse cookies across a session instead of arriving as a brand-new visitor every request, and don't pair one cookie jar with a rotating IP — that combination is itself a tell.
4. Improve IP quality only when the above is exhausted. Datacenter first, residential when datacenter keeps failing.

Where human-like timing does earn its keep is interaction-gated content — a search box you must type into, a dropdown that fires an XHR on change. There, the point isn't deception, it's that firing events faster than the page's own JavaScript can respond produces empty results. Playwright's pressSequentially handles this without any of the ceremony:

await page.locator('#search').pressSequentially('wireless headphones', { delay: 80 });
await page.waitForResponse(r => r.url().includes('/api/search'));

And a modest random delay between page loads is worth having, simply because a request exactly every 1000ms is a pattern nothing organic produces:

const jitter = (min, max) => new Promise(r => setTimeout(r, min + Math.random() * (max - min)));
await jitter(800, 2500);

If your target sits behind a serious anti-bot product, the realistic choices are a well-maintained residential proxy pool plus continuous fingerprinting work, or an API that owns that problem. Grinding on mouse curves is the option that feels productive and isn't.

Rate limiting and concurrency

Uncapped Promise.all over a thousand URLs is a self-inflicted denial of service — against the target and often against your own machine. Cap it with p-limit:

npm install p-limit

import pLimit from 'p-limit';

const limit = pLimit(5);            // at most 5 in flight

const results = await Promise.all(
  urls.map(url => limit(async () => {
    const res = await fetchWithRetry(url);
    return parse(await res.text());
  }))
);

Five concurrent requests is a sane starting point for a site you don't own. Raise it deliberately while watching for 429s, not optimistically on the first run.

Retries need backoff and jitter, and need to distinguish between "try again" and "this will never work":

async function fetchWithRetry(url, options = {}, attempts = 4) {
  for (let i = 0; i < attempts; i++) {
    const res = await fetch(url, options);

    if (res.ok) return res;
    if (res.status === 404 || res.status === 410) {
      throw new Error(`${res.status} — gone, not retrying: ${url}`);
    }

    if (res.status === 429 || res.status >= 500) {
      const retryAfter = Number(res.headers.get('retry-after'));
      const wait = Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter * 1000
        : 2 ** i * 1000 + Math.random() * 1000;   // backoff + jitter
      await new Promise(r => setTimeout(r, wait));
      continue;
    }

    throw new Error(`HTTP ${res.status} for ${url}`);
  }
  throw new Error(`Gave up after ${attempts} attempts: ${url}`);
}

Honouring Retry-After matters: it's the server telling you exactly how long to wait, and ignoring it is what turns a temporary throttle into a ban. The jitter matters too — without it, a batch of failures all retry in lockstep and hit the server as a second synchronised burst.

Storing what you scrape

Don't accumulate everything in an array and write once at the end. A crash at item 9,000 loses all of it, and large runs will exhaust memory.

JSON Lines is the right default for a scrape in progress — one JSON object per line, appendable, streamable, and readable even if the process dies mid-write:

import { createWriteStream } from 'node:fs';

const out = createWriteStream('books.jsonl', { flags: 'a' });
for (const book of books) out.write(JSON.stringify(book) + '\n');
out.end();

SQLite is worth it as soon as you need deduplication, resumability, or queries. better-sqlite3 is the practical choice — synchronous, fast, and simple. (Node 22 ships an experimental built-in node:sqlite; useful to know about, not yet what I'd build a production scraper on.)

import Database from 'better-sqlite3';

const db = new Database('books.db');
db.exec(`CREATE TABLE IF NOT EXISTS books (
  url TEXT PRIMARY KEY, title TEXT, price TEXT, scraped_at TEXT
)`);

const insert = db.prepare(`INSERT OR REPLACE INTO books
  VALUES (@url, @title, @price, datetime('now'))`);
const insertMany = db.transaction(rows => rows.forEach(r => insert.run(r)));

insertMany(books);

A PRIMARY KEY on the URL gives you idempotent re-runs for free: interrupt the scrape, restart it, and already-fetched rows are replaced rather than duplicated.

CSV only for handoff to a spreadsheet, and use a real serialiser rather than joining on commas — scraped text contains commas, quotes, and newlines, and hand-rolled CSV corrupts on all three.

What about PDFs?

If the data you need is behind a PDF link, that's a separate problem from HTML scraping. In Node, pdfjs-dist (Mozilla's library, the same engine Firefox uses) extracts text and metadata per page and handles the widest range of files. Expect layout to be the hard part rather than extraction: PDFs store positioned text runs, not paragraphs or table cells, so reconstructing a table means grouping items by their coordinates. Scanned PDFs contain no text at all and need OCR.

Which JavaScript scraping tool should you use?

For a deeper feature-by-feature comparison of the libraries themselves, see our breakdown of JavaScript web scraping libraries.

Can you scrape a website with browser-side JavaScript?

Not a third-party site, no. If you run fetch('https://example.com') from a script on your own page, the browser sends the request but refuses to let your code read the response unless example.com returns an Access-Control-Allow-Origin header permitting your origin. That's the same-origin policy, and it exists specifically to stop what you're trying to do.

The workarounds, in order of practicality:

1. Run in Node.js. No same-origin policy server-side. This is what everyone means by "web scraping with JavaScript".
2. Build a browser extension. Extensions with host permissions can make cross-origin requests, which is why scraper extensions exist.
3. Call a scraping API from your front end — a server fetches the page and returns the content to you with permissive CORS headers.
4. Use the DevTools console for one-offs. Code you paste into the console on a page runs as that origin, so document.querySelectorAll(...) works fine for scraping the page you're currently looking at. It just can't reach other domains.

How to avoid getting blocked

Blocks are diagnosable. Work through them in this order rather than immediately reaching for proxies:

1. Send real headers. A browser User-Agent and Accept-Language fix a surprising share of first-request 403s.
2. Slow down before you speed up. Get the scraper working sequentially with a delay, then raise concurrency deliberately with p-limit.
3. Read the status code. 403 on the very first request means your fingerprint is wrong (headers, TLS, or a headless-browser tell). 403 or 429 only after a burst means rate limiting or IP reputation. Honour Retry-After when it's present.
4. Retry selectively. Exponential backoff on 429 and 5xx; never retry a 404 or 410.
5. Escalate proxies only when needed. Datacenter IPs are cheap and fine for most targets; move to residential only when datacenter traffic keeps getting 403s, because residential costs several times more per request everywhere it's sold. The types of proxies guide covers the trade-offs.
6. Check robots.txt and the site's terms before you scale a crawl, not after.

Scraping JS-rendered pages without running browsers

Running headless Chrome in production is its own infrastructure project: memory ceilings, zombie processes, browser updates, and proxy rotation. If you'd rather not own that, WebScraping.AI does the rendering server-side and hands you HTML you can parse with the same Cheerio code you already wrote.

npm install webscraping-ai cheerio

import { WebScrapingAI } from 'webscraping-ai';
import * as cheerio from 'cheerio';

const client = new WebScrapingAI({ apiKey: process.env.WEBSCRAPING_AI_API_KEY });

// The same JS-rendered page from earlier — no Chromium on your machine
const html = await client.html({
  url: 'https://quotes.toscrape.com/js/',
  js: true,               // render with a headless browser (default)
  wait_for: 'div.quote',  // same waiting logic, server-side
  proxy: 'datacenter',    // 'residential' or 'stealth' if this gets blocked
});

const $ = cheerio.load(html);
console.log($('div.quote span.text').first().text());
// "The world as we have created it is a process of our thinking..."

Because the rendering happens in a real browser, shadow DOM and iframe content are already in the HTML you get back. And the headers parameter takes the session cookies from the login flow above, so an authenticated scrape doesn't have to run its own browser fleet:

const html = await client.html({
  url: 'https://example.com/dashboard',
  headers: { Cookie: cookieHeader },
});

If you'd rather skip selectors entirely, the AI extraction endpoints take a description of each field instead of a CSS path, which survives markup changes that would break a selector-based scraper:

const result = await client.fields({
  url: 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html',
  fields: {
    title: 'Book title',
    price: 'Price including the currency symbol',
    availability: 'Number of copies in stock',
  },
});

console.log(result); // { title: 'A Light in the Attic', price: '£51.77', ... }

Pricing is per credit and published up front: a datacenter request is 1 credit without JS rendering and 5 with it, residential is 10 and 25, stealth is 50, and AI extraction adds 5. Failed requests aren't charged. The free tier is 2,000 credits a month with no credit card, which is enough to test a real scraper end to end. Full parameter reference is in the API docs.

Is JavaScript or Python better for web scraping?

Neither wins outright, and the honest answer is "the one your project is already written in."

• JavaScript/Node is ahead on browser automation. Playwright and Puppeteer are first-class Node libraries, and if you're scraping a heavy SPA you're writing JavaScript in the page anyway ($$eval, page.evaluate), so there's no language switch.
• Python has the deeper parsing and data ecosystem: BeautifulSoup, lxml, Scrapy, and a direct path into pandas for analysis. If the scrape feeds a data pipeline, that matters. See our Python scraping guide for the equivalent workflow.
• Both have official Playwright bindings, so the browser half is a wash.

Pick by where the output goes, not by benchmark folklore.

Is web scraping with JavaScript legal?

The language you use has no bearing on legality. What matters is what you collect, from where, and what you do with it: public versus authenticated pages, personal data and the privacy law covering it, the site's terms of service, and copyright in the content itself. Rules differ by jurisdiction and the case law keeps moving. We wrote up the current picture in is web scraping legal — read that before scraping anything commercially sensitive, and don't treat any blog post (including ours) as legal advice.

Frequently asked questions

Can JavaScript do web scraping? Yes, in Node.js. fetch is built into the runtime and Cheerio parses the HTML, so a working scraper is about fifteen lines with one dependency. Browser-side JavaScript can only scrape the page it's already running on — the same-origin policy blocks it from reading any other site's response.

Do I still need axios or node-fetch? No. Node 22 and 24 ship a global fetch, and it covers everything a scraper needs including redirect control and streaming. Axios is still pleasant if you want interceptors or automatic JSON parsing, but it's a preference now, not a requirement. Remember that fetch doesn't throw on 4xx or 5xx — check res.ok yourself.

How do I get the final URL after redirects? res.url on the response. res.redirected tells you whether any hop happened. To see the individual hops, request with redirect: 'manual' and read the Location header — in Node that header is readable, unlike in a browser where a manual redirect response is opaque.

Why does querySelector return null for an element I can see? Three usual causes. The element is inside a shadow root, in which case use Playwright locators (they pierce open shadow roots) or Puppeteer's pierce/ prefix. It's inside an <iframe>, which needs frameLocator or contentFrame(). Or it hasn't rendered yet, which needs waitForSelector rather than a timer.

Can I scrape content inside a closed shadow root? Not through the DOM. attachShadow({ mode: 'closed' }) makes shadowRoot null for every caller, including Playwright and Puppeteer. Find the data in the network response that populated the component, or in a data-* attribute on the host element, instead.

How do I scrape a site that requires login? Log in once with Playwright, save the session with context.storageState({ path: 'session.json' }), and start later runs from that file. Then pull the cookies out and use plain fetch for the bulk requests — it's dramatically faster than driving a browser per page. Node's fetch has no cookie jar, so build the Cookie header yourself.

Can a scraper get past two-factor authentication? Not in any sense that means producing a factor you don't hold. Use the site's API token or app password if one exists; otherwise run headed, have a person complete the challenge once, and save the session. Automating TOTP with otplib is legitimate only for an account you own and enrolled — and storing the seed beside the password puts both factors in one place, which is most of what 2FA was protecting.

Do random delays and fake mouse movements defeat bot detection? Rarely. Commercial anti-bot systems score TLS fingerprints, canvas and WebGL output, and IP reputation, none of which your mouse path affects. Coherent headers, a slower request rate, stable sessions, and better IPs are what actually help. Human-like typing is still worth it for interaction-gated content, where the point is letting the page's own JavaScript keep up rather than fooling anyone.

How many requests can I run in parallel? Start at about five concurrent requests against a site you don't own, using p-limit, and raise it only while watching for 429s. Retry 429 and 5xx with exponential backoff plus jitter, honour Retry-After when it's present, and never retry a 404.

Where to go next

• Parsing HTML in depth: Cheerio guide
• Comparing libraries: JavaScript web scraping libraries
• Browser automation specifics: Puppeteer and Playwright
• Running browsers in production: headless browser guide
• Debugging a request before you write code: curl commands for web scraping
• A worked business case: price monitoring

If the scraper you're building keeps hitting 403s or drowning in Chromium processes, try WebScraping.AI free — 2,000 credits a month, no card, and the same Cheerio code on the other side.