n8n Web Scraping Guide: HTTP Requests, HTML Extraction, and Browser Automation

n8n is a workflow automation tool with a serious following among people who scrape: it has an HTTP client, an HTML parser, schedulers, loops, and connectors to every place scraped data ends up — Google Sheets, databases, Slack, S3 — all wired together visually. You can build a real scraper in n8n without writing a line of code, and drop into JavaScript exactly where you need to. This guide covers the full path: fetching pages, extracting data with CSS selectors, parsing JSON, handling pagination, scheduling, dealing with JavaScript-heavy sites, and exporting the results.

Key Takeaways

• The core static-site pattern is two nodes: HTTP Request fetches the page, HTML extracts values with CSS selectors (it runs cheerio under the hood).
• The HTML node speaks CSS selectors only — no XPath. Convert XPath expressions to CSS or handle exotic cases in a Code node.
• The HTTP Request node has built-in pagination; Loop Over Items batches multi-URL scrapes; Schedule Trigger turns any workflow into a recurring scraper.
• Headless browsers (Puppeteer/Playwright) are available as community nodes on self-hosted n8n only. On n8n Cloud, the practical route for JavaScript-rendered sites is a scraping API called from the HTTP Request node or the official WebScraping.AI community node.
• Export is where n8n shines: append rows to Google Sheets, convert to CSV/XLSX files, or write to any database without custom code.

The core pattern: HTTP Request + HTML node

Every static-site scraper in n8n is a variation of this workflow:

Schedule Trigger → HTTP Request → HTML → (transform) → Google Sheets

Step 1 — fetch the page. Add an HTTP Request node: method GET, the target URL, and set Response Format to Text (you want the raw HTML string, not n8n trying to parse it). Two options worth setting on real targets, under Options:

• Headers — send a realistic User-Agent; the default Node.js user agent is an instant block on many sites.
• Timeout — the default is generous; 10–15 seconds keeps a stuck target from stalling the whole workflow.

Step 2 — extract the data. Add an HTML node with operation Extract HTML Content. Point Source Data at the HTTP Request's output property (usually data), then define extraction values:

To scrape a list (search results, product grids), first extract the repeating container with Return Value: HTML and Return Array: on (e.g. selector div.product-card), then chain a second HTML node that runs per item and pulls out title, price, and url (with Attribute: href) from each fragment. Two HTML nodes in a row — container first, fields second — is the idiomatic n8n list-scraping pattern.

Step 3 — clean up. An Edit Fields (Set) node renames and trims values; expressions handle light cleanup inline:

{{ $json.price.replace(/[^0-9.]/g, "") }}

CSS selectors, XPath, and the Code node

The HTML node is cheerio, so it accepts standard CSS: descendants (.list article), attributes (a[href^="/product/"]), pseudo-classes (tr:nth-child(odd)). Our CSS selectors FAQ covers the syntax in depth.

What it does not accept is XPath. If you have existing XPath expressions, most convert mechanically (//div[@class="price"] → div.price); for the few that don't (text-content matching, parent axes), do the work in a Code node instead.

The Code node runs JavaScript per-item or once for all items. On self-hosted n8n you can allow external modules and use cheerio directly:

# in the n8n environment
NODE_FUNCTION_ALLOW_EXTERNAL=cheerio

// Code node, "Run Once for Each Item"
const cheerio = require('cheerio');
const $ = cheerio.load($input.item.json.data);

return {
  json: {
    title: $('h1').text().trim(),
    // things the HTML node can't express, e.g. filter by text content:
    inStock: $('.availability').filter((i, el) => $(el).text().includes('In stock')).length > 0,
  }
};

On n8n Cloud external modules aren't available in the Code node — use the HTML node for parsing and keep Code nodes for pure data reshaping.

Parsing JSON responses and embedded JSON

When the target is an API (or a site's own XHR endpoint — often the best scraping target of all, found via DevTools → Network), set the HTTP Request node's response format to JSON and the response is parsed automatically. Downstream nodes address it with expressions:

{{ $json.products[0].price }}

Two scraping-specific JSON tricks:

JSON-LD structured data. Many sites embed clean product/article data in <script type="application/ld+json"> — extract it with an HTML node (selector script[type="application/ld+json"], Return Value Text) and parse in a Code node:

const data = JSON.parse($input.item.json.jsonld);
return { json: { name: data.name, price: data.offers?.price } };

JSON buried in strings. JSON.parse() in a Code node handles API responses that double-encode payloads, and {{ $json.field.parseJson() }} works inline in expressions.

Pagination and multi-page scraping

Built-in pagination (simplest). The HTTP Request node has a Pagination option: it can increment a page parameter or follow a next URL from the response until a condition is met — no loop wiring needed. Use it when the site paginates predictably (?page=1, ?page=2, …) and set Max Pages so a selector change can't send you into an infinite crawl.

Loop Over Items. When you have a list of URLs (from a category page scrape, a sitemap, or a spreadsheet), the Loop Over Items (Split in Batches) node feeds them through the fetch-and-extract chain in controlled batches. Put a Wait node inside the loop — one to a few seconds between batches keeps your request rate polite and your scraper unblocked.

Crawl patterns. For "scrape the list page, then visit each detail page": list-page HTTP Request → HTML node extracting hrefs (Return Array on) → Loop Over Items → detail-page HTTP Request → HTML node. State that must survive between runs (last seen page, cursor) belongs in getWorkflowStaticData('global').

Scheduling and reliability

The Schedule Trigger node runs the workflow on an interval or full cron expression (0 6 * * * for daily at 06:00). Things that separate a scraper that runs for months from one that dies quietly:

• Deduplicate before writing: the Remove Duplicates node with "across executions" scope skips items you've already seen, so re-runs don't produce duplicate rows.
• Error workflow: set one in the workflow settings — it fires on failure and can alert you in Slack/email with the error and the failing item.
• Retry on fail: enable it on the HTTP Request node (2–3 tries with a wait) so one transient 502 doesn't kill the run.
• Timeout the workflow in settings, so a hung request doesn't hold your execution queue.

Scraping JavaScript-rendered sites

The HTTP Request node fetches HTML as served — if the content appears only after JavaScript runs (React/Vue apps, infinite scroll), the extraction comes back empty. Check first whether the data exists in the page source or an XHR endpoint (very often it does, and that's the cheaper path). When it genuinely needs a browser, you have two routes:

Community browser nodes (self-hosted only). n8n-nodes-puppeteer and Playwright equivalents give you a headless Chromium inside the workflow: navigate, wait for selectors, click, screenshot. The trade-offs are real — the n8n host needs Chrome dependencies and enough RAM, community nodes with native dependencies don't run on n8n Cloud, and you own the anti-bot arms race (fingerprints, proxies, CAPTCHAs). Between the two, Playwright is the more modern API (auto-waiting, cross-browser); Puppeteer's node is older and more widely deployed. For heavier browser work it's usually cleaner to run Playwright as a separate service n8n calls via webhook.

A scraping API (works everywhere, including Cloud). Rendering happens on the API's infrastructure — n8n just makes an HTTP call and gets back the final HTML:

GET https://api.webscraping.ai/html?api_key=YOUR_KEY&url=https://spa-site.com&js=true&proxy=residential

This is one HTTP Request node, no Chrome on your server, and the proxy/anti-bot problem is handled on the API side.

Proxies in n8n

For scraping at any volume from a fixed n8n server, requests need to come from more than one IP. The HTTP Request node accepts a proxy under Options → Proxy (http://user:pass@proxy-host:port), and self-hosted instances can set HTTP_PROXY/HTTPS_PROXY environment variables to route everything. That covers having a proxy — rotation, residential IPs, and retry-on-block are still yours to build. The alternative is pushing proxy selection to the API layer: WebScraping.AI rotates datacenter, residential, or stealth IPs per request with a proxy= parameter, which reduces the n8n side back to a single node.

Exporting scraped data

This is n8n's home turf — no CSV-writing code required:

• Google Sheets: the Google Sheets node with operation Append Row; map extracted fields to columns. The default choice for monitoring workflows humans read.
• CSV / Excel files: the Convert to File node turns items into CSV or XLSX binary, which a follow-up node emails, uploads to Drive/S3, or POSTs anywhere.
• Databases: Postgres/MySQL/MongoDB nodes insert items directly — the right target once volume outgrows spreadsheets.
• Webhooks: the workflow itself can be an API — a Webhook trigger that scrapes on demand and responds with JSON.

Using the WebScraping.AI node

WebScraping.AI ships an official n8n community node — n8n-nodes-webscraping-ai — that wraps the whole API as a single node with an operation dropdown. Install it from Settings → Community Nodes (search n8n-nodes-webscraping-ai), add your API key as a credential, and you get:

• HTML / Text / Selected: rendered page HTML, clean LLM-ready text, or specific elements by CSS selector — with JS rendering and rotating proxies handled per request.
• AI Question: ask a question about a page ("What is the delivery time?") and get the answer as text.
• AI Fields: extract structured fields without writing selectors at all — the piece that eliminates the most maintenance in scraping workflows:

Operation: AI Extract Fields
URL: https://competitor.com/product/123
Fields: { "name": "Product name", "price": "Price with currency", "in_stock": "Is it in stock, true/false" }

The node returns clean JSON ready for Google Sheets or your database, and when the target site redesigns, there are no selectors to fix. There's a free trial with no card required, and the same API works through a plain HTTP Request node if you prefer — see the integrations page for the full list of no-code surfaces.

Frequently asked questions

Can n8n scrape websites without code? Yes — HTTP Request + HTML node with CSS selectors covers static sites entirely through the UI, and Schedule Trigger plus Google Sheets makes it a recurring monitor. Code nodes are optional and only needed for unusual parsing or transformation.

Does n8n Cloud support Puppeteer or Playwright? No — browser automation community nodes need native dependencies and run only on self-hosted n8n. On Cloud, scrape JavaScript-rendered sites through a scraping API (one HTTP Request node with js=true, or the WebScraping.AI community node) instead.

How do I scrape multiple pages in n8n? Use the HTTP Request node's built-in Pagination option for numbered pages or next-links, and Loop Over Items for lists of URLs. Add a Wait node inside loops to control request rate, and cap max pages so a broken selector can't loop forever.

Can I use XPath in n8n? Not in the HTML node — it's cheerio-based and CSS-only. Convert XPath to CSS selectors (most convert directly), or use a Code node on self-hosted instances with an external module for genuine XPath needs. Our XPath cheat sheet has equivalence tables.

How do I avoid getting blocked while scraping with n8n? Set a realistic User-Agent, keep request rates low with Wait nodes, respect robots.txt, and route through rotating proxies. If a target still blocks you, move the fetch to a scraping API with residential/stealth proxies rather than escalating from a single server IP.

How do I export n8n scraped data to Google Sheets? Add a Google Sheets node with operation Append Row after your extraction, connect OAuth credentials, and map fields to columns with expressions like {{ $json.title }}. For files instead, Convert to File produces CSV/XLSX that any storage or email node can deliver.