LLM Web Scraping: How AI Extraction Pipelines Actually Work

An LLM cannot scrape a website. It has no network stack, no browser, no proxy pool — it reads text you hand it and writes text back. Every "AI scraper" you have seen is a conventional scraper with a language model bolted onto the last step, and understanding which step is which is the difference between a pipeline that works and a demo that breaks on the first site with a login wall.

This guide covers the whole pipeline: how LLM extraction works, which model to use in mid-2026 and what its context window actually buys you, what it costs per page, where accuracy fails, and which parts of the problem a model cannot solve at any price.

Key Takeaways

• LLM web scraping is a four-stage pipeline — fetch, render, clean, extract — and the model only does the last stage. Stages one and two are where scraping projects fail.
• Cleaning HTML to plain text before prompting is the single biggest cost lever, typically cutting input tokens by roughly 10x. Model choice is the second lever, worth another 10–25x between value and flagship tiers.
• Context windows stopped being the binding constraint. DeepSeek, Gemini, and current Claude models all carry 1M-token windows as of mid-2026; a cleaned page is 2,000–8,000 tokens. Most "context limit" errors are really "I sent raw HTML" errors.
• Constrain output with a strict JSON schema, not a polite request for JSON. JSON mode guarantees syntax, not completeness.
• An LLM cannot see data that lives only in markup — a rating encoded as class="star-rating Three" is invisible to a model reading page text, and it cannot solve a CAPTCHA or defeat bot detection either.
• Prompting beats fine-tuning for almost every extraction task. Fine-tuning pays off only at high volume on a stable, narrow schema.

What "LLM web scraping" actually means

Strip away the marketing and every AI scraping stack is the same four stages:

The model's job is stage four only. That sounds like a small share of the work, and in terms of code it is — but it replaces the part that historically caused the most maintenance: hand-written CSS or XPath selectors that break every time a site ships a redesign.

The trade you are making is explicit. Selectors are free to run, deterministic, and brittle. LLM extraction costs money per page, varies slightly run to run, and survives layout changes because it reads meaning rather than structure. Neither is universally right, which is why most production pipelines end up using both.

What changes when you swap selectors for a model

You stop writing per-site parsers. One prompt describing "product name, price, availability" works across a hundred stores whose HTML has nothing in common. For a comparison crawl across many domains, this is the entire value proposition.

You start paying per page. A parser costs nothing to run a million times. A model costs somewhere between a hundredth of a cent and fifteen cents per page depending on what you send and which tier you pick.

Your failure mode changes from loud to quiet. A broken selector returns nothing and your monitoring notices. A model handed an unexpected page returns something plausible, and nobody notices for three weeks. This is the risk that deserves the most engineering attention, and we come back to it below.

When LLM extraction earns its cost

A rough decision rule, from cheapest to most expensive approach:

The honest summary: LLMs are strongest where the structure varies and the meaning is stable, and weakest where the opposite is true.

Which model should you use?

Prices and context windows as of August 2026, per million tokens. These move constantly — check the linked pricing pages before you budget anything.

Sources: DeepSeek, Anthropic, OpenAI, Google.

The practical answer is that model choice matters much less than people expect. Schema-pinned extraction from cleaned text is a reading-comprehension task, not a reasoning task. Every model in that table handles it competently. We run Gemini 2.5 Flash Lite as the default behind our own AI endpoints for exactly this reason — on extraction workloads the accuracy gap to a flagship model is small and the price gap is over 50x.

Where the tier genuinely matters:

• Irregular, poorly structured pages — a value model starts dropping fields that a smarter one finds.
• Long documents needing synthesis, not field lookup.
• Once-per-site work like generating selectors or reverse-engineering an unfamiliar layout. Spend freely here; you do it once.

Benchmark on your own pages before committing. Layout variance across your actual targets moves accuracy far more than the difference between two adjacent model tiers.

Context windows and token limits, in practice

Context windows are the most over-worried number in this field. A cleaned web page is typically 2,000–8,000 tokens. Every model above holds at least 200,000. You are not close to the limit unless something has gone wrong.

Three limits get conflated, and only one of them usually bites:

• Context window — total input plus output. 200k to 1M on current models. Rarely the problem.
• Max output tokens — a separate, much smaller ceiling: 64k on Claude Haiku 4.5, 128k on current Claude models, 384k on DeepSeek V4. This is a real constraint when extracting hundreds of records from one page in a single call.
• Rate limits — requests or concurrent connections per account. This is what actually stops production crawls. DeepSeek, for instance, publishes concurrency limits (2,500 connections on Flash, 500 on Pro) and returns HTTP 429 above them.

If you hit a context error, check what you sent before you reach for a bigger model. Raw HTML is 10–50x the tokens of the same page as text, and the fix is a converter, not an upgrade.

When a page genuinely exceeds the window — a 5,000-row table, a full archive page — chunk on structural boundaries (headings, table rows, list items), never on a fixed character count that can split a record in half. Extract per chunk, merge, deduplicate on a stable key.

DeepSeek for scraping: models, tokens, and docs

DeepSeek gets asked about more than any other model in this category, mostly because of price. As of August 2026 its API exposes two models, deepseek-v4-flash and deepseek-v4-pro, both with a 1M-token context window and a 384k maximum output. Input costs $0.14 and $0.435 per million tokens respectively, output $0.28 and $0.87 — and cached input drops to roughly $0.003 per million, which matters a great deal if you send the same long system prompt on every page.

Practical notes for a scraping workload:

• Token estimation. DeepSeek's docs put it at roughly 0.3 tokens per English character and 0.6 per Chinese character. Estimate with that, then trust the usage object in the response for anything you bill against.
• Structured output. DeepSeek supports response_format: {"type": "json_object"}, with the documented caveat that you must also instruct the model to produce JSON in your system or user message. That is JSON mode, not schema-constrained output — it guarantees the response parses, not that every field you asked for is present. Validate accordingly.
• Function calling is supported, up to 128 tools per request, which is the cleaner route to a fixed shape.
• Rate limits are concurrency-based rather than requests-per-minute; a 429 means back off and retry, not that you have exhausted a quota.
• Data residency. DeepSeek is a China-based provider. If you are sending scraped pages that contain customer or personal data, read the terms and check your own compliance position before wiring it into a pipeline.

The official reference lives at api-docs.deepseek.com. The API is OpenAI-compatible, so any OpenAI SDK works by pointing base_url at https://api.deepseek.com and swapping the model id — which is also why migrating between DeepSeek and OpenAI costs almost nothing.

Claude for extraction

Anthropic's current lineup runs from Claude Haiku 4.5 ($1/$5 per MTok, 200k context) up through Sonnet, Opus, and Fable tiers at 1M context. For extraction the interesting properties are native tool-use-based structured output — which gives you schema enforcement without prompt gymnastics — and prompt caching, which cuts repeat input costs by roughly 10x when your system prompt is long and constant across pages.

Haiku is the tier that makes sense for per-page extraction work. The larger models are worth it for the once-per-site jobs, or when pages are genuinely irregular. Anthropic publishes a models overview with current ids, context windows, and output ceilings; it changes often enough that it is worth checking rather than trusting a blog post, including this one.

GPT and the OpenAI API

OpenAI's value tier is the most common starting point simply because the SDK is everywhere and the response_format schema support is the cleanest implementation of strict structured output in the field. The pattern ports directly: the same raw JSON Schema payload works against OpenRouter, Gemini's OpenAI-compatible endpoint, DeepSeek, and most gateways, which is a good reason to write it as plain JSON Schema rather than through a Pydantic or Zod wrapper.

If you are working through the mechanics of prompting these models — field lists, null rules, injection guards, retry strategy — our LLM web scraping prompts guide is the deep version, with measured token counts and the failure modes we hit in our own production logs.

Open models you can run yourself

Llama, Qwen, Mistral, and DeepSeek's open weights all run locally through Ollama or vLLM, and for extraction they are more than adequate. The reasons to do it are privacy (pages never leave your infrastructure) and marginal cost (electricity rather than tokens). The reasons not to: a GPU that can run a capable model costs more per month than a very large volume of API calls, throughput is a fraction of a hosted endpoint's, and structured-output support is less reliable — many local stacks fall back to "please return JSON" rather than true constrained decoding.

Run locally if data residency requires it or if you already own idle GPUs. Otherwise the economics favor a hosted value model by a wide margin.

Getting structured output that parses

Three mechanisms, in ascending order of reliability:

1. Prompt and hope — "return JSON". Works most of the time, fails unpredictably, needs defensive parsing.
2. JSON mode (response_format: {"type": "json_object"}) — the model is constrained to emit syntactically valid JSON. It is not constrained to include every field, or to finish.
3. Strict JSON Schema (response_format: {"type": "json_schema", "strict": true}) or function/tool calling — the model is constrained to your exact shape.

The gap between two and three is not academic. Our own /ai/fields endpoint originally used plain JSON mode; after a provider change the model began truncating responses mid-string, and we took roughly 300 parse errors before moving to strict schemas. JSON mode guaranteed the response started as JSON. Nothing guaranteed it finished.

Two details that make strict mode actually strict: list every key in required, and set additionalProperties: false. Without both, fields silently vanish instead of coming back null. And declare every field as ["string", "null"] rather than mixing types — a page reading "Contact for pricing" against a numeric price field forces the model to choose between violating the schema and inventing a number, and strict mode means it invents.

When a schema response still comes back malformed, retry with a fresh request rather than replying "that wasn't valid JSON" in the same conversation. The bad turn is now history the model conditions on, and it tends to reproduce it.

What it costs, and when it stops being economical

Assume 5,000 input tokens for a cleaned page and 200 output tokens per record. Per 1,000 pages, model inference only:

Two conclusions fall out. Cleaning the page saves roughly 10x — more than any model choice. And the spread between a value model on clean text and a flagship on raw HTML is about 500x: at 100,000 pages a month, $58 against $30,600.

Those numbers exclude fetching. Proxies, browser rendering, and retries against protected sites are usually the larger line item on real workloads; inference is often the cheap part of an AI scraping bill, which surprises people who budget only for tokens.

The architecture that beats per-page inference

If you are scraping the same site repeatedly, calling a model on every page is the expensive way to do it. Call a good model once per site to write CSS selectors, then run those selectors forever with Beautiful Soup or XPath, and re-run the prompt only when extraction starts coming back empty.

One frontier-model call per site per month costs less than one value-model call per page, and extraction becomes deterministic and free. For steady-state jobs like price monitoring, this is almost always the right architecture. Reserve per-page inference for the case it is actually good at: many sites, seen once each.

Accuracy: the failure that stays quiet

The characteristic LLM scraping bug is not a crash. It is a plausible, wrong value sitting in your database.

Defenses, roughly in order of effectiveness:

Make null explicit and mandatory. "Use null for any field not stated in the content. Never guess, infer, or calculate a value that is not written on the page." Models otherwise treat arithmetic on page values as fair game — computing a unit price you never asked for, inferring stock status from a button label.

Validate in code, not in the prompt. Range checks, enum membership, type casts, cross-field consistency. Anything a regex can verify should be verified by a regex.

Spot-check before trusting a run. Twenty records against their source pages, by eye, before a new target goes to production. This catches systematic misreads that per-record validation cannot.

Give the model an escape hatch. Without {"error": "not_a_product_page"} as an allowed answer, a model handed a 404 page will extract a product from it.

Treat page content as untrusted input. Scraped text can contain "ignore previous instructions". Wrap page content in explicit delimiters and state that it is data, not instructions. Our production prompts use XML-style tags for exactly this.

What the model genuinely cannot see

Some data lives only in markup. A star rating rendered as:

<p class="star-rating Three">

contains no text saying "3" anywhere. A pipeline that extracts page text and prompts a model over it cannot read that value, no matter how good the prompt is — and a well-behaved model returns null rather than guessing. That is the null rule working, not a model failure.

The fix is a selector or a JS snippet, not a better prompt:

curl -G "https://api.webscraping.ai/html" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" \
  --data-urlencode "js_script=document.querySelector('p.star-rating').className" \
  --data-urlencode "return_script_result=true"

Decision rule: if a field is visible as text, an LLM can extract it. If it is carried by a class, attribute, icon, or image, use a selector. Most real pipelines need both.

Fine-tuning versus prompting

Fine-tuning an LLM for extraction is almost never the right first move, and the reason is arithmetic rather than principle.

Prompting costs nothing up front, changes in seconds, and works across sites you have never seen. Fine-tuning costs a training run plus a labelled dataset (typically hundreds to thousands of examples), locks you to a base model that will be superseded, and needs redoing when your schema changes.

Fine-tuning pays off in one specific shape: very high volume, a narrow and stable schema, and a domain where a small model consistently underperforms. Then you can fine-tune a cheap model to match a much more expensive one's accuracy and win on unit cost. Below roughly a million pages a month with a settled schema, prompt engineering on a value model reaches the same place for less total effort.

There is a second, more common use of the same words: scraping the web to build training data for someone else's fine-tune. That is a genuinely large use case and a different problem — the bottleneck is volume, deduplication, and licensing rather than extraction accuracy. We have a page on LLM fine-tuning data collection covering that side.

Free AI scrapers: what is actually free

There are good free options, with a consistent catch worth stating plainly: the open-source tools are free, and the two things that cost money — model inference and getting past anti-bot systems — are not included.

Open-source frameworks:

• ScrapeGraphAI — Python, builds extraction pipelines as graphs; you describe the data in plain English and supply your own model key.
• Crawl4AI — Python, optimized for producing LLM-ready markdown from crawls; widely used as the ingest layer for RAG.
• llm-scraper — TypeScript, extracts structured data from a page against a schema you define.
• LangChain / LlamaIndex document loaders — less a scraper than a plumbing layer, but the loaders handle fetch-and-clean for many sources.

Free model capacity: most providers offer a free tier or trial credits, and local models via Ollama are free after hardware. Free tiers carry rate limits that make them fine for evaluation and unworkable for production crawls.

What free never covers: residential proxies, CAPTCHA-protected targets, and the operational work of keeping a browser fleet alive. On cooperative sites this does not matter at all. On retail, travel, and social targets it is the entire difference between a working pipeline and a folder full of 403s.

Our own free tier is 2,000 credits a month with no card, which is enough to test whether a target is reachable before you build anything around it.

Dynamic sites, and the part LLMs do not solve

Roughly half the pages worth scraping render their content client-side. A plain HTTP request returns an empty shell, and the model dutifully reports that the page contains no products.

The fix belongs entirely to stage two: run a real browser. Playwright or Puppeteer locally, or a rendering API that does it for you. Wait for a content selector rather than a fixed timeout, scroll to trigger lazy loading, and only then hand the DOM to the cleaner.

There is a shortcut worth checking first: many "dynamic" pages are fed by a JSON API you can call directly. Open the network tab, find the XHR that returns the data, and skip both the browser and the model. Clean JSON from the source beats extracted JSON from rendered text every time.

Can an LLM get past CAPTCHAs or bot detection?

No — and the reason is architectural, not a matter of the model refusing.

Bot detection operates on things a language model has no access to: TLS fingerprints, IP reputation, HTTP header ordering, and browser signals like canvas fingerprinting and mouse movement. The model never touches the network layer. By the time text reaches it, the block has already happened, and all it can tell you is "this page says access denied".

A model can help you understand a block — paste a response and ask what triggered it — and it can help you write better-behaved request code. It cannot solve the visual challenge, and it cannot change what your TCP connection looks like. Those are proxy and browser-fingerprint problems, solved by rotating residential IPs, realistic headers, and a genuine browser stack — or by using an API that has already solved them.

Building the pipeline, or skipping it

If you assemble this yourself, you own four things: an HTTP layer with proxy rotation and retry logic, a browser fleet, an HTML-to-text converter, and the model call with its schema and validation. Three of those four have nothing to do with AI.

Our API packages the first three, and optionally the fourth. Fetch and clean:

# Rendered page as clean text, ready to prompt
curl -G "https://api.webscraping.ai/text" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/product/123" \
  --data-urlencode "js=true"

Then prompt whichever model you prefer. Or skip the prompt layer — /ai/fields runs fetch, render, clean, schema-constrained extraction, and retry in one request, with your field descriptions acting as the prompt:

import requests

response = requests.get(
    "https://api.webscraping.ai/ai/fields",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/product/123",
        "fields[name]": "Product name",
        "fields[price]": "Numeric price without currency symbol",
        "fields[in_stock]": "true or false",
        "js": True,
    },
)
print(response.json()["result"])

For a one-off question rather than a fixed schema, /ai/question takes free-form instructions and returns an answer:

response = requests.get(
    "https://api.webscraping.ai/ai/question",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/product/123",
        "question": "Is this product in stock, and what is the price?",
    },
)
print(response.text)

Honest scope: AI extraction endpoints are table stakes in this category — Firecrawl, Apify, and most competitors ship an equivalent. What differs is billing. Ours is a flat +5 credits on top of the underlying request, published in the pricing table rather than metered by tokens, and failed requests are not billed at all. Everything is documented here.

Agents, MCP, and where this is heading

The newer pattern is not a pipeline at all: an agent decides what to fetch, fetches it through a tool, and extracts in the same loop. That is what the Model Context Protocol standardizes — scraping capabilities exposed as tools an agent can call, rather than functions you wire together in advance.

It is a genuinely different shape, and it suits exploratory work (research agents, one-off questions across many sites) far better than it suits a nightly crawl of 50,000 URLs, where a deterministic pipeline is cheaper and easier to monitor. Our guide to web scraping MCP servers covers the landscape, and we run an MCP server exposing the same endpoints as agent tools.

For what people are actually building on top of all this, AI web scraping use cases and ChatGPT use cases have concrete examples.

Frequently asked questions

Can ChatGPT or Claude scrape a website for me?

Not from the chat interface, in the sense people usually mean. The consumer apps can browse a page and summarize it, one URL at a time, interactively — useful for research, useless for collecting a thousand records. For a pipeline you need the API, and you still need to fetch the pages yourself and pass in the text. The model is an extraction step, not a crawler.

What is DeepSeek's token limit and context window?

As of August 2026, both deepseek-v4-flash and deepseek-v4-pro document a 1M-token context window with a maximum output of 384k tokens. Rate limiting is by concurrent connections (2,500 and 500 respectively) rather than tokens per minute, and exceeding it returns HTTP 429. The official docs are the place to confirm — these numbers have changed several times.

DeepSeek or OpenAI for scraping?

For schema-pinned extraction from cleaned text, both are competent and DeepSeek is cheaper, particularly on output tokens and cached input. OpenAI has stricter structured-output enforcement (true schema constraint rather than JSON mode) and a more mature SDK ecosystem. Since DeepSeek's API is OpenAI-compatible, you can benchmark both against your own pages by changing two lines, which is a better basis for the decision than anyone's ranking. Factor in data residency if the pages you scrape contain personal data.

What is the best LLM for data extraction?

Whichever value-tier model from a major lab you benchmark best on your own pages — the differences are small for this task. The bigger accuracy lever is what you send: cleaned text with a strict schema and an explicit null rule beats a smarter model reading raw HTML with a vague prompt, essentially every time.

How do I extract structured data from HTML with an LLM?

Strip scripts, styles, and navigation; convert to text or markdown; send it with a strict JSON schema declaring each field as nullable; validate the result in code. The prompts guide has copy-pasteable versions for tables, listings, classification, and pagination.

Are there free AI scrapers?

Yes — ScrapeGraphAI, Crawl4AI, and llm-scraper are open source and free to run, and local models via Ollama remove the inference bill. What no free tool provides is proxy infrastructure and anti-bot handling, which is what protected targets actually require. Free works well on cooperative sites and stalls on hard ones.

Should I fine-tune a model for scraping?

Almost certainly not. Prompting with a strict schema reaches the same accuracy for a tiny fraction of the effort, and it does not lock you to a base model that will be obsolete in six months. Fine-tuning only makes sense at very high volume against a schema that has stopped changing.

Is scraping pages to feed an LLM legal?

It depends on what you collect and how, not on whether a model is in the pipeline. Public non-personal facts at polite request rates sit on much safer ground than personal data (GDPR/CCPA), authenticated content, or wholesale republication of copyrighted text. Our web scraping legality guide covers the specifics.

Where to start

Build the fetch layer first. It is the part that determines whether the project works at all, and it is the part no model can substitute for. Once you can reliably get clean text out of your target pages, the extraction step is a schema and twenty lines of code — and swapping models later costs an afternoon.

If you would rather not build the first three stages, start with the free tier: 2,000 credits a month, no card, and /ai/fields pointed at a page you care about.