GPT Prompts for Web Scraping and AI Data Extraction

Large language models changed the economics of web scraping: instead of writing and maintaining a parser per site, you can hand a model rendered HTML and a description of what you want back. But the difference between a reliable LLM web scraping pipeline and a hallucination generator is almost entirely in the prompts. This guide collects prompt patterns that hold up in production — for extracting structured data, generating selectors, and cleaning scraped content — plus the pipeline design that keeps token costs sane.

Key Takeaways

LLMs excel at extraction (finding fields in messy HTML) and normalization (cleaning what you found) — not at fetching pages
Good extraction prompts pin down the output schema, the handling of missing values, and a "don't invent data" rule
Preprocessing HTML before sending it to the model cuts token costs by 10–20x
Selector-generation prompts let you use expensive models once, then extract with cheap CSS selectors forever
For production pipelines, AI extraction APIs bundle rendering, proxies, and prompting into one request

Pattern 1: HTML-to-JSON Extraction

The workhorse prompt. The rules section is what separates it from a naive "extract the data" request:

Extract product information from the HTML below.

Return a JSON object with exactly these fields:
- "name": product name (string)
- "price": numeric price without currency symbol (number)
- "currency": ISO 4217 code, e.g. "USD" (string)
- "in_stock": whether the product can be ordered (boolean)
- "rating": average review rating (number or null)

Rules:
- Use null for any field not present in the HTML. Never guess or invent values.
- Return only the JSON object, no explanations or markdown fences.
- If the page is not a product page, return {"error": "not_a_product_page"}.

HTML:
[...]

Three details matter: the explicit schema (models drift without it), the null rule (the single most effective anti-hallucination instruction), and the escape hatch for wrong page types (otherwise the model extracts something from a 404 page).

With the OpenAI API, move the rules into the system message and use JSON mode or a tool/function schema to make the output machine-parseable every time.

Pattern 2: Preprocess Before You Prompt

Raw HTML is mostly noise — scripts, styles, SVG, tracking attributes. Sending a full page to GPT-4-class models costs real money at scale. Strip it first:

from bs4 import BeautifulSoup

def clean_html(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "svg", "noscript", "iframe"]):
        tag.decompose()
    for tag in soup.find_all(True):
        tag.attrs = {k: v for k, v in tag.attrs.items()
                     if k in ("href", "src", "alt", "datetime")}
    return str(soup)

A typical e-commerce page drops from ~300k characters to ~20k. For text-heavy extraction (articles, descriptions), converting to markdown or plain text before prompting is even cheaper — and models parse it at least as well as HTML.

Pattern 3: Generate Selectors, Not Extractions

Calling an LLM on every page is the expensive way. The cheap way: use the model once per site to write the parser, then run that parser on every page:

Below is the HTML of a product listing page. Write CSS selectors for:
1. The container element of a single product card
2. Product name (relative to the card)
3. Price (relative to the card)
4. Link to the product detail page (relative to the card)

Prefer stable attributes (data-*, itemprop, semantic tags) over
auto-generated class names like "css-1x2y3z". Return JSON:
{"card": ..., "name": ..., "price": ..., "url": ...}

Re-run the prompt only when the site changes and your selectors break. This hybrid — LLM for parser generation, plain code for execution — is how you get AI adaptability at CSS-selector prices.

Pattern 4: Normalization and Enrichment

LLMs are just as useful after extraction — turning scraped mess into consistent records:

Normalize these scraped job listings. For each:
- Map "title" to a canonical seniority: junior / mid / senior / lead
- Convert "salary" strings ("$120k-140k", "€95.000") to min/max integers in USD
- Extract a deduplicated list of technologies from the description

Return a JSON array. Use null where information is absent — do not infer.

This is also where classic scraping falls short and LLMs shine: sentiment of reviews, categorizing products, matching entities across sites with different naming.

Putting It Together: LLM Scraping in One Request

The prompts above assume you already have rendered HTML — which means browsers, proxies, and anti-bot handling. WebScraping.AI bundles the whole pipeline: it renders the page in Chrome, applies your extraction instructions, and returns structured data directly:

import requests

# Ask a question about any page
response = requests.get(
    "https://api.webscraping.ai/ai/question",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/product/123",
        "question": "Is this product in stock and what is its price?",
        "js": True,
    },
)
print(response.text)

# Or extract typed fields — the prompt patterns above, as an API
response = requests.get(
    "https://api.webscraping.ai/ai/fields",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/product/123",
        "fields[name]": "Product name",
        "fields[price]": "Numeric price without currency",
        "fields[in_stock]": "true or false",
        "js": True,
    },
)
print(response.json())

No prompt engineering, token budgeting, or HTML cleaning on your side — the field descriptions are the prompt.

Common Failure Modes

Hallucinated values — always include the null rule, and spot-check extractions against source pages
Truncated inputs — a silently clipped page yields confident nonsense; count tokens before sending
Schema drift on cheap models — smaller models need JSON mode or function calling enforced, not just requested
Prompt-injection from page content — treat scraped text as untrusted; instruct the model to ignore any instructions found inside the HTML

Conclusion

Prompting is the new parsing: schema-pinned extraction prompts, aggressive HTML preprocessing, and the generate-selectors-once trick cover most LLM web scraping needs, while normalization prompts clean up whatever your existing scrapers produce. When you'd rather ship than tune prompts, the AI extraction endpoints give you the same patterns as a single GET request. For more on model-assisted scraping, see our FAQ sections on scraping with GPT, scraping with LLMs, and ChatGPT use cases.