How to Scrape Google Search Results

Google's search results page is the single most scraped and most defended page on the web. Every rank tracker, every SEO tool, every "what do people see when they search for us" dashboard is built on top of it — and Google would prefer that none of them existed. That tension is why almost every tutorial you'll find on this topic is quietly out of date: the ten-line requests + BeautifulSoup snippet that worked in 2020 now returns a consent wall or a CAPTCHA far more often than it returns results.

This guide covers what actually works in 2026, what it costs, where it breaks, and the legal ground you're standing on while you do it.

Key Takeaways

• Plain HTTP requests to google.com/search mostly fail now. You get a consent interstitial, a JavaScript shell, or a "unusual traffic" block. The old div.g / div.tF2Cxc selectors are obfuscated class names that rotate, so even a successful fetch often parses to zero results.
• A real browser (Playwright/Puppeteer) is the DIY approach that still works, but it needs consent handling, residential IPs, and structural selectors instead of class names. It does not scale cheaply — one SERP is one browser page load.
• &num=100 stopped working in September 2025. Google now serves ten results per page regardless, so "grab the top 100 in one request" pipelines had to be rewritten around &start= pagination — ten times the requests for the same data.
• Google's official Custom Search JSON API is real, cheap, and limited: 100 queries/day free, $5 per 1,000 after, 10 results per call, and it is a Programmable Search Engine — its rankings are not guaranteed to match what you see on google.com.
• The legal picture is not "scraping Google is illegal." Scraping public pages isn't a US computer-crime violation after hiQ, but Google's Terms of Service do prohibit automated access, and ToS is a contract question, not a criminal one. The two get conflated constantly.

Why Google is a hard target

Four separate things break naive scrapers, and they compound.

Anti-bot detection. Google fingerprints far more than your user agent: TLS handshake characteristics, HTTP/2 frame ordering, header order and casing, IP reputation, and browser-level signals like navigator.webdriver. A datacenter IP sending a header set that doesn't quite match a real Chrome build is flagged quickly, and the response is a "Our systems have detected unusual traffic from your computer network" page — served with a 200 OK, which is why scrapers that only check status codes think they're succeeding while writing garbage to disk.

JavaScript rendering. The modern SERP is assembled client-side. Fetch the raw HTML and much of what a user sees — some result blocks, People Also Ask expansions, AI Overviews — either isn't there or is buried in inline JSON payloads rather than semantic markup.

Consent screens. From EU IPs (and increasingly elsewhere) the first request lands on consent.google.com rather than results. Any scraper that doesn't handle the interstitial parses a cookie banner and reports zero results.

Deliberately unstable markup. Google's result containers carry generated class names — tF2Cxc, VwiC3b, yuRUbf — that change without notice. Every tutorial pinned to those strings has a shelf life measured in months. This is not incidental; obfuscated markup is an anti-scraping measure.

Is scraping Google legal?

Short version: it's a contract question, not a criminal one, and the honest answer is "it depends what you do with it."

Public pages are not a computer-crime violation in the US. hiQ Labs v. LinkedIn (9th Cir., 2022), read alongside Van Buren v. United States (2021), established that accessing a publicly available page without authentication doesn't violate the Computer Fraud and Abuse Act — there are, in the court's framing, "no gates to lift or lower." A Google SERP with no login is squarely a public page.

But Google's Terms of Service prohibit automated access, and hiQ is also the cautionary tale here: hiQ won the CFAA point and still lost on breach of contract. ToS violations are a real legal exposure — cease-and-desist letters, account termination, civil claims — they're just a different and less severe category than "illegal."

Practically, Google's enforcement against SERP scraping has overwhelmingly been technical rather than legal: blocks, CAPTCHAs, and IP bans, not lawsuits against small operators. That is an observation about historical behavior, not a promise, and it says nothing about what happens at scale or in a commercial product built on scraped rankings.

A few things do move you into worse territory regardless: scraping behind a Google account login (now you've agreed to the terms), harvesting personal data from results (GDPR and CCPA apply to scraped personal data exactly as they do to any other collection), and republishing snippets wholesale rather than using them as facts.

None of this is legal advice — we're engineers, not lawyers. Our complete guide to web scraping legality walks through the five separate legal questions with the case law behind each.

Approach 1: requests + BeautifulSoup (and why it usually fails)

This is the code every tutorial shows. It's worth seeing precisely because you need to understand its failure mode:

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

resp = requests.get(
    "https://www.google.com/search",
    params={"q": "web scraping api", "hl": "en", "gl": "us"},
    headers=headers,
    timeout=10,
)
soup = BeautifulSoup(resp.text, "html.parser")

for result in soup.select("div.g"):
    title = result.select_one("h3")
    link = result.select_one("a")
    if title and link:
        print(title.get_text(), link.get("href"))

From a residential connection this sometimes works. From a cloud IP — which is where your scraper actually runs — the usual outcomes are:

1. A 200 response containing a consent interstitial. Parses to zero results.
2. A 200 response containing "unusual traffic" and a reCAPTCHA. Parses to zero results.
3. A 429. At least this one is honest.
4. A real SERP that still parses to zero results, because div.g no longer wraps what you think it wraps.

Case 4 is the dangerous one. Silent zero-result parsing looks like "no results for this query" in your data, and you can run a broken rank tracker for weeks before noticing.

If you're going to parse SERP HTML at all, anchor on structure rather than class names:

# More durable: every organic result is a link containing an h3,
# inside the main results container.
for anchor in soup.select("#search a:has(h3)"):
    href = anchor.get("href")
    title = anchor.select_one("h3").get_text(strip=True)
    if href and href.startswith("http"):
        print(title, href)

Still brittle — just brittle along an axis Google changes less often. Whatever you write, assert on result count and alert when a query returns zero, rather than trusting the parse.

Approach 2: Playwright and headless browsers

A real browser solves the JavaScript problem and part of the detection problem, because you're running an actual Chrome rather than imitating one. This is the workhorse approach for small-to-medium DIY scraping.

import asyncio
from playwright.async_api import async_playwright

async def scrape_google(query, hl="en", gl="us"):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            locale="en-US",
            timezone_id="America/New_York",
            viewport={"width": 1366, "height": 768},
        )
        page = await context.new_page()

        url = f"https://www.google.com/search?q={query}&hl={hl}&gl={gl}"
        await page.goto(url, wait_until="domcontentloaded")

        # Consent interstitial: appears from EU IPs, sometimes elsewhere.
        # The button text is localized, so match on several forms.
        for label in ("Accept all", "I agree", "Alle akzeptieren"):
            button = page.get_by_role("button", name=label)
            if await button.count():
                await button.first.click()
                await page.wait_for_load_state("domcontentloaded")
                break

        # Bail out loudly instead of returning [] if we got blocked.
        if await page.locator("form#captcha-form").count():
            raise RuntimeError("Blocked: CAPTCHA challenge served")

        await page.wait_for_selector("#search", timeout=15000)

        results = await page.eval_on_selector_all(
            "#search a:has(h3)",
            """els => els.map((el, i) => ({
                position: i + 1,
                title: el.querySelector('h3').innerText,
                url: el.href,
            })).filter(r => r.url.startsWith('http'))""",
        )

        await browser.close()
        return results

print(asyncio.run(scrape_google("web scraping api")))

The Node equivalent is a near-direct translation:

const { chromium } = require('playwright');

async function scrapeGoogle(query) {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    locale: 'en-US',
    viewport: { width: 1366, height: 768 },
  });
  const page = await context.newPage();

  await page.goto(
    `https://www.google.com/search?q=${encodeURIComponent(query)}&hl=en&gl=us`,
    { waitUntil: 'domcontentloaded' }
  );

  const consent = page.getByRole('button', { name: 'Accept all' });
  if (await consent.count()) await consent.first().click();

  if (await page.locator('form#captcha-form').count()) {
    await browser.close();
    throw new Error('Blocked: CAPTCHA challenge served');
  }

  await page.waitForSelector('#search', { timeout: 15000 });

  const results = await page.$$eval('#search a:has(h3)', els =>
    els
      .map((el, i) => ({
        position: i + 1,
        title: el.querySelector('h3').innerText,
        url: el.href,
      }))
      .filter(r => r.url.startsWith('http'))
  );

  await browser.close();
  return results;
}

Four things about this code are load-bearing:

• It navigates straight to /search?q= rather than loading the homepage and typing into the search box. Fewer page loads, fewer chances to be fingerprinted, and no dependency on the homepage's markup.
• It fails loudly on a CAPTCHA instead of returning an empty list. This is the single most valuable line in the script.
• It sets locale and timezone_id. Mismatched locale/timezone/IP triples are a cheap detection signal.
• The selectors are structural. #search a:has(h3) will outlive div.tF2Cxc by years, but "outlive" is not "survive forever" — pin a test with a saved SERP fixture and run it in CI.

What this approach will not do is scale. Each query is a full browser page load, so a thousand keywords tracked daily is a thousand browser sessions a day from IPs Google will start recognizing. Which brings us to the part every tutorial skips.

Rate limits, CAPTCHAs, and getting blocked

There is no published rate limit for google.com/search, because it isn't an API — the limit is adaptive and depends on IP reputation, query patterns, and how much your traffic looks like a person. Practical observations from people who run this at scale:

• Datacenter IP ranges are heavily degraded. AWS, GCP, and Azure egress ranges are well known to Google. Residential and mobile IPs last far longer.
• A single IP running searches continuously gets throttled within tens to low hundreds of queries. There's no magic number to quote; it varies by range and by day.
• Blocks are usually IP-level and temporary, escalating with repetition.
• Pattern matters as much as volume. Perfectly regular intervals, no referrer variation, and identical viewports across sessions are all signals.

The honest engineering answer to rate limiting is fewer requests, spread wider: cache aggressively (a keyword's ranking does not change every hour), batch your checks, add real jitter rather than a fixed sleep(2), and distribute across residential IPs — which means paying for proxies. Our residential proxy provider comparison covers that market.

On CAPTCHAs specifically: when Google serves you a reCAPTCHA, that is Google telling you it has identified your traffic as automated. The right response is to back off and change your approach — not to defeat the challenge. This guide doesn't cover CAPTCHA-solving techniques, and the practical reality is that a pipeline depending on solving them is a pipeline that breaks constantly and costs more than the legitimate alternatives below.

Detect and stop, don't push through:

BLOCK_SIGNALS = ("captcha-form", "unusual traffic", "/sorry/index")

def is_blocked(html: str, final_url: str) -> bool:
    return any(s in html for s in BLOCK_SIGNALS) or "/sorry/" in final_url

How to scrape Google without an API key

This is the most common framing of the question, so it's worth answering directly rather than deflecting.

You can, and everything above is how. No API key is required to fetch google.com/search — it's a public URL. What you don't get without one is any reliability guarantee, and "no API key" doesn't mean "no cost": you pay in proxies, browser infrastructure, and maintenance time when selectors change.

Two things that come up constantly in this context and deserve correcting:

The num=100 parameter no longer works. Until September 2025, &num=100 returned 100 results in a single request, and a large share of rank-tracking tooling was built on it. Google stopped honoring it. You now get ten results per page and must paginate with &start=10, &start=20, and so on — the same data at roughly ten times the request volume. If you're following a tutorial that fetches the top 100 in one call, it predates that change and its cost estimates are wrong by an order of magnitude.

Unofficial "google search" pip packages are wrappers around the same scraping. Libraries like googlesearch-python fetch and parse google.com/search exactly as your own code would. They inherit every blocking problem described above, plus a dependency on someone else updating their selectors. Convenient for a one-off script; not a foundation.

The genuinely free, key-free options that don't fight Google:

• Google Search Console — for your own site's rankings, this is authoritative first-party data with no scraping involved. If your goal is "how do my pages rank," start here, not with a scraper.
• Bing and DuckDuckGo are meaningfully easier targets if search results in general — rather than Google's specifically — are what you need.

Approach 3: Google's official Custom Search JSON API

Google does sell programmatic access, and for a lot of use cases it's the right answer despite the constraints.

import requests

def custom_search(query, api_key, cx, start=1):
    resp = requests.get(
        "https://www.googleapis.com/customsearch/v1",
        params={"key": api_key, "cx": cx, "q": query, "num": 10, "start": start},
        timeout=10,
    )
    resp.raise_for_status()
    data = resp.json()
    return [
        {
            "title": item["title"],
            "url": item["link"],
            "snippet": item.get("snippet", ""),
        }
        for item in data.get("items", [])
    ]

The catch that trips people up: this is a Programmable Search Engine, not google.com. You configure a search engine (optionally set to search the entire web), and its ranking is Google's but not identical to the consumer SERP. It also returns no ads, no People Also Ask, no AI Overviews, no local pack, and no position data that you can safely present as "your Google ranking."

So: excellent for site search, retrieval for a RAG pipeline, or "find me pages about X." Not suitable for rank tracking or SERP-feature analysis, which is exactly why the scraping market exists.

Approach 4: SERP APIs

A whole vendor category exists to absorb this problem: you send a keyword, they handle proxies, rendering, blocking, and parsing, and you get structured JSON back. The established names are SerpApi, DataForSEO, Oxylabs, Bright Data, Scrapingdog, and Apify's Google Search actor, among others.

Being straight about it: WebScraping.AI does not sell a Google SERP endpoint, so we have no product interest in talking you out of these. If parsed SERP JSON with position data and SERP features is what you need, a dedicated SERP API is usually the cheapest path once you price in your own engineering time.

What to compare when you evaluate them:

• Price per 1,000 searches, and whether JavaScript rendering or "advanced" SERP features cost extra (they usually do).
• Geographic and language targeting granularity — city-level matters for local rank tracking; country-level is often not enough.
• Which SERP features are parsed — organic results are table stakes; AI Overviews, People Also Ask, local packs, and shopping results vary a lot between vendors.
• Whether failed requests are billed.
• Latency and whether the API is synchronous. Some vendors queue and call back, which changes how you build around them.

Approach 5: rendered HTML through a scraping API

The middle ground between "run your own browser fleet" and "buy parsed SERP JSON" is a general scraping API that handles rendering and proxies while you keep control of parsing. That's what WebScraping.AI does, and it's worth being precise about what that means for Google specifically.

Google is one of our hardest targets, and our service enforces that: requests to Google require proxy=residential or proxy=stealth and js=true — a datacenter-proxy request to Google is rejected outright rather than sold to you as a request that was never going to work.

# Rendered Google SERP HTML through a residential proxy
curl -G "https://api.webscraping.ai/html" \
  --data-urlencode "api_key=YOUR_KEY" \
  --data-urlencode "url=https://www.google.com/search?q=web+scraping+api&hl=en&gl=us" \
  --data-urlencode "js=true" \
  --data-urlencode "proxy=residential" \
  --data-urlencode "country=us"

You get the rendered HTML and parse it yourself with the structural selectors from earlier. If you'd rather skip parsing entirely, /ai/fields reads the page with an LLM and returns named values, which survives markup churn better than CSS selectors do:

curl -G "https://api.webscraping.ai/ai/fields" \
  --data-urlencode "api_key=YOUR_KEY" \
  --data-urlencode "url=https://www.google.com/search?q=web+scraping+api" \
  --data-urlencode "js=true" \
  --data-urlencode "proxy=residential" \
  --data-urlencode "fields[titles]=Titles of the organic search results, in order" \
  --data-urlencode "fields[urls]=Destination URLs of the organic search results, in order"

Honest limitations, so you can decide with clear eyes:

• There is no Google-specific endpoint and no parsed SERP JSON. We return the page; the SERP schema is your problem. A dedicated SERP API gives you positions and features out of the box.
• Expect failures on Google even with residential proxies. It is an adversarial target and nobody has a 100% success rate on it, whatever their marketing says. What we do promise is that you're not billed for failed requests — when our anti-bot detection spots a challenge page served as a 200 OK, the request is failed and the credit refunded rather than charged for a page of nothing.
• Locale control is country-level (country=us, gb, de, and others) plus Google's own hl/gl URL parameters. If you need city-level SERPs, that's a SERP API feature, not ours.

Where this shape wins is when Google is one of several sources in a pipeline — SERPs plus competitor pages plus product listings — and you'd rather run one API with one billing model than stitch together a SERP vendor and a general scraper. There's a free trial with no card required.

Parsing SERP features

Once you have the HTML, organic results are the easy part. The rest of the SERP is where the value usually is, and where the brittleness lives.

Two rules that save real debugging time. First, never key on generated class names. Anything that looks like tF2Cxc, VwiC3b, or yuRUbf is a build artifact and will change. Prefer semantic structure (h3 inside an anchor), stable IDs (#search, #rhs), or visible text labels (Sponsored, People also ask).

Second, save fixtures. Store the raw HTML of a handful of SERPs and run your parser against them in CI. When Google changes the markup, you find out from a red test rather than from a month of empty rows in a dashboard.

Building a SERP monitoring workflow

Scraping one SERP is a script. Tracking rankings over time is a system, and the difference is mostly in the boring parts:

1. Keep a keyword inventory with its intended locale and device. The same keyword in gl=us and gl=de is two different tracked items, not one.
2. Schedule realistically. Daily is plenty for most keywords; hourly is usually vanity and multiplies your blocking risk by 24.
3. Store snapshots, not just current positions. You want to answer "when did we drop?" — that requires history, plus the raw HTML for at least a short window so you can re-parse after a markup change.
4. Separate "we were blocked" from "we don't rank." A missing result and a failed fetch must never write the same row. This is the mistake that quietly ruins rank-tracking datasets.
5. Alert on movement, not on data. Position changes beyond a threshold, entering or leaving page one, a competitor appearing in a feature you owned.
6. Track SERP features alongside positions. Ranking #1 under an AI Overview is not the same #1 it was two years ago, and only feature-level tracking shows that.

Our SERP monitoring use case walks through this pattern with the API, and the same scheduling and change-detection machinery generalizes to competitor content analysis and backlink analysis.

Frequently asked questions

Is it legal to scrape Google search results? Scraping publicly accessible pages is not a US computer-crime violation — hiQ v. LinkedIn and Van Buren settled that. But Google's Terms of Service prohibit automated access, which makes it a contract question with real exposure (blocks, cease-and-desist, civil claims) rather than a criminal one. Logging into a Google account first, or scraping personal data from results, both make your position meaningfully worse. See our web scraping legality guide, and talk to a lawyer before building a business on it.

Can I scrape Google search results with Python and BeautifulSoup? Technically yes, practically not reliably. A plain requests fetch from a cloud IP typically returns a consent screen or a CAPTCHA page — both with 200 OK — and the div.g selectors in most tutorials no longer match Google's markup. You need at minimum a real browser and residential IPs; see the Playwright section above.

How do I scrape Google without an API key? Fetch google.com/search directly with a headless browser, handle the consent interstitial, and parse with structural selectors. No key is needed because it's a public URL — but you'll pay in proxies, browser infrastructure, and maintenance instead. Note that &num=100 stopped working in September 2025, so bulk result collection now requires paginating with &start=.

What's the difference between scraping Google and the Custom Search API? The Custom Search JSON API is sanctioned, stable, and cheap ($5 per 1,000 queries after 100/day free), but it's a Programmable Search Engine — 10 results per call, no ads, no People Also Ask, no AI Overviews, and rankings that aren't guaranteed to match google.com. Scraping gives you the real consumer SERP with all its features, at the cost of blocking, maintenance, and ToS exposure. Use the API for site search and retrieval; scrape (or buy a SERP API) for rank tracking.

How many Google searches can I scrape before getting blocked? There's no published limit and no number worth quoting — it's adaptive and depends on IP reputation, query patterns, and timing. Datacenter IPs degrade fast; residential IPs last much longer. Assume you'll be throttled sooner than you expect, cache aggressively, and treat a CAPTCHA as a signal to back off rather than an obstacle to defeat.

Does WebScraping.AI have a Google SERP API? No. We return rendered HTML for URLs you supply — including Google's, which requires proxy=residential or proxy=stealth with js=true — and you parse the SERP yourself, or let /ai/fields extract named values from it. If you want parsed positions and SERP features out of the box, a dedicated SERP API like SerpApi or DataForSEO is a better fit. Where we help is when Google is one source among several in a pipeline and you'd rather not run separate vendors, or when you want to pay only for requests that actually succeed.

Can I scrape Google's AI Overviews? Sometimes. They load asynchronously and are frequently absent from a headless fetch even when a real user sees them, so treat "no AI Overview found" as unreliable data rather than evidence of absence. If AI Overview presence is central to your tracking, verify against a vendor that explicitly parses them and states a success rate.