API Scraping: How to Find and Use a Website's Hidden APIs

Most modern websites don't ship data in their HTML. They ship an empty shell, then fetch JSON from an internal endpoint and render it in the browser. If you're parsing that rendered HTML with selectors, you're reading a downstream artifact of data that was already clean two steps earlier — and you're rewriting your parser every time a designer touches a CSS class.

API scraping is the practice of finding those endpoints and calling them directly. This guide covers how to locate them, how to get past the authentication that guards them, how to page through results, and — the part most tutorials skip — how to keep the whole thing running six months later when the endpoint changes without warning.

Key Takeaways

• Most dynamic sites load their data from internal JSON endpoints. Find the endpoint and you get structured data instead of HTML you have to parse.
• The browser's Network tab, filtered to Fetch/XHR, finds these endpoints in about thirty seconds. "Copy as cURL" carries the full request — headers, cookies, body — into something you can replay.
• Hidden endpoints are private APIs: undocumented, unversioned, and free to change without notice. That's the tradeoff for the cleaner data.
• Authentication is usually a bearer token, a CSRF token, a session cookie, or a header like X-Requested-With — all visible in the same request you just copied.
• Build for change. A private API will break on someone else's schedule, so validate the shape of every response and alert on drift rather than discovering it through empty output.
• When there's no API behind the page — server-rendered HTML, or an endpoint locked behind bot protection — fall back to fetching the rendered page. That's what WebScraping.AI is for.

API scraping vs. HTML scraping

The two approaches solve the same problem from different ends.

The efficiency gap is the real argument. A product listing page might be 400 KB of HTML wrapping twenty products; the endpoint behind it returns maybe 8 KB of JSON with more fields than the page displays — internal IDs, stock counts, and variant data that never made it into the markup. You also skip the browser entirely, which means no headless Chrome, no rendering time, and a fraction of the memory.

The catch is stability, and it cuts the other way than people expect. HTML breaks visibly and often — a class rename, a wrapper div — but it breaks in small ways you can patch. A private API breaks rarely and totally: it works perfectly for eight months and then returns a 404 because the team shipped /v3/. Plan for both failure modes, not just the one you've seen recently.

Public vs. private APIs — know which one you're using

This distinction determines your legal exposure, your maintenance burden, and whether you should be doing this at all.

Public APIs are the ones a company publishes on purpose. They have documentation, a registration flow, versioned URLs, published rate limits, and a support channel. Crucially, they come with a contract: the provider has told you what the response looks like and committed, at least loosely, to keeping it that way. Terms of use apply, and you agreed to them when you signed up for the key.

Private APIs — also called internal or undocumented APIs — are the endpoints a site's own frontend uses to talk to its own backend. They were never meant for you. There's no documentation, no version guarantee, no support, and no announcement when they change. The team that owns them assumes the only caller is their own JavaScript, which means they're free to rename a field on a Tuesday afternoon.

The practical implications:

Always check for a public API first. It takes two minutes — look for /developers, /api, or /docs on the site, or search "<sitename>" API documentation. A documented API with a free tier beats a reverse-engineered one on every axis that matters: it won't break silently, you won't get blocked for it, and nobody has to argue about whether you were allowed to call it. People skip this step constantly and end up maintaining a fragile scraper against an endpoint the company would have handed them a key for.

Finding endpoints with the Network tab

The browser DevTools Network tab is the primary tool, and it handles the overwhelming majority of cases. The workflow:

1. Open DevTools — F12, or Cmd+Opt+I on macOS.
2. Go to the Network tab and confirm recording is on (the circle is red).
3. Click the clear button to drop the noise from the initial page load.
4. Filter to Fetch/XHR. This is the step that makes the whole thing tractable — it hides images, fonts, CSS, and analytics beacons, leaving only the requests the page's JavaScript made for data.
5. Interact with the page to trigger the data you want: scroll to load more, click page 2, apply a filter, open a product. Watch what appears.
6. Click a promising request and read the Response tab. If it's JSON containing the values you can see on screen, you've found it.

Sorting by response size helps when a page fires dozens of requests — the endpoint carrying the actual content is usually one of the largest JSON responses. The Preview tab renders JSON as a collapsible tree, which is much faster to scan than the raw payload.

URL shapes worth recognizing while you scan the list:

/api/...            the obvious one
/api/v1/  /api/v2/  versioned REST
/graphql            a GraphQL endpoint — see below
/_next/data/...     Next.js page data (JSON for a server-rendered React page)
/__data.json        SvelteKit
/ajax/  /json/      older conventions, still common
/wp-json/wp/v2/     WordPress REST API — extremely common, and documented

That last one is worth internalizing. A large share of the web runs WordPress, and /wp-json/wp/v2/posts is a real, documented, usually-open REST API sitting on sites whose owners have no idea it's there.

Copy as cURL — the step that saves the afternoon

Once you've found the request, right-click it → Copy → Copy as cURL. This is the single highest-leverage move in API scraping. You get the complete request as the browser sent it: URL, method, every header, every cookie, and the request body. Paste it into a terminal and it runs.

curl 'https://example.com/api/v2/products?page=1&limit=20' \
  -H 'accept: application/json' \
  -H 'x-requested-with: XMLHttpRequest' \
  -H 'authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGci...' \
  -H 'referer: https://example.com/products' \
  -b 'session_id=abc123'

Now start deleting things. Remove a header, re-run, see if it still works. You will usually find that most of what the browser sent is irrelevant and two or three headers actually matter. Knowing which ones matter is what turns a 40-line curl command into six lines of maintainable code — and it tells you exactly what your scraper has to reproduce.

When you've minimized it, convert it to your language of choice. Our free cURL converter turns a curl command into Python, JavaScript, PHP, Go, and others, and our guide to cURL commands for web scraping covers the flags in depth.

When it isn't in the Network tab

Sometimes the Fetch/XHR filter comes up empty. That means one of three things, and each has a different answer:

• The page is server-rendered. The data arrived in the initial HTML document. There is no API to find — scrape the HTML. This is common and not a failure.
• The data is embedded in the HTML as JSON. Check the document response for <script type="application/json">, <script type="application/ld+json">, or a window.__INITIAL_STATE__ / window.__NUXT__ assignment. This is the best of both worlds: one request, and the payload is already structured. Search the page source for a distinctive value you can see on screen — a price, a product name — and see what wraps it.
• The endpoint is built in JavaScript. Open the Sources tab and search across all files (Cmd+Opt+F / Ctrl+Shift+F) for fetch(, axios., /api/, or a path fragment you noticed. Bundled JS is minified but string literals survive minification, so URL paths are usually readable.

One approach worth not reaching for: brute-forcing paths with directory fuzzing tools against someone else's server. It generates thousands of failed requests, looks exactly like an attack in their logs, and is a reliable way to get your IP blocked or worse. Everything you actually need is in the traffic the site already sends you.

GraphQL endpoints

If you see requests going to a single /graphql URL with a POST body containing a query string, the site uses GraphQL. This changes the shape of the work in a way that's worth understanding.

curl 'https://example.com/graphql' \
  -H 'content-type: application/json' \
  -d '{"query":"{ products(first: 10) { id name price } }"}'

The upside is real: you specify exactly which fields you want, so you can ask for more than the page displays and skip everything you don't need. Because the query lives in the request body, you can often widen first: 10 to first: 100 and cut your request count by an order of magnitude — though servers frequently cap this, and a cap is a deliberate signal about acceptable load.

Two complications. First, many production GraphQL servers disable introspection (the query that returns the full schema), so you can't enumerate what's available — you're limited to adapting queries you've observed the site itself make. Second, persisted queries: some sites send only a hash identifying a server-stored query rather than the query text. You can still replay those with the variables changed, but you can't modify the fields being requested, because the server only accepts hashes it already knows.

In both cases, the practical technique is the same as REST: copy the request the page made, then modify the variables.

Mobile app APIs

Mobile apps talk to backends too, and their APIs are often cleaner than the web ones — designed for a constrained client, so they tend to return flatter payloads with fewer presentational fields. Sometimes an app's API exposes data the website doesn't.

Finding them means inspecting the app's HTTPS traffic, which is done with an intercepting proxy — mitmproxy (open source, CLI) and Charles Proxy (commercial, GUI) are the standard tools. You route the device's traffic through the proxy and install its certificate so it can read TLS.

Two honest caveats before you go down this road:

Many apps pin their certificates. Certificate pinning means the app only trusts a specific certificate and will refuse to connect through an intercepting proxy — that's the entire point of the feature. When you hit a pinned app, treat it as the developer telling you plainly that this traffic isn't meant to be inspected. Working around pinning means modifying the app or its runtime, which typically breaches the app's terms and, depending on jurisdiction and method, can carry legal exposure well beyond ordinary scraping. This guide doesn't cover it.

Mobile endpoints are the most private of private APIs. They frequently require app-specific signing — a header derived from a secret compiled into the binary — so a captured request may not be replayable at all. The maintenance burden is higher than the web equivalent for the same data.

For most projects, the web endpoint is the better target. Reach for mobile only when the data genuinely doesn't exist on the website.

Authentication patterns

Whatever the page did to authenticate, your scraper has to do too. The copied cURL command already contains the answer; you just need to recognize which pattern you're looking at.

Bearer tokens are the most common. The frontend obtains a JWT at login (or on page load, for anonymous sessions) and sends it on every request:

headers = {"Authorization": "Bearer eyJ0eXAiOiJKV1QiLCJhbGci..."}

The problem is expiry. Tokens are short-lived — often 15 to 60 minutes — so a hardcoded token works during development and dies overnight in production. Find where the token is issued (look for a request to /auth/token, /session, or /oauth/token early in the page load) and reproduce that call, rather than pasting a token that's already ticking down.

CSRF tokens guard state-changing requests. The token is minted server-side and delivered in the page HTML (often a <meta name="csrf-token"> tag) or in a cookie, then echoed back in a header like X-CSRF-Token. It's paired with the session cookie — one without the other fails. The sequence is: fetch the page, extract the token, send it back with the cookie.

Session cookies are the classic pattern. Use a session object so cookies persist across requests automatically instead of managing the header yourself:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
    "Accept": "application/json",
})

session.get("https://example.com/products")           # picks up cookies
data = session.get("https://example.com/api/v2/products", params={"page": 1}).json()

Our Python Requests guide covers sessions, headers, and connection reuse in more detail.

Header-only gates are the easiest case and surprisingly common. Some endpoints check nothing more than X-Requested-With: XMLHttpRequest, a matching Referer, or an Accept: application/json. These aren't security — they're a filter against casual access — but omitting them gets you a 403 that looks like a hard block. When a copied cURL works and your code doesn't, the difference is almost always a header.

A note on hygiene: tokens and cookies you extract are credentials. Keep them in environment variables, not in source control, exactly as you would your own API keys.

Pagination

Endpoints return a slice, and you need the rest. Three patterns cover nearly everything.

Page-based — the most common and the easiest to parallelize:

/api/products?page=3&per_page=20

Offset/limit — equivalent, expressed as a record count:

/api/products?offset=40&limit=20

Cursor-based — the response includes an opaque token pointing at the next slice:

/api/products?after=eyJpZCI6MTIzfQ&limit=20

Page and offset pagination can be requested out of order, so you can fetch pages concurrently once you know the total. Cursor pagination is strictly sequential — you cannot construct the next cursor, only read it from the previous response — but it's more reliable on data that's actively changing, because it doesn't skip or duplicate records when rows are inserted mid-crawl.

def fetch_all(session, url, per_page=100):
    page, results = 1, []
    while True:
        response = session.get(url, params={"page": page, "per_page": per_page})
        response.raise_for_status()
        batch = response.json().get("data", [])
        if not batch:
            break
        results.extend(batch)
        page += 1
    return results

Three things that bite in production:

• Termination. Stop on an empty batch, not on a page count you computed once — totals shift while you crawl. If the response carries total or has_more, use it.
• Silent caps. Many endpoints accept per_page=1000 and quietly return 100. Always check len(batch) against what you asked for; assuming you got 1000 when you got 100 means silently dropping 90% of the data.
• Deep pagination. Some APIs refuse offsets past a few thousand records. If page 500 returns an error, filter the query (by date, category, or ID range) into narrower slices rather than paging deeper.

Rate limits, timeouts, and retries

Private endpoints have undocumented limits that are enforced anyway. You find them by hitting them, so it's cheaper to be conservative from the start.

Set timeouts on every request. A request without a timeout can hang indefinitely, and one stuck connection can stall an entire pipeline. There are two distinct values: how long to wait for the connection, and how long to wait for the response.

# (connect timeout, read timeout)
response = session.get(url, timeout=(5, 30))

Retry on transient failures only. A 500, a 502, a 429, or a connection reset is worth retrying — the request might succeed a second later. A 401 or a 404 will fail identically every time; retrying it just wastes requests and makes you look worse in their logs.

Back off exponentially, with jitter. Doubling the wait after each failure gives an overloaded server room to recover. The random jitter matters more than it looks: without it, a batch of workers that fail together will retry together, producing a synchronized thundering herd that keeps the server down.

import random, time
import requests

RETRYABLE = {429, 500, 502, 503, 504}

def get_with_retry(session, url, attempts=4, **kwargs):
    for attempt in range(attempts):
        try:
            response = session.get(url, timeout=(5, 30), **kwargs)
            if response.status_code not in RETRYABLE:
                response.raise_for_status()
                return response
        except requests.exceptions.RequestException:
            if attempt == attempts - 1:
                raise
        if attempt < attempts - 1:
            time.sleep(2 ** attempt + random.uniform(0, 1))   # 1s, 2s, 4s (+jitter)
    raise RuntimeError(f"{url} failed after {attempts} attempts")

The loop above is deliberately explicit so the logic is visible, but in production you'd usually mount an adapter and let the library handle it — the Python Requests guide covers HTTPAdapter with Retry and the exception hierarchy you'd catch.

Honor Retry-After. When a 429 response includes that header, the server has told you exactly how long to wait. Using it beats any backoff curve you'd invent, and ignoring it is what escalates a soft rate limit into a ban.

Add a circuit breaker for long runs. If an endpoint has failed twenty times in a row, it's down — continuing to hammer it for another hour helps nobody. Stop calling it, wait, then try a single probe request before resuming.

The steady-state rule is simpler than any of this: keep concurrency low and add a delay between requests. One to two requests per second against a private endpoint is polite and rarely triggers anything. Racing at fifty is how you discover the rate limit and get the endpoint tightened for everyone.

Surviving schema changes

This is where API scraping projects actually die, and it's the part that never appears in the tutorial. Your scraper works for months, someone renames price to unit_price, and your pipeline writes None into every row without raising a single exception. You find out from a dashboard weeks later.

The root problem is that a schema change usually isn't an error. The request returns 200. The JSON parses. Only the meaning is wrong — which is precisely the failure that generic error handling misses.

Validate the shape of every response. Don't reach blindly into a dict. Assert that the fields you depend on exist and hold the type you expect, and fail loudly when they don't:

REQUIRED = {"id": (int, str), "name": str, "price": (int, float)}

def parse_product(item):
    for field, expected in REQUIRED.items():
        if field not in item:
            raise SchemaError(f"missing field: {field}")
        if not isinstance(item[field], expected):
            raise SchemaError(f"{field}: expected {expected}, got {type(item[field])}")
    return {"id": item["id"], "name": item["name"], "price": float(item["price"])}

A schema validation library (pydantic, jsonschema, marshmallow) does this more thoroughly and is worth adopting once you have more than a couple of endpoints.

Alert on drift, not just on exceptions. The most valuable signal is statistical. Track the null rate per field and the record count per run, and alert when either moves sharply. A field that was 2% null yesterday and is 100% null today is a renamed field, and this catches it on the first run rather than the tenth.

Watch for new fields too. An unexpected key in the response often means the API is being actively worked on — useful advance warning that something is about to change, and occasionally a new field you want.

Fail partially, not totally. If one record in a batch fails validation, log it and keep the other ninety-nine. A single malformed row shouldn't abort a six-hour crawl. But if the failure rate crosses a threshold — say 10% of records — stop and alert, because that's not a bad row, that's a changed schema.

Store the raw response. Keeping the untouched JSON alongside your parsed output costs very little and means that when a change does slip through, you can diff yesterday's payload against today's and see precisely what moved. Without it you're guessing at what the response used to look like.

Version-pin where you can. If the endpoint has a version in the path, you're somewhat insulated — /api/v2/ shouldn't change under you, and when /v3/ appears you can migrate deliberately. Monitor for the version bump: when /v3/ starts appearing in the site's own traffic, /v2/ is on a deprecation clock even if nobody told you.

Legality and etiquette

Using an undocumented endpoint isn't inherently illegal, but "the JavaScript called it, so I can call it" isn't a complete argument either. The ground rules (this is not legal advice — see our full guide to web scraping legality):

• Public data is generally scrapable. U.S. courts, notably hiQ v. LinkedIn, have held that accessing publicly available data doesn't violate the CFAA. That reasoning covers an endpoint your browser can reach without logging in.
• Authentication changes the analysis. If you had to create an account to get the token, you accepted terms of service, and calling the endpoint from a script may breach a contract you agreed to. That's a genuinely different legal position from anonymous public access.
• Personal data triggers GDPR and CCPA regardless of how you obtained it. "It was publicly accessible" is not, by itself, a lawful basis for processing.
• Copyright applies to content, not facts. Prices, specs, and availability are much safer ground than republishing articles or images.
• Check robots.txt. It's a statement of what the operator considers acceptable. It rarely names API paths explicitly, but it tells you how they feel about automated access.

The etiquette matters as much as the law, because it determines whether you keep your access:

• Prefer the documented public API when one exists.
• Keep request rates low. You are hitting infrastructure sized for their own frontend.
• Identify yourself in the User-Agent for large crawls, with a contact address. Operators are far more likely to email you than block you.
• Cache aggressively — don't re-fetch data that hasn't changed.
• Don't route around a block. If you get a 403 and the message says stop, that's an answer.

When to fall back to HTML scraping

API scraping isn't always the right tool, and forcing it wastes more time than it saves. Go back to fetching pages when:

• There's no API. Server-rendered sites — plenty of e-commerce, most news, anything on classic WordPress or Rails — put the data in the HTML document. Nothing to find.
• The endpoint is harder to reach than the page. Signed requests, rotating tokens, or app-specific headers can make an endpoint more expensive to maintain than a selector.
• The endpoint is behind bot protection. Some sites protect API routes more aggressively than pages, because legitimate traffic to them is exclusively their own frontend.
• You need what's rendered. Prices assembled client-side from multiple calls, or anything where the visual result is the thing you're measuring.
• You're scraping many sites. Reverse-engineering a hundred private APIs is a hundred separate maintenance commitments. One HTML pipeline with per-site selectors — or AI extraction — scales better.

That last case is where a scraping API earns its keep. WebScraping.AI handles the fetch, the JavaScript rendering, and the proxy rotation, so you get the rendered page without running browser infrastructure:

# Rendered HTML, JavaScript executed, through a residential proxy
curl "https://api.webscraping.ai/html?api_key=YOUR_KEY&url=https://example.com/products&js=true&proxy=residential"

# Just the text, for LLM pipelines
curl "https://api.webscraping.ai/text?api_key=YOUR_KEY&url=https://example.com/products&js=true"

And when you'd rather not maintain selectors at all, /ai/fields extracts structured data by description — which sidesteps both failure modes discussed above, since neither a CSS class rename nor a JSON field rename breaks a natural-language field definition:

curl "https://api.webscraping.ai/ai/fields?api_key=YOUR_KEY&url=https://example.com/product/1&fields[name]=Product+name&fields[price]=Price+with+currency&fields[stock]=In+stock+or+not"

There's a free trial with no card required, and failed requests aren't billed.

A pragmatic hybrid works well in practice: use the private API where you've found one and it's stable, and route the sites where you haven't through a scraping API. You don't have to pick one approach for the whole project.

Frequently asked questions

What is API scraping? API scraping means calling a website's data endpoints directly instead of parsing its rendered HTML. Most modern sites load content by fetching JSON from an internal API, and that endpoint is visible in the browser's Network tab. Calling it yourself returns structured data with no parsing, no JavaScript rendering, and a fraction of the bandwidth.

Is API scraping legal? Calling a public, unauthenticated endpoint is generally treated like visiting a public page — U.S. courts have held that accessing publicly available data doesn't violate the CFAA. It gets more complicated if you had to log in (you accepted terms of service), if the data is personal (GDPR and CCPA apply), or if you're republishing copyrighted content. See our web scraping legality guide.

How do I find a website's hidden API? Open DevTools, go to the Network tab, filter to Fetch/XHR, clear the list, and interact with the page to trigger the data you want. Click through the requests and read the Response tab until you find JSON containing the values you can see on screen. Then right-click that request and choose "Copy as cURL" to get a replayable version with all headers and cookies.

Why does my code get a 403 when the copied cURL command works? Almost always a missing header. Browsers send headers your HTTP client doesn't — commonly Referer, X-Requested-With: XMLHttpRequest, Accept: application/json, or a realistic User-Agent. Start from the full copied cURL command, confirm it works, then remove headers one at a time until it breaks. The last one you removed is the one that matters.

How do I keep a scraper working when the API changes? Assume it will. Validate that every response contains the fields and types you expect, and raise an error rather than writing nulls when it doesn't. Track the null rate per field and the record count per run so a renamed field shows up as an alert on the first run instead of a gap you find weeks later. Store raw responses so you can diff them when something does change.

What's the difference between a public and a private API? A public API is documented, versioned, and offered deliberately, with published rate limits and a support channel. A private API is what a site's own frontend uses — undocumented, unversioned, and free to change without notice. Private APIs often expose more data, but they carry the full maintenance burden. Always check whether a public API exists first.

Can I scrape a mobile app's API? Sometimes. Apps talk to backends the same way websites do, and their APIs are often cleaner. But many apps pin their TLS certificates specifically to prevent traffic inspection, and working around that typically breaches the app's terms. Mobile endpoints also frequently require request signing derived from a secret in the binary, so captured requests may not be replayable at all. The web endpoint is usually the better target.

What if the site has no API at all? Then the data is in the HTML, either server-rendered or embedded as JSON in a <script> tag — check the page source for __INITIAL_STATE__ or application/json before assuming you need selectors. If it's genuinely server-rendered, scrape the HTML, and use a service like WebScraping.AI if the page needs JavaScript execution or proxy rotation to fetch reliably.