Real Estate Scraping: How to Collect Property Data in 2026

There is no self-serve API for US residential listings. Zillow retired its public Web Services API in September 2021, the MLS feeds that hold the authoritative data are gated behind a real estate licence, and everything else on the market is a scraper with an API-shaped wrapper. So real estate scraping is what most teams end up doing — and it works better than the anti-bot reputation suggests. When we tested on 28 July 2026, Zillow and Redfin both returned full listing pages through residential proxies on the first try. Realtor.com did not. Below: what we measured, where the data actually lives, and the licensing rules that make this area genuinely riskier than scraping product prices.

Key Takeaways

• Zillow and Redfin both loaded cleanly through residential proxies with JavaScript rendering on 28 July 2026. Realtor.com returned a 429 soft-block on both residential and stealth.
• Do not parse Zillow's HTML. Every listing on a search page sits in __NEXT_DATA__ under props.pageProps.searchPageState.cat1.searchResults.listResults — 41 fully structured records on the page we pulled, against a stated total of 5,858 for the metro.
• Zillow's robots.txt disallows /homes/ and /api/ but leaves city pages like /austin-tx/ and most of /homedetails/ crawlable — the opposite of what most people assume.
• MLS data is licensed, not public. RESO Web API 2.0 access requires broker or agent membership or an approved vendor agreement. There is no tier you can buy your way into as an unlicensed developer, and IDX display rules bind what you may show even when you have access.
• Pagination beats detail-fetching. A search page hands you price, beds, baths, area, coordinates, daysOnZillow and tax assessed value per listing — fetch detail pages only for the records that changed.
• Agent names and phone numbers are personal data. Aggregate market analytics carry a fraction of the legal exposure of a scraped agent contact list.

Is there a real estate data API?

Three routes exist, and only one of them is open to you.

MLS via the RESO Web API. The multiple listing services hold the authoritative data — full listing history, days on market, sold prices, agent remarks. RESO Web API 2.0 (OData over REST, OAuth 2.0) is the current standard; the older RETS protocol has been retired. Access requires that your business hold a real estate licence, affiliate with a licensed broker, or partner with an IDX vendor that holds one. This is the part people gloss over: it is a membership and licensing relationship, not an API subscription. If you have it, IDX policy also constrains display, attribution and refresh frequency on what you publish.

Zillow. The consumer Web Services API was retired on 30 September 2021. Bridge Interactive, its successor, is aimed at MLS-connected platforms and licensed brokers; access is an application with a use-case review, not a signup. Independent developers and early-stage proptech generally do not qualify.

Commercial data vendors. ATTOM, Datafiniti, CoreLogic and similar sell property datasets under licence. Real coverage, real cost, and the licence terms usually restrict redistribution — read them before you build a product on top.

Everything else marketed as a "Zillow API" or "real estate data API" is a scraper someone else operates. Which is a reasonable thing to buy or to build, as long as you are clear that is what it is.

Which real estate portals can you actually scrape?

Measured, not guessed. Each row is a live request through WebScraping.AI on 28 July 2026 with js=true and a 30-second timeout:

Three things follow. Portal posture varies enormously, so test each target rather than assuming a uniform "real estate sites are hard" — the reputation is mostly inherited from a few years ago. Realtor.com's block is a soft rate-limit page rather than a hard challenge, so lower concurrency and slower pacing are the first thing to try. And these are single-request results: sustained crawling at volume will hit different limits than a one-off fetch, which is why pacing matters more than proxy tier once you are past the first page.

If you are unsure where to start on proxy selection, our proxy types guide covers the datacenter-to-residential escalation path.

How to scrape Zillow listings

Zillow is a Next.js app, and the entire search result set is serialised into the page as JSON before any of it becomes HTML. Parsing the rendered cards is strictly worse: the class names churn, the JSON does not.

import json
import re
import requests

def zillow_search(url, api_key):
    resp = requests.get(
        "https://api.webscraping.ai/html",
        params={
            "api_key": api_key,
            "url": url,
            "js": "true",
            "proxy": "residential",
            "timeout": 30000,
        },
    )
    resp.raise_for_status()

    blob = re.search(
        r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>',
        resp.text,
        re.S,
    ).group(1)
    state = json.loads(blob)["props"]["pageProps"]["searchPageState"]

    total = state["categoryTotals"]["cat1"]["totalResultCount"]
    for home in state["cat1"]["searchResults"]["listResults"]:
        info = home.get("hdpData", {}).get("homeInfo", {})
        yield {
            "zpid": home["zpid"],
            "price": home.get("unformattedPrice"),
            "address": home.get("address"),
            "beds": home.get("beds"),
            "baths": home.get("baths"),
            "sqft": home.get("area"),
            "lat": home["latLong"]["latitude"],
            "lng": home["latLong"]["longitude"],
            "days_on_market": info.get("daysOnZillow"),
            "tax_assessed": info.get("taxAssessedValue"),
            "url": home["detailUrl"],
            "total_in_region": total,
        }

for home in zillow_search("https://www.zillow.com/austin-tx/", "YOUR_API_KEY"):
    print(home["address"], home["price"], home["days_on_market"])

Notes from the page we actually pulled. listResults held 41 records while categoryTotals.cat1.totalResultCount reported 5,858 for Austin — Zillow caps a search at roughly 20 pages, so wide markets need geographic subdivision (by ZIP, price band, or map bounding box) rather than deeper pagination. The same payload also reported restrictedListingCount: 120, listings withheld from the response entirely; your counts will not reconcile with the headline number, and that is expected rather than a scraping bug.

zpid is Zillow's stable property identifier. Use it as your primary key — addresses are not unique enough and their formatting drifts.

Extracting fields without maintaining selectors

The JSON approach is fastest when you have already reverse-engineered a portal. It also breaks the day the portal reshapes its state tree, and it needs redoing per site. For detail pages, or for a long tail of smaller portals where per-site parsers are not worth writing, describe the fields instead:

import requests

resp = requests.get(
    "https://api.webscraping.ai/ai/fields",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://www.zillow.com/homedetails/12821-Hughes-Park-Rd-Austin-TX-78732/70325038_zpid/",
        "fields[price]": "Listing price in USD, digits only",
        "fields[beds]": "Number of bedrooms",
        "fields[baths]": "Number of bathrooms",
        "fields[living_area_sqft]": "Living area in square feet, digits only",
        "fields[year_built]": "Year the home was built",
        "fields[days_on_market]": "Days on Zillow, digits only",
        "fields[hoa_fee]": "Monthly HOA fee in USD, empty string if none listed",
        "js": "true",
        "proxy": "residential",
    },
)
print(resp.json())

That exact call returned, on 28 July 2026:

{"result": {"price": "9900000", "beds": "6", "baths": "13",
            "living_area_sqft": "15394", "year_built": "2004",
            "days_on_market": "13", "hoa_fee": ""}}

The same field descriptions run unchanged against Redfin, a regional broker site, or a European portal — which is the point when you are normalising a multi-portal aggregator into one schema. AI extraction adds 5 credits per request on top of the fetch, so it is worth reserving for detail pages and the long tail rather than applying to every search result you already get structured for free. The AI scraping page has the full parameter reference.

Building a change-tracking pipeline

Snapshots are close to worthless; the analytical value is entirely in the deltas.

1. Crawl search pages on a schedule. Daily is enough for almost every market. Store zpid, price, status and timestamp. This is cheap — one request yields ~40 listings.
2. Diff against the previous run. New IDs are fresh supply. Missing IDs are delistings, the best available proxy for a sale or a let. Price changes are the market telling you something before any index does.
3. Fetch detail pages only for changed records. Detail fetches are where cost accumulates; gating them on a diff typically cuts request volume by an order of magnitude on a mature crawl.
4. Store history, not current state. A price-cut series and a days-on-market distribution are the products; a table of today's listings is not. This is the difference between a listings mirror and genuine rental market analysis.
5. Normalise early. One schema for price, area, rooms and status across every portal, applied at ingest. Retrofitting normalisation onto months of accumulated raw payloads is miserable.

The same shape works for commercial real estate, where listing counts are lower but time-on-market and asking-rent movements carry more signal per observation, and for alternative data built on housing supply.

What does Zillow's robots.txt say?

More permissive than its reputation, and specific about what it minds. Under User-agent: *:

• Disallow: /homes/ — the filtered search paths, with a narrow allowlist for national-level pages like /homes/for_sale/$ and their paginated *_p/$ forms
• Disallow: /api/ — the internal endpoints
• Disallow: /*_rect, /*_rid, /*_sort/ — map-bounds and sort permutations, i.e. the infinite-URL-space traps
• /homedetails/ is largely allowed, excluding only *view=owner, ?altId and rental-manager resources

City pages such as /austin-tx/ are not disallowed. The file also opens with a pointer to Zillow's Terms of Use — a reminder that robots.txt governs crawler etiquette while the ToS governs everything else, and the ToS is the document that prohibits automated access. Getting a 200 is not the same as being authorised.

The legal picture, honestly

Real estate is not the low-risk scraping target that price monitoring is. Three distinct constraints stack:

MLS licensing is real and enforced. This is the one most commonly ignored. MLS data is licensed to participants under agreements that restrict redistribution, display and derived use — including in many cases what you may do with data you obtained from a portal that got it from the MLS. Republishing listing content sourced downstream of an MLS does not escape the licence just because you scraped it from a public page. If your product touches MLS-derived content, this is a question for a lawyer, not a blog post.

Portal terms prohibit automated access. Zillow's, Redfin's and Realtor.com's terms all do. In the US, hiQ v. LinkedIn means scraping a public page is not by itself a Computer Fraud and Abuse Act violation, but breach of contract is a separate and live claim, and it is the one plaintiffs now bring. Republishing listings competes with the portal directly and reliably attracts attention; aggregate analytics rarely does.

Agent details are personal data. Names, direct phone numbers and email addresses are personal data under GDPR and under a growing set of US state privacy laws. A scraped agent contact list is a materially different risk profile from a price-and-attributes dataset, and it is the single easiest thing to drop from your schema. Do that unless you have a lawful basis you can articulate.

Our web scraping legality guide covers the case law behind this. It is worth twenty minutes before you commit engineering time.

Getting started

The practical path: pick one metro on one portal, confirm you can fetch it, parse the embedded JSON rather than the HTML, and get a daily diff running before you widen coverage. Most real estate scraping projects fail on normalisation and history, not on anti-bot — solving the interesting problem first is a trap.

WebScraping.AI's free tier is 2,000 credits a month with no credit card, and failed requests never consume credits, so probing a portal's current posture costs nothing. That is enough to run the Zillow pipeline above end to end across a small market before you decide it is worth paying for. The docs have the full endpoint and parameter reference.