How to Scrape Indeed in 2026: What an Indeed Scraper Can and Can't Do

Indeed has no self-serve API, and as of 28 July 2026 every request we sent to indeed.com — datacenter, residential, and stealth proxies, with JavaScript rendering, including the plain homepage — came back as HTTP 403 behind a Cloudflare "Additional Verification Required" interstitial. If you are planning an Indeed scraper, that is the fact to plan around. This guide covers what Indeed's own rules permit, where the data lives when a page does load, and the job-data routes that still work reliably enough to build a product on.

Key Takeaways

• There is no public Indeed API. The Publisher API closed to new applicants in October 2022 and was retired; the surviving Indeed Apply and Sponsored Jobs APIs are partner-only and require approval through partners.indeed.com.
• Indeed blocked us on every proxy tier we tried on 28 July 2026 — including the homepage — with a Cloudflare managed challenge, not a rate limit. This is a site-wide posture, not a per-endpoint one.
• Indeed's robots.txt disallows /viewjob? and /graphql for all user agents, while leaving /jobs?q= search pages allowed. A separate block bans GPTBot, ClaudeBot, CCBot, Diffbot, Bytespider and others from /jobs and /viewjob outright.
• When pages do render, the data is in JSON, not CSS classes: window.mosaic.providerData["mosaic-provider-jobcards"] on search pages, _initialData on job pages.
• Applicant tracking system boards are the route that actually works. Greenhouse, Ashby and Lever publish free, unauthenticated JSON endpoints — one request to Stripe's Greenhouse board returned 532 live postings when we tested it.
• Failed requests are free on WebScraping.AI, so probing a hostile target like Indeed to see whether the posture has changed costs you nothing.

Does Indeed have an API?

No — not one you can sign up for.

Indeed's Publisher API was the standard route for job-board aggregators for years. It stopped accepting new publishers in October 2022 and was subsequently retired; keys have not been minted since 2024. What remains under partners.indeed.com are the Indeed Apply and Sponsored Jobs APIs, which serve employers and ATS vendors pushing jobs into Indeed, not developers pulling search results out. Access is a partnership process, not a signup form, and it is scoped to your own postings.

So every "Indeed API" you find on a marketplace today is a third-party scraper with an API-shaped wrapper. That is worth knowing before you buy one, because the constraint below applies to them too.

Can you scrape Indeed in 2026?

Here is what we actually measured. Each row is a real request through WebScraping.AI on 28 July 2026, with js=true and a 30-second timeout:

The response body was a Cloudflare challenge page carrying a Ray ID and the text "Additional Verification Required", not a 429 or a throttle notice. That distinction matters: a rate limit means slow down, a managed challenge means the fingerprint was rejected before any content was served.

Two honest caveats. This is one provider's IP pools in one window — your results with a different vendor, a different country, or a residential ASN Indeed has not yet flagged may differ, and Indeed's posture has loosened and tightened repeatedly over the years. And an interstitial is not a permanent wall: the point is that in July 2026 you cannot treat Indeed as a target you fetch and parse, the way the older generation of tutorials assumed.

Re-test it yourself before writing off the target. Failed requests do not consume credits, so a probe costs nothing:

import requests

resp = requests.get(
    "https://api.webscraping.ai/html",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://www.indeed.com/jobs?q=python+developer&l=Remote",
        "js": "true",
        "proxy": "stealth",
        "timeout": 30000,
    },
)
print(resp.status_code)
print(resp.text[:500])   # a Cloudflare Ray ID here means still blocked

If you are new to the proxy-tier decision this implies, our guide to proxy types covers when datacenter is enough and when it isn't.

What does Indeed's robots.txt actually allow?

Worth reading before anyone tells you scraping Indeed is "just public data". Under User-agent: *, Indeed's robots.txt allows the root but explicitly disallows:

• /viewjob? and /m/viewjob? — the job detail pages, the ones carrying descriptions and salary
• /graphql — the internal API path
• /api/getrecjobs, /advanced_search, /alert, /company/*, and the localised /jobs/<COUNTRY>/ and /jobs/title paths

Search result pages at /jobs?q=...&l=... are not disallowed for generic agents. So Indeed's stated crawl policy is roughly: the search index is open, the individual postings are not.

A second block — covering GPTBot, CCBot, anthropic-ai, ClaudeBot, Bytespider, Baiduspider, Diffbot, AI2Bot, Meta-ExternalAgent and a dozen more — disallows /jobs and /viewjob entirely. If you are collecting training data or building a RAG knowledge base, Indeed has named your use case and declined it.

robots.txt is not law, and it is not Indeed's Terms of Service either — but it is the clearest public statement of intent you will get, and courts and counsel both read it.

Where does Indeed's data live on the page?

If you do get a page to render — via a partner arrangement, a residential ASN that still gets through, or a browser session you drive yourself — do not parse the HTML cards. Indeed's class names churn, but the embedded JSON has been stable for years:

• Search pages: window.mosaic.providerData["mosaic-provider-jobcards"], extracted with a regex like window\.mosaic\.providerData\["mosaic-provider-jobcards"\]=(\{.+?\});. It carries the full card payload — title, company, location, salary, and the jk job key.
• Job detail pages: an _initialData={...}; assignment, with the posting under jobInfoWrapperModel.jobInfoModel.

Use jk as your primary key. It is Indeed's stable job identifier, it survives repostings, and diffing on it is the only sane way to distinguish a genuinely new listing from the same role recycled.

We could not verify these keys first-hand in this round because we never got past the challenge — they are current as of third-party guides updated in April 2026. Confirm against a live page before you build on them.

The route that actually works: ATS job boards

Here is the part most Indeed tutorials skip. Indeed does not originate most of its postings — it ingests them from employers' applicant tracking systems. Those ATS platforms publish the same jobs through free, unauthenticated, documented JSON endpoints that nobody is trying to block.

We tested all three below on 28 July 2026. Stripe's Greenhouse board returned 532 postings in a single request; Ramp's Ashby board returned 118 with full descriptions attached.

import requests

def greenhouse_jobs(token):
    url = f"https://boards-api.greenhouse.io/v1/boards/{token}/jobs"
    for job in requests.get(url, timeout=30).json()["jobs"]:
        yield {
            "source": "greenhouse",
            "company": job.get("company_name"),
            "title": job["title"],
            "location": job["location"]["name"],
            "url": job["absolute_url"],
            "updated_at": job["updated_at"],
        }

def ashby_jobs(token):
    url = f"https://api.ashbyhq.com/posting-api/job-board/{token}"
    for job in requests.get(url, timeout=30).json()["jobs"]:
        yield {
            "source": "ashby",
            "title": job["title"],
            "location": job["location"],
            "remote": job["isRemote"],
            "url": job["jobUrl"],
            "description": job["descriptionPlain"],
        }

for job in greenhouse_jobs("stripe"):
    print(job["title"], "|", job["location"])

The trade-off is honest: you need the board token for every company you want to track, so this is a coverage problem rather than an access problem. But the tokens are usually just the company slug, they are discoverable from any careers page, and coverage you control beats coverage that 403s. For a job listing aggregator, a curated list of a few thousand ATS boards will outperform a fragile Indeed scraper on freshness, completeness of descriptions, and uptime.

Scraping careers pages that aren't on a known ATS

Plenty of employers run bespoke careers pages. Those are ordinary scraping targets — JavaScript-heavy, but not adversarial — and this is where a web scraping API earns its place. Describe the fields instead of writing per-site selectors, and the same code works across every layout:

import requests

resp = requests.get(
    "https://api.webscraping.ai/ai/fields",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/careers/senior-backend-engineer",
        "fields[title]": "Job title",
        "fields[location]": "Job location, or 'Remote'",
        "fields[salary]": "Salary or salary range, empty string if not listed",
        "fields[posted_at]": "Posting date in YYYY-MM-DD, empty string if absent",
        "fields[requirements]": "Key requirements as a comma-separated list",
        "js": "true",
        "proxy": "residential",
    },
)
print(resp.json())

Because the extraction is described rather than selector-bound, a careers page redesign does not break your pipeline — the usual failure mode for a salary benchmarking dataset that has to stay comparable across quarters. See the AI scraping docs for the full parameter set.

What about Indeed's mobile GraphQL endpoint?

You will find open-source job scrapers — JobSpy is the best known — that skip the website entirely and call apis.indeed.com with headers lifted from Indeed's iOS app, including a hard-coded Indeed-API-Key extracted from the binary. It works, which is why it is popular.

Be clear about what it is. That key was not issued to you, the endpoint is not documented for third-party use, Indeed's robots.txt disallows /graphql, and Indeed's Terms of Service prohibit automated access to the service. Using an app's private credentials to pull data at scale is a materially weaker position than scraping a public page, and it is not one we would build a commercial product on. We are describing the technique because you will encounter it, not recommending it.

Is scraping Indeed legal?

Scraping publicly accessible data has generally fared well in US courts — hiQ v. LinkedIn established that accessing a public website does not by itself violate the Computer Fraud and Abuse Act. That is a narrower holding than it is usually made out to be, and it does not clear the specific hazards here:

• Indeed's Terms of Service prohibit automated access. Breach of contract is a separate claim from CFAA, and it is the one that actually gets litigated against scrapers now. Circumventing a bot-management challenge weakens any argument that access was authorised.
• Salary and recruiter data edges into personal data. Named recruiter contacts are personal data under GDPR and, depending on the field combination, under US state privacy laws too. Aggregate salary statistics are a much safer product than a contact database — relevant if you are using job data for sales intelligence.
• Republishing full postings competes with Indeed directly. Derived analytics rarely draws legal attention; a mirror of the listings reliably does.

Our guide to web scraping legality covers the case law in detail. Read it before you commit engineering time, and do not treat "it's public data" as a defence on its own.

What to build instead

If you need job market data in 2026, the honest ordering is:

1. ATS boards first (Greenhouse, Ashby, Lever, Workday, SmartRecruiters). Free, structured, permitted, and the original source of the data Indeed aggregates.
2. Direct careers pages for employers not on a known ATS — a straightforward headless browser or scraping-API job.
3. Job boards that don't run a managed challenge. Coverage varies by market, and posture changes; test before you commit.
4. Indeed last, and only with a partner agreement or a lawyer's sign-off.

Building the first two well gets you a better dataset than an Indeed scraper that spends its life in a retry loop. WebScraping.AI's free tier is 2,000 credits a month with no credit card, and failed requests never cost anything — enough to run the ATS pipeline above end to end and confirm the coverage before you pay for anything. The docs have the full endpoint reference.