How to Scrape Indeed: Build an Indeed Job Scraper

Indeed is the largest job board in the world, which makes it the default data source for anyone building a job listing aggregator, running salary benchmarking, or tracking hiring signals for sales and investment research. Indeed discontinued its public job search API years ago, so today the only way to get that data at scale is scraping. This guide covers what you can extract, why naive scrapers get blocked within minutes, and the approaches that work in production.

Key Takeaways

• Indeed has no public API for job search data — the former Publisher API is closed to new applicants
• Job listings are public and don't require login, but the site sits behind Cloudflare with aggressive bot detection
• Search result pages embed structured JSON, which is far more reliable than parsing HTML cards
• Datacenter proxies get blocked almost immediately; residential proxies with JavaScript rendering are the dependable path
• Hiring data has real business value: aggregation, salary analytics, lead generation, and labor market research

What Data Can You Get From Indeed?

Each job posting exposes a consistent set of fields:

• Job title, company, and location (including remote flags)
• Salary range — either employer-provided or Indeed's estimate
• Full job description with requirements and benefits
• Posting date and application volume hints
• Company rating pulled from Indeed's employer reviews

Search result URLs are easy to construct (https://www.indeed.com/jobs?q=data+engineer&l=Austin%2C+TX), and pagination is a simple start parameter incremented by 10 — the crawling logic is trivial. Getting the pages to load is the hard part.

The Cloudflare Problem

Indeed protects everything behind Cloudflare's bot management. A plain requests.get() returns a 403 challenge page, and even well-configured HTTP clients with browser-like headers fail once the JavaScript challenge kicks in. Headless browsers pass the challenge but get fingerprinted and rate-limited per IP shortly after.

In practice this means an Indeed scraper needs three things: JavaScript execution to pass challenges, residential proxies so each request comes from a plausible consumer IP, and retry logic for requests that still get flagged.

Scraping Indeed With a Web Scraping API

The most maintainable setup is to let a scraping API handle the anti-bot layer. Here's a working example with WebScraping.AI:

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://api.webscraping.ai/html",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://www.indeed.com/jobs?q=python+developer&l=Remote",
        "js": True,
        "proxy": "residential",
    },
)
soup = BeautifulSoup(response.text, "html.parser")

for card in soup.select("div.job_seen_beacon"):
    title = card.select_one("h2.jobTitle")
    company = card.select_one("[data-testid='company-name']")
    location = card.select_one("[data-testid='text-location']")
    print(title.get_text(strip=True), "|",
          company.get_text(strip=True), "|",
          location.get_text(strip=True))

For the full job description, follow each card's link to the viewjob page and request it the same way.

Skipping Selector Maintenance With AI Extraction

Indeed changes its markup regularly, and every change breaks CSS selectors. The AI field extraction endpoint avoids that by describing the data instead of locating it:

response = requests.get(
    "https://api.webscraping.ai/ai/fields",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://www.indeed.com/viewjob?jk=abc123",
        "fields[title]": "Job title",
        "fields[salary]": "Salary or salary range, empty if not listed",
        "fields[requirements]": "Key requirements as a comma-separated list",
        "js": True,
        "proxy": "residential",
    },
)
print(response.json())

Practical Tips

• Query the JSON island. Search pages embed a mosaic-provider-jobcards JSON blob with all card data — parsing it is more stable than HTML selectors
• Respect pacing. Even with residential proxies, keep concurrent requests per search moderate; Indeed throttles suspiciously fast crawls
• Deduplicate by job key. The jk parameter is Indeed's stable job ID — use it as your primary key across runs
• Track posting freshness. Listings expire and repost constantly; scrape on a schedule and diff by jk to detect real changes

Is Scraping Indeed Legal?

Job postings are public business information, and courts have generally been favorable to scraping publicly accessible data. That said, Indeed's terms of service prohibit automated access, salary and recruiter contact data can edge into personal-data territory, and the legal picture differs by jurisdiction. Our guide to web scraping legality covers the current case law — read it before building a commercial product on scraped job data.

Conclusion

Indeed is a high-value, high-friction target: the data is public and well-structured, but Cloudflare stands between you and it. Building and babysitting your own proxy-plus-headless-browser stack is a full-time job; a web scraping API with JavaScript rendering and residential proxies reduces the problem to writing parsers. WebScraping.AI's free trial is enough to test a full search-results-to-job-descriptions pipeline before committing.