Web scraping is used to collect data that exists publicly on the web but isn't available in any structured form: competitor prices, job postings, property listings, reviews, search rankings, regulatory filings. Companies scrape when the information they need to make a decision is visible on a page but not downloadable from an API.
This page is an index of the applications and what each one actually collects. For how scraping works mechanically — the fetch, parse, extract, store cycle — see what is web scraping.
The 12 applications
| # | Application | Data collected | Primary users |
| 1 | Price monitoring | Competitor prices, discounts, promotions | E-commerce, retail |
| 2 | Stock and inventory tracking | Availability, restock timing, depth | Retail ops, resellers |
| 3 | Product data aggregation | Specs, images, descriptions, categories | Marketplaces, comparison sites |
| 4 | B2B lead generation | Company names, roles, contact details | Sales teams |
| 5 | CRM enrichment | Firmographics appended to existing records | RevOps |
| 6 | Job listing aggregation | Titles, employers, locations, salary ranges | Job boards, recruiters |
| 7 | Salary benchmarking | Posted compensation by role and geography | HR, compensation teams |
| 8 | SERP monitoring | Rankings, snippets, competitor ad copy | SEO teams |
| 9 | Brand sentiment tracking | Reviews, ratings, mentions across platforms | Marketing, PR |
| 10 | Property listing aggregation | Prices, size, days on market | Real estate, proptech |
| 11 | Alternative data | Hiring velocity, store counts, inventory signals | Hedge funds, equity research |
| 12 | RAG knowledge bases | Clean article and documentation text | AI engineering teams |
What these have in common
Three patterns explain why these particular applications recur:
The data is public but unstructured. Every item above is visible to anyone with a browser. The value isn't access, it's assembling thousands of scattered observations into something queryable.
Freshness beats completeness. A price snapshot from last quarter is worthless; today's is actionable. Most scraping pipelines are scheduled jobs, not one-off extractions.
The alternative is manual work that doesn't scale. Checking fifty competitor SKUs daily is a person's entire job. Checking fifty thousand isn't possible at all without automation.
The newest application: data for AI
RAG knowledge bases and fine-tuning datasets are the fastest-growing use of scraped data. Retrieval systems need current text from sources a model wasn't trained on, which turns "convert this URL into clean readable text" into a routine engineering requirement. Our machine learning data guide covers building that pipeline.
Adjacent applications
Several narrower variants are worth knowing about: competitor content analysis for editorial planning, SEC filing monitoring for compliance and research, travel price tracking for fare and rate data, MAP compliance monitoring for brands enforcing minimum advertised pricing, and social media monitoring for reach and engagement measurement. The full use case index lists all of them.
Before you start
Two things to settle first. Check whether the source offers an API — if it does, use it, since it's faster and unambiguously permitted. And understand where the legal lines sit, particularly around authentication, personal data, and copyright; see is web scraping legal.
Building one of these? Get a free API key — 2,000 credits a month, no credit card — and browse the API documentation.
