# WebScraping.AI

> Web scraping API: send a URL, get back rendered HTML, clean text, CSS-selected fragments, or AI answers and structured JSON extracted from the page. JavaScript rendering, rotating datacenter/residential/stealth proxies and retries are handled server-side. Also parsed Google search results (`/serp`) and structured data for supported social sites (`/data`).

- API base URL: `https://api.webscraping.ai`; every endpoint is a GET authenticated with the `api_key` query parameter.
- Pricing is in credits per successful request (failed requests are free). The free plan includes 2,000 credits per month (2 concurrent requests); see [Rules & Pricing](https://webscraping.ai/docs/rules.md) for per-option costs.
- Machine-readable spec: [OpenAPI (YAML)](https://webscraping.ai/openapi.yml), [OpenAPI (JSON)](https://webscraping.ai/openapi.json).
- Remote MCP server for AI assistants: `https://mcp.webscraping.ai/mcp` (Streamable HTTP, OAuth sign-in, no API key needed).
- Official SDKs for Python, JavaScript, PHP, Ruby, Go, Java and C# / .NET, an n8n node, and a CLI with an agent skill: see [SDKs](https://webscraping.ai/docs/sdks.md) and [Using with AI Tools](https://webscraping.ai/docs/ai-tools.md).
- Every page on this site is available as Markdown: append `.md` to its path (`/index.md` for the home page) or request it with `Accept: text/markdown`.

## Docs

- [Full API documentation](https://webscraping.ai/docs.md): every endpoint, parameter and error on one page
- [Full documentation as one file](https://webscraping.ai/llms-full.txt): the API docs plus the main product pages
- [Quick Start](https://webscraping.ai/docs/quick-start.md)
- [Authentication](https://webscraping.ai/docs/authentication.md)
- [Base URL](https://webscraping.ai/docs/base-url.md)
- [Rules & Pricing](https://webscraping.ai/docs/rules.md)
- [Success Rates & Retrying](https://webscraping.ai/docs/success-rates.md)
- [Best Practices](https://webscraping.ai/docs/best-practices.md)
- [Sending Cookies](https://webscraping.ai/docs/cookies.md)
- [Official SDKs](https://webscraping.ai/docs/sdks.md)
- [Using with AI Tools](https://webscraping.ai/docs/ai-tools.md)
- [Proxy Mode](https://webscraping.ai/docs/proxy-mode.md)
- [Ask Questions About a Page](https://webscraping.ai/docs/ai-question.md)
- [Extract Structured Fields](https://webscraping.ai/docs/ai-fields.md)
- [Get Page HTML](https://webscraping.ai/docs/html.md)
- [Get Page Text](https://webscraping.ai/docs/text.md)
- [Get Selected HTML](https://webscraping.ai/docs/selected.md)
- [Get Multiple Selected Areas](https://webscraping.ai/docs/selected-multiple.md)
- [Search Results (SERP)](https://webscraping.ai/docs/serp.md)
- [Structured Data](https://webscraping.ai/docs/data.md)
- [Account Information](https://webscraping.ai/docs/account.md)
- [Error Codes](https://webscraping.ai/docs/errors.md)
- [Parameters Reference](https://webscraping.ai/docs/parameters.md)
- [Full API Reference](https://webscraping.ai/docs/openapi.md)
- [Home page and pricing](https://webscraping.ai/index.md)

## Scraping APIs

- [AI Web Scraping API - Extract Data from Any Website with AI](https://webscraping.ai/ai-web-scraping.md): AI web scraping in one API call: ask questions about any page or extract typed JSON fields. JS rendering and rotating proxies included. From $29/mo.
- [Amazon Scraping API - Product Data as JSON with AI Extraction](https://webscraping.ai/amazon-scraping-api.md): Scrape Amazon product pages, search results, and reviews into JSON you define. AI extraction, rotating residential proxies, no parser to maintain. From $29/mo.
- [Google SERP API - Parsed Search Results as JSON](https://webscraping.ai/google-serp-api.md): Google SERP API: send a query, get parsed organic results, related searches, and pagination as JSON. Flat 15 credits per search; failed searches are free.
- [Instagram Scraper API - Profiles, Posts & Reels](https://webscraping.ai/instagram-scraper-api.md): Instagram scraper API: send a profile, post or reel URL, get followers, captions, likes, comments and media as JSON. No login. 15 credits per page, free tier.
- [LinkedIn Scraper API - Profiles, Companies & Jobs](https://webscraping.ai/linkedin-scraper-api.md): LinkedIn scraper API: send a public profile, company or job URL, get structured JSON. No cookies or LinkedIn account. 15 credits per page, free tier.
- [Reddit Scraper API - Posts, Comments & Subreddits](https://webscraping.ai/reddit-scraper-api.md): Reddit scraper API: send a post, subreddit or user URL, get posts and nested comment trees as JSON. No Reddit API key. 50 credits per page, free tier.
- [Social Media Scraper API - YouTube, TikTok, Instagram & More](https://webscraping.ai/social-media-scraper-api.md): Social media scraper API: send a YouTube, TikTok, X, LinkedIn, Instagram or Reddit URL, get structured JSON. 15 credits per page (Reddit 50), free tier.
- [TikTok Scraper API - Scrape TikTok Videos & Profiles](https://webscraping.ai/tiktok-scraper-api.md): TikTok scraper API: send a TikTok video or profile URL, get likes, plays, shares, music and follower counts as JSON. 15 credits per page, free tier.
- [Twitter Scraper API - Scrape Tweets & X Profiles](https://webscraping.ai/twitter-scraper-api.md): Twitter scraper API for X: send a tweet or profile URL, get text, likes, reposts, views, media and author stats as JSON. No X API plan. 15 credits a page.
- [YouTube Scraper API - Videos, Channels & Transcripts](https://webscraping.ai/youtube-scraper-api.md): YouTube scraper API: send a video, channel or playlist URL, get views, likes, subscribers and transcripts as JSON. 15 credits per page, free tier.

## Integrations

- [Web Scraping Integrations - Zapier, n8n, MCP, Make & More](https://webscraping.ai/integrations.md): Integrate WebScraping.AI with Zapier, n8n, MCP, Make, Pipedream and more. Automate web scraping workflows with no-code tools and AI assistants.
- [Web Scraping MCP Server for Claude, Cursor, and AI Agents](https://webscraping.ai/integrations/mcp-server.md): Give AI assistants web scraping, Google search and site data tools via MCP: JS rendering, proxies, AI extraction. Hosted server with OAuth, no API key needed.
- [n8n Web Scraping - JS Rendering, Proxies, AI Extraction](https://webscraping.ai/integrations/n8n.md): Scrape any website in n8n with the official WebScraping.AI node: JavaScript rendering, rotating proxies, and AI extraction — no brittle CSS selectors.

## Free tools

- [Free Web Scraping Tools](https://webscraping.ai/tools.md): Free developer tools for web scraping: selector tester, curl converter, HTML to Markdown converter, MCP server, CLI, and n8n node. No signup required.
- [curl Converter - Convert curl Commands to Code Online](https://webscraping.ai/tools/curl-converter.md): Free online curl converter. Turn curl commands into Python, JavaScript, Node.js, PHP, Go, Java, Ruby, or C# code — right in your browser, nothing uploaded.
- [HTML to Markdown Converter - Free Online Tool & API](https://webscraping.ai/tools/html-to-markdown.md): Free HTML to Markdown converter. Paste HTML or convert any URL to clean Markdown for LLMs and RAG pipelines — in your browser or via the URL-to-Markdown API.
- [XPath Tester & CSS Selector Tester - Free Online Tool](https://webscraping.ai/tools/selector-tester.md): Free online XPath tester and CSS selector tester. Run selectors against pasted HTML or a live URL rendered with headless Chrome — no signup.

## Use cases

- [Web Scraping API Use Cases by Industry](https://webscraping.ai/use-cases.md): See how teams use WebScraping.AI for price monitoring, lead generation, real estate, SEO tracking, and more - web scraping use cases by industry.
- [Alternative Data for Investment: Web Data Collection API](https://webscraping.ai/use-cases/alternative-data-investment.md): Collect alternative data for investment research. Extract web-based signals, sentiment, and hiring data as structured JSON with WebScraping.AI.
- [Lead Scraping API for B2B Lead Generation](https://webscraping.ai/use-cases/b2b-lead-generation.md): Scrape B2B leads from directories and company websites. Extract company data, contact details, and emails as structured JSON with WebScraping.AI.
- [Backlink Monitoring & Link Analysis API for SEO](https://webscraping.ai/use-cases/backlink-analysis.md): Monitor backlink pages, extract outbound links and anchor text, and find link building prospects. Pull link data from any page with WebScraping.AI.
- [Brand Sentiment Analysis API - Brand Monitoring Tool](https://webscraping.ai/use-cases/brand-sentiment-tracking.md): Brand sentiment analysis from social posts, reviews, and forums. Monitor public perception and catch reputation shifts early with WebScraping.AI.
- [Commercial Real Estate Data API - CRE Property Intelligence](https://webscraping.ai/use-cases/commercial-real-estate.md): Collect commercial real estate listings, lease rates, and property data. Build comprehensive CRE databases with WebScraping.AI.
- [Review Monitoring API - Google Reviews, Glassdoor & More](https://webscraping.ai/use-cases/company-reviews-monitoring.md): Review monitoring across Google Reviews, Glassdoor, Trustpilot, and more. Extract ratings and review sentiment as structured JSON with WebScraping.AI.
- [Competitor Monitoring & Content Analysis API](https://webscraping.ai/use-cases/competitor-content-analysis.md): Competitor monitoring for content and messaging. Track blog posts, positioning changes, and content gaps. Extract competitor data with WebScraping.AI.
- [CRM Enrichment API - Automated Lead Enrichment](https://webscraping.ai/use-cases/crm-enrichment.md): Enrich CRM and lead data with company details, firmographics, and contacts pulled from company websites. Fill data gaps with WebScraping.AI.
- [Hotel Review Aggregation & Analysis API](https://webscraping.ai/use-cases/hotel-review-aggregation.md): Aggregate and analyze hotel reviews, ratings, and amenities from booking platforms. Build hospitality datasets as structured JSON with WebScraping.AI.
- [Influencer Analytics API - Track Social Media Influencers](https://webscraping.ai/use-cases/influencer-analytics.md): Monitor influencer metrics, engagement rates, and content performance. Build influencer marketing platforms with WebScraping.AI.
- [Job Scraping API - Aggregate Job Listings & Postings](https://webscraping.ai/use-cases/job-listing-aggregation.md): Job scraping API to aggregate job postings from any board or careers page. Build job boards and analyze hiring trends with WebScraping.AI.
- [LLM Training Data Collection API for Fine-tuning](https://webscraping.ai/use-cases/llm-fine-tuning-data.md): Collect LLM training data for fine-tuning: Q&A pairs, instruction datasets, and domain corpora as structured JSON with WebScraping.AI.
- [MAP Compliance Monitoring API - Track Minimum Advertised Price](https://webscraping.ai/use-cases/map-compliance-monitoring.md): MAP monitoring across your retailers and marketplaces. Detect minimum advertised price violations and collect evidence with WebScraping.AI.
- [Stock Sentiment Analysis API - Market & News Sentiment](https://webscraping.ai/use-cases/market-sentiment-analysis.md): Stock sentiment analysis from financial news, forums, and social media. Extract bullish and bearish signals as structured data with WebScraping.AI.
- [Competitor Price Monitoring API - Price Scraper for E-Commerce](https://webscraping.ai/use-cases/price-monitoring.md): Scrape competitor prices from any e-commerce site. A price monitoring and tracking API that extracts prices, discounts, and stock for smarter pricing.
- [Product Data API - Aggregate Product Catalogs at Scale](https://webscraping.ai/use-cases/product-data-aggregation.md): A product data API for e-commerce. Handle product data extraction across any store - specs, descriptions, images, and reviews - as clean structured JSON.
- [Property Data API - Aggregate Real Estate Listings](https://webscraping.ai/use-cases/property-listing-aggregation.md): A property data API for real estate scraping. Aggregate listings from multiple platforms and extract price, beds, baths, photos, and agent details as JSON.
- [RAG Knowledge Base API - Build AI Knowledge Bases](https://webscraping.ai/use-cases/rag-knowledge-base.md): Build knowledge bases for retrieval-augmented generation (RAG). Extract and structure web content for AI chatbots and assistants with WebScraping.AI.
- [Rental Market Analysis API - Monitor Rental Prices & Trends](https://webscraping.ai/use-cases/rental-market-analysis.md): Rental market analysis from live listings. Scrape rents, deposits, amenities, and availability across rental platforms to track rental market data over time.
- [Restaurant Data API - Menus, Reviews & Dining Data](https://webscraping.ai/use-cases/restaurant-data-aggregation.md): A restaurant data API for dining platforms. Scrape menus, prices, reviews, ratings, and hours into structured JSON for food discovery apps and market research.
- [Salary Benchmarking API - Compensation Data Collection](https://webscraping.ai/use-cases/salary-benchmarking.md): Salary benchmarking data from live job postings. Scrape salary ranges, bonus, and equity from job boards to build current compensation benchmarks.
- [Sales Intelligence Tools API - Prospect & Competitor Data](https://webscraping.ai/use-cases/sales-intelligence.md): Build sales intelligence tools on a scraping API. Extract competitor pricing, prospect research, and hiring signals from any website as structured data.
- [SEC EDGAR API - Extract Data from SEC Filings](https://webscraping.ai/use-cases/sec-filing-monitoring.md): Use a SEC EDGAR API to extract financial data from filings. Pull revenue, earnings, and risk factors from 10-K, 10-Q, and 8-K reports on EDGAR.
- [SERP Monitoring API - Track Search Engine Rankings](https://webscraping.ai/use-cases/serp-monitoring.md): SERP monitoring API: track Google rankings by country and language with parsed organic results, then scrape the ranking pages or extract SERP features with AI.
- [Social Media Monitoring API - Track Brand Mentions & Sentiment](https://webscraping.ai/use-cases/social-media-monitoring.md): Build social media monitoring and social listening tools on a scraping API. Extract public posts, brand mentions, and sentiment from social platforms as JSON.
- [Stock & Inventory Monitoring API - Track Product Availability](https://webscraping.ai/use-cases/stock-inventory-monitoring.md): An inventory monitoring API for retail. Track stock status and availability across competitor sites, then trigger your own restock and stockout alerts.
- [ML Training Data Collection API - Build AI Datasets](https://webscraping.ai/use-cases/training-data-collection.md): Training data collection for machine learning. Scrape clean text and structured web data to build ML training data for NLP, classification, and RAG.
- [Flight & Hotel Price API - Travel Price Tracking](https://webscraping.ai/use-cases/travel-price-tracking.md): A flight price API and hotel price API in one. Scrape fares and nightly rates from travel sites to build deal alerts and analyze pricing trends.

## Comparisons

- [Web Scraping API Comparisons & Alternatives](https://webscraping.ai/alternatives.md): Honest, side-by-side comparisons of WebScraping.AI against ScrapingBee, ScraperAPI, Firecrawl, and other scraping APIs — features, pricing, and who each fits.
- [Apify Alternative — Simple Web Scraping API, Predictable Pricing](https://webscraping.ai/apify-alternative.md): An Apify alternative that's a simple HTTP scraping API, not a platform: flat per-request pricing, pay only for successful requests, AI extraction. From $29/mo.
- [Bright Data Alternative — Self-Serve Web Scraping API](https://webscraping.ai/bright-data-alternative.md): A self-serve Bright Data alternative with transparent per-request pricing, no KYC or sales calls, and AI extraction. From $29/mo — start scraping in minutes.
- [Browse AI Alternative for Developers](https://webscraping.ai/browse-ai-alternative.md): A Browse AI alternative built for developers: one API call with AI extraction, JS rendering, and rotating proxies — no robots to retrain. From $29/mo.
- [Browserless Alternative Without Managing Browsers](https://webscraping.ai/browserless-alternative.md): A Browserless alternative for scraping: skip Puppeteer scripts and browser-time billing — one call returns rendered HTML, text, or AI JSON. From $29/mo.
- [Diffbot Alternative — Affordable Scraping API with AI Extraction](https://webscraping.ai/diffbot-alternative.md): A Diffbot alternative from $29/mo (no $299 floor): general scraping with residential and stealth proxies, JS rendering, and AI Q&A and field extraction.
- [Firecrawl Alternative for Hard-to-Scrape Sites](https://webscraping.ai/firecrawl-alternative.md): A Firecrawl alternative with residential and stealth proxies designed for protected sites, AI extraction, and pay-only-for-success billing — from $29/mo.
- [Jina AI Alternative for Production Web Scraping](https://webscraping.ai/jina-ai-alternative.md): A Jina AI (Reader) alternative with fixed per-request pricing, residential and stealth proxies, and AI extraction endpoints — from $29/mo.
- [Octoparse Alternative — a Developer API, Not a No-Code App](https://webscraping.ai/octoparse-alternative.md): An Octoparse alternative: a stateless HTTP scraping API with proxies, JS rendering, and AI extraction — no desktop app, no task-building. From $29/mo.
- [Oxylabs Alternative — Self-Serve Scraping API from $29/mo](https://webscraping.ai/oxylabs-alternative.md): An Oxylabs alternative: transparent per-request pricing, no KYC, and AI extraction — enterprise-grade scraping without the overhead. From $29/mo.
- [ParseHub Alternative — a Modern Scraping API for Developers](https://webscraping.ai/parsehub-alternative.md): A ParseHub alternative for developers: a fast HTTP scraping API with proxies, JS rendering, and AI extraction — no desktop app or visual projects. From $29/mo.
- [ScraperAPI Alternative with Flat Per-Request Pricing](https://webscraping.ai/scraperapi-alternative.md): A ScraperAPI alternative with flat per-request pricing and no per-domain surcharges — the same volume for less ($99 vs $149 for 1M credits). From $29/mo.
- [ScrapFly Alternative with Simpler, Cheaper Credits](https://webscraping.ai/scrapfly-alternative.md): A ScrapFly alternative with a flatter credit model: residential proxies at 10 credits instead of 25, AI extraction, 7 SDKs, free failed requests. From $29/mo.
- [ScrapingBee Alternative for Web Scraping](https://webscraping.ai/scrapingbee-alternative.md): A ScrapingBee alternative: the same credit-based scraping API for less ($29 vs $49 for 250k credits), with cheaper stealth proxies and 7 official SDKs.
- [ScrapingBee vs ScraperAPI: 2026 Comparison](https://webscraping.ai/scrapingbee-vs-scraperapi.md): ScrapingBee vs ScraperAPI compared honestly: pricing models, credit multipliers, JS rendering, AI features, and which scraping API fits which workload.
- [Scrapingdog Alternative — Scraping API with AI & 7 SDKs](https://webscraping.ai/scrapingdog-alternative.md): A Scrapingdog alternative from $29/mo with the same low-cost credit model, plus AI Q&A and field extraction, stealth proxies, and 7 official SDKs.
- [SerpApi Alternative — Google SERP API + Web Scraping](https://webscraping.ai/serpapi-alternative.md): A SerpApi alternative: parsed Google results as JSON for 15 credits per search, plus scraping and AI extraction of any page — one API, one bill. From $29/mo.
- [ZenRows Alternative — Lower-Priced Scraping API with 7 SDKs](https://webscraping.ai/zenrows-alternative.md): A ZenRows alternative with the same anti-bot credit model for less ($29 vs $69 for 250k credits), plus AI extraction, an MCP server, and 7 official SDKs.
- [Zyte Alternative — Predictable Scraping API Pricing](https://webscraping.ai/zyte-alternative.md): A Zyte alternative with published, fixed per-request pricing instead of variable per-tier costs — plus AI extraction, an MCP server, and 7 SDKs. From $29/mo.

## Blog

- [Web Scraping Blog: Guides, Tutorials & Comparisons](https://webscraping.ai/blog.md): Web scraping guides, tutorials, and library comparisons for Python, JavaScript, Ruby, PHP, and more, plus proxy and anti-bot best practices.
- [AI Scraping Use Cases: Where AI Data Extraction Pays Off](https://webscraping.ai/blog/ai-use-cases.md): AI scraping use cases that hold up in production: when LLM extraction beats CSS selectors, what it costs per page, and where it still fails.
- [API Scraping: How to Find and Use a Website's Hidden APIs](https://webscraping.ai/blog/api-scraping-guide.md): How to find a website's hidden JSON APIs with the DevTools Network tab, then handle auth, pagination, rate limits, and schema changes in production.
- [Beautiful Soup Web Scraping in Python: The Complete Guide](https://webscraping.ai/blog/beautiful-soup-python.md): Web scraping with Beautiful Soup and Python: install, find() and find_all(), CSS selectors, text extraction, dynamic pages, and saving data to CSV/JSON.
- [Best Datacenter Proxies for Web Scraping: 6 Providers Compared](https://webscraping.ai/blog/best-datacenter-proxy-providers-for-web-scraping.md): The best datacenter proxies compared on price per IP, price per GB, pool size, and endpoint format. 6 providers, verified Sept 2026 pricing, working code.
- [Free Proxy List APIs and Scrapers for Web Scraping](https://webscraping.ai/blog/best-free-proxy-lists.md): Nine free proxy list APIs and sources, each verified live in July 2026: endpoints, formats, measured working rates, rotation code, and when free stops paying.
- [Best Proxies for Web Scraping: 7 Providers Compared (2026)](https://webscraping.ai/blog/best-proxy-providers-for-web-scraping.md): The best proxies for web scraping, compared: verified 2026 pricing for 7 providers, a datacenter-vs-residential decision rule, and working Python code.
- [Best Residential Proxies for Web Scraping: 6 Providers Compared](https://webscraping.ai/blog/best-residential-proxy-providers-for-web-scraping.md): Pick residential proxies for web scraping on pool size, sourcing transparency, sticky sessions, and geo granularity. 6 providers, September 2026 pricing.
- [ChatGPT Web Scraping: Use Cases That Actually Work](https://webscraping.ai/blog/chatgpt-use-cases.md): Can ChatGPT scrape websites? What ChatGPT web scraping really does, four workflows that work, and where you still need a scraping API.
- [Cheapest Residential Proxies in 2026: Verified Price per GB](https://webscraping.ai/blog/cheapest-residential-proxies.md): Verified Sept 2026 price-per-GB for 9 residential proxy providers, plus the minimums, expiry rules and retries that make cheap proxies expensive.
- [Cheerio Web Scraping: Parsing HTML in Node.js](https://webscraping.ai/blog/cheerio-web-scraping.md): Cheerio in Node.js: install, load HTML, CSS selectors, .each/.map loops, attributes, tables, TypeScript, plus Cheerio vs jsdom vs Puppeteer.
- [C# HttpClient: The Complete Guide with Examples](https://webscraping.ai/blog/csharp-httpclient-guide.md): Everything about HttpClient in C#/.NET with copy-paste examples: GET/POST, headers, auth, JSON, timeouts, proxies, file downloads, and IHttpClientFactory.
- [C# Web Scraping: The Complete Guide](https://webscraping.ai/blog/csharp-web-scraping.md): Web scraping in C#: picking between HttpClient, AngleSharp, Html Agility Pack, Playwright for .NET and Selenium, with async patterns and working examples.
- [CSS Selectors Cheat Sheet: Syntax, Examples, and Scraping Patterns](https://webscraping.ai/blog/css-selectors-cheat-sheet.md): A practical CSS selectors cheat sheet: combinators, attribute selectors, nth-child, :not and :has, text-matching limits, and per-tool syntax for web scraping.
- [cURL Commands and Options for Web Scraping: Complete Guide](https://webscraping.ai/blog/curl-commands-for-web-scraping.md): Every cURL command and option you need for web scraping, with copy-paste examples: GET/POST requests, redirects, cookies, auth, proxies, retries, and downloads.
- [Firecrawl Guide: Self-Hosting, Pricing, API Keys, and Limits](https://webscraping.ai/blog/firecrawl-guide.md): What Firecrawl is and how it works: getting an API key, credit pricing, self-hosting with Docker, rate limits, robots.txt, legality, and alternatives.
- [Go Web Scraping: net/http, goquery, Colly, and chromedp](https://webscraping.ai/blog/golang-web-scraping.md): Golang web scraping guide: net/http headers and cookies, goquery parsing, Colly crawlers, chromedp for JavaScript, and goroutine concurrency patterns.
- [GPT and LLM Web Scraping Prompts That Hold Up in Production](https://webscraping.ai/blog/gpt-prompts-for-web-scraping.md): Copy-pasteable GPT and LLM web scraping prompts, measured token costs per page, and fixes for the failures you actually hit: bad JSON and hallucinated fields.
- [What Is a Headless Browser? Headless Chrome for Scraping and Testing](https://webscraping.ai/blog/headless-browser-guide.md): Headless browsers explained: running Chrome and Chromium headless, essential flags, Puppeteer/Playwright/Selenium setup, memory tuning, and detection.
- [How to Scrape Indeed in 2026: What an Indeed Scraper Can and Can't Do](https://webscraping.ai/blog/how-to-scrape-indeed.md): We tested Indeed with datacenter, residential and stealth proxies in July 2026 — all blocked. What an Indeed scraper can still do, and what works instead.
- [Real Estate Scraping: How to Collect Property Data in 2026](https://webscraping.ai/blog/how-to-scrape-real-estate-data.md): Real estate scraping in 2026: which portals we could actually fetch, Zillow's embedded JSON schema, working Python, and the MLS licensing rules people ignore.
- [TikTok Scraper: How to Scrape TikTok Data in 2026](https://webscraping.ai/blog/how-to-scrape-tiktok.md): Build a TikTok scraper that survives: official APIs vs. scraping, why plain HTTP fails, and working Python code for profiles, videos, hashtags, and comments.
- [Html Agility Pack: C# HTML Parsing and Web Scraping Guide](https://webscraping.ai/blog/html-agility-pack-csharp.md): Parse HTML in C# with Html Agility Pack: XPath and LINQ selection, class/id matching, text extraction, fixing malformed HTML, and AngleSharp comparison.
- [HtmlUnit Web Scraping in Java: Setup, Forms, and Limits](https://webscraping.ai/blog/htmlunit-web-scraping-java.md): HtmlUnit 5.x for Java web scraping: Maven setup, JDK 17 requirement, forms and sessions, JavaScript support, and when to use Playwright or an API instead.
- [HTTP Headers for Web Scraping: Anatomy of a Real Browser Request](https://webscraping.ai/blog/http-headers-for-web-scraping.md): Every header a real browser sends, how to match them in Python, Node, and curl, plus compression, chunked encoding, auth, TLS certificates and 403 fixes.
- [Is Web Scraping Legal? What the Case Law Actually Says](https://webscraping.ai/blog/is-web-scraping-legal.md): Is web scraping legal? It depends on five separate questions: CFAA, contract, copyright, personal data, and trespass. What the case law says in 2026.
- [JavaScript Web Scraping Libraries: What to Use in 2026](https://webscraping.ai/blog/javascript-web-scraping-libraries.md): Every JavaScript web scraping library compared: Cheerio, Puppeteer, Playwright, Crawlee. Verified 2026 versions, what each breaks on, and which are now dead.
- [jsoup Java Web Scraping: HTML Parsing and CSS Selectors](https://webscraping.ai/blog/jsoup-java-guide.md): Java web scraping with jsoup: Maven setup, CSS selectors, tables, encoding, proxies and country targeting, caching, login sessions, and JavaScript pages.
- [LLM Web Scraping: How AI Extraction Pipelines Actually Work](https://webscraping.ai/blog/llm-web-scraping.md): How LLM web scraping works end to end: model comparison for extraction, context windows and token limits, real cost per page, and where LLMs cannot help.
- [n8n Web Scraping Guide: HTTP Requests, HTML Extraction, and Browser Automation](https://webscraping.ai/blog/n8n-web-scraping.md): Build web scrapers in n8n: HTTP Request and HTML nodes, CSS selectors, JSON parsing, pagination, scheduling, proxies, exports to Sheets/CSV, and dynamic sites.
- [PHP Guzzle Guide: Requests, Async Concurrency, Retries, and Proxies](https://webscraping.ai/blog/php-guzzle-guide.md): Practical Guzzle tutorial for PHP: installing, JSON and multipart requests, timeouts, error handling, retry middleware, async pools, proxies, and SSL.
- [Playwright Web Scraping: The Complete Guide (Python & Node.js)](https://webscraping.ai/blog/playwright-web-scraping.md): How to scrape websites with Playwright: install, locators, waits, headless mode, proxies, screenshots, Docker, and how it compares to Selenium and Puppeteer.
- [Puppeteer Web Scraping: The Complete Node.js Guide](https://webscraping.ai/blog/puppeteer-web-scraping.md): How to scrape websites with Puppeteer: setup, selectors, waiting, clicks, AJAX, cookies, PDFs, proxies, stealth, Docker, and Puppeteer vs Playwright.
- [PuppeteerSharp Web Scraping: Headless Chrome in C#](https://webscraping.ai/blog/puppeteersharp-web-scraping.md): How to scrape websites with PuppeteerSharp: NuGet setup, selectors, timeouts, logins, downloads, PDFs, Docker, and PuppeteerSharp vs Playwright for .NET.
- [Python Requests Library: The Complete Guide](https://webscraping.ai/blog/python-requests-guide.md): Complete Python requests guide: GET and POST, json vs data, sessions, headers, timeouts, retries, proxies, redirects, streaming, SSL, and error handling.
- [Selenium Web Scraping: The Complete Python Guide](https://webscraping.ai/blog/python-selenium.md): How to scrape websites with Selenium in Python: setup, selectors, waits, proxies, profiles, downloads, screenshots, exceptions, Grid, and bot detection.
- [Python urllib3 Guide: PoolManager, Retries, Timeouts, and SSL](https://webscraping.ai/blog/python-urllib3-guide.md): Practical urllib3 tutorial: urllib vs urllib3 vs requests, PoolManager and connection pooling, retry strategies, timeouts, SSL verification, and proxies.
- [Python Web Scraping Libraries: Which One to Use in 2026](https://webscraping.ai/blog/python-web-scraping-libraries.md): Compare Python web scraping libraries — requests, httpx, Beautiful Soup, lxml, Scrapy, Selenium, Playwright, curl_cffi — with a 2026 decision table.
- [Python XML Parsing: ElementTree, lxml, and How to Choose](https://webscraping.ai/blog/python-xml-parsing.md): Parse XML and HTML in Python: ElementTree vs lxml, XPath, namespaces, iterparse for huge files, modifying and saving documents, and XXE-safe parsing.
- [Web Scraping with Ruby: Nokogiri, HTTParty, and the Rest of the 2026 Toolkit](https://webscraping.ai/blog/ruby-web-scraping-libraries.md): Web scraping with Ruby in 2026: HTTParty timeouts and SSL options, Nokogiri parsing and install fixes, Ferrum for JavaScript, PDFs, and sessions. Working code.
- [Rust Reqwest Guide: HTTP Requests, JSON, Timeouts, and Retries](https://webscraping.ai/blog/rust-reqwest-guide.md): Practical reqwest tutorial for Rust: async and blocking clients, JSON with serde, timeouts, error handling, retries, streaming, mocking, and debugging.
- [How to Scrape Google Search Results](https://webscraping.ai/blog/scrape-google-search-results.md): Why requests and BeautifulSoup no longer work on Google, a Playwright approach that does, consent screens, CAPTCHAs, SERP APIs, and the legal ground rules.
- [Scrapy Web Scraping: The Complete Guide](https://webscraping.ai/blog/scrapy-web-scraping.md): How to scrape with Scrapy: spiders, selectors, items, pipelines, the settings that matter, middlewares, proxies, JavaScript rendering, and deployment.
- [Swift Web Scraping: URLSession, Alamofire, SwiftSoup, and Kanna](https://webscraping.ai/blog/swift-web-scraping.md): Scrape sites in Swift: URLSession vs Alamofire, JSON with Codable, HTML parsing via SwiftSoup and Kanna, redirects, WKWebView limits, and App Store rules.
- [Proxies for Web Scraping: Types, Costs, and How to Choose](https://webscraping.ai/blog/types-of-proxies-for-web-scraping.md): Datacenter, ISP, residential, and mobile proxies for web scraping compared: how each is priced, where each breaks, and which one to reach for first.
- [User Agent List for Web Scraping (2026) and How to Rotate Them](https://webscraping.ai/blog/user-agent-rotation-for-web-scraping.md): Current user agent strings verified Sept 2026, how to rotate them with matching headers and Client Hints, and where user agent rotation stops working.
- [Web Scraping for Machine Learning: Building LLM Training Data](https://webscraping.ai/blog/web-scraping-for-machine-learning.md): Build LLM training data from the web: scrape, clean, chunk, deduplicate, and track provenance, with working Python and the legal rules that actually apply.
- [Web Scraping with MCP Servers: Setup, Workflows, and Troubleshooting](https://webscraping.ai/blog/web-scraping-mcp-servers.md): Set up MCP servers for web scraping in Claude, Cursor, and Claude Code: Playwright MCP, fetch and hosted servers, pagination, resources, and connection fixes.
- [Web Scraping with JavaScript and Node.js](https://webscraping.ai/blog/web-scraping-with-javascript.md): Node.js scraping end to end: fetch and Cheerio, Playwright for rendered pages, plus redirects, shadow DOM, iframes, logins, 2FA, and rate limiting.
- [Mechanize Web Scraping in Ruby and Python](https://webscraping.ai/blog/web-scraping-with-mechanize.md): Mechanize for web scraping in 2026: working Ruby and Python code for logins and pagination, honest maintenance status, and what to use instead of each.
- [Web Scraping with PHP: The Complete Guide](https://webscraping.ai/blog/web-scraping-with-php.md): How to scrape websites with PHP: cURL and SSL, DOMDocument and XPath, Simple HTML DOM, Symfony DomCrawler, Panther for JavaScript pages, and anti-blocking.
- [Web Scraping with Python: Complete Tutorial](https://webscraping.ai/blog/web-scraping-with-python.md): Web scraping with Python end to end: requests, Beautiful Soup, pagination, JavaScript pages, logins, redirects, encodings, tables, 403 errors, and storage.
- [Rust Web Scraping: the scraper Crate, TLS, PDFs, XML, and Headless Chrome](https://webscraping.ai/blog/web-scraping-with-rust.md): Rust web scraping with current crates: the scraper crate, TLS certificates, XML and PDF parsing, headless Chrome, tokio concurrency, and error handling.
- [What is Data Scraping? Types, Tools, and Legal Limits](https://webscraping.ai/blog/what-is-data-scraping.md): Data scraping explained: how it differs from web scraping, the four main types, whether it is legal, and which tools fit each source. Written for practitioners.
- [What is Web Scraping? How It Works and What It's Used For](https://webscraping.ai/blog/what-is-web-scraping.md): Web scraping explained: how a scraper fetches and parses a page, what companies use the data for, how it differs from crawling and APIs, and the legal lines.
- [What is Web Scraping Used For? 12 Business Applications](https://webscraping.ai/blog/what-is-web-scraping-used-for.md): The 12 business applications of web scraping, what data each one collects, and who uses it — from price monitoring to AI retrieval datasets.
- [XPath Cheat Sheet: Syntax, Functions, and Examples for Web Scraping](https://webscraping.ai/blog/xpath-cheat-sheet.md): A practical XPath cheat sheet: syntax, contains() and text matching, axes, position, and/or predicates, escaping — with copy-paste examples for scraping.

## Optional

- [About us](https://webscraping.ai/about.md): Learn about WebScraping.AI and our mission to make reliable, scalable web data extraction accessible to developers and businesses.
- [Affiliate Program - Earn 25% Recurring Commission](https://webscraping.ai/affiliates.md): Join WebScraping.AI's affiliate program and earn 25% recurring commission for every customer you refer. No limits, lifetime commissions.
- [Privacy Policy](https://webscraping.ai/privacy.md): Read the WebScraping.AI Privacy and Cookies Policy to learn how we collect, use, protect, retain, and share personal data.
- [Terms Of Service](https://webscraping.ai/terms.md): Read the WebScraping.AI Terms of Service, including account, billing, acceptable use, data processing, and service conditions.
- [curl to Go Converter - net/http Code Generator](https://webscraping.ai/tools/curl-converter/go.md): Convert curl commands to Go net/http code online. Headers, cookies, bodies, auth, and proxies translated in your browser. Free, no signup.
- [curl to Java Converter - HttpClient Code Generator](https://webscraping.ai/tools/curl-converter/java.md): Convert curl commands to Java HttpClient code online. Headers, cookies, bodies, auth, and redirects translated in your browser. Free, no signup.
- [curl to JavaScript Converter - fetch Code Generator](https://webscraping.ai/tools/curl-converter/javascript.md): Convert curl commands to JavaScript fetch code online. Headers, cookies, JSON bodies, and FormData translated in your browser. Free, no signup.
- [curl to Node.js Converter - Axios Code Generator](https://webscraping.ai/tools/curl-converter/node.md): Convert curl commands to Node.js axios code online. Headers, cookies, auth, proxies, and form-data uploads translated in your browser. Free, no signup.
- [curl to PHP Converter - PHP cURL Code Generator](https://webscraping.ai/tools/curl-converter/php.md): Convert curl commands to PHP cURL code online. Headers, cookies, POST bodies, auth, and proxies mapped to curl_setopt in your browser. Free, no signup.
- [curl to Python Converter - requests Code Generator](https://webscraping.ai/tools/curl-converter/python.md): Convert curl commands to Python requests code online. Headers, cookies, JSON, form data, auth, and proxies translated in your browser. Free, no signup.