---
title: "Python Web Scraping Libraries: Which One to Use in 2026"
description: "Compare Python web scraping libraries — requests, httpx, Beautiful Soup, lxml, Scrapy, Selenium, Playwright, curl_cffi — with a 2026 decision table."
url: https://webscraping.ai/blog/python-web-scraping-libraries
markdown_index: https://webscraping.ai/llms.txt
---
# Python Web Scraping Libraries: Which One to Use in 2026

Most Python scraping projects need exactly two libraries: one that fetches HTML and one that parses it. `requests` + Beautiful Soup covers static pages. Everything else on this page — Playwright, Scrapy, curl\_cffi — exists to solve one specific failure that happens _after_ those two stop working. This guide gives you the decision table: what each library is for, the exact symptom that should make you switch, and which widely-recommended libraries are now dead.

## Key Takeaways

- **Two libraries, not seven.** One HTTP client plus one parser handles the majority of scraping jobs. Add a browser only when the data you see in DevTools is missing from `response.text`.
- **`requests` is still the default** (2.34.2, May 2026) and still actively developed. `httpx` is the async alternative — note it has sat on 0.28.1 since December 2024, though its repo is still active.
- **Installing `lxml` speeds up Beautiful Soup too.** `BeautifulSoup(html, "lxml")` uses lxml as the parsing backend, so it isn't an either/or choice.
- **For a new browser-based scraper in 2026, pick Playwright** (1.61.0). Selenium 4.46 wins only when you inherit an existing suite or need its wider browser/grid matrix.
- **Scrapy pays for itself past a few thousand URLs.** Below that, its project scaffolding costs more time than its concurrency saves.
- **Delete these from your notes:** `requests-html` (no release since 2019), `selenium-wire` (archived January 2024), `pyppeteer` (no commits since June 2024). A 2026 tutorial recommending them is stale.
- **No pure-Python HTTP client defeats TLS fingerprinting.** `curl_cffi` is the only one on this list that tries.

## The 2026 Python scraping library decision table

Versions and dates verified on PyPI and GitHub, 2026-07-28.

| Library | What it's for | When it breaks | Learning curve | Latest (2026-07) |
| --- | --- | --- | --- | --- |
| **requests** | Synchronous HTTP: the default fetcher | Blocked by TLS fingerprinting; no async, no JS | Trivial | 2.34.2 (May 2026) |
| **httpx** | Async HTTP with a `requests`-shaped API, HTTP/2 | Same blocks as requests; async adds complexity you may not need | Easy | 0.28.1 (Dec 2024) |
| **urllib3** | The connection-pooling layer under requests | Verbose for everyday scraping — you rarely want it directly | Medium | 2.7.0 (May 2026) |
| **curl\_cffi** | HTTP that impersonates a real browser's TLS/JA3 fingerprint | Still no JavaScript execution; impersonation targets go stale | Easy | 0.15.0 (Apr 2026) |
| **Beautiful Soup** | Forgiving HTML/XML parsing with a readable API | Slow on very large documents; can't fetch or run JS | Trivial | 4.15.0 (Jun 2026) |
| **lxml** | Fast C-backed parsing, full XPath 1.0 | Stricter on broken markup; XPath is a second syntax to learn | Medium | 6.1.1 (May 2026) |
| **selectolax** | Very fast CSS-selector parsing for high-volume pipelines | No XPath, smaller ecosystem and community | Easy | 0.4.11 (Jul 2026) |
| **Scrapy** | Full crawling framework: scheduling, concurrency, pipelines | Overkill for one-off scripts; no JS without a plugin | Steep | 2.17.0 (Jul 2026) |
| **Selenium** | Real browser control, W3C WebDriver, Grid | Slow, resource-heavy; explicit waits are on you | Medium | 4.46.0 (Jul 2026) |
| **Playwright** | Real browser control with auto-waiting and bundled browsers | Same speed/RAM cost as any browser; heavier install | Medium | 1.61.0 (Jun 2026) |

## Which Python web scraping library should I use?

Read this top to bottom and stop at the first line that matches:

- **You need one page, or a few hundred, from a site that renders server-side** → `requests` + Beautiful Soup. Nothing else. This is the correct answer far more often than blog posts admit.
- **You're parsing thousands of documents and profiling shows parsing is the bottleneck** → keep `requests`, swap the parser for `lxml` or selectolax.
- **You need hundreds of concurrent fetches** → `httpx` with `asyncio`, or Scrapy if you also need crawling logic.
- **`response.text` is missing data that's visible in your browser** → first check the Network tab for a JSON API you can call directly. If there isn't one, use Playwright.
- **You get 403s on the first request even with correct headers** → the site is fingerprinting your TLS handshake. Try `curl_cffi`, then a [scraping API](https://webscraping.ai/) that handles proxies and rendering.
- **You're crawling a whole site with link-following, dedup, retries, and export pipelines** → Scrapy.
- **You need form logins and session handling on plain HTML pages** → `requests.Session()` covers most of it; MechanicalSoup (1.4.0, still maintained) wraps it more neatly.

For a start-to-finish walkthrough rather than a comparison, see our [web scraping with Python tutorial](https://webscraping.ai/blog/web-scraping-with-python).

## Which HTTP client: requests, httpx, urllib3, or curl\_cffi?

These four do the same job — get bytes over the wire — and differ on concurrency and on how much they look like a browser.

**`requests`** remains the default and is not going anywhere: 2.34.2 shipped in May 2026, and it now requires Python 3.10+. Use a `Session` so connections are pooled and headers persist:

```python
import requests
from bs4 import BeautifulSoup

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
})

response = session.get("https://example.com/products", timeout=10)
response.raise_for_status()

soup = BeautifulSoup(response.text, "lxml")
for card in soup.select(".product-card"):
    print(card.select_one("h2").get_text(strip=True),
          card.select_one(".price").get_text(strip=True))
```

**`httpx`** is what you reach for when you want `asyncio` and HTTP/2. The API is deliberately close to `requests`, so porting is mostly mechanical. Be aware of the maintenance picture before you standardise on it: the repo is active, but PyPI still shows 0.28.1 from December 2024 as the current release.

```python
import asyncio
import httpx
from selectolax.parser import HTMLParser

async def fetch_title(client, url):
    response = await client.get(url, timeout=10)
    return HTMLParser(response.text).css_first("h1").text(strip=True)

async def main(urls):
    limits = httpx.Limits(max_connections=20)
    async with httpx.AsyncClient(limits=limits, follow_redirects=True) as client:
        return await asyncio.gather(*(fetch_title(client, u) for u in urls))

print(asyncio.run(main(["https://example.com/a", "https://example.com/b"])))
```

**`urllib3`** sits underneath `requests` and gives you direct control of pools and retries. You want it when you're building a client library or tuning retry behaviour precisely — see the [urllib3 guide](https://webscraping.ai/blog/python-urllib3-guide) for that. For scraping, `requests` on top of it is the better ergonomic trade.

**`curl_cffi`** is the newest of the four and the only one that addresses the reason modern blocks happen. Sites increasingly fingerprint your TLS handshake (JA3/JA4) and HTTP/2 frame ordering; a Python client sending a Chrome User-Agent with a Python TLS signature is trivially detectable. `curl_cffi` binds to curl-impersonate to reproduce a real browser's handshake:

```python
from curl_cffi import requests as cffi_requests

response = cffi_requests.get("https://example.com", impersonate="chrome")
print(response.status_code)
```

It still can't run JavaScript, and the impersonation targets need updating as browsers ship new versions — but when a page returns 403 to `requests` and 200 to your browser with identical headers, this is the cheapest thing to try next.

## Which parser: Beautiful Soup, lxml, or selectolax?

These do not compete the way most listicles imply. **Beautiful Soup is a friendly API that sits on top of a parser backend, and lxml is one of the backends it can use.** The idiomatic 2026 install is both:

```bash
pip install requests beautifulsoup4 lxml
```

Use **Beautiful Soup** (4.15.0) when you value readable selector code and forgiving behaviour on malformed markup, which describes most scraping. `find()`, `find_all()`, and `select()` cover nearly everything; our [Beautiful Soup guide](https://webscraping.ai/blog/beautiful-soup-python) goes through them properly.

Use **lxml** directly (6.1.1) when you want XPath — for anything involving axes, text-node matching, or "the sibling after the label that says Price," XPath expresses in one line what CSS selectors can't express at all:

```python
from lxml import html as lxml_html

tree = lxml_html.fromstring(response.text)
price = tree.xpath('//dt[text()="Price"]/following-sibling::dd[1]/text()')
```

Our [lxml and XML parsing guide](https://webscraping.ai/blog/python-xml-parsing) covers the API, and the [XPath cheat sheet](https://webscraping.ai/blog/xpath-cheat-sheet) covers the expression syntax.

Use **selectolax** (0.4.11) only when you have profiled and parsing is genuinely your bottleneck — millions of documents, not thousands. You trade XPath support and ecosystem breadth for raw speed.

## Playwright vs Selenium: which should you use in 2026?

Both drive a real browser, so both pay the same cost: hundreds of megabytes of RAM per instance and page loads measured in seconds rather than milliseconds. The differences that matter in practice:

| | Playwright 1.61 | Selenium 4.46 |
| --- | --- | --- |
| **Browser setup** | `playwright install` downloads pinned browser builds | Selenium Manager resolves drivers for your installed browsers |
| **Waiting** | Auto-waits for actionability before every interaction | Explicit `WebDriverWait` conditions are your responsibility |
| **Async** | First-class `async` and sync APIs in one package | Sync API; async via third-party wrappers |
| **Network control** | Request interception and route mocking built in | Requires CDP access or external tooling |
| **Standard** | Chrome DevTools Protocol / its own protocol | W3C WebDriver — the actual standard |
| **Ecosystem** | Newer, growing fast | Two decades of Grid, cloud vendors, and enterprise tooling |

**Pick Playwright for new scraping work.** Auto-waiting removes the single largest source of flaky scrapers — hand-written sleeps and waits — and bundled browsers make CI reproducible:

```python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/dashboard", wait_until="domcontentloaded")
    page.wait_for_selector(".result-row")
    rows = page.locator(".result-row").all_text_contents()
    browser.close()
```

**Pick Selenium** when you already have a Selenium suite, when you need to test against a browser matrix that Playwright doesn't bundle, or when your team's cloud grid speaks WebDriver. The equivalent script is longer because the waiting is explicit:

```python
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options) # Selenium Manager resolves the driver
try:
    driver.get("https://example.com/dashboard")
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".result-row"))
    )
    rows = [e.text for e in driver.find_elements(By.CSS_SELECTOR, ".result-row")]
finally:
    driver.quit()
```

Depth on either: the [Playwright scraping guide](https://webscraping.ai/blog/playwright-web-scraping) and the [Python Selenium guide](https://webscraping.ai/blog/python-selenium). And before you launch a browser at all, check the Network tab — a page that fetches its data from a JSON endpoint can usually be scraped with `requests` against that endpoint, at a fraction of the cost. Our [headless browser guide](https://webscraping.ai/blog/headless-browser-guide) covers when the browser is genuinely unavoidable.

## When is Scrapy worth the setup?

Scrapy 2.17 is not a library you drop into a script — it's a framework with its own project layout, settings module, and CLI. That cost buys you scheduling, concurrent requests, automatic retries, deduplication, throttling, and item pipelines that you would otherwise reimplement badly.

```python
import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/catalog"]
    custom_settings = {"DOWNLOAD_DELAY": 0.5, "CONCURRENT_REQUESTS": 16}

    def parse(self, response):
        for card in response.css(".product-card"):
            yield {
                "title": card.css("h2::text").get(),
                "price": card.css(".price::text").get(),
            }

        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)
```

The rule of thumb: **under a few thousand URLs from a single site, a `requests` loop is faster to write and easier to debug.** Past that, or as soon as you need to crawl link graphs rather than a known URL list, Scrapy wins. Scrapy has no JavaScript rendering of its own; the `scrapy-playwright` plugin (0.0.48, July 2026) is the maintained way to add it.

Scrapy's selector layer is worth knowing separately: `parsel` (1.11.0) is that layer extracted as a standalone package, giving you `.css()` and `.xpath()` chaining outside a Scrapy project.

## Which Python scraping recommendations are now stale?

Library listicles get copied for years after the libraries stop shipping. Checked on 2026-07-28:

- **`requests-html`** — last PyPI release 0.10.0 in February 2019; the repository's last commit is from 2024. Still recommended constantly. Don't. Use `httpx` for the async part and Playwright for the rendering part.
- **`selenium-wire`** — the go-to for request interception and authenticated proxies in Selenium. Its repository was **archived in January 2024**. Use Playwright's built-in `route()` interception instead.
- **`pyppeteer`** — the Python Puppeteer port. No commits since June 2024. Playwright is the maintained successor and covers the same ground.
- **Scrapy + Splash** — the old JavaScript-rendering combination. `scrapy-playwright` is where the maintenance activity is now.
- **MechanicalSoup** is the exception to this list: 1.4.0 (May 2025) with an actively maintained repository. It's niche — form-heavy, JavaScript-free sites — but it isn't abandoned. If you're comparing it to older options, our [Mechanize post](https://webscraping.ai/blog/web-scraping-with-mechanize) has the history.

The other stale recommendation is implicit: any 2024 guide that treats "rotate your User-Agent" as anti-block advice. Header rotation stopped being sufficient once TLS fingerprinting became common — see [User-Agent rotation](https://webscraping.ai/blog/user-agent-rotation-for-web-scraping) for what it does and doesn't still buy you.

## What no Python library can solve on its own

Every library above assumes it can reach the page. In production, the failures that consume your time aren't parsing bugs — they're 403s, CAPTCHAs, geo-restricted content, and JavaScript that only renders behind a residential IP. Solving those in-house means running browser infrastructure and buying proxies, which is a separate product from the scraper you wanted to write.

That's the gap [WebScraping.AI](https://webscraping.ai/) fills: you keep your parser and swap the fetch layer for an API call that handles rendering, proxy rotation, and retries.

```python
import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://api.webscraping.ai/html",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/products",
        "js": "true", # render JavaScript (default)
        "proxy": "residential", # datacenter | residential | stealth
        "wait_for": ".product-card",
        "country": "us",
    },
    timeout=60,
)

soup = BeautifulSoup(response.text, "lxml")
for card in soup.select(".product-card"):
    print(card.select_one("h2").get_text(strip=True))
```

If you'd rather skip selectors entirely — useful when page structure changes often — the `/ai/fields` endpoint returns structured JSON from a plain-English description of each field:

```python
fields = requests.get(
    "https://api.webscraping.ai/ai/fields",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/products/1",
        "fields[title]": "Product title",
        "fields[price]": "Current price including currency symbol",
        "fields[in_stock]": "Whether the item is in stock, true or false",
    },
    timeout=60,
).json()
```

Costs are published rather than estimated: 1 credit for a datacenter request without JavaScript, 5 with JavaScript, 10 and 25 for residential, 50 for stealth, plus 5 for AI extraction. **Failed requests are free.** The free tier is 2,000 credits per month with no credit card, and there are official SDKs for Python, Ruby, PHP, JavaScript, Go, Java, and C#, plus an [MCP server](https://webscraping.ai/integrations/mcp-server) and an [n8n node](https://webscraping.ai/integrations/n8n) if you're wiring scraping into an agent or workflow. Full parameter reference is in the [docs](https://webscraping.ai/docs).

Common destinations for this stack: [price monitoring](https://webscraping.ai/use-cases/price-monitoring), [job listing aggregation](https://webscraping.ai/use-cases/job-listing-aggregation), and [RAG knowledge bases](https://webscraping.ai/use-cases/rag-knowledge-base).

Whatever you build, the legal picture is worth understanding before you scale it up — [is web scraping legal?](https://webscraping.ai/blog/is-web-scraping-legal) covers the actual case law rather than the usual hand-waving.

## Frequently Asked Questions

### Which Python library is best for web scraping?

There is no single best one, because they occupy different layers. For a static site, `requests` + Beautiful Soup is the best answer and the shortest code. For a JavaScript-rendered site, Playwright. For crawling thousands of pages across a site, Scrapy. Start with `requests` + Beautiful Soup and change only when a specific symptom forces you to.

### Is Scrapy better than Beautiful Soup?

They aren't substitutes. Beautiful Soup parses HTML you already have; Scrapy fetches, schedules, retries, and pipelines the data, and uses its own `parsel` selectors for parsing. Choose Scrapy when the crawling logic is the hard part. Choose Beautiful Soup when fetching is trivial and you just need to pull fields out of a page.

### Should I use httpx or requests?

Use `requests` unless you need `asyncio` or HTTP/2 — those are the two reasons `httpx` exists. `httpx` mirrors the `requests` API closely enough that switching later is cheap, so there's no penalty for starting simple. One consideration: `requests` shipped 2.34.2 in May 2026, while `httpx` has been on 0.28.1 since December 2024.

### Do I still need lxml if I use Beautiful Soup?

Yes, in most cases. `BeautifulSoup(html, "lxml")` uses lxml as its backend parser, which is faster than the standard library's `html.parser`. Install both. You only work with lxml's own API directly when you need XPath.

### Can Python libraries scrape JavaScript-rendered sites?

Not `requests`, `httpx`, Beautiful Soup, or lxml — none of them execute JavaScript. Playwright and Selenium do, because they drive a real browser. Before reaching for either, open the Network tab: if the page loads its data from a JSON endpoint, calling that endpoint with `requests` is faster and far more stable than rendering the page.

### Which Python scraping libraries handle anti-bot blocking?

None of them fully. `curl_cffi` addresses TLS fingerprinting, which is the most common cause of instant 403s, and Playwright's real browser handles JavaScript-based checks. Neither gives you IP rotation, which is what most large-scale blocking actually keys on. At that point you're choosing between running proxy infrastructure yourself and [using an API that includes it](https://webscraping.ai/).

### What's the difference between web scraping and web crawling?

Scraping extracts specific data from pages you already have URLs for. Crawling discovers those URLs by following links. Scrapy does both; a `requests` script usually only does the first. If you can't enumerate your target URLs in advance, you need a crawler.