Python Web Scraping Libraries: Which One to Use in 2026

Most Python scraping projects need exactly two libraries: one that fetches HTML and one that parses it. requests + Beautiful Soup covers static pages. Everything else on this page — Playwright, Scrapy, curl_cffi — exists to solve one specific failure that happens after those two stop working. This guide gives you the decision table: what each library is for, the exact symptom that should make you switch, and which widely-recommended libraries are now dead.

Key Takeaways

• Two libraries, not seven. One HTTP client plus one parser handles the majority of scraping jobs. Add a browser only when the data you see in DevTools is missing from response.text.
• requests is still the default (2.34.2, May 2026) and still actively developed. httpx is the async alternative — note it has sat on 0.28.1 since December 2024, though its repo is still active.
• Installing lxml speeds up Beautiful Soup too. BeautifulSoup(html, "lxml") uses lxml as the parsing backend, so it isn't an either/or choice.
• For a new browser-based scraper in 2026, pick Playwright (1.61.0). Selenium 4.46 wins only when you inherit an existing suite or need its wider browser/grid matrix.
• Scrapy pays for itself past a few thousand URLs. Below that, its project scaffolding costs more time than its concurrency saves.
• Delete these from your notes: requests-html (no release since 2019), selenium-wire (archived January 2024), pyppeteer (no commits since June 2024). A 2026 tutorial recommending them is stale.
• No pure-Python HTTP client defeats TLS fingerprinting. curl_cffi is the only one on this list that tries.

The 2026 Python scraping library decision table

Versions and dates verified on PyPI and GitHub, 2026-07-28.

Which Python web scraping library should I use?

Read this top to bottom and stop at the first line that matches:

• You need one page, or a few hundred, from a site that renders server-side → requests + Beautiful Soup. Nothing else. This is the correct answer far more often than blog posts admit.
• You're parsing thousands of documents and profiling shows parsing is the bottleneck → keep requests, swap the parser for lxml or selectolax.
• You need hundreds of concurrent fetches → httpx with asyncio, or Scrapy if you also need crawling logic.
• response.text is missing data that's visible in your browser → first check the Network tab for a JSON API you can call directly. If there isn't one, use Playwright.
• You get 403s on the first request even with correct headers → the site is fingerprinting your TLS handshake. Try curl_cffi, then a scraping API that handles proxies and rendering.
• You're crawling a whole site with link-following, dedup, retries, and export pipelines → Scrapy.
• You need form logins and session handling on plain HTML pages → requests.Session() covers most of it; MechanicalSoup (1.4.0, still maintained) wraps it more neatly.

For a start-to-finish walkthrough rather than a comparison, see our web scraping with Python tutorial.

Which HTTP client: requests, httpx, urllib3, or curl_cffi?

These four do the same job — get bytes over the wire — and differ on concurrency and on how much they look like a browser.

requests remains the default and is not going anywhere: 2.34.2 shipped in May 2026, and it now requires Python 3.10+. Use a Session so connections are pooled and headers persist:

import requests
from bs4 import BeautifulSoup

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
})

response = session.get("https://example.com/products", timeout=10)
response.raise_for_status()

soup = BeautifulSoup(response.text, "lxml")
for card in soup.select(".product-card"):
    print(card.select_one("h2").get_text(strip=True),
          card.select_one(".price").get_text(strip=True))

httpx is what you reach for when you want asyncio and HTTP/2. The API is deliberately close to requests, so porting is mostly mechanical. Be aware of the maintenance picture before you standardise on it: the repo is active, but PyPI still shows 0.28.1 from December 2024 as the current release.

import asyncio
import httpx
from selectolax.parser import HTMLParser

async def fetch_title(client, url):
    response = await client.get(url, timeout=10)
    return HTMLParser(response.text).css_first("h1").text(strip=True)

async def main(urls):
    limits = httpx.Limits(max_connections=20)
    async with httpx.AsyncClient(limits=limits, follow_redirects=True) as client:
        return await asyncio.gather(*(fetch_title(client, u) for u in urls))

print(asyncio.run(main(["https://example.com/a", "https://example.com/b"])))

urllib3 sits underneath requests and gives you direct control of pools and retries. You want it when you're building a client library or tuning retry behaviour precisely — see the urllib3 guide for that. For scraping, requests on top of it is the better ergonomic trade.

curl_cffi is the newest of the four and the only one that addresses the reason modern blocks happen. Sites increasingly fingerprint your TLS handshake (JA3/JA4) and HTTP/2 frame ordering; a Python client sending a Chrome User-Agent with a Python TLS signature is trivially detectable. curl_cffi binds to curl-impersonate to reproduce a real browser's handshake:

from curl_cffi import requests as cffi_requests

response = cffi_requests.get("https://example.com", impersonate="chrome")
print(response.status_code)

It still can't run JavaScript, and the impersonation targets need updating as browsers ship new versions — but when a page returns 403 to requests and 200 to your browser with identical headers, this is the cheapest thing to try next.

Which parser: Beautiful Soup, lxml, or selectolax?

These do not compete the way most listicles imply. Beautiful Soup is a friendly API that sits on top of a parser backend, and lxml is one of the backends it can use. The idiomatic 2026 install is both:

pip install requests beautifulsoup4 lxml

Use Beautiful Soup (4.15.0) when you value readable selector code and forgiving behaviour on malformed markup, which describes most scraping. find(), find_all(), and select() cover nearly everything; our Beautiful Soup guide goes through them properly.

Use lxml directly (6.1.1) when you want XPath — for anything involving axes, text-node matching, or "the sibling after the label that says Price," XPath expresses in one line what CSS selectors can't express at all:

from lxml import html as lxml_html

tree = lxml_html.fromstring(response.text)
price = tree.xpath('//dt[text()="Price"]/following-sibling::dd[1]/text()')

Our lxml and XML parsing guide covers the API, and the XPath cheat sheet covers the expression syntax.

Use selectolax (0.4.11) only when you have profiled and parsing is genuinely your bottleneck — millions of documents, not thousands. You trade XPath support and ecosystem breadth for raw speed.

Playwright vs Selenium: which should you use in 2026?

Both drive a real browser, so both pay the same cost: hundreds of megabytes of RAM per instance and page loads measured in seconds rather than milliseconds. The differences that matter in practice:

Pick Playwright for new scraping work. Auto-waiting removes the single largest source of flaky scrapers — hand-written sleeps and waits — and bundled browsers make CI reproducible:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/dashboard", wait_until="domcontentloaded")
    page.wait_for_selector(".result-row")
    rows = page.locator(".result-row").all_text_contents()
    browser.close()

Pick Selenium when you already have a Selenium suite, when you need to test against a browser matrix that Playwright doesn't bundle, or when your team's cloud grid speaks WebDriver. The equivalent script is longer because the waiting is explicit:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)  # Selenium Manager resolves the driver
try:
    driver.get("https://example.com/dashboard")
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".result-row"))
    )
    rows = [e.text for e in driver.find_elements(By.CSS_SELECTOR, ".result-row")]
finally:
    driver.quit()

Depth on either: the Playwright scraping guide and the Python Selenium guide. And before you launch a browser at all, check the Network tab — a page that fetches its data from a JSON endpoint can usually be scraped with requests against that endpoint, at a fraction of the cost. Our headless browser guide covers when the browser is genuinely unavoidable.

When is Scrapy worth the setup?

Scrapy 2.17 is not a library you drop into a script — it's a framework with its own project layout, settings module, and CLI. That cost buys you scheduling, concurrent requests, automatic retries, deduplication, throttling, and item pipelines that you would otherwise reimplement badly.

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/catalog"]
    custom_settings = {"DOWNLOAD_DELAY": 0.5, "CONCURRENT_REQUESTS": 16}

    def parse(self, response):
        for card in response.css(".product-card"):
            yield {
                "title": card.css("h2::text").get(),
                "price": card.css(".price::text").get(),
            }

        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

The rule of thumb: under a few thousand URLs from a single site, a requests loop is faster to write and easier to debug. Past that, or as soon as you need to crawl link graphs rather than a known URL list, Scrapy wins. Scrapy has no JavaScript rendering of its own; the scrapy-playwright plugin (0.0.48, July 2026) is the maintained way to add it.

Scrapy's selector layer is worth knowing separately: parsel (1.11.0) is that layer extracted as a standalone package, giving you .css() and .xpath() chaining outside a Scrapy project.

Which Python scraping recommendations are now stale?

Library listicles get copied for years after the libraries stop shipping. Checked on 2026-07-28:

• requests-html — last PyPI release 0.10.0 in February 2019; the repository's last commit is from 2024. Still recommended constantly. Don't. Use httpx for the async part and Playwright for the rendering part.
• selenium-wire — the go-to for request interception and authenticated proxies in Selenium. Its repository was archived in January 2024. Use Playwright's built-in route() interception instead.
• pyppeteer — the Python Puppeteer port. No commits since June 2024. Playwright is the maintained successor and covers the same ground.
• Scrapy + Splash — the old JavaScript-rendering combination. scrapy-playwright is where the maintenance activity is now.
• MechanicalSoup is the exception to this list: 1.4.0 (May 2025) with an actively maintained repository. It's niche — form-heavy, JavaScript-free sites — but it isn't abandoned. If you're comparing it to older options, our Mechanize post has the history.

The other stale recommendation is implicit: any 2024 guide that treats "rotate your User-Agent" as anti-block advice. Header rotation stopped being sufficient once TLS fingerprinting became common — see User-Agent rotation for what it does and doesn't still buy you.

What no Python library can solve on its own

Every library above assumes it can reach the page. In production, the failures that consume your time aren't parsing bugs — they're 403s, CAPTCHAs, geo-restricted content, and JavaScript that only renders behind a residential IP. Solving those in-house means running browser infrastructure and buying proxies, which is a separate product from the scraper you wanted to write.

That's the gap WebScraping.AI fills: you keep your parser and swap the fetch layer for an API call that handles rendering, proxy rotation, and retries.

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://api.webscraping.ai/html",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/products",
        "js": "true",              # render JavaScript (default)
        "proxy": "residential",    # datacenter | residential | stealth
        "wait_for": ".product-card",
        "country": "us",
    },
    timeout=60,
)

soup = BeautifulSoup(response.text, "lxml")
for card in soup.select(".product-card"):
    print(card.select_one("h2").get_text(strip=True))

If you'd rather skip selectors entirely — useful when page structure changes often — the /ai/fields endpoint returns structured JSON from a plain-English description of each field:

fields = requests.get(
    "https://api.webscraping.ai/ai/fields",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/products/1",
        "fields[title]": "Product title",
        "fields[price]": "Current price including currency symbol",
        "fields[in_stock]": "Whether the item is in stock, true or false",
    },
    timeout=60,
).json()

Costs are published rather than estimated: 1 credit for a datacenter request without JavaScript, 5 with JavaScript, 10 and 25 for residential, 50 for stealth, plus 5 for AI extraction. Failed requests are free. The free tier is 2,000 credits per month with no credit card, and there are official SDKs for Python, Ruby, PHP, JavaScript, Go, Java, and C#, plus an MCP server and an n8n node if you're wiring scraping into an agent or workflow. Full parameter reference is in the docs.

Common destinations for this stack: price monitoring, job listing aggregation, and RAG knowledge bases.

Whatever you build, the legal picture is worth understanding before you scale it up — is web scraping legal? covers the actual case law rather than the usual hand-waving.

Frequently Asked Questions

Which Python library is best for web scraping?

There is no single best one, because they occupy different layers. For a static site, requests + Beautiful Soup is the best answer and the shortest code. For a JavaScript-rendered site, Playwright. For crawling thousands of pages across a site, Scrapy. Start with requests + Beautiful Soup and change only when a specific symptom forces you to.

Is Scrapy better than Beautiful Soup?

They aren't substitutes. Beautiful Soup parses HTML you already have; Scrapy fetches, schedules, retries, and pipelines the data, and uses its own parsel selectors for parsing. Choose Scrapy when the crawling logic is the hard part. Choose Beautiful Soup when fetching is trivial and you just need to pull fields out of a page.

Should I use httpx or requests?

Use requests unless you need asyncio or HTTP/2 — those are the two reasons httpx exists. httpx mirrors the requests API closely enough that switching later is cheap, so there's no penalty for starting simple. One consideration: requests shipped 2.34.2 in May 2026, while httpx has been on 0.28.1 since December 2024.

Do I still need lxml if I use Beautiful Soup?

Yes, in most cases. BeautifulSoup(html, "lxml") uses lxml as its backend parser, which is faster than the standard library's html.parser. Install both. You only work with lxml's own API directly when you need XPath.

Can Python libraries scrape JavaScript-rendered sites?

Not requests, httpx, Beautiful Soup, or lxml — none of them execute JavaScript. Playwright and Selenium do, because they drive a real browser. Before reaching for either, open the Network tab: if the page loads its data from a JSON endpoint, calling that endpoint with requests is faster and far more stable than rendering the page.

Which Python scraping libraries handle anti-bot blocking?

None of them fully. curl_cffi addresses TLS fingerprinting, which is the most common cause of instant 403s, and Playwright's real browser handles JavaScript-based checks. Neither gives you IP rotation, which is what most large-scale blocking actually keys on. At that point you're choosing between running proxy infrastructure yourself and using an API that includes it.

What's the difference between web scraping and web crawling?

Scraping extracts specific data from pages you already have URLs for. Crawling discovers those URLs by following links. Scrapy does both; a requests script usually only does the first. If you can't enumerate your target URLs in advance, you need a crawler.