Web Scraping with Python: The Complete Tutorial

A working Python scraper is about fifteen lines: requests fetches the HTML, Beautiful Soup finds the elements, a loop writes them to a file. The other 95% of the work is everything that happens after that — pages two through two hundred, data that only appears after JavaScript runs, the 403 that shows up on request 300, and the run that dies at 2 a.m. with nothing saved.

This tutorial covers all of it in order, with code you can run. Start at the top if you're new; jump to pagination, JavaScript pages, or 403s and blocking if you already have a scraper that broke.

Key Takeaways

• pip install requests beautifulsoup4 lxml covers 90% of real scraping jobs — reach for a browser only after you've confirmed the data isn't in the raw HTML
• Before writing a selector, run curl -s URL | grep "some visible text". If it returns nothing, the page is JavaScript-rendered and requests alone will never see the data
• Sites that render client-side usually call a JSON API you can hit directly — that endpoint is faster, more stable, and cheaper than driving a browser
• A 403 on request 1 is a headers problem; a 403 on request 300 is a rate-limit or IP problem. The fixes are completely different
• Write each record to a JSONL file or SQLite as you go, never to an in-memory list — a crash on page 180 of 200 should cost you one page, not the whole run
• Roll your own until you're spending more time on proxies and browser infrastructure than on the data; at that point a scraping API is cheaper than the maintenance

What you need to get started

Python 3.10 or newer, and a virtual environment so library versions stay per-project:

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install requests beautifulsoup4 lxml

Three packages, three jobs:

If you want the full field of options — Scrapy, Selenium, httpx, Scrapling — see our comparison of the best Python web scraping libraries. For this tutorial, requests plus Beautiful Soup is the right stack, and it stays the right stack far longer than most beginners expect.

Your first Python web scraper

This scrapes book titles, prices, and ratings from books.toscrape.com, a sandbox site built for practice:

import requests
from bs4 import BeautifulSoup

url = "http://books.toscrape.com/catalogue/page-1.html"
response = requests.get(url, timeout=20)
response.raise_for_status()

soup = BeautifulSoup(response.text, "lxml")

for card in soup.select("article.product_pod"):
    title = card.select_one("h3 a")["title"]
    price = card.select_one("p.price_color").get_text(strip=True)
    rating = card.select_one("p.star-rating")["class"][1]
    print(f"{title} | {price} | {rating}")

Run it and you get 20 books. Four things in there are worth internalizing, because they're what separates a script that works once from one that works every day:

• timeout=20 — without it, requests waits forever on a hung server and your scraper silently stalls
• raise_for_status() — turns a 404 or 500 into an exception instead of letting you parse an error page as if it were data
• select() / select_one() — CSS selectors, the same ones you use in the browser console, rather than nested find() calls
• Scoping to the card — card.select_one(...) searches inside one product, so titles and prices can never drift out of sync the way two separate find_all() lists can

How does web scraping actually work?

Every scraper, from a 15-line script to a distributed crawler, is the same four steps:

1. Fetch — send an HTTP request, get HTML (or JSON) back
2. Parse — turn that text into a tree you can query
3. Select — pull out the specific values you want
4. Store — write them somewhere that survives the process exiting

Most scraping problems are misdiagnosed because people skip step 0: confirm the data is actually in the response. Your browser shows you the page after JavaScript has run, after XHR requests have resolved, after the framework has hydrated. requests shows you what the server sent. Those are often different documents.

Check before you write a single selector:

curl -s "https://example.com/products" | grep -i "some text you can see on the page"

Nothing returned? The page is rendered client-side, and no amount of Beautiful Soup will help. Skip ahead.

How do I find the right CSS selector?

Open the page, right-click the element you want, choose Inspect. In DevTools, right-click the highlighted node and pick Copy → Copy selector as a starting point — then shorten it by hand. Browser-generated selectors like body > div:nth-child(3) > div > div.container > article > h3 break the instant anyone touches the layout.

Write selectors that describe meaning, not position:

# Fragile: breaks when a wrapper div is added
soup.select_one("body > div:nth-child(3) > article > h3 > a")

# Durable: survives layout changes
soup.select_one("article.product_pod h3 a")

# Most durable: test IDs and semantic attributes, when the site has them
soup.select_one("[data-testid='product-title']")
soup.select_one("[itemprop='price']")

Test a selector in the browser console with document.querySelectorAll("article.product_pod h3 a") before putting it in code — instant feedback, no request wasted.

Beautiful Soup also offers find() and find_all() with keyword filters (soup.find_all("p", class_="price_color")), which are easier to build dynamically. Our Beautiful Soup guide covers the full API — navigating siblings and parents, matching multiple classes, extracting attributes. If you prefer XPath, lxml gives you the same power with different syntax; see the XPath cheat sheet and our Python XML parsing guide.

How do I scrape multiple pages? (pagination patterns)

Almost every real job is multi-page. There are three patterns, and identifying which one you're facing takes ten seconds of clicking "next" and watching the URL bar.

1. Numbered URLs — the page number is in the path or query string:

import requests
from bs4 import BeautifulSoup

def scrape_page(page_num):
    url = f"http://books.toscrape.com/catalogue/page-{page_num}.html"
    response = requests.get(url, timeout=20)
    if response.status_code == 404:
        return None                      # ran off the end of the list
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "lxml")
    return [
        {
            "title": card.select_one("h3 a")["title"],
            "price": card.select_one("p.price_color").get_text(strip=True),
        }
        for card in soup.select("article.product_pod")
    ]

page = 1
while (books := scrape_page(page)):
    print(f"page {page}: {len(books)} books")
    page += 1

Stop on an empty result or a 404 — never on a hardcoded page count, which silently truncates your data the day the catalog grows.

2. "Next" links — follow the link rather than guessing the URL shape:

from urllib.parse import urljoin

url = "http://books.toscrape.com/catalogue/page-1.html"
while url:
    response = requests.get(url, timeout=20)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "lxml")

    # ... extract records here ...

    next_link = soup.select_one("li.next a")
    url = urljoin(url, next_link["href"]) if next_link else None

urljoin is doing real work: pagination hrefs are usually relative (page-2.html), and string concatenation gets it wrong the moment you're not at the site root.

3. Infinite scroll / "load more" — there is no next URL, because the page calls an API. Open DevTools → Network → Fetch/XHR, scroll, and watch what fires. You'll almost always find a JSON endpoint with ?page=2 or ?offset=40 that you can call directly:

records = []
for offset in range(0, 500, 50):
    data = requests.get(
        "https://example.com/api/products",
        params={"offset": offset, "limit": 50},
        timeout=20,
    ).json()
    if not data["items"]:
        break
    records.extend(data["items"])

That's the single highest-leverage trick in this article. A JSON API gives you clean typed data, no parsing, no broken selectors when the design changes, and one request per 50 records instead of one per page-load.

The one-line shortcut for tables

If the data you want is an HTML <table>, don't write selectors at all:

from io import StringIO
import pandas as pd
import requests

html = requests.get("https://example.com/stats", timeout=20).text
tables = pd.read_html(StringIO(html))   # list of DataFrames, one per <table>
df = tables[0]
df.to_csv("stats.csv", index=False)

pandas.read_html finds every table on the page and hands back DataFrames. (Pass a StringIO, not a raw string — pandas 2.x deprecated literal-string input.) It needs lxml or html5lib installed, which you already have.

Why is the data missing from the HTML?

You confirmed with curl that the values you want aren't in the response. Three options, in order of what you should try first:

Find the JSON API first. DevTools → Network → Fetch/XHR, reload, and look at the responses. Sites built on React, Vue, or Next.js are fetching their data from somewhere; that somewhere is usually a documented-enough endpoint. Also check for a <script id="__NEXT_DATA__" type="application/json"> blob in the HTML — Next.js sites embed the entire page state there, and it's parseable with requests alone:

import json

soup = BeautifulSoup(requests.get(url, timeout=20).text, "lxml")
blob = soup.select_one("script#__NEXT_DATA__")
data = json.loads(blob.string)

If there's genuinely no API, drive a real browser. Playwright is the modern default; Selenium is the incumbent with the larger ecosystem:

from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/products", wait_until="networkidle")
    page.wait_for_selector(".product-card")
    soup = BeautifulSoup(page.content(), "lxml")
    browser.close()

for card in soup.select(".product-card"):
    print(card.select_one(".title").get_text(strip=True))

Note the pattern: the browser's only job is to produce HTML, then Beautiful Soup does the extraction exactly as before. Everything you learned above still applies.

Full setups for both are in our Playwright web scraping guide and Python Selenium guide; the headless browser guide covers the tradeoffs between them.

The third option is having something else run the browser. js=true is the default on our /html endpoint, so you get rendered HTML back from a plain requests.get — no Chromium on your machine, no memory leaks in a long-running job:

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://api.webscraping.ai/html",
    params={
        "api_key": API_KEY,
        "url": "https://example.com/products",
        "wait_for": ".product-card",   # wait for this selector before returning
    },
    timeout=60,
)
soup = BeautifulSoup(response.text, "lxml")

Why am I getting 403 Forbidden?

Two very different failures wear the same status code, and the timing tells you which one you have.

403 on the first request = a headers problem. The default python-requests/2.x User-Agent is an announcement that you're a bot. Send a realistic browser header set — and send the whole set, since a Chrome User-Agent with no Accept-Language is its own kind of tell:

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "https://www.google.com/",
    "Upgrade-Insecure-Requests": "1",
}

session = requests.Session()
session.headers.update(HEADERS)
response = session.get(url, timeout=20)

Keep that User-Agent current — a Chrome version from three years ago is as suspicious as no Chrome at all. Copy the real string from your own browser's console (navigator.userAgent). If you're rotating across many requests, our guide to user agent rotation covers doing it without creating inconsistent fingerprints.

Using a Session matters beyond convenience: it reuses the TCP connection (faster) and persists cookies across requests, which many sites require after the first page sets one.

Still 403 with perfect headers? The site is fingerprinting your TLS handshake. Python's requests has a cipher-suite signature no browser produces, and services like Cloudflare check it. curl_cffi speaks the exact TLS profile of a real browser:

from curl_cffi import requests as cffi_requests

response = cffi_requests.get(url, impersonate="chrome", timeout=20)

It's a near drop-in replacement for requests and it solves a category of block that no amount of header tweaking will.

403 (or 429) after N successful requests = rate limiting or IP reputation. Headers won't fix this. Slow down first, then change IPs — see the next two sections.

How do I avoid getting blocked or rate limited?

Politeness is also self-interest: a scraper that gets banned collects no data. Start with the site's own rules:

from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse, urljoin

def allowed(url, user_agent="*"):
    root = urlparse(url)
    rp = RobotFileParser()
    rp.set_url(urljoin(f"{root.scheme}://{root.netloc}", "/robots.txt"))
    rp.read()
    return rp.can_fetch(user_agent, url)

robots.txt also often publishes a Crawl-delay, which is a free answer to "how fast is acceptable here?"

Then build retries in properly. Hand-rolled time.sleep() loops miss the important cases; urllib3's Retry handles them, including honoring the server's own Retry-After header on a 429:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=5,
    backoff_factor=1,                                  # 1s, 2s, 4s, 8s, 16s
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods={"GET", "HEAD"},
    respect_retry_after_header=True,
)

session = requests.Session()
session.headers.update(HEADERS)
adapter = HTTPAdapter(max_retries=retry, pool_connections=10, pool_maxsize=10)
session.mount("https://", adapter)
session.mount("http://", adapter)

Concrete rules that keep scrapers alive:

• 1–2 requests per second per domain is a safe default for a site you don't own. Faster only if robots.txt or an API's documented limits say so
• Randomize your delay (time.sleep(random.uniform(1, 3))). Requests exactly 1.000s apart are a machine signature
• Never retry a 403 or 404 — retrying a block just deepens it. Retry 429s and 5xx only
• Back off on the first 429, don't wait for the ban. Double your delay and keep it doubled for the rest of the run
• Cache during development. Save responses to disk so that debugging a selector costs zero requests

When slowing down isn't enough, you need different IPs. Datacenter proxies are cheap and fast — start there. Move to residential only once datacenter IPs are getting blocked, because they cost several times more. Our guide to proxy types covers the tradeoffs; in requests the mechanics are trivial:

proxies = {
    "http": "http://user:pass@proxy.example.com:8000",
    "https": "http://user:pass@proxy.example.com:8000",
}
response = session.get(url, proxies=proxies, timeout=20)

Being blocked is rarely a legal question, but it's worth knowing where the lines are — scraping public data is generally lawful in the US, while bypassing authentication or ignoring a contract you accepted is a different matter. We cover the specifics in is web scraping legal?.

How do I store scraped data?

The mistake almost everyone makes once: accumulate everything in a Python list, write it at the end, and lose 40 minutes of work when page 180 throws an exception.

Write as you go. For a few thousand records, JSONL — one JSON object per line — is the best default. It's append-only, survives a crash mid-run, handles nested data, and streams:

import json

with open("products.jsonl", "a", encoding="utf-8") as f:
    for record in records:
        f.write(json.dumps(record, ensure_ascii=False) + "\n")

That also gives you free resumability:

import os

seen = set()
if os.path.exists("products.jsonl"):
    with open("products.jsonl", encoding="utf-8") as f:
        seen = {json.loads(line)["url"] for line in f}

for url in all_urls:
    if url in seen:
        continue
    # ... scrape and append ...

For anything you'll re-run on a schedule, use SQLite. A unique key plus ON CONFLICT turns a scraper into an idempotent job you can run daily without duplicates:

import sqlite3

conn = sqlite3.connect("products.db")
conn.execute("""
    CREATE TABLE IF NOT EXISTS products (
        url   TEXT PRIMARY KEY,
        title TEXT,
        price REAL,
        seen_at TEXT DEFAULT CURRENT_TIMESTAMP
    )
""")

conn.executemany("""
    INSERT INTO products (url, title, price) VALUES (:url, :title, :price)
    ON CONFLICT(url) DO UPDATE SET
        title = excluded.title,
        price = excluded.price,
        seen_at = CURRENT_TIMESTAMP
""", records)
conn.commit()

Clean on write, not later. Prices as floats, dates as ISO strings, whitespace normalized with " ".join(text.split()) — a "$1,299.99" string in your database is a bug you'll pay for during analysis.

How do I make my scraper faster?

Sequential requests spends most of its life waiting on the network. httpx plus asyncio fetches concurrently, with a semaphore capping how hard you hit one host:

import asyncio
import httpx
from bs4 import BeautifulSoup

async def fetch(client, sem, url):
    async with sem:
        try:
            response = await client.get(url, timeout=20)
            response.raise_for_status()
            return url, response.text
        except httpx.HTTPError as e:
            return url, None

async def scrape_all(urls, concurrency=5):
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient(headers=HEADERS, follow_redirects=True) as client:
        return await asyncio.gather(*(fetch(client, sem, u) for u in urls))

results = asyncio.run(scrape_all(urls))
for url, html in results:
    if html is None:
        continue
    soup = BeautifulSoup(html, "lxml")
    # ... extract ...

The Semaphore is not optional. asyncio.gather over 2,000 URLs without one fires 2,000 simultaneous requests, which is functionally a denial-of-service attempt and will get you blocked in seconds. Five to ten concurrent requests per domain is a reasonable ceiling.

Two other speedups worth more than concurrency, in order of impact:

1. Hit the JSON API instead of HTML. 50 records per request beats 20, with no parsing at all
2. Use lxml as your parser, and parse only the fragment you need (SoupStrainer) on large documents

If you're crawling tens of thousands of pages, stop hand-rolling and use Scrapy — it gives you scheduling, deduplication, retries, and concurrency as framework features. Our Python scraping libraries comparison and the Scrapy FAQ cover when that switch pays off.

When should you stop rolling your own?

Requests plus Beautiful Soup is genuinely the right answer for most jobs, and the honest advice is to stay there as long as it works. The switch to a managed API makes sense when your bug reports stop being about data and start being about infrastructure:

• You're maintaining proxy rotation, and a meaningful share of your week goes to swapping providers
• You're running headless Chrome in production and fighting memory leaks, zombie processes, or container images
• The same scraper works locally and fails in your datacenter, because of the IP, not the code
• You're solving CAPTCHAs, or your success rate quietly dropped and you can't tell which layer broke

That's the point where a scraping API is cheaper than the maintenance. It handles rendering, proxies, and retries behind one HTTP call — and your parsing code doesn't change at all:

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://api.webscraping.ai/html",
    params={
        "api_key": API_KEY,
        "url": "https://example.com/products",
        "js": "true",
        "proxy": "residential",     # datacenter (default) | residential | stealth
        "country": "us",
    },
    timeout=60,
)
soup = BeautifulSoup(response.text, "lxml")   # identical to everything above

When the page structure changes often enough that maintaining selectors is the real cost, describe the fields instead of locating them. The /ai/fields endpoint returns structured JSON from a plain-English description:

data = requests.get(
    "https://api.webscraping.ai/ai/fields",
    params={
        "api_key": API_KEY,
        "url": "https://example.com/products/123",
        "fields[title]": "Product title",
        "fields[price]": "Current price as a number, no currency symbol",
        "fields[in_stock]": "Whether the product is in stock, true or false",
    },
    timeout=60,
).json()
# {"title": "...", "price": "129.00", "in_stock": "true"}

Costs are published per request type: 1 credit for a datacenter request without JavaScript, 5 with it, 10/25 for residential, 50 for stealth, and +5 for AI extraction. Failed requests are free, so blocks don't burn quota. The free tier is 2,000 credits a month with no credit card, which is enough to test whether the infrastructure problem you're fighting actually goes away.

Teams typically arrive here through a specific project — price monitoring across retailers that block datacenter IPs, job listing aggregation across boards with different layouts, or building a RAG knowledge base from thousands of documentation pages.

Common errors and what they actually mean

Where to go deeper

This post is the map; these are the territories:

• Beautiful Soup guide — the full parsing API: find, select, navigation, attribute extraction
• Python web scraping libraries — Scrapy vs. Selenium vs. requests vs. the rest, with selection criteria
• Python Selenium and Playwright — browser automation in depth
• XPath cheat sheet — the other selector language, and when it beats CSS
• Python XML parsing — lxml, sitemaps, feeds, and namespaced documents
• urllib3 guide — connection pooling and retries one layer below requests
• Requests FAQ and Python FAQ — specific how-do-I questions

Start with the 15-line scraper at the top of this page against a site you actually care about. You'll hit pagination within an hour and JavaScript rendering within a day, and by then you'll know exactly which section to come back to.

Ready to skip the proxy and browser infrastructure entirely? Get 2,000 free API credits — no credit card required.