Beautiful Soup Web Scraping in Python: The Complete Guide

Beautiful Soup is Python's most popular HTML parsing library, and for good reason: it turns messy real-world HTML into a tree you can search with three or four intuitive methods, and it forgives markup that would crash stricter parsers. This guide is a complete, current tutorial — installing Beautiful Soup 4 with the right parser, extracting data with find(), find_all(), and CSS selectors, handling the pages where Beautiful Soup alone isn't enough, and saving results to CSV and JSON.

Key Takeaways

• Install with pip install beautifulsoup4 lxml — the lxml parser is markedly faster than the built-in html.parser
• find() returns the first match, find_all() returns them all, select() takes CSS selectors — those three methods cover 90% of scraping work
• Beautiful Soup parses HTML; it doesn't fetch it — pair it with requests (static pages) or a browser/rendering API (JavaScript pages)
• Match multi-class elements with class_="badge" (any class matches) or select(".badge.sale") (all must match)
• Beautiful Soup cannot execute JavaScript — if data is missing from page.text, the site rendered it client-side
• "Beautiful Soup" today means Beautiful Soup 4 (bs4); BS3 has been unmaintained since 2012 and is Python 2 only

What is Beautiful Soup?

Beautiful Soup is a Python library that parses HTML and XML documents into a navigable object tree. You hand it markup — including broken, unclosed, real-world markup — and query it by tag name, attributes, CSS class, or text content. It sits on top of a parser (lxml, html.parser, or html5lib) and adds the ergonomic search API those parsers lack.

What it deliberately doesn't do: make HTTP requests, execute JavaScript, or crawl links. Beautiful Soup is the parsing layer; you bring the HTML. That separation is why it pairs so cleanly with everything — requests, httpx, Selenium, Playwright, or a scraping API all just produce HTML for Beautiful Soup to dissect.

Installing Beautiful Soup

pip install beautifulsoup4 requests lxml

That's the package (beautifulsoup4), an HTTP client (requests), and the recommended parser (lxml). The import name differs from the package name — a classic first-day stumble:

from bs4 import BeautifulSoup   # package: beautifulsoup4, module: bs4

If pip isn't found or installs to the wrong Python, python -m pip install beautifulsoup4 removes the ambiguity. In a fresh project, use a virtual environment (python -m venv venv) so library versions stay per-project.

Choosing a parser

Beautiful Soup delegates actual parsing to one of three backends, chosen in the BeautifulSoup() call:

soup = BeautifulSoup(html, "lxml")          # recommended
soup = BeautifulSoup(html, "html.parser")   # zero-dependency fallback
soup = BeautifulSoup(html, "html5lib")      # browser-identical tree building

Always pass the parser name explicitly — if you omit it, Beautiful Soup picks the "best" installed parser, so the same code can build different trees on different machines. The differences only surface on malformed HTML, which is exactly when they're hardest to debug. Use lxml unless a page's broken markup parses wrong, then try html5lib, whose browser-grade error recovery matches what you see in DevTools.

Your first scraper

Fetch a page with requests, parse it, extract structured data:

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/"
response = requests.get(url, timeout=30,
                        headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"})
response.raise_for_status()

soup = BeautifulSoup(response.text, "lxml")

for book in soup.find_all("article", class_="product_pod"):
    title = book.h3.a["title"]
    price = book.find("p", class_="price_color").get_text(strip=True)
    in_stock = "In stock" in book.find("p", class_="instock").get_text()
    print(f"{title} — {price} — {'✓' if in_stock else '✗'}")

Three things worth copying into every scraper: a timeout (requests hangs forever without one), raise_for_status() (fail loudly on 403/500 instead of parsing an error page), and a real User-Agent header (the default python-requests/2.x is the first thing basic bot filters block).

find() and find_all()

The core search API. find() returns the first matching element (or None); find_all() returns a list of all matches:

soup.find("h1")                          # first h1
soup.find("div", id="main")              # by id
soup.find("p", class_="intro")           # by class (note the underscore)
soup.find_all("a")                       # every link
soup.find_all("a", href=True)            # links that have an href
soup.find_all("span", attrs={"data-price": True})   # any attribute
soup.find_all("p", limit=5)              # cap the result count
soup.find_all(["h1", "h2", "h3"])        # any of several tags

class_ has a trailing underscore because class is a Python keyword. Beyond strings, filters accept regular expressions and functions:

import re
soup.find_all("a", href=re.compile(r"^/product/"))       # regex on attribute
soup.find_all(string=re.compile("free shipping", re.I))  # regex on text
soup.find_all(lambda tag: tag.name == "div" and len(tag.get("class", [])) > 2)

A find() that matches nothing returns None, so chained access like soup.find("div", class_="price").text throws AttributeError: 'NoneType' object has no attribute 'text' — the most common Beautiful Soup exception. Guard the chain:

price_tag = soup.find("div", class_="price")
price = price_tag.get_text(strip=True) if price_tag else None

CSS selectors with select()

If you think in CSS, select() and select_one() accept full CSS selector syntax:

soup.select("div.product > h2")          # direct child
soup.select("#reviews .stars")           # id and class nesting
soup.select("a[href^='https://']")       # attribute prefix
soup.select("tr:nth-of-type(odd)")       # positional pseudo-class
soup.select_one("meta[property='og:price']")["content"]

select() is also the cleanest answer to a subtle multi-class question. HTML class is a list of classes, and the two APIs treat it differently:

# <span class="badge sale featured">

soup.find_all("span", class_="sale")          # matches — ANY class equals "sale"
soup.select("span.badge.sale")                # matches — has BOTH classes
soup.find_all("span", class_="badge sale")    # fragile: exact string, order-dependent

Use class_="x" when one class is enough to identify the element; use select(".x.y") when you need the combination. For matching exactly a set of classes regardless of order, compare the list: [el for el in soup.find_all("span") if set(el.get("class", [])) == {"badge", "sale"}].

CSS selectors can't match by text content — that's find_all(string=...) territory, or XPath via lxml directly when you need text matching plus tree traversal in one expression.

Extracting text and attributes

Element text comes from .get_text(), attributes via dictionary access:

el = soup.find("div", class_="description")

el.get_text()                        # all nested text, whitespace preserved
el.get_text(strip=True)              # trimmed
el.get_text(" ", strip=True)         # nested fragments joined by spaces
el["href"]                           # attribute — KeyError if absent
el.get("href")                       # attribute — None if absent
el.get("class", [])                  # class is always a list

get_text(" ", strip=True) is the everyday form — without the separator, <p>New<br>York</p> extracts as "NewYork". For machine-readable data, check attributes before parsing visible text: prices and dates often hide in content, datetime, or data-* attributes in cleaner formats than the display text (<span class="price" data-amount="1999">$19.99</span>).

Two related APIs worth knowing: .string returns the text only when an element has a single text child (otherwise None — prefer get_text()), and .stripped_strings iterates every text fragment in a subtree, pre-trimmed.

Navigating the tree

When the target element has no useful class of its own, navigate from one that does:

el.parent                              # up one level
el.find_parent("table")                # nearest ancestor by tag
el.find_next_sibling("td")             # sideways, elements only
el.find_previous_sibling()
el.contents                            # direct children (list)
el.children                            # direct children (iterator)
el.descendants                         # everything beneath

The label/value pattern — find the cell whose text you know, take its sibling — handles most spec tables:

label = soup.find("th", string="ISBN")
isbn = label.find_next_sibling("td").get_text(strip=True) if label else None

Prefer find_next_sibling() over .next_sibling: the latter returns nodes, including the whitespace text node between tags, and iterating it correctly is fiddly.

Cleaning and modifying the tree

Beautiful Soup can edit the parse tree — the scraping use case is cleanup before text extraction:

for tag in soup(["script", "style", "noscript"]):
    tag.decompose()                    # delete element and its contents

article_text = soup.get_text(" ", strip=True)   # now free of JS/CSS junk

decompose() destroys an element permanently; extract() removes and returns it; unwrap() removes a tag but keeps its children (useful for stripping inline formatting like <b> from text). These mutate the soup in place — extract data you need before aggressive cleanup passes.

Malformed HTML

Real pages ship unclosed tags, misnested lists, and orphaned table cells. Beautiful Soup's tolerance for this is its founding feature — every parser repairs broken markup into some valid tree, they just disagree on how:

broken = "<ul><li>One<li>Two<td>stray cell</ul>"
BeautifulSoup(broken, "lxml")       # one repair strategy
BeautifulSoup(broken, "html5lib")   # the browser's repair strategy

Practical consequences: when your selector matches in DevTools but not in code, the parser may have repaired the tree differently than the browser did — switch to html5lib to get the browser's tree. And when scraping tables specifically, verify the repaired structure (print(table.prettify())) before trusting row/cell indexing; stray </tr> tags relocate cells in parser-dependent ways.

JavaScript pages: when Beautiful Soup sees nothing

Beautiful Soup parses the HTML the server sent. If a site builds its content client-side — React, Vue, infinite scroll, prices loaded by API call — that HTML is a nearly empty shell, and no parser can extract what isn't there. Diagnose it in two seconds: view the page source (Ctrl+U, not DevTools) and search for your data. Missing → it's rendered by JavaScript.

The options, in order of preference:

1. Find the underlying JSON API. Open DevTools → Network → XHR while the page loads. Sites that render client-side fetch their data from somewhere, and consuming that JSON directly beats HTML parsing entirely.
2. Check for embedded JSON. Frameworks often ship data in <script type="application/ld+json"> or a __NEXT_DATA__ blob — parseable with json.loads(soup.find("script", id="__NEXT_DATA__").string).
3. Render the page, then parse. Drive a real browser with Playwright or Selenium and feed the rendered HTML to Beautiful Soup — BeautifulSoup(page.content(), "lxml"). Your parsing code doesn't change.
4. Use a rendering API. Offload the browser fleet (see the last section) and keep the same two-library stack.

Sessions, logins, and headers

For pages behind a login, requests.Session persists cookies across requests — log in once, scrape with the session:

session = requests.Session()
session.headers["User-Agent"] = "Mozilla/5.0 (X11; Linux x86_64)"

login_page = BeautifulSoup(session.get("https://example.com/login").text, "lxml")
token = login_page.find("input", {"name": "csrf_token"})["value"]   # hidden CSRF field

session.post("https://example.com/login",
             data={"user": "me", "password": "secret", "csrf_token": token})

dashboard = BeautifulSoup(session.get("https://example.com/account").text, "lxml")

The CSRF-token dance shown above is the step most login scrapers miss: modern forms reject POSTs without the hidden token from the form page. Sites with JavaScript-driven logins (SSO redirects, CAPTCHA at signin) are better handled by logging in with a browser and exporting its cookies into the session.

Be polite while you're in there: throttle with time.sleep() between requests, respect robots.txt, and back off on 429 responses. Beyond etiquette, aggressive scraping gets IPs banned — the legality guide covers where the actual legal lines run.

Saving scraped data to CSV and JSON

Collect rows as dictionaries, then write once at the end:

import csv, json

books = [{"title": t, "price": p} for t, p in scraped_items]

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price"])
    writer.writeheader()
    writer.writerows(books)

with open("books.json", "w", encoding="utf-8") as f:
    json.dump(books, f, indent=2, ensure_ascii=False)

newline="" prevents blank lines in Windows CSVs; ensure_ascii=False keeps non-English text readable; and encoding="utf-8" on every file handle avoids the platform-default-encoding surprises. For larger jobs, pandas.DataFrame(books).to_csv(...) adds dedup, sorting, and Excel export in one dependency.

Beautiful Soup 4 vs Beautiful Soup 3

All modern code uses Beautiful Soup 4 (bs4), and every current tutorial means BS4 when it says "Beautiful Soup." BS3 matters only as a historical artifact you might meet in decade-old code:

Porting BS3 code is mostly mechanical: change the import, swap findAll → find_all (BS4 keeps the old names as deprecated aliases), and pick an explicit parser. If you're on Python 3, BS3 won't even install — there is no version decision to make, only old code to migrate.

Beautiful Soup vs Scrapy vs Selenium

The perennial "which tool" question is really three different jobs:

• Beautiful Soup is a parsing library — smallest learning curve, perfect for scripts, notebooks, and anything up to thousands of pages. It has no crawling engine, no concurrency, no JavaScript.
• Scrapy is a crawling framework — request scheduling, concurrent fetching, pipelines, retries, and export built in. Worth its steeper setup once a project means "crawl this whole site on a schedule," not "parse these pages."
• Selenium/Playwright are browser automation — the JavaScript-execution layer. They're not parsers; teams routinely feed browser page source into Beautiful Soup.

They compose: requests + Beautiful Soup for static sites, Playwright + Beautiful Soup when rendering is needed, Scrapy when crawl orchestration dominates. Start with Beautiful Soup; graduate when a specific limitation — not fashion — forces it. Our Python web scraping libraries roundup compares the full landscape.

Scaling past the blocking wall

Beautiful Soup never gets blocked — but requests does. At small scale, headers and delays suffice; past that, sites answer with 403s, CAPTCHAs, and JavaScript challenges that no parsing library can address. WebScraping.AI solves the fetch side — rotating residential proxies, headless Chrome rendering, anti-bot handling — and returns HTML straight into the Beautiful Soup code you already have:

import requests
from bs4 import BeautifulSoup

html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/products",
    "js": "true",                     # render JavaScript before returning
}).text

soup = BeautifulSoup(html, "lxml")    # your parsing code is unchanged

There's also a shortcut past parsing altogether: the /ai/fields endpoint takes field descriptions ("product name", "price with currency") and returns structured JSON from any page — useful when a site's markup churns too fast for selectors to keep up.

Frequently asked questions

Is Beautiful Soup good for web scraping in 2026? Yes — it remains the standard Python HTML parser and the right starting point for most projects. What's changed is the web around it: more sites render client-side and run bot protection, so Beautiful Soup increasingly pairs with a rendering layer (Playwright or a scraping API) rather than plain requests alone.

Why is my element missing even though I see it in the browser? Almost always JavaScript rendering: the browser shows the rendered DOM, Beautiful Soup parses the served HTML. Confirm with View Source. The fixes, in order: find the site's JSON API, look for embedded __NEXT_DATA__/JSON-LD, or render with a browser first. Occasionally it's parser repair instead — retry with html5lib.

What's the difference between find_all() and select()? Capability overlap, different syntax: find_all() uses keyword arguments and accepts regexes/functions as filters; select() uses CSS selector strings and handles nesting/combinators more naturally. Use whichever reads cleaner per query — mixing them in one scraper is normal.

Does Beautiful Soup work with XML? Yes — install lxml and pass features="xml", which parses strictly and preserves case-sensitive tag names. For huge XML files or XPath queries, use lxml.etree directly; Beautiful Soup's convenience layer costs memory at that scale.

How fast is Beautiful Soup? With the lxml backend, parsing a typical page takes single-digit milliseconds — network latency will dominate your scraper's runtime by 100×. If parsing genuinely becomes the bottleneck (millions of documents), dropping to raw lxml/XPath roughly halves parse time at the cost of the friendlier API.

Can Beautiful Soup handle infinite scroll or pagination? Pagination, yes — follow the next link in a loop, fetching each page URL. Infinite scroll, no — that's JavaScript loading data as you scroll; capture the underlying API calls from the Network tab or scroll a real browser with Playwright, then parse the accumulated HTML.