--- title: "Beautiful Soup Web Scraping in Python: The Complete Guide" description: "Web scraping with Beautiful Soup and Python: install, find() and find_all(), CSS selectors, text extraction, dynamic pages, and saving data to CSV/JSON." url: https://webscraping.ai/blog/beautiful-soup-python markdown_index: https://webscraping.ai/llms.txt --- # Beautiful Soup Web Scraping in Python: The Complete Guide Beautiful Soup is Python's most popular HTML parsing library, and for good reason: it turns messy real-world HTML into a tree you can search with three or four intuitive methods, and it forgives markup that would crash stricter parsers. This guide is a complete, current tutorial — installing Beautiful Soup 4 with the right parser, extracting data with `find()`, `find_all()`, and CSS selectors, handling the pages where Beautiful Soup alone isn't enough, and saving results to CSV and JSON. ## Key Takeaways - Install with `pip install beautifulsoup4 lxml` — the `lxml` parser is markedly faster than the built-in `html.parser` - `find()` returns the first match, `find_all()` returns them all, `select()` takes CSS selectors — those three methods cover 90% of scraping work - Beautiful Soup parses HTML; it doesn't fetch it — pair it with `requests` (static pages) or a browser/rendering API (JavaScript pages) - Match multi-class elements with `class_="badge"` (any class matches) or `select(".badge.sale")` (all must match) - Beautiful Soup cannot execute JavaScript — if data is missing from `page.text`, the site rendered it client-side - "Beautiful Soup" today means Beautiful Soup 4 (`bs4`); BS3 has been unmaintained since 2012 and is Python 2 only ## What is Beautiful Soup? Beautiful Soup is a Python library that parses HTML and XML documents into a navigable object tree. You hand it markup — including broken, unclosed, real-world markup — and query it by tag name, attributes, CSS class, or text content. It sits on top of a parser (`lxml`, `html.parser`, or `html5lib`) and adds the ergonomic search API those parsers lack. What it deliberately doesn't do: make HTTP requests, execute JavaScript, or crawl links. Beautiful Soup is the _parsing_ layer; you bring the HTML. That separation is why it pairs so cleanly with everything — `requests`, `httpx`, Selenium, Playwright, or a scraping API all just produce HTML for Beautiful Soup to dissect. ## Installing Beautiful Soup ```bash pip install beautifulsoup4 requests lxml ``` That's the package (`beautifulsoup4`), an HTTP client (`requests`), and the recommended parser (`lxml`). The import name differs from the package name — a classic first-day stumble: ```python from bs4 import BeautifulSoup # package: beautifulsoup4, module: bs4 ``` If `pip` isn't found or installs to the wrong Python, `python -m pip install beautifulsoup4` removes the ambiguity. In a fresh project, use a virtual environment (`python -m venv venv`) so library versions stay per-project. ### Choosing a parser Beautiful Soup delegates actual parsing to one of three backends, chosen in the `BeautifulSoup()` call: | Parser | Install | Speed | Behavior | | --- | --- | --- | --- | | `lxml` | `pip install lxml` | Fastest (C-based) | Lenient, the default choice for scraping | | `html.parser` | Built into Python | Moderate | No dependencies, slightly different error recovery | | `html5lib` | `pip install html5lib` | Slowest (pure Python) | Parses exactly like a web browser | ```python soup = BeautifulSoup(html, "lxml") # recommended soup = BeautifulSoup(html, "html.parser") # zero-dependency fallback soup = BeautifulSoup(html, "html5lib") # browser-identical tree building ``` Always pass the parser name explicitly — if you omit it, Beautiful Soup picks the "best" installed parser, so the same code can build _different trees on different machines_. The differences only surface on malformed HTML, which is exactly when they're hardest to debug. Use `lxml` unless a page's broken markup parses wrong, then try `html5lib`, whose browser-grade error recovery matches what you see in DevTools. ## Your first scraper Fetch a page with `requests`, parse it, extract structured data: ```python import requests from bs4 import BeautifulSoup url = "https://books.toscrape.com/" response = requests.get(url, timeout=30, headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"}) response.raise_for_status() soup = BeautifulSoup(response.text, "lxml") for book in soup.find_all("article", class_="product_pod"): title = book.h3.a["title"] price = book.find("p", class_="price_color").get_text(strip=True) in_stock = "In stock" in book.find("p", class_="instock").get_text() print(f"{title} — {price} — {'✓' if in_stock else '✗'}") ``` Three things worth copying into every scraper: a `timeout` (requests hangs forever without one), `raise_for_status()` (fail loudly on 403/500 instead of parsing an error page), and a real `User-Agent` header (the default `python-requests/2.x` is the first thing basic bot filters block). ## find() and find\_all() The core search API. `find()` returns the first matching element (or `None`); `find_all()` returns a list of all matches: ```python soup.find("h1") # first h1 soup.find("div", id="main") # by id soup.find("p", class_="intro") # by class (note the underscore) soup.find_all("a") # every link soup.find_all("a", href=True) # links that have an href soup.find_all("span", attrs={"data-price": True}) # any attribute soup.find_all("p", limit=5) # cap the result count soup.find_all(["h1", "h2", "h3"]) # any of several tags ``` `class_` has a trailing underscore because `class` is a Python keyword. Beyond strings, filters accept regular expressions and functions: ```python import re soup.find_all("a", href=re.compile(r"^/product/")) # regex on attribute soup.find_all(string=re.compile("free shipping", re.I)) # regex on text soup.find_all(lambda tag: tag.name == "div" and len(tag.get("class", [])) > 2) ``` A `find()` that matches nothing returns `None`, so chained access like `soup.find("div", class_="price").text` throws `AttributeError: 'NoneType' object has no attribute 'text'` — the most common Beautiful Soup exception. Guard the chain: ```python price_tag = soup.find("div", class_="price") price = price_tag.get_text(strip=True) if price_tag else None ``` ## CSS selectors with select() If you think in CSS, `select()` and `select_one()` accept full CSS selector syntax: ```python soup.select("div.product > h2") # direct child soup.select("#reviews .stars") # id and class nesting soup.select("a[href^='https://']") # attribute prefix soup.select("tr:nth-of-type(odd)") # positional pseudo-class soup.select_one("meta[property='og:price']")["content"] ``` `select()` is also the cleanest answer to a subtle multi-class question. HTML `class` is a _list_ of classes, and the two APIs treat it differently: ```python # soup.find_all("span", class_="sale") # matches — ANY class equals "sale" soup.select("span.badge.sale") # matches — has BOTH classes soup.find_all("span", class_="badge sale") # fragile: exact string, order-dependent ``` Use `class_="x"` when one class is enough to identify the element; use `select(".x.y")` when you need the combination. For matching _exactly_ a set of classes regardless of order, compare the list: `[el for el in soup.find_all("span") if set(el.get("class", [])) == {"badge", "sale"}]`. CSS selectors can't match by text content — that's `find_all(string=...)` territory, or [XPath](https://webscraping.ai/blog/xpath-cheat-sheet) via `lxml` directly when you need text matching plus tree traversal in one expression. ## Extracting text and attributes Element text comes from `.get_text()`, attributes via dictionary access: ```python el = soup.find("div", class_="description") el.get_text() # all nested text, whitespace preserved el.get_text(strip=True) # trimmed el.get_text(" ", strip=True) # nested fragments joined by spaces el["href"] # attribute — KeyError if absent el.get("href") # attribute — None if absent el.get("class", []) # class is always a list ``` `get_text(" ", strip=True)` is the everyday form — without the separator, `

New
York

` extracts as `"NewYork"`. For machine-readable data, check attributes before parsing visible text: prices and dates often hide in `content`, `datetime`, or `data-*` attributes in cleaner formats than the display text (`$19.99`). Two related APIs worth knowing: `.string` returns the text only when an element has a single text child (otherwise `None` — prefer `get_text()`), and `.stripped_strings` iterates every text fragment in a subtree, pre-trimmed. ## Navigating the tree When the target element has no useful class of its own, navigate from one that does: ```python el.parent # up one level el.find_parent("table") # nearest ancestor by tag el.find_next_sibling("td") # sideways, elements only el.find_previous_sibling() el.contents # direct children (list) el.children # direct children (iterator) el.descendants # everything beneath ``` The label/value pattern — find the cell whose text you know, take its sibling — handles most spec tables: ```python label = soup.find("th", string="ISBN") isbn = label.find_next_sibling("td").get_text(strip=True) if label else None ``` Prefer `find_next_sibling()` over `.next_sibling`: the latter returns _nodes_, including the whitespace text node between tags, and iterating it correctly is fiddly. ## Cleaning and modifying the tree Beautiful Soup can edit the parse tree — the scraping use case is cleanup before text extraction: ```python for tag in soup(["script", "style", "noscript"]): tag.decompose() # delete element and its contents article_text = soup.get_text(" ", strip=True) # now free of JS/CSS junk ``` `decompose()` destroys an element permanently; `extract()` removes and returns it; `unwrap()` removes a tag but keeps its children (useful for stripping inline formatting like `` from text). These mutate the soup in place — extract data you need _before_ aggressive cleanup passes. ## Malformed HTML Real pages ship unclosed tags, misnested lists, and orphaned table cells. Beautiful Soup's tolerance for this is its founding feature — every parser repairs broken markup into _some_ valid tree, they just disagree on how: ```python broken = "" BeautifulSoup(broken, "lxml") # one repair strategy BeautifulSoup(broken, "html5lib") # the browser's repair strategy ``` Practical consequences: when your selector matches in DevTools but not in code, the parser may have repaired the tree differently than the browser did — switch to `html5lib` to get the browser's tree. And when scraping tables specifically, verify the repaired structure (`print(table.prettify())`) before trusting row/cell indexing; stray `` tags relocate cells in parser-dependent ways. ## JavaScript pages: when Beautiful Soup sees nothing Beautiful Soup parses the HTML the server sent. If a site builds its content client-side — React, Vue, infinite scroll, prices loaded by API call — that HTML is a nearly empty shell, and no parser can extract what isn't there. Diagnose it in two seconds: view the page source (Ctrl+U, not DevTools) and search for your data. Missing → it's rendered by JavaScript. The options, in order of preference: 1. **Find the underlying JSON API.** Open DevTools → Network → XHR while the page loads. Sites that render client-side fetch their data from somewhere, and consuming that JSON directly beats HTML parsing entirely. 2. **Check for embedded JSON.** Frameworks often ship data in `