Beautiful Soup is a Python library for pulling data out of HTML and XML files. It creates a parse tree from page source code that can be used to extract data easily. The main methods provided by Beautiful Soup for navigating the parse tree are as follows:
Navigating using tag names
You can navigate the parse tree by calling a tag name as an attribute. It gives you the first occurrence of that tag:
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.title) # Returns the first <title> tag
Navigating using .find() and .find_all()
These methods allow you to search for all instances of a tag, and .find() returns the first match, while .find_all() returns a list of all matches:
first_paragraph = soup.find('p') # Finds the first <p> tag
all_paragraphs = soup.find_all('p') # Finds all <p> tags
Navigating using CSS selectors
The .select() method allows you to use CSS selectors to find elements:
all_paragraphs = soup.select('p') # Select all <p> tags
first_paragraph = soup.select_one('p') # Select the first <p> tag
Accessing tag contents and attributes
You can access a tag’s children by using .contents to get a list of a tag’s children or .children to iterate over them. You can access attributes like accessing dictionary values:
tag = soup.p
print(tag['class']) # Returns the class attribute
print(tag.attrs) # Returns all attributes as a dictionary
Navigating the tree
Beautiful Soup provides several ways to navigate the tree:
.parent: Accesses the parent of a tag..parents: Iterates over all parents of a tag..next_siblingand.previous_sibling: Navigate between page elements that are on the same level of the parse tree..next_siblingsand.previous_siblings: Iterators to loop over a tag’s siblings..next_elementand.previous_element: Navigate to the next or previous element in the tree, not just direct siblings..next_elementsand.previous_elements: Iterators to loop over a tag’s next or previous elements.
Here's an example of navigating siblings:
sibling = soup.p.next_sibling.next_sibling # Gets the second sibling of the first <p> tag
Extracting all text
The .get_text() method extracts all the text in a document or under a tag:
text = soup.get_text() # Retrieves all text within the HTML document
These methods provide a powerful and intuitive way of navigating and searching the parse tree created by Beautiful Soup. They make it easy to extract information from HTML documents efficiently.
Here's a full example that uses a combination of the above methods:
from bs4 import BeautifulSoup
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
# Using tag names
print(soup.head)
print(soup.title)
# Using find/find_all
print(soup.find_all('a'))
# Using CSS selectors
print(soup.select('p.story'))
# Accessing tag contents and attributes
print(soup.a['href'])
# Navigating the tree
print(soup.body.p.next_sibling.next_sibling)
# Extracting all text
print(soup.get_text())
Note that Beautiful Soup is a Python-only library, so there are no JavaScript equivalents for these methods. However, similar functionality can be achieved in JavaScript using libraries such as Cheerio or using the native DOM API (e.g., document.querySelector, document.querySelectorAll, etc.).
