Python XML Parsing: ElementTree, lxml, and How to Choose

Python ships with an XML parser in the standard library, and for many jobs that's all you need. But the moment you hit a huge feed that doesn't fit in memory, a document with namespaces, broken HTML, or an XPath expression more ambitious than a tag name, you'll want lxml. This guide covers both: xml.etree.ElementTree for the simple cases, lxml for everything else, and the operational details — encoding, memory, thread safety, and the security flags you should never leave at their defaults when parsing untrusted input.

Key Takeaways

• xml.etree.ElementTree is built in and fine for small, well-formed XML; lxml is a C-backed superset with full XPath, HTML repair, XSLT, and validation
• lxml parses the same API shape as ElementTree — most code migrates by changing one import
• For files bigger than RAM, use iterparse() and clear elements as you go; that keeps memory flat regardless of file size
• lxml is also one of the best HTML parsers in Python — lxml.html handles real-world broken markup
• Never parse untrusted XML with entity resolution enabled — XXE attacks are a parser configuration problem, not an exotic edge case
• For web scraping, pair lxml with an HTTP client or a web scraping API and query with XPath or CSS selectors

Python's XML parsers at a glance

The practical decision tree: stdlib ElementTree if the file is small and clean, lxml if you need speed, XPath, HTML, or streaming — and defusedxml wrappers if the input comes from strangers (more on that below).

ElementTree: the standard library baseline

import xml.etree.ElementTree as ET

tree = ET.parse("catalog.xml")          # from a file
root = tree.getroot()

root = ET.fromstring(xml_string)        # from a string

for book in root.iter("book"):          # every <book>, any depth
    title = book.find("title").text
    price = book.get("price")           # attribute
    print(title, price)

find() returns the first match, findall() a list, iter() walks the whole subtree. ElementTree supports only a small path subset — .//book[@lang='en'] works, but functions like contains(), axes like following-sibling, or selecting attributes directly do not. When your query outgrows that, it's lxml time.

Installing lxml

pip install lxml

Binary wheels exist for every major platform, so on modern Python this installs in seconds with no compiler — the old advice about needing libxml2-dev/libxslt-dev system packages only applies when pip falls back to building from source (unusual architectures, brand-new Python releases).

Pinning a specific version works like any package:

pip install lxml==5.4.0            # exact version
pip install "lxml>=5,<6"           # range
pip show lxml                      # what's installed now

If a legacy project needs an old lxml on a new Python, check PyPI's release history for a version with wheels for your interpreter — very old lxml releases predate wheels for current Pythons and will try (and often fail) to compile.

Parsing XML with lxml

lxml deliberately mirrors the ElementTree API:

from lxml import etree

tree = etree.parse("catalog.xml")       # file or URL-like object
root = tree.getroot()

root = etree.fromstring(xml_bytes)      # from bytes

for book in root.iter("book"):
    print(book.findtext("title"), book.get("price"))

One encoding gotcha: etree.fromstring() refuses a str that carries an XML declaration like <?xml version="1.0" encoding="UTF-8"?> — the declaration promises bytes in a specific encoding, and a decoded string contradicts it. Pass bytes (xml_text.encode("utf-8")) or strip the declaration. When parsing files, lxml reads the declaration itself and handles the encoding for you; trust it rather than decoding manually.

Iterating the tree

for el in root.iter():                   # every element, document order
    print(el.tag, (el.text or "").strip())

for child in root:                       # direct children only
    ...

for para in root.iter("para"):           # filter by tag
    ...

for el in root.iterchildren("item"):     # lxml extras: children/descendants
    ...
for el in root.iterdescendants("price"):
    ...

Elements behave like lists of their children, so slicing (root[0], root[-1], root[1:3]) works too. lxml also gives every element getparent(), getprevious(), and getnext() — navigation ElementTree simply doesn't have.

XPath: lxml's superpower

xpath() evaluates full XPath 1.0 and is the main reason to choose lxml:

# All English-language book titles as strings
titles = root.xpath("//book[@lang='en']/title/text()")

# Attribute values directly
isbns = root.xpath("//book/@isbn")

# Text matching and counting
cheap = root.xpath("//book[number(price) < 20]")
n = root.xpath("count(//book)")

find()/findall() still exist and are slightly faster for trivial paths, but xpath() handles predicates, functions, axes, and returns strings/numbers/booleans, not just elements. For the expression syntax itself — contains(), following-sibling, position tricks, quote escaping — see our XPath cheat sheet.

Parsing HTML (including broken HTML)

Real-world HTML has unclosed tags, stray ampersands, and misnested elements. lxml.html repairs all of it, the same way browsers do:

import lxml.html

doc = lxml.html.fromstring(html_text)

# CSS selectors or XPath — your choice
for link in doc.cssselect("a.result"):        # pip install cssselect
    print(link.get("href"), link.text_content())

prices = doc.xpath("//span[contains(@class, 'price')]/text()")

text_content() returns all text inside an element, tags stripped — the workhorse for scraping. For HTML tables, pandas.read_html() uses lxml under the hood, or extract manually:

rows = doc.xpath("//table[@id='data']//tr")
data = [[td.text_content().strip() for td in row.xpath("./td")] for row in rows]

(Note the leading dot in ./td — inside a loop, an absolute //td would search the whole document from every row, a classic bug.)

Finding elements by class or id

doc.get_element_by_id("main")                  # lxml.html only
doc.xpath("//div[@id='main']")

# @class is a space-separated list — match one class safely:
doc.xpath("//div[contains(concat(' ', normalize-space(@class), ' '), ' product ')]")
doc.cssselect("div.product")                   # …or let CSS do it readably

Namespaces

Namespaced XML (<soap:Envelope>, Atom feeds, sitemaps) trips everyone up: //url finds nothing because every element actually lives in a namespace. Map prefixes explicitly:

ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
locs = root.xpath("//sm:url/sm:loc/text()", namespaces=ns)

The prefix (sm) is your choice — only the URI must match the document. For a default (unprefixed) namespace you still must register a prefix in the map; XPath has no way to say "the default namespace". The blunt instrument when you just don't care: match on local name, //*[local-name()='loc'].

Modifying documents and saving them

lxml trees are mutable:

for book in root.xpath("//book[@status='discontinued']"):
    book.getparent().remove(book)              # delete an element

for price in root.iter("price"):
    price.text = str(round(float(price.text) * 1.1, 2))
    price.set("currency", "USD")               # set/replace an attribute

new = etree.SubElement(root, "book", attrib={"isbn": "978-0"})
etree.SubElement(new, "title").text = "New Arrival"

Write the result back:

tree.write("catalog-updated.xml", encoding="utf-8",
           xml_declaration=True, pretty_print=True)

xml_bytes = etree.tostring(root, encoding="utf-8", pretty_print=True)

pretty_print only re-indents nodes that don't already have surrounding whitespace text; when re-indenting a parsed file, parse it with etree.XMLParser(remove_blank_text=True) so lxml is free to lay it out cleanly. To build documents from scratch, etree.Element() + SubElement() as above, or the E-factory (lxml.builder) for declarative construction.

Huge files: iterparse and flat memory

A DOM parse holds the whole tree in RAM — typically 5–10× the file size (this behavior on multi-GB documents is the most-cited performance complaint about lxml, and it's inherent to any in-memory tree). The fix is streaming with iterparse(), clearing elements once processed:

from lxml import etree

def fast_iter(path, tag):
    context = etree.iterparse(path, events=("end",), tag=tag)
    for _, elem in context:
        yield elem
        # Free the element and its preceding siblings
        elem.clear()
        while elem.getprevious() is not None:
            del elem.getparent()[0]
    del context

for record in fast_iter("huge-feed.xml", "product"):
    process(record.findtext("sku"))

Memory stays constant whether the file is 10 MB or 50 GB. The clear()-plus-delete-siblings dance matters: without it, processed elements stay attached to the root and memory grows anyway. For line-of-business sizes (under ~100 MB) skip the ceremony — a plain parse() is simpler and plenty fast: lxml parses on the order of hundreds of MB/s, typically several times faster than ElementTree and far ahead of minidom.

CDATA, comments, entities, and other special constructs

# Comments and processing instructions appear in iteration unless you opt out
parser = etree.XMLParser(remove_comments=True, remove_pis=True)
root = etree.fromstring(xml_bytes, parser)

# CDATA text is just .text when parsed; to *write* CDATA:
el.text = etree.CDATA("if (a < b) { ... }")

Documents with custom entity definitions (<!ENTITY co "Example Corp"> in an internal DTD) fail by default with "Entity 'co' not defined". Tell the parser to resolve them:

parser = etree.XMLParser(resolve_entities=True, load_dtd=True)
root = etree.parse("with-entities.xml", parser).getroot()

Only do this for XML you trust. Entity resolution is also the attack surface for XXE (XML External Entity) injection — a malicious document can define an entity that reads /etc/passwd or makes network requests. For untrusted input: leave resolve_entities=False, set no_network=True, or parse through the defusedxml wrappers, which disable the dangerous features across stdlib parsers and lxml alike.

Validation and XSLT

lxml validates against DTD, XML Schema (XSD), and RelaxNG:

schema = etree.XMLSchema(etree.parse("catalog.xsd"))
doc = etree.parse("catalog.xml")
if not schema.validate(doc):
    for error in schema.error_log:
        print(error.line, error.message)

And it's the only mainstream Python library with built-in XSLT 1.0:

transform = etree.XSLT(etree.parse("to-html.xsl"))
html_result = transform(doc)
print(str(html_result))

Thread safety

Safe: parsing in multiple threads concurrently — lxml releases the GIL inside libxml2, so parsing actually parallelizes across threads. Not safe: two threads touching the same tree (even reads can move memory in dicts under the hood). Rules of thumb: one tree per thread; don't share XMLParser, XSLT, or schema objects across threads without a lock (or make them thread-local); for CPU-bound pipelines, multiprocessing sidesteps the question entirely.

lxml vs ElementTree vs BeautifulSoup

Use ElementTree for small clean XML with zero dependencies, lxml when performance or query power matters, BeautifulSoup when you value its ergonomic API for scraping and speed is secondary. They compose, too: BeautifulSoup(html, "lxml") uses lxml as the engine.

XML parsing in web scraping

For scraping, lxml is usually the parsing half of a two-part pipeline — something fetches the page, lxml extracts the data:

import requests, lxml.html

html = requests.get("https://example.com/products", timeout=30).text
doc = lxml.html.fromstring(html)
names = doc.xpath("//h2[@class='product-name']/text()")

The fetching half is where scraping actually gets hard: JavaScript rendering, IP blocks, CAPTCHAs. WebScraping.AI handles that half — the /html endpoint returns fully rendered HTML through rotating residential proxies, and your lxml code stays exactly the same:

import requests, lxml.html

html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/products",
    "js": "true",                     # rendered in a real browser
}).text

doc = lxml.html.fromstring(html)      # your parsing, unchanged
names = doc.xpath("//h2[@class='product-name']/text()")

Or skip selector-writing entirely and let /ai/fields return structured data from a plain-English description of what you want.

Frequently asked questions

Should I use lxml or ElementTree? Start with ElementTree if the XML is small, well-formed, and your queries are simple — it's built in. Switch to lxml when you hit any of: XPath beyond basic paths, files over ~100 MB, HTML, namespaces-heavy documents, XSLT, or schema validation. The APIs are close enough that switching later is cheap.

Is lxml included with Python? No — it's a third-party package (pip install lxml). The confusion comes from how ubiquitous it is: pandas, Scrapy, and BeautifulSoup all use it as an optional or default backend. The built-in module with the similar API is xml.etree.ElementTree.

What's the fastest way to parse XML in Python? lxml for tree parsing — its libxml2 core parses hundreds of MB/s, typically several times faster than ElementTree. For files too big for memory, lxml.etree.iterparse() with element clearing gives near-flat memory at similar speed. If you only need a few values from a huge document, streaming wins even when a full parse would fit.

How do I convert XML to a Python dict? For quick jobs, the xmltodict package (xmltodict.parse(xml_string)) mirrors the document as nested dicts. It gets awkward with repeated elements, mixed content, and attributes — for anything beyond config-file-shaped XML, extracting exactly the fields you need with lxml XPath produces cleaner code than transforming the whole document.

Can lxml parse HTML with JavaScript-rendered content? No parser can — lxml sees only the HTML string you give it, and JavaScript-rendered content isn't in the initial HTML. Fetch the page with a headless browser (see our headless browser guide) or a rendering API like WebScraping.AI's /html endpoint with js=true, then parse the result with lxml as usual.

How do I fix "Unicode strings with encoding declaration are not supported"? You passed a decoded str containing <?xml ... encoding=...?> to etree.fromstring(). Either pass the raw bytes (response.content instead of response.text, or s.encode("utf-8")), or remove the declaration. Bytes are the better habit — the parser then honors whatever encoding the document declares.