---
title: "Python XML Parsing: ElementTree, lxml, and How to Choose"
description: "Parse XML and HTML in Python: ElementTree vs lxml, XPath, namespaces, iterparse for huge files, modifying and saving documents, and XXE-safe parsing."
url: https://webscraping.ai/blog/python-xml-parsing
markdown_index: https://webscraping.ai/llms.txt
---
# Python XML Parsing: ElementTree, lxml, and How to Choose

Python ships with an XML parser in the standard library, and for many jobs that's all you need. But the moment you hit a huge feed that doesn't fit in memory, a document with namespaces, broken HTML, or an XPath expression more ambitious than a tag name, you'll want [lxml](https://lxml.de/). This guide covers both: `xml.etree.ElementTree` for the simple cases, lxml for everything else, and the operational details — encoding, memory, thread safety, and the security flags you should never leave at their defaults when parsing untrusted input.

## Key Takeaways

- `xml.etree.ElementTree` is built in and fine for small, well-formed XML; lxml is a C-backed superset with full XPath, HTML repair, XSLT, and validation
- lxml parses the same API shape as ElementTree — most code migrates by changing one import
- For files bigger than RAM, use `iterparse()` and clear elements as you go; that keeps memory flat regardless of file size
- lxml is also one of the best HTML parsers in Python — `lxml.html` handles real-world broken markup
- Never parse untrusted XML with entity resolution enabled — XXE attacks are a parser configuration problem, not an exotic edge case
- For web scraping, pair lxml with an HTTP client or a [web scraping API](https://webscraping.ai/) and query with XPath or CSS selectors

## Python's XML parsers at a glance

| Parser | In stdlib? | Speed | XPath | Best for |
| --- | --- | --- | --- | --- |
| `xml.etree.ElementTree` | Yes | Good (C accelerator) | Tiny subset | Small/medium well-formed XML |
| `lxml.etree` | No (`pip install lxml`) | Fastest | Full XPath 1.0 | Everything: big files, namespaces, XSLT, validation |
| `lxml.html` | No | Fastest | Full XPath 1.0 | Real-world (broken) HTML |
| `xml.dom.minidom` | Yes | Slow | No | Pretty-printing tiny documents; legacy DOM code |
| `xml.sax` / `pulldom` | Yes | Low memory | No | Event-driven parsing (mostly superseded by `iterparse`) |
| `BeautifulSoup` | No | Slower (wrapper) | No (CSS via soupsieve) | Forgiving HTML scraping — see our [Beautiful Soup guide](https://webscraping.ai/blog/beautiful-soup-python) |

The practical decision tree: stdlib `ElementTree` if the file is small and clean, `lxml` if you need speed, XPath, HTML, or streaming — and `defusedxml` wrappers if the input comes from strangers (more on that below).

## ElementTree: the standard library baseline

```python
import xml.etree.ElementTree as ET

tree = ET.parse("catalog.xml") # from a file
root = tree.getroot()

root = ET.fromstring(xml_string) # from a string

for book in root.iter("book"): # every <book>, any depth
    title = book.find("title").text
    price = book.get("price") # attribute
    print(title, price)
```

`find()` returns the first match, `findall()` a list, `iter()` walks the whole subtree. ElementTree supports only a small path subset — `.//book[@lang='en']` works, but functions like `contains()`, axes like `following-sibling`, or selecting attributes directly do not. When your query outgrows that, it's lxml time.

## Installing lxml

```bash
pip install lxml
```

Binary wheels exist for every major platform, so on modern Python this installs in seconds with no compiler — the old advice about needing `libxml2-dev`/`libxslt-dev` system packages only applies when pip falls back to building from source (unusual architectures, brand-new Python releases).

Pinning a specific version works like any package:

```bash
pip install lxml==6.1.3 # exact version
pip install "lxml>=6,<7" # range
pip show lxml # what's installed now
```

If a legacy project needs an old lxml on a new Python, check [PyPI's release history](https://pypi.org/project/lxml/#history) for a version with wheels for your interpreter — very old lxml releases predate wheels for current Pythons and will try (and often fail) to compile.

## Parsing XML with lxml

lxml deliberately mirrors the ElementTree API:

```python
from lxml import etree

tree = etree.parse("catalog.xml") # file or URL-like object
root = tree.getroot()

root = etree.fromstring(xml_bytes) # from bytes

for book in root.iter("book"):
    print(book.findtext("title"), book.get("price"))
```

One encoding gotcha: `etree.fromstring()` refuses a _str_ that carries an XML declaration like `<?xml version="1.0" encoding="UTF-8"?>` — the declaration promises bytes in a specific encoding, and a decoded string contradicts it. Pass bytes (`xml_text.encode("utf-8")`) or strip the declaration. When parsing files, lxml reads the declaration itself and handles the encoding for you; trust it rather than decoding manually.

### Iterating the tree

```python
for el in root.iter(): # every element, document order
    print(el.tag, (el.text or "").strip())

for child in root: # direct children only
    ...

for para in root.iter("para"): # filter by tag
    ...

for el in root.iterchildren("item"): # lxml extras: children/descendants
    ...
for el in root.iterdescendants("price"):
    ...
```

Elements behave like lists of their children, so slicing (`root[0]`, `root[-1]`, `root[1:3]`) works too. lxml also gives every element `getparent()`, `getprevious()`, and `getnext()` — navigation ElementTree simply doesn't have.

## XPath: lxml's superpower

`xpath()` evaluates full XPath 1.0 and is the main reason to choose lxml:

```python
# All English-language book titles as strings
titles = root.xpath("//book[@lang='en']/title/text()")

# Attribute values directly
isbns = root.xpath("//book/@isbn")

# Text matching and counting
cheap = root.xpath("//book[number(price) < 20]")
n = root.xpath("count(//book)")
```

`find()`/`findall()` still exist and are slightly faster for trivial paths, but `xpath()` handles predicates, functions, axes, and returns strings/numbers/booleans, not just elements. For the expression syntax itself — `contains()`, `following-sibling`, position tricks, quote escaping — see our [XPath cheat sheet](https://webscraping.ai/blog/xpath-cheat-sheet).

## Parsing HTML (including broken HTML)

Real-world HTML has unclosed tags, stray ampersands, and misnested elements. `lxml.html` repairs all of it, the same way browsers do:

```python
import lxml.html

doc = lxml.html.fromstring(html_text)

# CSS selectors or XPath — your choice
for link in doc.cssselect("a.result"): # pip install cssselect
    print(link.get("href"), link.text_content())

prices = doc.xpath("//span[contains(@class, 'price')]/text()")
```

`text_content()` returns all text inside an element, tags stripped — the workhorse for scraping. For HTML tables, `pandas.read_html()` uses lxml under the hood, or extract manually:

```python
rows = doc.xpath("//table[@id='data']//tr")
data = [[td.text_content().strip() for td in row.xpath("./td")] for row in rows]
```

(Note the leading dot in `./td` — inside a loop, an absolute `//td` would search the whole document from every row, a classic bug.)

### Finding elements by class or id

```python
doc.get_element_by_id("main") # lxml.html only
doc.xpath("//div[@id='main']")

# @class is a space-separated list — match one class safely:
doc.xpath("//div[contains(concat(' ', normalize-space(@class), ' '), ' product ')]")
doc.cssselect("div.product") # …or let CSS do it readably
```

## Namespaces

Namespaced XML (`<soap:Envelope>`, Atom feeds, sitemaps) trips everyone up: `//url` finds nothing because every element actually lives in a namespace. Map prefixes explicitly:

```python
ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
locs = root.xpath("//sm:url/sm:loc/text()", namespaces=ns)
```

The prefix (`sm`) is your choice — only the URI must match the document. For a default (unprefixed) namespace you still must register a prefix in the map; XPath has no way to say "the default namespace". The blunt instrument when you just don't care: match on local name, `//*[local-name()='loc']`.

## Modifying documents and saving them

lxml trees are mutable:

```python
for book in root.xpath("//book[@status='discontinued']"):
    book.getparent().remove(book) # delete an element

for price in root.iter("price"):
    price.text = str(round(float(price.text) * 1.1, 2))
    price.set("currency", "USD") # set/replace an attribute

new = etree.SubElement(root, "book", attrib={"isbn": "978-0"})
etree.SubElement(new, "title").text = "New Arrival"
```

Write the result back:

```python
tree.write("catalog-updated.xml", encoding="utf-8",
           xml_declaration=True, pretty_print=True)

xml_bytes = etree.tostring(root, encoding="utf-8", pretty_print=True)
```

`pretty_print` only re-indents nodes that don't already have surrounding whitespace text; when re-indenting a parsed file, parse it with `etree.XMLParser(remove_blank_text=True)` so lxml is free to lay it out cleanly. To build documents from scratch, `etree.Element()` + `SubElement()` as above, or the `E-factory` (`lxml.builder`) for declarative construction.

## Huge files: iterparse and flat memory

A DOM parse holds the whole tree in RAM — typically 5–10× the file size (this behavior on multi-GB documents is the most-cited performance complaint about lxml, and it's inherent to any in-memory tree). The fix is streaming with `iterparse()`, clearing elements once processed:

```python
from lxml import etree

def fast_iter(path, tag):
    context = etree.iterparse(path, events=("end",), tag=tag)
    for _, elem in context:
        yield elem
        # Free the element and its preceding siblings
        elem.clear()
        while elem.getprevious() is not None:
            del elem.getparent()[0]
    del context

for record in fast_iter("huge-feed.xml", "product"):
    process(record.findtext("sku"))
```

Memory stays constant whether the file is 10 MB or 50 GB. The `clear()`-plus-delete-siblings dance matters: without it, processed elements stay attached to the root and memory grows anyway. For line-of-business sizes (under ~100 MB) skip the ceremony — a plain `parse()` is simpler and plenty fast: lxml parses on the order of hundreds of MB/s, typically several times faster than ElementTree and far ahead of minidom.

## CDATA, comments, entities, and other special constructs

```python
# Comments and processing instructions appear in iteration unless you opt out
parser = etree.XMLParser(remove_comments=True, remove_pis=True)
root = etree.fromstring(xml_bytes, parser)

# CDATA text is just .text when parsed; to *write* CDATA:
el.text = etree.CDATA("if (a < b) { ... }")
```

Internal entity definitions (`<!ENTITY co "Example Corp">` in the document's own DTD) are expanded by default — since lxml 5.0 the parser default is `resolve_entities="internal"`. External entities, which pull in a file or URL, are not. If you genuinely need those, opt in:

```python
parser = etree.XMLParser(resolve_entities=True, load_dtd=True)
root = etree.parse("with-entities.xml", parser).getroot()
```

**Only do this for XML you trust.** Entity resolution is also the attack surface for XXE (XML External Entity) injection — a malicious document can define an entity that reads `/etc/passwd` or makes network requests. For untrusted input: set `resolve_entities=False`, keep `no_network=True` (the default), and don't enable `load_dtd`. For the stdlib parsers, the [defusedxml](https://pypi.org/project/defusedxml/) wrappers disable the dangerous features (its `defusedxml.lxml` module is deprecated — harden the lxml parser yourself as above).

## Validation and XSLT

lxml validates against DTD, XML Schema (XSD), and RelaxNG:

```python
schema = etree.XMLSchema(etree.parse("catalog.xsd"))
doc = etree.parse("catalog.xml")
if not schema.validate(doc):
    for error in schema.error_log:
        print(error.line, error.message)
```

And it's the standard way to run XSLT 1.0 from Python:

```python
transform = etree.XSLT(etree.parse("to-html.xsl"))
html_result = transform(doc)
print(str(html_result))
```

## Thread safety

Safe: parsing in multiple threads concurrently — lxml releases the GIL inside libxml2, so parsing actually parallelizes across threads. Not safe: two threads touching the _same tree_ (even reads can move memory in dicts under the hood). Rules of thumb: one tree per thread; don't share `XMLParser`, `XSLT`, or schema objects across threads without a lock (or make them thread-local); for CPU-bound pipelines, `multiprocessing` sidesteps the question entirely.

## lxml vs ElementTree vs BeautifulSoup

| | ElementTree | lxml | BeautifulSoup |
| --- | --- | --- | --- |
| Install | built-in | `pip install lxml` | `pip install beautifulsoup4` |
| Speed | good | fastest | slowest (it wraps another parser — often lxml itself) |
| XPath | tiny subset | full 1.0 | none (CSS selectors instead) |
| Broken HTML | no | yes (`lxml.html`) | yes, most forgiving |
| XSLT / validation | no | yes | no |
| API feel | procedural tree | same, plus extras | pythonic, beginner-friendly |

Use ElementTree for small clean XML with zero dependencies, lxml when performance or query power matters, BeautifulSoup when you value its ergonomic API for scraping and speed is secondary. They compose, too: `BeautifulSoup(html, "lxml")` uses lxml as the engine.

## XML parsing in web scraping

For scraping, lxml is usually the parsing half of a two-part pipeline — something fetches the page, lxml extracts the data:

```python
import requests, lxml.html

html = requests.get("https://example.com/products", timeout=30).text
doc = lxml.html.fromstring(html)
names = doc.xpath("//h2[@class='product-name']/text()")
```

The fetching half is where scraping actually gets hard: JavaScript rendering, IP blocks, CAPTCHAs. [WebScraping.AI](https://webscraping.ai/) handles that half — the [/html endpoint](https://webscraping.ai/docs) returns fully rendered HTML through rotating proxies (datacenter by default, `proxy=residential` or `proxy=stealth` when a target needs it), and your lxml code stays exactly the same:

```python
import requests, lxml.html

html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/products",
    "js": "true", # rendered in a real browser
}).text

doc = lxml.html.fromstring(html) # your parsing, unchanged
names = doc.xpath("//h2[@class='product-name']/text()")
```

Or skip selector-writing entirely and let [/ai/fields](https://webscraping.ai/ai-web-scraping) return structured data from a plain-English description of what you want — the difference matters most on long-running feeds where XPath rot is the real cost, like [SEC filing monitoring](https://webscraping.ai/use-cases/sec-filing-monitoring) or [product data aggregation](https://webscraping.ai/use-cases/product-data-aggregation).

## Frequently asked questions

**Should I use lxml or ElementTree?** Start with ElementTree if the XML is small, well-formed, and your queries are simple — it's built in. Switch to lxml when you hit any of: XPath beyond basic paths, files over ~100 MB, HTML, namespaces-heavy documents, XSLT, or schema validation. The APIs are close enough that switching later is cheap.

**Is lxml included with Python?** No — it's a third-party package (`pip install lxml`). The confusion comes from how ubiquitous it is: pandas, Scrapy, and BeautifulSoup all use it as an optional or default backend. The built-in module with the similar API is `xml.etree.ElementTree`.

**What's the fastest way to parse XML in Python?** lxml for tree parsing — its libxml2 core parses hundreds of MB/s, typically several times faster than ElementTree. For files too big for memory, `lxml.etree.iterparse()` with element clearing gives near-flat memory at similar speed. If you only need a few values from a huge document, streaming wins even when a full parse would fit.

**How do I convert XML to a Python dict?** For quick jobs, the `xmltodict` package (`xmltodict.parse(xml_string)`) mirrors the document as nested dicts. It gets awkward with repeated elements, mixed content, and attributes — for anything beyond config-file-shaped XML, extracting exactly the fields you need with lxml XPath produces cleaner code than transforming the whole document.

**Can lxml parse HTML with JavaScript-rendered content?** No parser can — lxml sees only the HTML string you give it, and JavaScript-rendered content isn't in the initial HTML. Fetch the page with a headless browser (see our [headless browser guide](https://webscraping.ai/blog/headless-browser-guide)) or a rendering API like [WebScraping.AI's /html endpoint](https://webscraping.ai/) with `js=true`, then parse the result with lxml as usual.

**How do I fix "Unicode strings with encoding declaration are not supported"?** You passed a decoded `str` containing `<?xml ... encoding=...?>` to `etree.fromstring()`. Either pass the raw bytes (`response.content` instead of `response.text`, or `s.encode("utf-8")`), or remove the declaration. Bytes are the better habit — the parser then honors whatever encoding the document declares.