XPath Cheat Sheet: Syntax, Functions, and Examples for Web Scraping

XPath is a query language for selecting nodes in HTML and XML documents, and it remains the most expressive way to target elements when scraping: it can match by text content, walk up the tree to parents, and filter by conditions CSS selectors simply can't express. This cheat sheet collects the syntax, functions, and axes you'll actually use — each with a copy-paste example — plus the sharp edges (quoting, whitespace, case sensitivity) that break real-world scrapers.

Key Takeaways

• //tag[@attr='value'] is the workhorse pattern — // searches the whole document, [...] filters
• Use contains(), starts-with(), and text() to match elements by their visible text — the thing CSS selectors can't do
• Indexing is 1-based: (//li)[1] is the first match, (//li)[last()] the last — and the parentheses matter
• Axes like parent::, following-sibling::, and ancestor:: navigate in directions CSS has no syntax for
• normalize-space() fixes the #1 cause of "my XPath doesn't match": invisible leading/trailing whitespace
• Test expressions in your browser first: $x("//h2") in Chrome/Firefox DevTools console evaluates XPath live

What is XPath?

XPath (XML Path Language) is a W3C-standard query language for selecting nodes from XML and HTML documents. An XPath expression describes a path through the document tree — elements, attributes, and text — optionally filtered by predicates in square brackets.

Every major scraping stack speaks it: lxml and Scrapy in Python, Selenium and Playwright in browser automation, document.evaluate() in plain JavaScript. HTML parsers evaluate XPath 1.0 in practice — the 2.0+ features (regex matching, lower-case()) exist in XML tooling but not in browsers or lxml, so this guide sticks to what actually runs.

The cheat sheet

The rest of the guide unpacks each group with working examples.

XPath syntax basics

An XPath expression is a series of steps separated by /. Each step names a node, and each step can carry predicates in [...]:

//div[@class='product']/h2/text()
└┬┘└──────┬───────────┘└┬┘└──┬──┘
 │        │             │    └─ text node of that h2
 │        │             └─ direct child h2
 │        └─ predicate: only divs with this class
 └─ search the whole document

• / — child step (or document root at the start of an expression)
• // — descendant step: search at any depth
• . — current node, .. — parent node
• @name — an attribute
• * — any element, @* — any attribute

Absolute paths (/html/body/div[2]/div/p) — the kind browser "Copy XPath" produces — break the moment the page layout shifts. Prefer relative paths anchored on stable attributes: //article[@data-id='42']//p.

Selecting by attribute

The [@attr] and [@attr='value'] predicates handle most scraping selectors:

//img[@alt]                          # images that have alt text
//input[@name='email']               # exact attribute match
//a[contains(@href, '/product/')]    # attribute contains a substring
//div[starts-with(@id, 'result-')]   # attribute prefix match
//a[@href='/pricing']/@href          # select the attribute value itself

One classic trap: @class matches the entire attribute string. //div[@class='card'] does not match <div class="card featured">. For multi-class elements, the robust idiom is:

//div[contains(concat(' ', normalize-space(@class), ' '), ' card ')]

Plain contains(@class, 'card') is shorter but also matches class="cardholder" — fine when you know the page, risky in general.

Matching text: contains(), starts-with(), and text()

Text matching is XPath's headline feature over CSS selectors. Three building blocks:

//button[text()='Add to cart']          # exact text
//h2[contains(text(), 'Review')]        # substring
//li[starts-with(text(), 'Step')]       # prefix

text() selects the element's direct text nodes only. When the text you see is split across nested tags — <a>View <b>all</b> deals</a> — match against the full string value with . instead:

//a[contains(., 'View all deals')]

As a rule: use . when matching what a human sees, text() when you specifically want the element's own text nodes. There's no case-insensitive match in XPath 1.0; the workaround is translate():

//a[contains(translate(., 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'), 'sign in')]

Position: first, last, and ranges

Indexes are 1-based, and the parentheses change the meaning:

(//li)[1]           # the first li in the whole document
//ul/li[1]          # the first li OF EACH ul  ← different!
(//li)[last()]      # the last li in the document
(//li)[last()-1]    # second to last
//tr[position() > 1]            # skip a table's header row
//tr[position() >= 2 and position() <= 6]   # rows 2–6

Without parentheses, //li[1] applies the predicate per parent — on a page with five lists you get five "first" items. Wrapping in (...) first collects all matches into one set, then indexes it. This is the single most common XPath position bug.

Combining conditions: and, or, not(), union

Predicates take full boolean expressions:

//input[@type='text' and @required]        # both conditions
//div[@class='ad' or @class='promo']       # either condition
//div[not(contains(@class, 'hidden'))]     # negation
//a[@target and not(@rel)]                 # has one attribute, lacks another

Chained predicates are an AND that reads left to right — //tr[td][position() < 5] filters rows that contain cells, then takes the first four. The union operator | merges result sets from separate expressions:

//h1 | //h2 | //h3        # all headings, in document order

Union works on whole expressions, not inside predicates — //div[a | b] means "has an a or a b child", which is the one place | acts like OR.

Axes: parent, siblings, ancestors

Axes select relative to the current node in directions CSS can't go. The everyday ones:

The following-sibling pattern is the key to scraping label/value layouts — find the node whose text you know, then step sideways to the value you want:

//th[normalize-space()='Price']/following-sibling::td[1]
//dt[contains(., 'Publisher')]/following-sibling::dd[1]

And ancestor:: powers "find the container from a fragment": locate a product card by its title text, then climb to the card element to extract the rest of its fields.

Counting and child-count predicates

count() returns a number — useful both as an extracted value and inside predicates:

count(//div[@class='result'])       # how many results (returns a float)
//ul[count(li) > 10]                # lists with more than 10 items
//table[count(.//tr) = 1]           # tables with exactly one row
//div[li]                           # divs that have at least one li child
//div[not(node())]                  # elements with no children at all

In predicates, a bare node test is an implicit existence check: [li] means count(li) >= 1.

Whitespace, normalize-space(), and empty elements

HTML is full of invisible whitespace, and text()='Log in' fails when the markup is <a> Log in </a>. normalize-space() strips leading/trailing whitespace and collapses internal runs to single spaces:

//a[normalize-space()='Log in']                 # exact match, whitespace-proof
//span[normalize-space(@data-label)='Total']    # works on attributes too
//p[normalize-space()='']                       # paragraphs that are visually empty
//div[not(normalize-space())]                   # empty or whitespace-only divs

Make normalize-space() your default for exact text matches — it turns the most fragile XPath pattern into a reliable one.

Escaping quotes and special characters

XPath 1.0 has no escape character for quotes. The rules:

• String contains single quotes → delimit with double quotes: //a[text()="it's here"]
• String contains double quotes → delimit with single quotes: //a[text()='say "hi"']
• Contains both → concatenate pieces: //a[text()=concat('it', "'", 's "fine"')]

In your own code, the outer language's quoting stacks on top — in Python, wrap the whole expression in the quote style you're not using inside it. Other characters (/, [, @, .) never need escaping inside string literals; they only have meaning outside them.

Case sensitivity in HTML

XPath itself is case-sensitive, but HTML parsers normalize tag names to lowercase — so write //div, never //DIV, and it will match <DIV> too. Attribute values keep their original case: @class='Nav' and @class='nav' are different. For case-insensitive value matching, use the translate() idiom from the text-matching section. XML documents (feeds, sitemaps) get no normalization at all — there, tag case must match the source exactly.

Scraping an HTML table with XPath

Tables map naturally to row/cell paths. In Python with lxml:

from lxml import html
import requests

tree = html.fromstring(requests.get("https://example.com/stats").text)

headers = tree.xpath("//table[@id='data']//th/text()")
for row in tree.xpath("//table[@id='data']/tbody/tr"):
    cells = row.xpath("./td//text()")           # note the leading dot
    values = [c.strip() for c in cells if c.strip()]
    print(dict(zip(headers, values)))

The leading . in ./td//text() is load-bearing: inside a loop, //td would search the whole document from the root, silently returning every cell of every row each iteration. Relative XPath inside iteration is another top-three XPath bug.

XPath and iframes

An iframe embeds a separate document — XPath cannot cross that boundary, because the outer document tree simply ends at the <iframe> element. You can select the frame element itself (//iframe[@id='checkout']), but to query inside it you must switch documents in your driver:

# Selenium
driver.switch_to.frame(driver.find_element(By.XPATH, "//iframe[@id='checkout']"))
total = driver.find_element(By.XPATH, "//span[@class='total']").text

# Playwright
total = page.frame_locator("#checkout").locator("xpath=//span[@class='total']").inner_text()

If an "element exists on the page but XPath finds nothing," an iframe is the first thing to check — followed by content that JavaScript rendered after your HTML snapshot was taken.

XPath vs CSS selectors

CSS selectors are shorter for everyday class/id targeting; XPath is strictly more powerful. What only XPath can do:

• Match by text content — //button[contains(., 'Next')]; CSS has no text selector
• Walk upward — parent::, ancestor::; CSS can't select an element by its descendants (:has() finally helps, but isn't available in most scraping parsers)
• Sideways navigation in both directions — CSS has ~ and + for following siblings only
• Conditions and functions — count(), not(), position() ranges, value comparisons
• Select attributes and text nodes directly — //a/@href returns strings, no .get_attribute() loop

The pragmatic split most scrapers land on: CSS for simple structural selection, XPath the moment text matching, upward traversal, or label/value layouts appear. Performance differences are negligible at scraping scale; robustness of the expression matters far more than the query engine. (For the CSS side of the comparison, see our CSS selectors FAQ.)

Using XPath in Python, Selenium, and JavaScript

Python + lxml (also what Scrapy uses under the hood):

from lxml import html
tree = html.fromstring(page_source)
prices = tree.xpath("//span[@class='price']/text()")

Selenium:

from selenium.webdriver.common.by import By
element = driver.find_element(By.XPATH, "//button[normalize-space()='Accept']")

Playwright — CSS is the default, prefix with xpath=:

page.locator("xpath=//th[text()='SKU']/following-sibling::td").inner_text()

Plain JavaScript in the browser or Puppeteer's page.evaluate:

const result = document.evaluate(
  "//a[contains(@href, '/download/')]",
  document, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null
);
for (let i = 0; i < result.snapshotLength; i++) {
  console.log(result.snapshotItem(i).href);
}

Testing expressions: the fastest XPath tester is already in your browser — open DevTools and use $x("//h2") in the console (Chrome and Firefox both support it), or Ctrl+F in the Elements panel, which accepts XPath and highlights matches live. Iterate there, then paste the working expression into your scraper.

When good XPath isn't enough

A correct XPath expression still returns nothing when the HTML it targets never arrives: JavaScript-rendered content, anti-bot walls, and IP blocks all happen before your parser runs. WebScraping.AI handles that fetch layer — headless Chrome rendering with rotating proxies behind a single API call — and returns HTML your XPath can rely on:

import requests
from lxml import html

# Fully rendered HTML — JavaScript executed, proxies and blocks handled
resp = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/products",
    "js": "true",
})
tree = html.fromstring(resp.text)
names = tree.xpath("//div[@class='product']//h2/text()")

And when a site reshuffles its markup weekly and no selector stays valid, the /ai/fields endpoint skips selectors entirely — describe the fields you want and let the model extract them from the rendered page.

Frequently asked questions

What's the difference between //div/p and //div//p? / selects direct children only; // selects descendants at any depth. //div/p matches a <p> immediately inside a <div>; //div//p also matches one nested three wrappers deeper.

Why does my XPath work in DevTools but not in my scraper? DevTools queries the live DOM after JavaScript ran; your scraper usually parses the raw HTML response. View the page source (Ctrl+U) — if your target element isn't in it, you need JS rendering (Selenium, Playwright, or a rendering API), not a different XPath. Iframes cause the identical symptom.

How do I select the first element with XPath? (//div[@class='result'])[1] — with the parentheses. //div[@class='result'][1] means "first such div within each parent," which can return many elements. Indexes start at 1, not 0.

Is XPath faster or slower than CSS selectors? For scraping workloads the difference is noise — both evaluate in microseconds against a parsed tree, and network time dominates by orders of magnitude. Old advice about CSS being dramatically faster dates from Selenium's IE driver era. Choose by expressiveness, not speed.

Does XPath work with JSON APIs? No — XPath operates on XML/HTML node trees. For JSON, use JSONPath or jq, which borrow XPath's ideas. If a site serves JSON, that's good news: skip HTML parsing entirely and consume the JSON directly.

Can I use XPath 2.0 functions like matches() or lower-case()? Not in browsers, lxml, or Selenium — they all implement XPath 1.0. Regex-style matching needs contains()/starts-with() combinations or post-filtering in your programming language; case-insensitivity needs the translate() trick.