` in the document | | `/html/body/div` | `

` children of `` (absolute path) | | `//div/p` | `

` elements that are direct children of a `

` | | `//div//p` | `

` elements anywhere inside a `

` | | `//*[@id='main']` | Any element with `id="main"` | | `//a[@href]` | `` elements that have an `href` attribute | | `//a[@class='btn']` | `` with class exactly equal to `btn` | | `//a[contains(@class, 'btn')]` | `` whose class attribute contains `btn` | | `//a[starts-with(@href, 'https')]` | Links whose URL starts with `https` | | `//h2[text()='Pricing']` | `

` whose text is exactly `Pricing` | | `//h2[contains(text(), 'Price')]` | `

` whose text contains `Price` | | `(//li)[1]` | First `
` match in the document | | `(//li)[last()]` | Last `
` match in the document | | `//tr[position() <= 3]` | First three rows of each table | | `//input[@type='text' and @required]` | Multiple conditions (AND) | | `//h1 \ | //h2` | | `//p/text()` | Text nodes inside `
` (not the element) | | `//a/@href` | The `href` attribute values themselves | | `count(//img)` | Number of images (returns a number, not nodes) | | `//div[not(@class)]` | `
` elements without a class attribute | The rest of the guide unpacks each group with working examples. ## XPath syntax basics An XPath expression is a series of steps separated by `/`. Each step names a node, and each step can carry predicates in `[...]`: ```text //div[@class='product']/h2/text() └┬┘└──────┬───────────┘└┬┘└──┬──┘ │ │ │ └─ text node of that h2 │ │ └─ direct child h2 │ └─ predicate: only divs with this class └─ search the whole document ``` - `/` — child step (or document root at the start of an expression) - `//` — descendant step: search at any depth - `.` — current node, `..` — parent node - `@name` — an attribute - `` — any element, `@` — any attribute Absolute paths (`/html/body/div[2]/div/p`) — the kind browser "Copy XPath" produces — break the moment the page layout shifts. Prefer relative paths anchored on stable attributes: `//article[@data-id='42']//p`. ## Selecting by attribute The `[@attr]` and `[@attr='value']` predicates handle most scraping selectors: ```text //img[@alt] # images that have alt text //input[@name='email'] # exact attribute match //a[contains(@href, '/product/')] # attribute contains a substring //div[starts-with(@id, 'result-')] # attribute prefix match //a[@href='/pricing']/@href # select the attribute value itself ``` One classic trap: `@class` matches the _entire_ attribute string. `//div[@class='card']` does not match `
`. For multi-class elements, the robust idiom is: ```text //div[contains(concat(' ', normalize-space(@class), ' '), ' card ')] ``` Plain `contains(@class, 'card')` is shorter but also matches `class="cardholder"` — fine when you know the page, risky in general. ## Matching text: contains(), starts-with(), and text() Text matching is XPath's headline feature over CSS selectors. Three building blocks: ```text //button[text()='Add to cart'] # exact text //h2[contains(text(), 'Review')] # substring //li[starts-with(text(), 'Step')] # prefix ``` `text()` selects the element's _direct_ text nodes only. When the text you see is split across nested tags — `View all deals` — match against the full string value with `.` instead: ```text //a[contains(., 'View all deals')] ``` As a rule: use `.` when matching what a human sees, `text()` when you specifically want the element's own text nodes. There's no case-insensitive match in XPath 1.0; the workaround is `translate()`: ```text //a[contains(translate(., 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'), 'sign in')] ``` ## Position: first, last, and ranges Indexes are 1-based , and the parentheses change the meaning: ```text (//li)[1] # the first li in the whole document //ul/li[1] # the first li OF EACH ul ← different! (//li)[last()] # the last li in the document (//li)[last()-1] # second to last //tr[position() > 1] # skip a table's header row //tr[position() >= 2 and position() <= 6] # rows 2–6 ``` Without parentheses, `//li[1]` applies the predicate per parent — on a page with five lists you get five "first" items. Wrapping in `(...)` first collects all matches into one set, then indexes it. This is the single most common XPath position bug. ## Combining conditions: and, or, not(), union Predicates take full boolean expressions: ```text //input[@type='text' and @required] # both conditions //div[@class='ad' or @class='promo'] # either condition //div[not(contains(@class, 'hidden'))] # negation //a[@target and not(@rel)] # has one attribute, lacks another ``` Chained predicates are an AND that reads left to right — `//tr[td][position() < 5]` filters rows that contain cells, _then_ takes the first four. The union operator `|` merges result sets from separate expressions: ```text //h1 | //h2 | //h3 # all headings, in document order ``` Union works on whole expressions, not inside predicates — `//div[a | b]` means "has an `a` or a `b` child", which is the one place `|` acts like OR. ## Axes: parent, siblings, ancestors Axes select relative to the current node in directions CSS can't go. The everyday ones: | Axis | Meaning | Example | | --- | --- | --- | | `parent::` | Direct parent (shorthand `..`) | `//input[@id='q']/parent::form` | | `ancestor::` | All ancestors, nearest last | `//span[@class='price']/ancestor::article` | | `following-sibling::` | Later siblings, same parent | `//dt[text()='SKU']/following-sibling::dd[1]` | | `preceding-sibling::` | Earlier siblings | `//td[@class='total']/preceding-sibling::td[1]` | | `descendant::` | All descendants (what `//` uses) | `//nav/descendant::a` | | `following::` | Everything after in the document | `//h2[@id='specs']/following::table[1]` | The `following-sibling` pattern is the key to scraping label/value layouts — find the node whose _text you know_, then step sideways to the value you want: ```text //th[normalize-space()='Price']/following-sibling::td[1] //dt[contains(., 'Publisher')]/following-sibling::dd[1] ``` And `ancestor::` powers "find the container from a fragment": locate a product card by its title text, then climb to the card element to extract the rest of its fields. That pair — `following-sibling` for spec tables, `ancestor` for cards — is most of what a [price monitoring](https://webscraping.ai/use-cases/price-monitoring) or [product data](https://webscraping.ai/use-cases/product-data-aggregation) scraper does to a retail listing page. ## Counting and child-count predicates `count()` returns a number — useful both as an extracted value and inside predicates: ```text count(//div[@class='result']) # how many results (returns a float) //ul[count(li) > 10] # lists with more than 10 items //table[count(.//tr) = 1] # tables with exactly one row //div[li] # divs that have at least one li child //div[not(node())] # elements with no children at all ``` In predicates, a bare node test is an implicit existence check: `[li]` means `count(li) >= 1`. ## Whitespace, normalize-space(), and empty elements HTML is full of invisible whitespace, and `text()='Log in'` fails when the markup is ` Log in `. `normalize-space()` strips leading/trailing whitespace and collapses internal runs to single spaces: ```text //a[normalize-space()='Log in'] # exact match, whitespace-proof //span[normalize-space(@data-label)='Total'] # works on attributes too //p[normalize-space()=''] # paragraphs that are visually empty //div[not(normalize-space())] # empty or whitespace-only divs ``` Make `normalize-space()` your default for exact text matches — it turns the most fragile XPath pattern into a reliable one. ## Escaping quotes and special characters XPath 1.0 has no escape character for quotes. The rules: - String contains single quotes → delimit with double quotes: `//a[text()="it's here"]` - String contains double quotes → delimit with single quotes: `//a[text()='say "hi"']` - Contains both → concatenate pieces: `//a[text()=concat('it', "'", 's "fine"')]` In your own code, the outer language's quoting stacks on top — in Python, wrap the whole expression in the quote style you're not using inside it. Other characters (`/`, `[`, `@`, `.`) never need escaping inside string literals; they only have meaning outside them. ## Case sensitivity in HTML XPath itself is case-sensitive, but HTML parsers normalize tag names to lowercase — so write `//div`, never `//DIV`, and it will match `
` too. Attribute _values_ keep their original case: `@class='Nav'` and `@class='nav'` are different. For case-insensitive value matching, use the `translate()` idiom from the text-matching section. XML documents (feeds, sitemaps) get no normalization at all — there, tag case must match the source exactly. ## Scraping an HTML table with XPath Tables map naturally to row/cell paths. In Python with `lxml`: ```python from lxml import html import requests tree = html.fromstring(requests.get("https://example.com/stats").text) headers = tree.xpath("//table[@id='data']//th/text()") for row in tree.xpath("//table[@id='data']/tbody/tr"): cells = row.xpath("./td//text()") # note the leading dot values = [c.strip() for c in cells if c.strip()] print(dict(zip(headers, values))) ``` The leading `.` in `./td//text()` is load-bearing: inside a loop, `//td` would search the _whole document_ from the root, silently returning every cell of every row each iteration. Relative XPath inside iteration is another top-three XPath bug. ## XPath and iframes An iframe embeds a separate document — XPath cannot cross that boundary, because the outer document tree simply ends at the `` element. You can select the frame element itself (`//iframe[@id='checkout']`), but to query inside it you must switch documents in your driver: ```python # Selenium driver.switch_to.frame(driver.find_element(By.XPATH, "//iframe[@id='checkout']")) total = driver.find_element(By.XPATH, "//span[@class='total']").text # Playwright total = page.frame_locator("#checkout").locator("xpath=//span[@class='total']").inner_text() ``` If an "element exists on the page but XPath finds nothing," an iframe is the first thing to check — followed by content that JavaScript rendered after your HTML snapshot was taken. ## XPath vs CSS selectors CSS selectors are shorter for everyday class/id targeting; XPath is strictly more powerful. What only XPath can do: - Match by text content — `//button[contains(., 'Next')]`; CSS has no text selector - Walk upward — `parent::`, `ancestor::`; CSS can only approximate this with `:has()` (now supported in browsers, Soup Sieve, Cheerio, and `cssselect`), and it can't climb to an arbitrary ancestor - Sideways navigation in both directions — CSS has `~` and `+` for _following_ siblings only - Conditions and functions — `count()`, `not()`, `position()` ranges, value comparisons - Select attributes and text nodes directly — `//a/@href` returns strings, no `.get_attribute()` loop The pragmatic split most scrapers land on: CSS for simple structural selection, XPath the moment text matching, upward traversal, or label/value layouts appear. Performance differences are negligible at scraping scale; robustness of the expression matters far more than the query engine. (For the CSS side of the comparison, see our [CSS selectors cheat sheet](https://webscraping.ai/blog/css-selectors-cheat-sheet).) ## Using XPath in Python, Selenium, and JavaScript Python + lxml (also what Scrapy uses under the hood): ```python from lxml import html tree = html.fromstring(page_source) prices = tree.xpath("//span[@class='price']/text()") ``` Selenium: ```python from selenium.webdriver.common.by import By element = driver.find_element(By.XPATH, "//button[normalize-space()='Accept']") ``` Playwright — CSS is the default, prefix with `xpath=`: ```python page.locator("xpath=//th[text()='SKU']/following-sibling::td").inner_text() ``` Plain JavaScript in the browser or Puppeteer's `page.evaluate`: ```javascript const result = document.evaluate( "//a[contains(@href, '/download/')]", document, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null ); for (let i = 0; i < result.snapshotLength; i++) { console.log(result.snapshotItem(i).href); } ``` Testing expressions: the fastest XPath tester is already in your browser — open DevTools and use `$x("//h2")` in the console (Chrome and Firefox both support it), or Ctrl+F in the Elements panel, which accepts XPath and highlights matches live. Iterate there, then paste the working expression into your scraper. ## When good XPath isn't enough A correct XPath expression still returns nothing when the HTML it targets never arrives: JavaScript-rendered content, anti-bot walls, and IP blocks all happen before your parser runs. [WebScraping.AI](https://webscraping.ai/) handles that fetch layer — headless Chrome rendering with rotating proxies behind a single API call — and returns HTML your XPath can rely on: ```python import requests from lxml import html # Fully rendered HTML — JavaScript executed, proxies and blocks handled resp = requests.get("https://api.webscraping.ai/html", params={ "api_key": API_KEY, "url": "https://example.com/products", "js": "true", }) tree = html.fromstring(resp.text) names = tree.xpath("//div[@class='product']//h2/text()") ``` Note the division of labor: the API returns rendered HTML, and XPath runs in your code. (The [`/selected` and `/selected-multiple` endpoints](https://webscraping.ai/docs) can trim the response server-side, but they take _CSS_ selectors — for anything needing text matching or axes, fetch the HTML and evaluate XPath locally as above.) And when a site reshuffles its markup weekly and no selector stays valid, the [/ai/fields endpoint](https://webscraping.ai/ai-web-scraping) skips selectors entirely — describe the fields you want and let the model extract them from the rendered page. ## Frequently asked questions What's the difference between `//div/p` and `//div//p`? `/` selects direct children only; `//` selects descendants at any depth. `//div/p` matches a `<p>` immediately inside a `<div>`; `//div//p` also matches one nested three wrappers deeper. Why does my XPath work in DevTools but not in my scraper? DevTools queries the _live DOM_ after JavaScript ran; your scraper usually parses the raw HTML response. View the page source (Ctrl+U) — if your target element isn't in it, you need JS rendering (Selenium, Playwright, or a [rendering API](https://webscraping.ai/)), not a different XPath. Iframes cause the identical symptom. How do I select the first element with XPath? `(//div[@class='result'])[1]` — with the parentheses. `//div[@class='result'][1]` means "first such div _within each parent_," which can return many elements. Indexes start at 1, not 0. Is XPath faster or slower than CSS selectors? For scraping workloads the difference is noise — both evaluate in microseconds against a parsed tree, and network time dominates by orders of magnitude. Old advice about CSS being dramatically faster dates from Selenium's IE driver era. Choose by expressiveness, not speed. Does XPath work with JSON APIs? No — XPath operates on XML/HTML node trees. For JSON, use JSONPath or `jq`, which borrow XPath's ideas. If a site serves JSON, that's good news: skip HTML parsing entirely and consume the JSON directly. Can I use XPath 2.0 functions like matches() or lower-case()? Not in browsers, `lxml`, or Selenium — they all implement XPath 1.0. For regex, `lxml` and Scrapy support the EXSLT extension (`//a[re:test(@href, '\d+$')]`, with the `re` namespace registered in lxml; Scrapy registers it for you); elsewhere use `contains()`/`starts-with()` combinations or post-filter in your programming language. Case-insensitivity needs the `translate()` trick.

` whose text is exactly `Pricing` | | `//h2[contains(text(), 'Price')]` | `

` whose text contains `Price` | | `(//li)[1]` | First `` match in the document | | `(//li)[last()]` | Last `