XPath

XPath#

XPath is a query language for navigating XML (and HTML) documents. Despite XML’s decline as an interchange format, XPath survives because it’s still the standard way to query HTML for scraping, test automation, and SVG manipulation.

Tree Model#

An XML document is a tree of nodes:

Document, the root.
Element, <a>...</a>.
Attribute, href="...".
Text, text content.
Comment, Processing instruction.

XPath queries select a set of nodes matching a path expression.

Core Syntax#

XPath expressions read like filesystem paths over the document tree. / walks the root downward, // matches at any depth, predicates in brackets filter nodes, and @ selects attributes. The table below covers the patterns that show up in nearly every scraping or XML-querying task.

Path	Selects
`/`	document root
`/html`	root element `html`
`/html/body`	body element
`//a`	any `a` element at any depth
`//a[@href]`	`a` with an `href` attribute
`//a[@href='/home']`	`a` whose `href` is `/home`
`//a/@href`	`href` attribute values
`//a/text()`	text inside `a` elements
`//ul/li[1]`	first `li` inside `ul` (1-indexed)
`//ul/li[last()]`	last `li`
`//div[@class='post']`	`div` with class `post`
`//*[@id='main']`	any element with id `main`
`//book[year > 2000]`	books with year > 2000
`//author \| //editor`	union of two paths

Axes#

XPath has 13 axes describing direction relative to a node. The default is child::.

Axis	Selects
`child::`	immediate children (default)
`descendant::`	all descendants
`parent::`	the parent
`ancestor::`	all ancestors
`following::`	everything after in document order
`preceding::`	everything before
`following-sibling::`	siblings after
`preceding-sibling::`	siblings before
`attribute::` (`@`)	attributes
`self::`	the node itself
`descendant-or-self::`	`//` is shorthand for this
`ancestor-or-self::`	this and ancestors
`namespace::`	namespace nodes (rarely used)

//li/following-sibling::li[1]    # the next li after each li
//a/ancestor::section[1]         # the closest section ancestor

Predicates#

Filters in [ ] after a step:

//book[@lang='en' and price > 10]
//div[@class='post' and not(@hidden)]
//tr[position() > 1]              # skip the header row
//li[contains(@class, 'active')]

Functions#

The XPath function library covers string manipulation, counting, position, and a handful of node-set operations. Most queries lean on the same dozen functions over and over (contains, starts-with, normalize-space, position), worth memorizing.

text()                            # text content
name()                            # element name
count(...)                        # number of nodes
contains(haystack, needle)
starts-with(s, prefix)
ends-with(s, suffix)              # XPath 2.0+
normalize-space(s)                # collapse whitespace
string-length(s)
substring(s, start, length)
concat(a, b, ...)
not(...) / true() / false()
position() / last()

XPath 1.0 vs. 2.0+ vs. 3.x#

XPath 1.0, ubiquitous; most engines support only this. No ends-with, no regex functions, no maps / arrays.
XPath 2.0, adds rich type system, sequences, regex (matches / replace / tokenize), much more.
XPath 3.0 / 3.1, maps, arrays, higher-order functions, string-join.

In browsers and most Java / .NET / Python (lxml 1.0 mode) you get 1.0. Saxon is the standard 2.0+ engine.

HTML#

Browsers and HTML-specific scrapers run XPath against the DOM. Useful in test automation.

// Playwright / Selenium-like
page.locator('xpath=//button[contains(@class, "primary") and text()="Save"]')

CSS selectors are usually preferred for readability; XPath shines when you need.

Text matching, //button[text()="Save"].
Ancestor traversal, //input[@id='x']/ancestor::form.
Following-sibling, //label[text()='Email']/following-sibling::input[1].

Things CSS does poorly or not at all.

Tooling#

The CLIs and language libraries that run XPath against documents. xmllint is the everyday command line for XML; xmlstarlet is the jq-like alternative; Saxon covers XPath 3.x; browser DevTools’ $x() is the on-the-fly sandbox for trying queries against a live page.

xmllint:

$ xmllint --xpath '//book/title/text()' library.xml
$ xmllint --html --xpath '//a/@href' page.html 2>/dev/null

xmlstarlet, jq-like CLI for XML.
Saxon, XPath 3.x engine and command line.
Browser DevTools, $x("xpath here") in the console.
lxml (Python), Nokogiri (Ruby), xmldom/fast-xml-parser (Node), XPathExpression (JS DOM).

XSLT#

XSLT (eXtensible Stylesheet Language Transformations) uses XPath in templates to transform XML to other XML, HTML, or text. Once everywhere; now mostly seen in publishing pipelines and legacy systems.

Modern transformations more often use a general-purpose language with an XML library, but XSLT is still in use for high-volume publishing workflows.

Common Mistakes#

The traps that catch XPath users. 1-indexed positions, child as the default axis, the element-versus-text distinction, namespace handling, and result ordering all surface as “my query returns nothing” or “my query returns the wrong thing” until the rule is internalized.

1-indexed, [1] is the first, not the second.
Default axis is child, a/b means immediate children, not any descendant.
Element vs. text, //a selects elements; //a/text() selects the text inside them. //a and //a/text() are different node sets.
Namespaces, //book doesn’t match <my:book> in a namespaced document. You have to bind the prefix and use it: //my:book.
Ordering, result sets are typically in document order, but only for some axes.

When to Use#

The kinds of work where XPath is the right reach. HTML scraping with structural rules, XML querying that can’t be escaped, and test automation locators that exceed CSS’s reach. For JSON, use jq or JMESPath; for tabular data, use SQL; for modern DOMs, prefer CSS selectors when they suffice.

Scraping HTML with structural rules (text content, ancestor relationships).
Querying XML you can’t avoid (SOAP, RSS / Atom, OOXML, SVG).
Test automation locators when CSS selectors don’t reach.
XSLT pipelines (legacy or publishing-heavy systems).

When not to.

Querying JSON, use jq or JMESPath.
Anything where the data is in a real database, use SQL.
Browser DOM in modern code, prefer CSS selectors and ARIA roles when they suffice.