Quick answer: XPath is a query language for selecting elements in HTML and XML documents, and for scraping you only need a small subset. The workhorses: //tag finds elements anywhere, [@attr="value"] filters by attribute, contains() does partial matching, text() grabs text content, and .. walks up to a parent, which is the one trick CSS selectors can't do. The cheat sheet below covers what real scraping work actually requires.
Bookmark the tables, then read the last two sections, because knowing when to use XPath (and when not to) saves more time than any individual expression.
The Core Cheat Sheet
| Expression | Selects |
|---|---|
//div |
Every div anywhere in the document |
/html/body/div |
Absolute path from the root (brittle, avoid) |
//div/p |
p elements that are direct children of a div |
//div//p |
p elements anywhere inside a div |
//div[@class="price"] |
div with that exact class attribute |
//a[@href] |
Every a that has an href at all |
//ul/li[1] |
First li in each list (XPath counts from 1) |
//ul/li[last()] |
Last li in each list |
//tr[position() <= 3] |
First three rows |
(//h2)[2] |
The second h2 in the whole document |
Matching Text and Partial Values
Real pages have messy class names and dynamic fragments, so partial matching does the heavy lifting:
| Expression | Selects |
|---|---|
//button[text()="Next"] |
Button whose text is exactly "Next" |
//h2[contains(text(), "Review")] |
Headings containing "Review" |
//div[contains(@class, "product")] |
Class attribute containing "product" |
//a[starts-with(@href, "/item/")] |
Links whose href starts with /item/ |
//p[normalize-space()="Done"] |
Text match ignoring stray whitespace |
That contains(@class, ...) pattern is the single most-used expression in practical scraping, because modern sites generate classes like product-card__price--3xk9f.
Extracting Values (Not Just Elements)
| Expression | Returns |
|---|---|
//h1/text() |
The heading's text node |
//a/@href |
The link URL itself |
//img/@src |
The image URL |
string(//div[@class="desc"]) |
All descendant text, concatenated |
The Two Tricks CSS Selectors Can't Do
Walking up the tree. You found the price; now you want the product card that contains it:
//span[contains(text(), "$")]/..
//span[contains(text(), "$")]/ancestor::div[contains(@class, "card")]
Selecting by an element's content. "The td after the td that says Email":
//td[text()="Email"]/following-sibling::td[1]
These two patterns are why scraping veterans keep XPath in the toolbox even when they prefer CSS selectors day to day. Key-value tables, in particular, are miserable with CSS and trivial with following-sibling.
Using It in Practice
Test expressions live in your browser first: DevTools console, $x('//h2') runs XPath directly. Then in code:
# Python with lxml
from lxml import html
tree = html.fromstring(page_source)
prices = tree.xpath('//div[contains(@class, "price")]/text()')
// Playwright
const price = await page.locator('xpath=//td[text()="Price"]/following-sibling::td[1]').textContent();
Scrapy, Selenium, and Playwright all take XPath natively. BeautifulSoup notably does not; that's the main reason its users default to CSS selectors.
XPath or CSS Selectors?
The pragmatic answer: CSS for the easy 80 percent, XPath for the hard 20. CSS selectors (div.price, ul > li:first-child) are shorter and more readable for straightforward selection. Reach for XPath when you need text-content matching, parent traversal, or sibling logic. Every framework that accepts one accepts the other, so this isn't a commitment, just a per-query choice.
One warning that applies to both: selectors are the part of your scraper that rots. Sites redesign and your beautiful expressions silently match nothing (see our note on validating what a 200 response actually contains). Prefer expressions anchored to stable semantics (itemprop attributes, data-* attributes, visible text) over generated class names, and validate extraction output so failures are loud.
And sometimes the right amount of XPath is none: if you're feeding pages to an LLM, fetching them as clean Markdown via link.sc and letting the model pull the fields replaces selector maintenance entirely. Selector-based extraction is cheaper per page; Markdown-plus-LLM is cheaper per site redesign. Pick per project.
The Bottom Line
Twenty expressions cover essentially all scraping XPath: //, attribute filters, contains(), positional indexes, text() and @attr extraction, and the parent and sibling axes. Test in DevTools with $x(), anchor to stable attributes, and keep XPath for the queries CSS can't express. That's the whole discipline.
Prefer skipping selectors altogether? Get a free link.sc API key and fetch any page as LLM-ready Markdown instead.