Quick answer: to scrape a website with XPath, fetch the HTML with requests, parse it with lxml or parsel, and pull data out with expressions like //article[@class="product_pod"]//h3/a/@title. Test every expression in your browser DevTools first with $x(), then move it into Python. The whole workflow fits in about 20 lines of code, and we'll build it below against a real site.
We'll scrape books.toscrape.com, a site that exists specifically for practicing this, so you can run every snippet as-is without worrying about terms of service. The steps are identical for real targets: inspect, test, extract, paginate.
If you want a reference for the expressions themselves, keep our XPath cheat sheet open in another tab. This post is about the workflow around them.
Step 1: Inspect the page before writing any code
Open books.toscrape.com, right-click a book, and hit Inspect. You'll see each book lives in a structure like this:
<article class="product_pod">
<h3><a href="catalogue/a-light-in-the-attic_1000/index.html"
title="A Light in the Attic">A Light in the ...</a></h3>
<p class="star-rating Three">...</p>
<div class="product_price">
<p class="price_color">£51.77</p>
<p class="instock availability">In stock</p>
</div>
</article>
Two things worth noticing before touching Python. The visible link text is truncated ("A Light in the ..."), but the title attribute holds the full name, so we'll extract the attribute, not the text. And the star rating isn't text at all; it's encoded in the class name star-rating Three. Pages hide data in attributes constantly, and XPath is unusually good at digging it out.
Step 2: Test expressions in DevTools, not in Python
This is the step most tutorials skip, and it's the one that saves the most time. Your browser console runs XPath natively via $x(). Open the console on the books page and try:
$x('//article[@class="product_pod"]') // 20 elements
$x('//article[@class="product_pod"]//h3/a/@title') // 20 title attributes
$x('//p[@class="price_color"]/text()') // 20 prices
Each call returns the matched nodes instantly. If an expression comes back empty, you find out in two seconds instead of after another edit-run-scroll cycle in your script. Only move an expression into Python once $x() proves it matches what you expect.
One caution: DevTools queries the rendered DOM, including anything JavaScript added after load. Your Python script sees only the raw HTML. If $x() finds elements but your script gets an empty list, the data is probably injected by JavaScript, which we'll deal with at the end.
Step 3: Extract with lxml
Install the two libraries:
pip install requests lxml
Now the scraper:
import requests
from lxml import html
resp = requests.get("https://books.toscrape.com/")
tree = html.fromstring(resp.text)
books = tree.xpath('//article[@class="product_pod"]')
print(f"found {len(books)} books")
for book in books:
title = book.xpath('.//h3/a/@title')[0]
price = book.xpath('.//p[@class="price_color"]/text()')[0]
stock = book.xpath('normalize-space(.//p[contains(@class, "availability")])')
print(f"{title} | {price} | {stock}")
Run it and you get 20 rows of clean data.
The pattern here matters more than the specific site: select the repeating container first (//article[...]), then run relative expressions inside each one. That keeps each book's title paired with its own price. If you extracted all titles and all prices as two flat lists instead, one missing price anywhere on the page would silently misalign every row after it.
Note the leading dot in .//h3/a/@title. That dot means "search within this element." Drop it and //h3 searches the whole document from the root, so every iteration returns the first book's title 20 times. In my experience this is the single most common XPath scraping bug, and it's miserable to spot because the code runs without errors and the output looks plausible.
The normalize-space() call handles the availability text, which is padded with newlines and spaces in the raw HTML. It's built into XPath, so you don't need .strip() gymnastics in Python.
Step 4: The same scraper with parsel
parsel is the selector library that powers Scrapy, and it wraps lxml with a friendlier API:
pip install parsel
import requests
from parsel import Selector
resp = requests.get("https://books.toscrape.com/")
sel = Selector(text=resp.text)
for book in sel.xpath('//article[@class="product_pod"]'):
rating_class = book.xpath('./p[contains(@class, "star-rating")]/@class').get()
print({
"title": book.xpath('.//h3/a/@title').get(),
"price": book.xpath('.//p[@class="price_color"]/text()').get(),
"rating": rating_class.replace("star-rating ", ""),
})
The practical difference is .get(), which returns the first match or None instead of raising an IndexError on an empty list the way lxml's [0] does. On a 500-page crawl, one malformed listing shouldn't kill the whole run, so I reach for parsel on anything bigger than a one-off script. It also lets you mix CSS and XPath on the same selector, and chaining .getall() reads better than list indexing.
Notice we also grabbed the rating by reading the @class attribute and stripping the prefix, turning star-rating Three into Three. Attribute extraction like this is where XPath earns its keep.
Step 5: Handle pagination
The catalogue has 50 pages, and each page links to the next with <li class="next">. That gives us a clean loop: scrape, look for the next link, follow it, stop when it's gone.
from urllib.parse import urljoin
import requests
from parsel import Selector
url = "https://books.toscrape.com/catalogue/page-1.html"
results = []
while url:
sel = Selector(text=requests.get(url).text)
for book in sel.xpath('//article[@class="product_pod"]'):
results.append({
"title": book.xpath('.//h3/a/@title').get(),
"price": book.xpath('.//p[@class="price_color"]/text()').get(),
})
next_href = sel.xpath('//li[@class="next"]/a/@href').get()
url = urljoin(url, next_href) if next_href else None
print(len(results)) # 1000
Two details do real work here. urljoin resolves the relative href against the current URL, which matters because the site's links are relative and change shape between the homepage and the catalogue pages. And driving the loop off the presence of the next button, rather than hardcoding "50 pages," means the scraper keeps working when the catalogue grows.
On a real site, add a time.sleep(1) inside that loop. Hammering a server is how you get rate limited, and it's rude besides.
When this stops working
This workflow assumes the data is in the HTML that the server sends. Increasingly, it isn't. If your DevTools test matches elements but your script sees nothing, the page is rendering content with JavaScript, and you need a different approach for JS-rendered sites. If you're getting 403s before you can even parse anything, you're being fingerprinted, and swapping in curl_cffi for requests usually fixes it.
There's also a maintenance question nobody warns beginners about: selectors rot. Sites redesign, class names change, and your scraper silently returns garbage. For scrapers feeding LLMs or agents, I've largely stopped writing selectors at all and instead fetch pages as clean Markdown through link.sc, letting the model pull the fields. XPath is cheaper per page; Markdown plus an LLM is cheaper per redesign. For a stable site and a tight budget, though, the 20 lines above are hard to beat.
Want the page contents without maintaining selectors? Grab a free link.sc API key and fetch any URL as LLM-ready Markdown.