← All posts

Free XPath Tester: Check Any Selector Against a Live Page

ENGINEERING · JUL 19, 2026 · 5 MIN READ

Quick answer: An XPath tester lets you type a selector, point it at a page, and see exactly which elements match before you bake it into a scraper. The fastest options are already on your machine: the browser DevTools console runs $x('//h2') against the page you are looking at, and a five-line Python script runs XPath against any URL you fetch. Below is a tester you can copy, plus the one thing most testers get wrong: they test against the HTML your browser rendered, not the HTML your scraper will actually receive.

The gap between "works in DevTools" and "works in my scraper" is where most selector bugs live. A good XPath tester closes that gap by evaluating your expression against the same bytes your code will see.

The 30-Second Browser Tester

You do not need to install anything to test XPath against the page in front of you. Open DevTools (F12), go to the Console tab, and use the built-in $x() function:

// Returns an array of matching nodes
$x('//h2')

// Count matches quickly
$x('//div[contains(@class, "price")]').length

// See the text of the first match
$x('//h1')[0].textContent

For CSS selectors, $$('div.price') does the same job. Both update live as you tweak the expression, which makes the console the fastest place to iterate on a selector.

The catch: this tests against the DOM after JavaScript has run and after the browser has repaired malformed HTML. Your scraper, if it uses requests or curl, sees the raw HTML instead. The two can differ enough that a selector matching ten elements in the console matches zero in your code.

A Real XPath Tester You Can Run

To test against the HTML your scraper actually receives, run the expression in the same environment. This ten-line script is a complete XPath tester: give it a URL and a selector, and it prints what matches.

import sys
import requests
from lxml import html

url, expr = sys.argv[1], sys.argv[2]
page = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
tree = html.fromstring(page.content)
matches = tree.xpath(expr)

print(f"{len(matches)} match(es)\n")
for m in matches[:10]:
    text = m if isinstance(m, str) else m.text_content()
    print(repr(text.strip()[:120]))

Run it like this:

python tester.py "https://example.com" '//h1/text()'

Because it uses lxml, it accepts the same XPath your Scrapy or Selenium code uses, so a passing test here means a passing test in production. Swap page.content for a saved file and you can test offline against a captured snapshot too.

If you want to check a CSS selector in the same script, lxml has cssselect:

from lxml.cssselect import CSSSelector
sel = CSSSelector("div.price")
matches = sel(tree)

Why Your Selector Passes in DevTools but Fails in Code

Three things routinely cause a selector to match in the browser and miss in a script:

JavaScript-rendered content. The console sees the hydrated page. A plain requests.get() sees the initial HTML, which for many modern sites is a nearly empty shell with the data loaded later by JavaScript. If your tester returns zero matches but DevTools returns ten, this is the usual culprit. You either need a rendering fetch or you need to find the underlying API call. We cover the tradeoffs in scraping JavaScript-rendered websites.

Browser HTML repair. Browsers silently fix unclosed tags and misnested elements, which can change the tree structure your XPath walks. lxml is more forgiving than a strict parser but still will not reproduce every browser fixup, so //table/tr might work in the console (the browser inserted a tbody for you) while your code needs //table//tr.

Getting blocked instead of the page. If the site returns a challenge page, a 403, or an empty body to your script, your selector is being tested against the wrong document entirely. A 200 status code does not guarantee real content, so always print the response length before trusting a zero-match result.

Testing Against the HTML Your Scraper Will See

The cleanest way to eliminate the render-and-block problem is to test against a fetch that already handles both. Fetch the page through link.sc, which renders JavaScript and returns clean content, then run your XPath against that:

import requests
from lxml import html

resp = requests.post(
    "https://api.link.sc/v1/fetch",
    headers={"Authorization": "Bearer YOUR_KEY"},
    json={"url": "https://example.com", "format": "html"},
)
tree = html.fromstring(resp.json()["content"])
print(tree.xpath('//h2/text()'))

Now your tester and your scraper share the same input, so "it worked in the tester" actually means something. This is the difference between a toy tester and one you can trust: it evaluates the expression against the exact bytes production will handle, rendering and anti-bot included. The developer quickstart walks through the request shape and key setup.

If you would rather skip HTML entirely, request format: "markdown" and let a model pull the fields. That trades per-page cost for zero selector maintenance, which is worth it on sites that redesign often.

Tester Habits That Save Hours

A few practices turn a quick check into a reliable one:

  • Test the count, not just the presence. len(matches) catches the case where a loose selector grabs 200 elements instead of the 12 you wanted. Silent over-matching is as common as under-matching.
  • Anchor to stable attributes. Test expressions built on itemprop, data-* attributes, or visible text rather than generated classes like product-card__price--3xk9f. The XPath cheat sheet lists the contains() and following-sibling patterns that survive redesigns.
  • Print the raw response length first. One line, print(len(page.content)), tells you instantly whether you got the real page or a stub.
  • Save a snapshot. Write the fetched HTML to a file so you can re-run the tester after tweaking a selector without hitting the site again. This also gives you a fixture for a regression test.

XPath or CSS in the Tester

Both belong in your tester because both break in the same ways. Use CSS for straightforward selection (div.price, ul > li:first-child) and XPath for the queries CSS cannot express: text-content matching, parent traversal, and sibling logic. Every scraping framework accepts either, so testing is a per-query choice, not a commitment. The rule of thumb from the BeautifulSoup versus Selenium comparison holds here too: pick the selector language your parser supports and test in that exact parser.

The Bottom Line

The best XPath tester is the one that evaluates your selector against the same HTML your scraper receives. Iterate fast in the DevTools console with $x(), then confirm in lxml against a real fetch before you ship. When rendering or blocking muddies the input, fetch through a service that hands you clean, JavaScript-rendered content so the tester and the scraper agree. Test the count, anchor to stable attributes, and check the response length, and your selectors will fail loudly in the tester instead of silently in production.

Want your tester and your scraper to see the same page? Get a free link.sc API key and fetch any URL as rendered HTML or clean Markdown.