Quick answer: Scrapers break silently because the sites they target change and the scraper keeps returning something, just the wrong something. Test a web scraper with saved HTML fixtures (fast, offline unit tests of your parser), snapshot tests that flag when extracted fields change, and contract checks that assert the shape of the data. Then monitor scrapers in production on three signals: success rate per domain, field fill rate, and schema drift, with alerts when any of them drops. The goal is to find out from a dashboard, not from a user, that a selector died.
A scraper is the only kind of software that breaks without you touching it. You ship it working, a target site quietly reships their markup, and your div.price selector now matches nothing or matches a cookie banner. The code did not change; the world did. Here is how to catch that fast instead of discovering it in a stale report three weeks later.
Why Scrapers Rot
Every scraper carries an implicit dependency on someone else's HTML, and that HTML is not a stable API. It drifts in a few predictable ways:
- Selector drift. A class renames from
product-pricetopdp-price__valueand your extractor silently returns null. - Layout changes. A site moves a field into a different container, or splits one element into two.
- A/B tests. The same URL serves different markup to different visitors, so your scraper works half the time.
- Anti-bot changes. A site adds a challenge and your fetch starts returning a block page that still parses as valid HTML.
The dangerous part is that most of these do not throw an error. Your code runs, returns a row, and writes it. The row is just empty or wrong. Testing and monitoring exist to convert silent wrongness into a loud signal.
Testing: Freeze the HTML, Test the Parser
The single most valuable test is also the cheapest: save real HTML to a file and assert your parser extracts the right fields from it. This is a fixture test. It runs offline, in milliseconds, with no network, and it pins your parsing logic against a known page.
# tests/test_parser.py
from pathlib import Path
from myscraper.parse import parse_product
def load(name: str) -> str:
return Path(f"tests/fixtures/{name}").read_text()
def test_parses_product_page():
html = load("product_2026-07-18.html")
product = parse_product(html)
assert product["title"] == "Wireless Headphones"
assert product["price"] == 79.99
assert product["in_stock"] is True
def test_handles_out_of_stock():
html = load("product_oos.html")
product = parse_product(html)
assert product["in_stock"] is False
assert product["price"] is None
Keep fixtures for the happy path and every weird case you have hit: out of stock, missing image, sale price, sold-out variant. When a real page changes, save the new HTML as a fixture, watch the test fail, fix the parser, and you have both fixed the bug and captured a regression test in one move.
Snapshot Tests
A snapshot test records the full extracted output and flags any change on the next run. It is not asserting correctness; it is asserting stability. When the snapshot changes, a human looks and decides "expected" (accept the new snapshot) or "the site changed under us" (fix the parser). This catches the subtle drifts a hand-written assertion would miss.
Contract Checks
A contract check validates the shape of what you extract, independent of the specific values. Every product must have a non-empty title, a price that is a positive number, and a boolean stock flag. Encode it once and run it against every scrape, in tests and in production.
def validate(product: dict) -> list[str]:
errors = []
if not product.get("title"):
errors.append("missing title")
price = product.get("price")
if price is not None and (not isinstance(price, (int, float)) or price <= 0):
errors.append(f"bad price: {price!r}")
if not isinstance(product.get("in_stock"), bool):
errors.append("in_stock not boolean")
return errors
The same validate function is a test helper and a production guardrail. That reuse is the point: your contract is defined in exactly one place.
Monitoring in Production
Tests catch what you anticipated. Monitoring catches what you did not. Watch three signals per domain and alert when they move.
| Signal | What it means | Alert when |
|---|---|---|
| Success rate | Fraction of fetches returning usable pages | Drops sharply for one domain |
| Field fill rate | Fraction of rows with each key field populated | A field's fill rate falls off a cliff |
| Schema drift | New or missing keys vs the expected contract | Any unexpected schema change |
Success rate per domain is the headline metric. A global success rate hides the failure; per-domain, you see that one site went from 98% to 4% overnight, which almost always means a layout change or a new block, not a bug in your code.
Field fill rate catches the silent partial break. Your fetch succeeds, the page parses, but price is suddenly null on 90% of rows because the price selector broke. Success rate looks fine; fill rate screams.
Schema drift catches structural change. Log the set of keys you extract and compare it to the contract. A field that vanishes or a new one that appears is worth a human glance.
import logging
logger = logging.getLogger("scraper.metrics")
def record(domain: str, product: dict):
errors = validate(product)
logger.info("scrape", extra={
"domain": domain,
"ok": not errors,
"has_price": product.get("price") is not None,
"has_title": bool(product.get("title")),
"errors": errors,
})
Ship those structured logs to whatever you already run (a metrics backend, a log aggregator, even a nightly query over a table) and put an alert on each signal. The specific tooling matters far less than having the numbers at all.
Separate "Fetch Failed" From "Parse Failed"
A block page and a layout change look identical in your data (empty fields) but need opposite fixes. Distinguish them early. If the fetch layer returns a 403, a challenge page, or a suspiciously short body, that is a fetch problem, and the playbook is how sites detect and block scrapers. If the fetch returned a full, valid page but your selectors found nothing, that is a parse problem, and you need a new fixture and a selector fix.
Using a managed fetch layer helps here because it collapses the fetch-failure surface. link.sc handles proxies, rendering, and the block ladder server-side and returns clean content or a clear error, so a null field is almost always a parse problem, not a mystery block:
import os, requests
def fetch(url: str) -> str:
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": os.environ["LINKSC_KEY"]},
json={"url": url, "format": "markdown"},
timeout=60,
)
resp.raise_for_status() # fetch problems surface as HTTP errors
return resp.json()["content"] # a clean body means null fields are on you
Wire It Into Your Automation
Testing and monitoring are not a separate project; they belong in the scheduled pipeline you already run. If you have not built that scheduling and orchestration yet, how to automate web scraping covers where the fixture tests run in CI and where the production metrics get emitted. The short version: run fixture and contract tests on every deploy, and emit the three production signals on every run.
The Ethics Note
Monitoring is also how you stay a good citizen at scale. Watch your own request rate per domain and alert if a bug sends it spiking, because a runaway scraper is a self-inflicted denial-of-service against someone else's site. Keep to public data, respect robots.txt, and treat a sudden success-rate drop as a possible signal that a site owner would rather you slowed down. Reliable scraping and polite scraping turn out to be the same discipline.
Build the fixture tests first, add the three production signals, and separate fetch failures from parse failures. Do that and your scrapers stop breaking silently, which is the only kind of breaking that actually hurts.
Want null fields to mean a parse bug, not a mystery block? Get a free link.sc key and let the fetch layer return clean content every time.