Quick answer: Most competitor price scraping fails for boring reasons: prices rendered by JavaScript, variant-dependent pricing, and brittle CSS selectors. The reliable approach is to check for schema.org/JSON-LD structured data first (most stores have it), fall back to LLM extraction when they don't, and fetch through an API that handles rendering and blocking for you. Then schedule it and diff the results.
I've built price monitors that ran for years and price monitors that died in a week. The difference was never cleverness. It was picking extraction methods that don't depend on the retailer's CSS staying still.
What You're Actually Extracting
For competitor price tracking, you need surprisingly few fields per product page:
| Field | Why it matters |
|---|---|
| Product title | Confirming you're on the right product |
| Price | The whole point |
| Currency | Multi-region stores lie to you otherwise |
| Availability | A price on an out-of-stock item is noise |
| Variant / SKU | "The price" often depends on size or color |
| Timestamp | Prices without time context are useless |
Resist the urge to grab everything. Every extra field is another thing that breaks when the page changes.
The Shortcut Almost Everyone Misses: JSON-LD
Before you write a single CSS selector, view the source of the product page and search for application/ld+json. Most ecommerce platforms (Shopify, WooCommerce, BigCommerce, Magento and the big custom builds) embed schema.org Product markup so Google can show rich results. That markup contains exactly what you want, in stable, machine-readable form:
{
"@type": "Product",
"name": "Trail Runner GTX",
"offers": {
"@type": "Offer",
"price": "129.95",
"priceCurrency": "USD",
"availability": "https://schema.org/InStock"
}
}
Retailers keep this accurate because their Google Shopping visibility depends on it. It changes far less often than the page's HTML. When it's present, parse it and ignore the rest of the page:
import json, re, requests
def price_from_jsonld(html: str):
for block in re.findall(
r'<script type="application/ld\+json">(.*?)</script>', html, re.S
):
try:
data = json.loads(block)
except json.JSONDecodeError:
continue
items = data if isinstance(data, list) else [data]
for item in items:
if item.get("@type") == "Product":
offer = item.get("offers", {})
if isinstance(offer, list):
offer = offer[0]
return {
"name": item.get("name"),
"price": offer.get("price"),
"currency": offer.get("priceCurrency"),
"availability": offer.get("availability"),
}
return None
Handling JavaScript-Rendered Prices
The catch: on plenty of modern storefronts, the raw HTML you get from a plain requests.get doesn't contain the price at all. It arrives later via an API call the browser makes, or the JSON-LD itself is injected client-side. You have two options.
Option one is running your own headless browser fleet. It works, and it's a part-time job: Chrome updates, memory leaks, proxy rotation, fingerprint tuning. Option two is letting a fetch API do the rendering. This is the core of what link.sc does: you request a URL, it handles rendering and anti-bot friction, and you get back the final page as markdown, HTML, or extracted JSON.
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_your_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://shop.example.com/products/trail-runner-gtx",
"format": "json",
"schema": {
"title": "product title",
"price": "current price as a number",
"currency": "ISO currency code",
"in_stock": "true if the product can be purchased now"
}
}'
The LLM extraction path costs a bit more per request than parsing JSON-LD yourself, so my rule is: JSON-LD first, LLM extraction as the fallback for stores that don't expose it. If the store sits behind aggressive bot protection, that's a separate problem I covered in scraping Cloudflare-protected sites.
Variants Will Ruin Your Data If You Let Them
A product page is often not one price. It's a matrix: size 10 in blue costs more than size 9 in black, and the "default" price shown on load is whichever variant the store chose to feature. If you track "the price of the page," your history will show phantom price changes that are really just default-variant changes.
Two fixes:
- Track variant-specific URLs where the store supports them (
?variant=123on Shopify, for example). - Extract the variant identifier alongside the price and store price history per SKU, not per URL.
If you can only do one, do the second. Per-SKU history is what makes the data trustworthy.
Scheduling and Change Detection
Price scraping is only useful as a time series. A cron job, a fetch per product, and a diff against the last stored value gets you 90% of the way:
def check(product):
current = fetch_price(product["url"]) # link.sc call from above
last = db.latest_price(product["sku"])
if last is not None and current["price"] != last:
alert(product["sku"], last, current["price"])
db.insert(product["sku"], current)
For cadence: daily is enough for most catalogs. Hourly only makes sense for fast-moving categories, and even then, only for your top SKUs. Every extra check multiplies your cost and the load you put on the retailer for data that mostly hasn't changed. I went deeper on the alerting side in monitor competitor pricing changes.
The Legal and ToS Note You Shouldn't Skip
Pricing data on a public product page is generally about as safe as scraped data gets: it's public, factual, and not personal data. That said:
- Many retailers' terms of service prohibit automated access. A ToS violation is usually a contract issue rather than a crime, but it can get your access blocked and, in aggressive cases, lawyered. Read the terms of anyone you monitor at scale.
- Respect robots.txt and keep your request rate low. One request per product per day is invisible; hammering a site every minute is how you end up IP-banned and on someone's radar.
- Never scrape prices from behind a login you agreed to terms for.
- Don't republish a retailer's full catalog as your own product.
None of this is legal advice; if the data is core to your business, spend an hour with a lawyer before you spend a month on infrastructure.
The Bottom Line
Ecommerce price scraping in 2026 is a solved problem if you stack it right: JSON-LD when available, LLM extraction as fallback, a fetch API to absorb the rendering and blocking headaches, per-SKU storage, and a polite daily schedule. Spend your creativity on what you do with the data, not on re-solving extraction every time a retailer redesigns.
Want price extraction without babysitting headless browsers? link.sc fetches any product page and returns clean JSON in one call. Get 500 free credits a month.