You want to know the moment a competitor changes their prices. Sounds simple: grab the page every morning, compare it to yesterday, alert on a difference. Then you actually build it, and the alerts never stop firing. The page changed, but not the prices. Welcome to the gap between "detect any change" and "detect the change I care about."
This post walks through the naive approach, why it breaks, and how to get to something you can actually trust.
The Naive Approach and Why It Falls Apart
The first version everyone writes is a cron job plus a diff:
#!/bin/bash
curl -s https://competitor.example/pricing > new.html
diff old.html new.html && echo "no change" || echo "CHANGED"
mv new.html old.html
This is a good instinct and a bad implementation. Three things will make your life miserable:
JS-rendered prices. A growing share of pricing pages ship an empty shell of HTML and hydrate the numbers with JavaScript. curl sees <div id="price"></div> and never the $49. Your diff is comparing skeletons, so you either see no changes ever or you see changes that have nothing to do with price.
Layout and markup noise. Even when the price is in the HTML, it is buried in a page full of things that change on every request: a CSRF token, a build hash in an asset URL, a rotating testimonial, an ad slot, a "trusted by 4,312 teams" counter. A raw byte diff treats all of it as signal.
Timestamp and personalization churn. Server-rendered timestamps, A/B test variants, and geo-based banners mean two requests seconds apart already differ. You will get paged at 3am because a marketing carousel rotated.
The root problem is that you are diffing the whole document when you only care about a handful of numbers. Fix that and most of the pain disappears.
Extract Just the Price with Structured Extraction
Instead of comparing pages, compare facts. Define the exact shape of the data you care about and pull only that. link.sc takes a JSON schema and returns structured data, so it renders the JavaScript, finds the fields, and hands you clean values.
import requests
schema = {
"type": "object",
"properties": {
"plans": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"monthly_price_usd": {"type": "number"},
"annual_price_usd": {"type": "number"},
"included_seats": {"type": "integer"}
}
}
}
}
}
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_..."},
json={
"url": "https://competitor.example/pricing",
"format": "json",
"schema": schema,
},
)
snapshot = resp.json()["data"]
Now snapshot is something like {"plans": [{"name": "Pro", "monthly_price_usd": 49, ...}]}. Notice what is not in there: no ad copy, no counters, no build hashes, no timestamps. You have reduced a 400KB HTML document to the dozen values that actually matter. Because link.sc handles rendering, proxies, and retries, the JS-rendered prices show up too.
Schedule the Checks
Once extraction is a pure function from URL to a small JSON object, scheduling is boring in the good way. A daily cron entry is plenty for pricing, which rarely changes more than a few times a year:
# crontab -e — run at 08:00 daily
0 8 * * * /usr/bin/python3 /opt/pricewatch/check.py >> /var/log/pricewatch.log 2>&1
A few practical notes. Store each snapshot with a timestamp so you have history, not just "current vs. previous" — you will want to answer "when did this change?" later. Keep the raw markdown or HTML alongside the structured JSON too ("format": "markdown" in a second call, or fetch both), so when something looks off you can see the page as it was. And do not poll every minute; you gain nothing and it is rude.
Diff Semantically, Not Textually
With snapshots as normalized JSON, the diff becomes a comparison of values, not text. Load the previous snapshot, compare the fields you care about, and only flag real differences:
import json, pathlib
prev = json.loads(pathlib.Path("last.json").read_text() or "{}")
curr = snapshot
def index_by_name(snap):
return {p["name"]: p for p in snap.get("plans", [])}
old, new = index_by_name(prev), index_by_name(curr)
changes = []
for name, plan in new.items():
if name not in old:
changes.append(f"NEW PLAN: {name} at ${plan['monthly_price_usd']}/mo")
else:
for field in ("monthly_price_usd", "annual_price_usd", "included_seats"):
if old[name].get(field) != plan.get(field):
changes.append(
f"{name}.{field}: {old[name].get(field)} -> {plan.get(field)}"
)
for name in old.keys() - new.keys():
changes.append(f"REMOVED PLAN: {name}")
pathlib.Path("last.json").write_text(json.dumps(curr, indent=2))
This is the whole trick. You are no longer asking "did any byte change?" You are asking "did the Pro plan's monthly price change from 49 to 59?" That question has almost no false positives, because you stripped the noise out before you ever compared anything. Adding or removing a plan is caught explicitly, which is often the more strategically interesting event than a price tweak.
Alert on Real Changes
Only send a notification when changes is non-empty, and put the actual before/after in the message so the reader does not have to go digging:
if changes:
body = "Competitor pricing changed:\n" + "\n".join(f" - {c}" for c in changes)
requests.post(
"https://hooks.slack.example/services/XXX",
json={"text": body},
)
Because the payload only fires on genuine value changes, people will actually read these alerts instead of muting the channel after week one. That is the real payoff: a monitor that stays useful.
Wrapping Up
The naive cron-and-diff approach fails not because the idea is wrong but because it diffs the wrong thing. Extract the specific values first, then diff those. link.sc handles the ugly parts — JS rendering, proxies, retries, and turning a messy page into the exact JSON schema you asked for — so your monitoring code stays a dozen lines of comparison logic.
You get 500 free credits every month, which is more than enough to watch a handful of pricing pages daily. Grab a key at link.sc/register or read the docs to see the full schema options.