Quick answer: To monitor a web page for changes, fetch it on a schedule (cron or a scheduled function), normalize the content to strip noise, compare it against the last stored snapshot, and push an alert to Slack, email, or a webhook when the diff matters. The naive approach (hash the raw HTML) fires constantly on ad rotations and timestamps, so the real engineering is in deciding what counts as a change: normalized-content diffs for most cases, CSS-selector watching for specific fields, and LLM diffs when you care about meaning.
I've built page monitors that paged me at 3am because a site rotated its CSRF token. Here's the ladder of approaches, from naive to smart, and where each one belongs.
Level 0: hash the page (and regret it)
The obvious first version: fetch the page, SHA-256 the body, compare to last time.
This fails on almost any real page. Session tokens, cache-buster query strings in asset URLs, rotating ads, "23 people viewing this" widgets, and server timestamps all change on every single request. Your monitor cries wolf hourly and you stop reading the alerts within a week.
Raw hashing is only viable for static files: a PDF, a robots.txt, a JSON endpoint. For HTML pages, skip straight to level 1.
Level 1: normalize, then diff
The fix is to compare content, not bytes. Extract the main content, convert it to a stable text form, and diff that. Fetching as markdown does most of the normalization for you, because boilerplate, scripts, and markup noise are already gone.
Here's a complete monitor in Python using link.sc for the fetch and a Slack webhook for alerts:
import difflib
import hashlib
import json
import pathlib
import requests
WATCH_URL = "https://example.com/changelog"
SNAP_DIR = pathlib.Path("snapshots")
SLACK_WEBHOOK = "https://hooks.slack.com/services/T000/B000/XXXX"
def fetch_markdown(url: str) -> str:
resp = requests.get(
"https://link.sc/v1/fetch",
headers={"Authorization": "Bearer lsc_your_key"},
params={"url": url, "format": "markdown"},
timeout=60,
)
resp.raise_for_status()
return resp.json()["markdown"]
def run():
SNAP_DIR.mkdir(exist_ok=True)
current = fetch_markdown(WATCH_URL)
key = hashlib.sha256(WATCH_URL.encode()).hexdigest()[:16]
snap_file = SNAP_DIR / f"{key}.md"
previous = snap_file.read_text() if snap_file.exists() else None
snap_file.write_text(current)
if previous is None or previous == current:
return
diff = "\n".join(difflib.unified_diff(
previous.splitlines(), current.splitlines(),
lineterm="", n=2,
))[:3000]
requests.post(SLACK_WEBHOOK, json={
"text": f"Page changed: {WATCH_URL}\n```{diff}```"
})
if __name__ == "__main__":
run()
Schedule it with cron:
# Check every hour, log output
0 * * * * /usr/bin/python3 /opt/monitor/watch.py >> /var/log/watch.log 2>&1
This version is already good enough for changelogs, docs pages, terms-of-service pages, and job boards. Markdown normalization kills most of the false positives that plague raw-HTML approaches.
Level 2: watch specific fields with selectors
Sometimes you don't care about "the page changed," you care about one value: a price, a stock status, a version number. Extract just that field and compare it.
from bs4 import BeautifulSoup
def extract_price(html: str) -> str | None:
soup = BeautifulSoup(html, "html.parser")
node = soup.select_one("span.price")
return node.get_text(strip=True) if node else None
Store the extracted value instead of the whole page, and alert only when it changes. This gives you zero false positives and alerts that say something useful ("price went from $49 to $59") instead of dumping a diff.
The tradeoff is brittleness: when the site redesigns, your selector returns None. Treat a sudden None as its own alert, not as "no change." This is the exact pattern behind price tracking, which I covered end to end in how to monitor competitor pricing changes.
Level 3: semantic diffs with an LLM
Normalized diffs still fire on changes you don't care about: a fixed typo, a reworded sentence, a reordered FAQ. When you only want alerts for changes in meaning, put an LLM in the loop.
The pattern: when the normalized diff is non-empty, send both versions (or just the diff) to a small model and ask a yes/no question with a summary.
from openai import OpenAI
client = OpenAI()
def is_meaningful(diff: str, context: str) -> dict:
resp = client.chat.completions.create(
model="gpt-4o-mini",
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": (
"You review diffs of a monitored web page. Decide whether the "
"change is meaningful for someone tracking: " + context + ". "
'Reply as JSON: {"meaningful": bool, "summary": "one sentence"}'
)},
{"role": "user", "content": diff},
],
)
return json.loads(resp.choices[0].message.content)
Because you only call the LLM when a diff exists, and only send the diff, the token cost stays tiny. This is the difference between "page changed" and "the vendor removed the free tier from their pricing page," and the second alert is the one that gets read.
Alerting sinks: where the alert should go
| Sink | Best for | Notes |
|---|---|---|
| Slack webhook | Team visibility | One POST, five minutes of setup |
| Non-technical stakeholders | Use a service (SES, Resend); don't run your own SMTP | |
| Webhook to your app | Triggering automation | Alert becomes an event other systems consume |
| PagerDuty/Opsgenie | Genuinely urgent changes | Only if a change requires waking someone up |
My advice: start with Slack, and put the summary in the message itself. An alert that requires a click to understand is an alert that gets ignored.
Storing snapshots properly
Keep history, not just the latest snapshot. When an alert fires at 9am, you'll want to know if the change happened at 8am or Friday night.
Practical rules: store the normalized markdown (small, diffable), keyed by URL hash and timestamp; a snapshots/ directory in a git repo is a surprisingly good poor-man's history (every check is a commit, diffs come free). Move to S3 plus a database row per check once you're watching more than a few dozen pages.
One warning on politeness: checking a page every minute is rarely justified and looks like abuse to the target site. Hourly is right for most business monitoring. If you start hitting 429s, back off; I wrote up the full playbook in HTTP 429 explained.
The build vs. buy line
The cron-plus-script version above genuinely works, and for a handful of stable pages I'd just run it. The pain arrives with scale and with hostile pages: JS-rendered content, bot walls, and sites that block your server's IP.
That fetch layer is the part worth outsourcing. The link.sc fetch endpoint handles rendering and access and returns normalized markdown, so your monitor stays twenty lines of diff logic. The free tier covers 500 fetches a month, which is hourly monitoring of a page with room to spare.
Want change monitoring without babysitting a scraper? Get a free link.sc key and have the fetch-and-normalize layer done in one API call.