Quick answer: Competitive intelligence with web scraping means watching the public surfaces your competitors control (pricing pages, feature and docs pages, changelogs, job listings, blog and press) on a schedule, diffing each fetch against the last, and letting an LLM summarize what actually changed. The output is a short digest a human reads in two minutes, not a dashboard nobody opens. Stick to public data and respect each site's terms.
I have watched teams build elaborate scraping rigs that collect everything and surface nothing. The value is not in the collection. It is in the diff and the summary: knowing a competitor cut their entry price or shipped a feature you were about to build, the morning it happens.
What is actually worth tracking
Not every page is worth a crawl. These five signals carry most of the value, because each one is a competitor telling the market something about their strategy.
| Signal | Source page | Tells you |
|---|---|---|
| Pricing | Pricing / plans page | Positioning, discounting, new tiers |
| Features | Product, docs, changelog | Roadmap direction, gaps closed |
| Hiring | Careers / job board | Where they are investing next |
| News | Blog, press, newsroom | Launches, funding, partnerships |
| Messaging | Homepage hero, taglines | Who they think their buyer is |
Hiring is the underrated one. A company that just posted five roles for a payments team is telling you exactly what they are about to build, months before it ships.
The pipeline: search, fetch, diff, summarize
The architecture is four stages, and each is a small, boring component. Boring is the goal; competitive intelligence has to run unattended for months.
1. Discover. For news and mentions you do not already have URLs for, search. Each result in serpData.results carries a targetUrl, and the interesting ones feed straight into the fetch stage.
curl https://api.link.sc/v1/search \
-H "x-api-key: lsc_..." \
-H "Content-Type: application/json" \
-d '{
"q": "Acme Corp pricing OR funding OR launch",
"engine": "google"
}'
2. Fetch. For the pages you already track (their pricing page, their changelog), fetch each to clean markdown on a schedule. Clean text is what makes the diff meaningful; a raw-HTML diff lights up on every rotated session token and ad tag.
curl https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_..." \
-H "Content-Type: application/json" \
-d '{
"url": "https://competitor.example.com/pricing",
"format": "markdown"
}'
3. Diff. Compare today's markdown against the stored copy. A plain text diff is enough to detect that something changed and to isolate the changed lines.
import difflib
def changed_lines(old: str, new: str) -> list[str]:
diff = difflib.unified_diff(
old.splitlines(), new.splitlines(), lineterm="", n=0
)
return [l for l in diff if l.startswith(("+", "-"))
and not l.startswith(("+++", "---"))]
4. Summarize. Hand the changed lines (not the whole page) to an LLM and ask for a one-line, plain-language description of what moved. Feeding only the diff keeps it cheap and keeps the model focused on the delta.
prompt = f"""These lines changed on a competitor's {page_type} page.
In one sentence, describe what changed in business terms. If the
change is cosmetic (typo, reformatting), reply exactly "no signal".
Changed lines:
{chr(10).join(changed_lines(old, new))}
"""
That "no signal" escape hatch matters. Most diffs are noise, and you want the model to say so rather than manufacture significance for a fixed footer date.
Turning it into a digest
Raw change events are still noise. Batch them. Run the crawl overnight, collect every change that came back with a real signal, group by competitor, and send one digest.
Competitive digest, 2026-07-18
Acme Corp
• Pricing: entry plan dropped from $29 to $19/mo
• Careers: 3 new roles on an "AI infra" team
Globex
• Changelog: shipped SSO on the Business plan
A person reads that in under a minute and knows what changed across every competitor. Compare it to a scraping setup that dumps 4,000 rows into a table: same data, but only one of them changes a decision.
For the deeper mechanics of scheduling fetches and detecting changes reliably, see monitor web page changes and get alerts. If pricing is your main interest, monitor competitor pricing changes narrows the same pipeline to that one signal.
Cadence: match the crawl to the page
Crawling everything hourly wastes credits and annoys the sites you depend on. Match frequency to how fast each page actually moves.
| Page type | Sensible cadence |
|---|---|
| Pricing | Daily |
| Changelog / releases | Daily |
| Job listings | Weekly |
| Blog / news | Daily via search |
| Homepage messaging | Weekly |
The ethics and the public-data boundary
Competitive intelligence has a clear line, and staying on the right side of it is not just legal caution, it is what keeps the practice defensible.
Everything above is public data: pages a competitor publishes for anyone to read. That is fair game to observe, and observing it is what analysts have always done, just faster. What is not fair game: anything behind a login you are not entitled to, anything that requires evading authentication, scraping personal data about individuals, or hammering a site hard enough to degrade it. Read each site's terms and robots.txt, keep your rate polite, identify your crawler honestly, and stop where a site tells you to.
The simple test: if a competitor watched your logs, would your activity look like a curious visitor or like an intruder? Keep it firmly in the first category. For the wider legal picture, is web scraping legal is worth reading before you scale anything up.
Start small
You do not need to track ten competitors across twenty pages on day one. Pick two competitors and their pricing and changelog pages. Get the diff-and-digest loop working and useful. Then add signals. A small pipeline that reliably tells you when a competitor cuts their price beats an ambitious one that breaks in a month and gets ignored.
The whole point is a two-minute morning read that occasionally changes what your team does that day. Build toward that, not toward a data lake.
Watching competitors' public pages? link.sc gives you search and clean-markdown fetch in one API, so your diff-and-summarize loop stays simple. Start free.