Quick answer: A price comparison tool has five parts: product matching across retailers, price fetching, normalization, storage, and a presentation layer. Fetching prices is the easy part in 2026; an API call gets you clean data from most product pages. The genuinely hard part is matching, deciding that retailer A's "Sony WH1000XM5B" and retailer B's "Sony WH-1000XM5 Wireless Headphones, Black" are the same product. Budget most of your effort there.
I want to be upfront about the difficulty curve here, because most tutorials get it backwards. They spend 90% of the words on scraping and one paragraph on matching. In a real system, the ratio of pain is reversed.
The Architecture at a Glance
Catalog (canonical products)
|
Matcher <--- Retailer product URLs (discovery)
|
Fetcher ---> link.sc /v1/fetch (price, stock, title)
|
Normalizer ---> currency, units, variant resolution
|
Storage ---> per (product, retailer) price history
|
UI / API ---> "cheapest now" + history charts
Five components, and each one can be embarrassingly simple at first. The one you cannot fake is the matcher.
Product Matching: The Actual Hard Part
Every retailer describes products differently. Titles differ, model numbers get mangled, bundles muddy the water ("with carry case"), and variants multiply everything. If you match wrong, your tool confidently tells users that a refurbished unit is the cheapest "new" price, and you lose their trust permanently.
Signals, in descending order of reliability:
| Signal | Reliability | Catch |
|---|---|---|
| GTIN / EAN / UPC | Excellent | Many pages don't expose it |
| MPN (manufacturer part number) | Good | Formatting varies (XM5 vs XM-5) |
| Brand + normalized model | Decent | Bundles and refurbs slip through |
| Title similarity (embeddings/fuzzy) | Fair | Needs human review queue |
| Price proximity | Weak | Only as a sanity check |
The practical strategy: extract identifiers first. Schema.org Product markup often includes gtin13 or mpn, and it's sitting in the page's JSON-LD waiting for you. When identifiers are missing, fall back to brand plus normalized model number, and route low-confidence matches to a manual review queue. Yes, a human queue. Every serious comparison site has one, whatever their marketing says.
Be honest with yourself about scope, too. Matching 500 electronics SKUs across 6 retailers is a weekend of tooling plus ongoing review. Matching "everything" across "everywhere" is the business model of companies with data teams. Start narrow.
Fetching Prices
Once you have matched URLs per retailer, fetching is the commodity step. Product pages are JavaScript-heavy and often bot-protected, so rather than run a browser farm, I fetch through link.sc and ask for structured output directly:
import requests
def fetch_offer(url: str) -> dict:
r = requests.post(
"https://link.sc/v1/fetch",
headers={"Authorization": "Bearer lsc_your_key"},
json={
"url": url,
"format": "json",
"schema": {
"title": "product title",
"price": "current price as a number",
"currency": "ISO 4217 currency code",
"in_stock": "boolean, true if purchasable now",
"condition": "new, used, or refurbished",
"gtin": "GTIN/EAN/UPC if present",
"mpn": "manufacturer part number if present",
},
},
timeout=60,
)
r.raise_for_status()
return r.json()
Note that I extract gtin and mpn in the same call. Every fetch is a chance to strengthen your matching data. The general technique is the one from extract structured JSON from any webpage.
Normalization: Small, Boring, Essential
Raw extracted offers disagree about everything. Normalize before storage:
- Currency. Store the original price and currency, plus a converted value at a timestamped exchange rate. Never overwrite the original.
- Condition. Map retailer language ("open box", "renewed", "grade A") onto a small enum. Comparing new against refurb is the classic comparison-site failure.
- Units. For consumables, compute price per unit (per 100g, per tablet). "Cheapest" is meaningless across pack sizes otherwise.
- Shipping. Decide early whether you compare item price or landed price, and label it clearly in the UI. Users assume landed; most data is item-only.
Storage and Freshness
The schema is small. Something like:
CREATE TABLE offers (
product_id TEXT NOT NULL,
retailer TEXT NOT NULL,
url TEXT NOT NULL,
price NUMERIC,
currency TEXT,
in_stock BOOLEAN,
fetched_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY (product_id, retailer, fetched_at)
);
Append-only history per (product, retailer), and a materialized "latest offer" view for the UI. That gives you price charts for free later.
On freshness: show the fetch timestamp in your UI and let users request an on-demand refresh for a product they're actively looking at. A daily background sweep plus on-demand refresh beats an hourly sweep of everything, both on cost and on how fresh the data feels where it matters. For products you refresh on a schedule, the mechanics are the same as any change monitor; I wrote that up in monitor web page changes and get alerts.
The UI Is Mostly About Honesty
The presentation layer is standard web work, but three product decisions matter more than the framework:
- Show when each price was last checked. Stale prices presented as current is how comparison tools die.
- Link out to the retailer rather than pretending to be a store. Users want the source.
- Surface condition and shipping assumptions next to the price, not in a footnote.
Compliance Note
You're building on other people's product pages, so hold yourself to some rules: fetch public pages only, respect robots.txt, keep per-retailer request rates polite (your daily sweep should be indistinguishable from one shopper), and check retailer terms; some explicitly prohibit automated access, and some offer affiliate or product feeds that are both allowed and better than scraping. When a feed exists, use it. Factual price data is low-risk territory, but low-risk is not no-risk; the longer discussion lives in is web scraping legal.
The Bottom Line
Build the matcher first, with a review queue, on a deliberately narrow catalog. Treat fetching as a commodity you buy per call instead of infrastructure you maintain. Normalize ruthlessly, store history append-only, and be honest in the UI about freshness and condition. That's the whole recipe, and the order matters.
Ready to build the fetching layer in an afternoon? link.sc turns any product page into clean JSON with one API call. Start with 500 free credits a month.