Quick answer: point a SERP API at the Google Shopping surface, pull the product listings and their per-merchant prices, then run each result through structured extraction so you end up with clean rows you can compare and store. The hard parts (rendering, anti-bot, proxies) get handled by the API, and your code stays a short normalization step.
Google Shopping is not the same target as an Amazon product page or a generic ecommerce listing. It is a comparison surface: one product, many merchants, many prices, all rendered into a grid that changes shape constantly. That makes it uniquely valuable and uniquely annoying to scrape. This post walks through a pipeline that treats Shopping as what it is, a price-comparison feed, and turns it into structured data.
Why Google Shopping Is Its Own Problem
If you have scraped a single retailer before, you might assume Shopping is just more of the same. It is not, for a few reasons.
It aggregates merchants. A search for "sony wh-1000xm5" returns one product card backed by a dozen sellers, each with a different price, shipping cost, and condition. The value is in the spread across merchants, not in any single number. You are collecting a distribution, not a data point.
It is heavily JavaScript-rendered. The Shopping grid hydrates client-side. A plain curl or a requests.get on the search URL hands you a skeleton with no prices in it. If you have fought this before on other sites, the JS-rendered pages guide covers why the HTML you download is not the HTML a user sees.
It is aggressively bot-protected. Google fingerprints browsers, rate-limits by IP, and throws JavaScript challenges. Datacenter IPs get flagged fast. Running your own headless browser fleet through clean residential proxies is a real project, not a weekend script.
Its layout drifts. Google reshapes the Shopping SERP often. Parsers pinned to specific CSS class names break on a schedule you do not control.
The takeaway: do not build a fragile HTML parser against Google directly. Let an API absorb the rendering and anti-bot fight, and spend your effort on the data shape you actually want.
Step 1: Pull the Shopping Results
The link.sc Search API queries the search surface and returns results with content extraction already done. Point it at a product query and ask for structured output:
import linksc
client = linksc.Client(api_key="lsc_YOUR_KEY")
results = client.search(
q="sony wh-1000xm5 headphones",
format="json",
num_results=10,
country="us",
)
for r in results.results:
print(r.title, r.url)
Setting country matters more here than on most scrapes. Shopping prices, currency, and even which merchants appear are geo-specific, so a query without a target market gives you a blurry average of nowhere. Pin it to the market you care about.
You can also work at the raw HTTP level if you would rather not add a dependency. The response comes back as a JSON object with a content field and, for search-surface queries, a serpData block holding the structured results:
curl -X POST https://api.link.sc/v1/search \
-H "x-api-key: lsc_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"q": "sony wh-1000xm5 headphones",
"format": "json",
"num_results": 10,
"country": "us"
}'
Step 2: Normalize Into Price Rows
Search results give you the listings and the pages behind them. To build a real price comparison you want a flat table: product, merchant, price, currency, condition, link. The cleanest way to get there is structured extraction against each listing page, where you hand the Fetch API a JSON schema and it returns exactly those fields.
schema = {
"type": "object",
"properties": {
"product_title": {"type": "string"},
"offers": {
"type": "array",
"items": {
"type": "object",
"properties": {
"merchant": {"type": "string"},
"price": {"type": "number"},
"currency": {"type": "string"},
"shipping": {"type": "number"},
"condition": {"type": "string"},
"url": {"type": "string"},
},
},
},
},
}
rows = []
for r in results.results:
page = client.fetch(url=r.url, format="json", schema=schema)
data = page.data
for offer in data.get("offers", []):
rows.append({
"product": data.get("product_title"),
"merchant": offer.get("merchant"),
"price": offer.get("price"),
"shipping": offer.get("shipping") or 0,
"condition": offer.get("condition", "new"),
"url": offer.get("url"),
})
Notice what this buys you. You never touch a CSS selector, so a layout change on Google's side does not break your code. You get numbers as numbers, not strings with a currency glyph glued to the front. And because the schema is explicit, the API knows to render the page and dig the values out of whatever markup they happen to live in today. This is the same structured-extraction trick that makes competitor price monitoring reliable, applied to a comparison surface instead of a single vendor.
Step 3: Build the Comparison
Now that every offer is a uniform row, comparison is arithmetic. Fold shipping into a landed price so you are comparing what a buyer actually pays, then sort:
for row in rows:
row["total"] = round(row["price"] + row["shipping"], 2)
rows.sort(key=lambda x: x["total"])
cheapest = rows[0]
spread = rows[-1]["total"] - rows[0]["total"]
print(f"Best: {cheapest['merchant']} at ${cheapest['total']}")
print(f"Spread across {len(rows)} merchants: ${round(spread, 2)}")
The merchant spread is usually the interesting signal. A wide gap means there is margin to undercut or a mispriced seller to watch. A tight gap means the market is efficient and you compete on shipping or availability instead of headline price. Filtering by condition also matters: a refurbished unit dragging down the "cheapest" line is not really the same product, so split new and used before you draw conclusions.
Step 4: Track It Over Time
A single snapshot answers "what does this cost right now." The strategic questions are about movement: who dropped price this week, when did a new merchant enter, is the whole category trending down. So store each run with a timestamp rather than overwriting.
import json, datetime, pathlib
snapshot = {
"captured_at": datetime.datetime.utcnow().isoformat(),
"q": "sony wh-1000xm5 headphones",
"rows": rows,
}
out = pathlib.Path(f"snapshots/{snapshot['captured_at']}.json")
out.parent.mkdir(exist_ok=True)
out.write_text(json.dumps(snapshot, indent=2))
Point a daily cron job at this and you have a price history you can diff. Shopping prices move more than pricing-page prices do, but they still do not change minute to minute, so a once- or twice-daily cadence captures the signal without hammering anything. If you widen this to a catalog of products, the SERP scraping guide has notes on batching and caching that keep large keyword sets affordable.
A Few Practical Notes
Match products carefully. Google groups offers under a product it identifies, but for niche items the grouping can be loose. Key your rows on a stable identifier (a GTIN or model number when present) rather than the display title, which varies by merchant.
Respect the cadence. You are reading a public results surface, but you are still a guest. Daily or hourly is plenty for pricing. There is no prize for polling every minute, and it only raises your odds of getting throttled.
Keep the raw output too. Alongside the structured rows, save the markdown or JSON the API returned. When a number looks wrong later, you want the page as it was, not a guess.
Wrapping Up
Scraping Google Shopping is really three moves: pull the listings from the search surface, normalize each into flat price rows with a schema, and compare or track the spread across merchants. The reason this stays a short script instead of a headless-browser saga is that the API takes the rendering, proxy rotation, and anti-bot work off your plate. You describe the shape you want and get clean data back.
You get 500 free credits every month, enough to track a basket of products daily while you tune the pipeline. Grab a key and try it against a product you actually care about.
Ready to turn Google Shopping into a clean price feed? Get your free link.sc API key and make your first call in under two minutes.