Quick answer: Web scraping for ecommerce means collecting public product data (prices, stock, catalog details, reviews) from retailer and marketplace sites, then turning it into a time series you can act on. The reliable stack is structured data first (schema.org/JSON-LD), LLM extraction as a fallback, a fetch API to handle rendering and blocking, and a scheduler that diffs results over time. Below are the main use cases and how to build the pipeline without getting blocked.
I have watched teams spend months building a scraping stack when their real problem was that they never decided what data they needed. So before any code, let me map the terrain.
The Use Cases That Actually Pay Off
Ecommerce is one of the highest-return verticals for scraping because the data is public, factual, and directly tied to revenue decisions. Here are the ones worth building for.
| Use case | What you collect | Who uses it |
|---|---|---|
| Price monitoring | Competitor prices per SKU over time | Retailers, D2C brands, marketplaces |
| MAP enforcement | Reseller prices vs your minimum advertised price | Manufacturers, distributors |
| Catalog and assortment | What products competitors carry, and gaps | Category managers, buyers |
| Review mining | Ratings, review text, sentiment | Product teams, brand analysts |
| Availability tracking | In-stock vs out-of-stock signals | Ops, dropshippers, resellers |
| Competitor analysis | New launches, promotions, bundles | Strategy and marketing |
Notice that most of these are the same extraction problem (grab a few fields from a product page) applied to different questions. That is good news. You build the extraction once and point it at different queries.
Pricing and MAP monitoring
This is the flagship use case. You track competitor prices per SKU, and if you are a brand, you also watch whether resellers are advertising below your minimum advertised price. The trap is treating a product page as one price when it is really a matrix of variants. Size 10 in blue is not size 9 in black. Track price history per SKU, not per URL, or your history fills with phantom changes. I go deep on the extraction mechanics in the ecommerce price scraping guide.
Catalog and assortment intelligence
Instead of one product, you crawl a category or a competitor's full catalog to answer questions like: what do they stock that we do not, how wide is their assortment, and where are the gaps we could fill. This is slower and heavier than price checks, so run it weekly, not hourly.
Review and sentiment mining
Public reviews are a goldmine for product teams. Collecting rating distributions and review text across your own and competitor listings tells you what customers complain about before it shows up in your returns data. Keep this to public review text and aggregate signals, and stay away from anything that identifies individual reviewers.
Availability and competitor moves
Stock status and promotional changes are leading indicators. A competitor going out of stock on a hot SKU is a demand signal. A sudden price drop across a category is a promotion you may need to match.
The Pipeline, End to End
Every ecommerce scraping project has the same four stages. The complexity lives in stages two and three.
1. Discover URLs -> sitemaps, category pages, search results
2. Fetch pages -> render JS, rotate IPs, absorb anti-bot friction
3. Extract fields -> JSON-LD first, LLM extraction as fallback
4. Store + diff -> per-SKU time series, alert on change
Stage 1: Discovery
Start with the sitemap. Most stores publish sitemap.xml, and it is the polite, low-cost way to enumerate product URLs. Category pages and on-site search fill in the rest.
Stage 2: Fetch
This is where naive scrapers die. On plenty of modern storefronts, a plain HTTP GET returns HTML with no price in it, because the price arrives later from a client-side API call. You have two options: run your own headless browser fleet (a part-time job of Chrome updates, proxy rotation, and fingerprint tuning) or call a fetch API that does the rendering for you. This is the core of what link.sc does. You request a URL and get back the final rendered page.
curl https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_your_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://shop.example.com/products/trail-runner-gtx",
"format": "markdown",
"render_js": true
}'
The response comes back as { "content": "..." } with the rendered page as clean markdown.
Stage 3: Extraction
Before writing a single CSS selector, search the page source for application/ld+json. Most platforms embed schema.org Product markup so Google can show rich results, and it contains exactly what you want in a stable, machine-readable form. Retailers keep it accurate because their Google Shopping visibility depends on it.
import json, re
def price_from_jsonld(html: str):
for block in re.findall(
r'<script type="application/ld\+json">(.*?)</script>', html, re.S
):
try:
data = json.loads(block)
except json.JSONDecodeError:
continue
items = data if isinstance(data, list) else [data]
for item in items:
if item.get("@type") == "Product":
offer = item.get("offers", {})
if isinstance(offer, list):
offer = offer[0]
return {
"name": item.get("name"),
"price": offer.get("price"),
"currency": offer.get("priceCurrency"),
"availability": offer.get("availability"),
}
return None
When JSON-LD is missing, fall back to LLM-based extraction, which reads the rendered page and returns structured fields regardless of markup. The rule of thumb is JSON-LD first because it is cheaper and more stable, LLM extraction second because it is more flexible. I covered the general version of this in extract structured JSON from any webpage.
Stage 4: Store and diff
Scraped prices are only useful as a time series. Store each observation per SKU with a timestamp, diff against the last stored value, and alert on change.
def check(product):
current = fetch_price(product["url"])
last = db.latest_price(product["sku"])
if last is not None and current["price"] != last:
alert(product["sku"], last, current["price"])
db.insert(product["sku"], current)
Compliance and Ethics
Ecommerce pricing is close to the safest data you can scrape, because it is public and factual rather than personal. That does not mean anything goes.
- Respect robots.txt and rate limits. One request per product per day is invisible. Hammering a site every minute is how you get IP-banned and noticed.
- Read the terms of service. Many retailers prohibit automated access. A ToS violation is usually a contract matter, not a crime, but it can get your access cut off and, in aggressive cases, lawyered.
- Do not scrape behind a login you agreed to terms for, and never collect personal data on reviewers or buyers.
- Do not republish a competitor's full catalog as your own.
None of this is legal advice. If the data is core to your business, spend an hour with a lawyer before you spend a month on infrastructure. For a broader treatment, see whether web scraping is legal.
When to Use an API Instead of Building
Building your own scraper makes sense when you have a small, stable set of targets and engineering time to spare. An API earns its keep the moment you hit rendered prices, bot protection, or dozens of different site layouts, because you stop maintaining browser infrastructure and start spending your time on the analysis that actually differentiates you. link.sc offers 500 free credits a month, and paid plans start at $19 a month, so you can prototype the whole pipeline before committing. Pricing is at link.sc/pricing.
The takeaway: pick your use case first, extract with JSON-LD before anything clever, store per SKU, and let a fetch API absorb the rendering and blocking headaches so you can focus on the decisions the data drives.
Want product data without babysitting headless browsers? link.sc fetches any ecommerce page and returns clean markdown or JSON in one call. Get 500 free credits a month.