How to Extract Structured JSON From Any Webpage

Quick answer: There are three reliable ways to extract structured JSON from a webpage: CSS/XPath selectors (fast and cheap but breaks when the layout changes), reading embedded JSON-LD structured data (free and clean, but only present on some sites), and LLM extraction against a JSON schema (survives redesigns, costs tokens). For production pipelines, the winning move is a hybrid: check for JSON-LD first, fall back to selectors for high-volume known sites, and use an LLM for everything else.

I've built extraction pipelines using all three approaches, and each one has failed me in a different way. Here's what actually happens with each, with code you can run today.

Approach 1: CSS and XPath selectors

The classic. You inspect the page, find that the price lives in span.price-tag, and write a selector.

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example-store.com/product/123")
soup = BeautifulSoup(resp.text, "html.parser")

product = {
    "name": soup.select_one("h1.product-title").get_text(strip=True),
    "price": soup.select_one("span.price-tag").get_text(strip=True),
    "in_stock": soup.select_one(".availability") is not None,
}
print(product)

This is fast (milliseconds), free after the fetch, and deterministic. If you're scraping one site you know well, at high volume, selectors are still the right tool. Keep an XPath cheat sheet handy for the gnarly cases.

The problem is brittleness. The site ships a redesign, price-tag becomes pdp-price, and your pipeline silently returns None for every product until someone notices the dashboard looks weird. Selector-based scrapers don't fail loudly. They fail quietly, which is worse.

My rule: selectors are fine when you control the monitoring. Add an assertion that key fields are non-null, and alert when the null rate spikes.

Approach 2: JSON-LD and schema.org (the free lunch)

A huge fraction of commercial sites embed machine-readable structured data for Google. Product pages, recipes, job postings, articles, events: they often ship a <script type="application/ld+json"> block containing exactly the JSON you were about to painfully reconstruct.

import json
import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example-store.com/product/123")
soup = BeautifulSoup(resp.text, "html.parser")

for tag in soup.find_all("script", type="application/ld+json"):
    try:
        data = json.loads(tag.string)
    except (json.JSONDecodeError, TypeError):
        continue
    items = data if isinstance(data, list) else [data]
    for item in items:
        if item.get("@type") == "Product":
            print(item["name"], item.get("offers", {}).get("price"))

When it's there, this is the best possible source: it's the site telling you, in a standard vocabulary, what the page is about. No layout parsing, no guessing, and it survives redesigns because it's decoupled from the visual HTML.

The catch is coverage. Sites that care about SEO rich results have it. Internal tools, small business sites, forums, and most B2B pages don't. And even when present, it's sometimes stale or incomplete (a price in JSON-LD that doesn't match the rendered page is a thing I have personally debugged).

So: always check for it first, never rely on it alone.

Approach 3: LLM extraction with a JSON schema

This is the approach that changed the game. Fetch the page, convert it to clean markdown, and hand it to an LLM with a schema.

The markdown conversion matters more than people think. Raw HTML is 5-20x more tokens than the equivalent markdown, and most of those tokens are nav menus and div soup. I wrote more about this in token optimization for LLM web data.

Here's the full pipeline using link.sc for the fetch-to-markdown step:

import json
import requests
from openai import OpenAI

# 1. Fetch the page as clean markdown
page = requests.get(
    "https://link.sc/v1/fetch",
    headers={"Authorization": "Bearer lsc_your_key"},
    params={"url": "https://example-store.com/product/123", "format": "markdown"},
).json()

# 2. Extract against a schema
client = OpenAI()
schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "price": {"type": "number"},
        "currency": {"type": "string"},
        "in_stock": {"type": "boolean"},
    },
    "required": ["name", "price"],
}

result = client.chat.completions.create(
    model="gpt-4o-mini",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": f"Extract product data matching this JSON schema: {json.dumps(schema)}. Return only JSON."},
        {"role": "user", "content": page["markdown"]},
    ],
)
print(json.loads(result.choices[0].message.content))

The LLM doesn't care what the CSS classes are called. Redesigns, A/B tests, different templates across a site: it handles all of them with the same prompt. It also handles the long tail, meaning you can point one extractor at a thousand different sites instead of writing a thousand scrapers.

The costs are tokens and latency. A small model on a markdown-converted page is cheap per page, but it's not free, and at millions of pages per day it adds up. LLMs can also hallucinate fields that aren't on the page, so validate the output against your schema and spot-check.

The honest comparison

	Selectors	JSON-LD	LLM extraction
Cost per page	Near zero	Near zero	Tokens (small but real)
Speed	Milliseconds	Milliseconds	Seconds
Survives redesigns	No	Yes	Yes
Works on any site	Yes (with per-site work)	Only where present	Yes
Per-site engineering	High	None	None
Failure mode	Silent nulls	Missing or stale data	Hallucinated fields
Best at	One site, high volume	SEO-optimized pages	Long tail, many sites

The hybrid strategy I actually recommend

If I were building this today, the decision tree would be:

1. Parse JSON-LD first. It's free, it's authoritative, and checking for it costs one regex over the HTML.
2. Use selectors for your top sites. If 80% of your volume is five domains, hand-write those five extractors and monitor null rates.
3. LLM for everything else. The long tail is where LLM extraction earns its token cost, because the alternative is engineering time you don't have.
4. Validate every path against the same schema. One schema, three extractors, one validation layer. When the LLM output fails validation, retry once, then flag for review.

One more practical note: none of this works if you can't get the page in the first place. JS-rendered pages need a real browser, and plenty of sites block datacenter IPs outright (I covered the tradeoffs in curl vs headless vs stealth browsers). The link.sc fetch endpoint handles rendering and access for you and returns markdown that's ready to drop into step 3, which is exactly why we built it that way.

Start with JSON-LD, keep selectors for the sites that pay the bills, and let an LLM sweep up the rest.

---

Want the fetch-and-clean half of this pipeline handled for you? Sign up for link.sc and get 500 free credits a month.