Working With JSON Data From the Web: Download, Parse, Convert

Quick answer: The cleanest way to get JSON from the web is to hit a JSON endpoint directly and save the response body to a .json file. When there is no endpoint, extract the embedded JSON-LD from the page, and when even that is missing, convert the scraped HTML into structured JSON with a schema. Below are runnable Python and JavaScript examples for each case, plus how to handle files too large to fit in memory.

JSON is the format most people actually want when they say they are "scraping" a site. It is already structured, it maps directly to dicts and objects, and it skips the messy step of parsing prose. So before you reach for an HTML parser, check whether the data is available as JSON somewhere. It usually is.

Four ways to get JSON, from best to last resort

Source	When it applies	Effort	Reliability
Direct JSON endpoint	Site has a public or internal API	Low	High
Embedded JSON-LD	Page has <script type="application/ld+json">	Low	Medium
Inline state JSON	Page ships a __NEXT_DATA__ or Redux blob	Medium	Medium
HTML converted to JSON	No JSON exists anywhere	High	Depends on schema

Work down that list in order. Every row down costs more code and breaks more often.

1. Hit the JSON endpoint directly (best)

Open your browser dev tools, go to the Network tab, filter by Fetch/XHR, and reload the page. Most modern sites load their data from a JSON endpoint that the page then renders. If you find one, you can skip HTML entirely.

Downloading a JSON file in Python is three lines:

import requests

resp = requests.get("https://api.example.com/v1/products?page=1")
resp.raise_for_status()
data = resp.json()  # already a dict/list

# Save it to disk
import json
with open("products.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)

In Node:

import { writeFile } from "node:fs/promises";

const resp = await fetch("https://api.example.com/v1/products?page=1");
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
const data = await resp.json();

await writeFile("products.json", JSON.stringify(data, null, 2));

If the endpoint is behind rendering, geoblocking, or bot protection, that is exactly where a fetch API earns its keep. You can pull the underlying content and ask for JSON back without running a browser yourself:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/product/42", "format": "json"}'

2. Parse embedded JSON-LD

Many pages carry structured data for search engines inside a <script type="application/ld+json"> tag. This is often the richest, cleanest data on the page: product prices, article metadata, event times, recipe ingredients. It is designed to be machine readable, so use it.

import requests
from bs4 import BeautifulSoup
import json

html = requests.get("https://example.com/product/42").text
soup = BeautifulSoup(html, "html.parser")

blocks = []
for tag in soup.find_all("script", type="application/ld+json"):
    try:
        blocks.append(json.loads(tag.string))
    except (json.JSONDecodeError, TypeError):
        continue  # some sites ship malformed JSON-LD

print(json.dumps(blocks, indent=2))

A gotcha: JSON-LD is frequently invalid. Trailing commas, unescaped quotes, and multiple objects concatenated together are all common in the wild. Wrap every parse in a try/except and never assume a page has exactly one block.

3. Convert scraped HTML into structured JSON

When there is no endpoint and no JSON-LD, you have to build the JSON yourself from the rendered page. The naive approach is a pile of brittle CSS selectors. The durable approach is to define the schema you want and extract against it, so a layout change breaks one field instead of the whole scraper.

Here is the pattern with a fetch API that returns clean content plus a schema-guided extraction:

import requests

payload = {
    "url": "https://example.com/product/42",
    "format": "json",
    "schema": {
        "title": "string",
        "price": "number",
        "in_stock": "boolean",
        "reviews": "number",
    },
}

resp = requests.post(
    "https://link.sc/v1/fetch",
    headers={"Authorization": "Bearer lsc_..."},
    json=payload,
)
product = resp.json()
print(product["price"], product["in_stock"])

If you would rather roll it yourself, the general flow is: strip boilerplate, isolate the main content region, then map elements to fields. We go deeper on the cleanup step in remove boilerplate from scraped pages, and the full field-mapping approach lives in extract structured JSON from any webpage.

4. Saving and streaming large JSON

Small responses fit in memory and json.load is fine. Once a file crosses a few hundred megabytes, loading the whole thing will blow up your process. Two techniques help.

For a JSON array you control the writing of, stream records to a JSON Lines file (one JSON object per line). It is trivial to append to and to read back a line at a time:

import json

with open("records.jsonl", "w", encoding="utf-8") as f:
    for record in fetch_records():  # a generator
        f.write(json.dumps(record, ensure_ascii=False) + "\n")

# Reading back without loading everything
with open("records.jsonl", encoding="utf-8") as f:
    for line in f:
        obj = json.loads(line)
        process(obj)

For a large single JSON document you did not write, use a streaming parser like ijson so you pull one item at a time instead of materializing the whole tree:

import ijson

with open("huge.json", "rb") as f:
    for item in ijson.items(f, "results.item"):
        process(item)  # constant memory, one record at a time

In Node, the equivalent is a streaming parser such as stream-json, or writing NDJSON with a Writable stream. The principle is identical: never hold more than one record in memory when the file is bigger than your RAM budget.

Common pitfalls

Pitfall	Symptom	Fix
Assuming resp.json() always works	Crash on HTML error pages	Check Content-Type and status first
Encoding mismatch	Garbled accents, mojibake	Write with encoding="utf-8", ensure_ascii=False
Loading giant files whole	Out-of-memory kills	Stream with ijson / JSON Lines
Trusting JSON-LD validity	Random parse failures	try/except each block
Numbers as strings	"19.99" not 19.99	Cast explicitly, do not rely on the source

Which approach should you pick

If the site exposes JSON, use it directly. That is the whole game. Every layer below (JSON-LD, inline state, HTML conversion) is a workaround for the absence of a clean endpoint, and each one is more fragile than the last. The value of a fetch API here is that it collapses the messy layers: it handles rendering and access for you and can hand back structured JSON against your schema, so your code stays on the "get JSON, use JSON" happy path even when the site was never built to cooperate.

Whatever route you take, respect the source. Read the site's terms and robots.txt, keep request rates polite, and do not try to reach data that sits behind authentication you were not granted. Public data collected considerately is the goal.

---

Need clean JSON from any URL without babysitting browsers and proxies? Try link.sc free.