How to Scrape a Single-Page App (React, Vue, Angular)

Quick answer: A single-page app returns a nearly empty HTML shell because the content is built in the browser by JavaScript. You have two ways to scrape it: find the JSON API the app calls behind the scenes and hit that directly (fastest and cleanest), or render the page with a headless browser and read the DOM after it finishes loading. The hidden-API route is almost always the better one when it exists.

The first time you scrape a React site and print the HTML, you get a shock: a <div id="root"></div> and a script tag. No article, no product, no prices. Nothing you came for.

That is not a bug. That is how SPAs work, and once you understand it, scraping them becomes straightforward.

Why the HTML is empty

A traditional website sends fully-formed HTML from the server. A single-page app (built with React, Vue, Angular, Svelte, and friends) sends a bare shell plus a JavaScript bundle. The browser downloads the bundle, runs it, and the JavaScript fetches data and builds the page in the DOM.

Your HTTP client (requests, fetch, curl) downloads the shell and stops. It does not run JavaScript, so it never sees the content. That is the whole problem in one sentence.

import requests
html = requests.get("https://some-react-app.com/products").text
print(len(html))   # tiny
print("Product" in html)  # False

If that check comes back False, you are looking at an SPA. Now pick a path.

Path 1: Find the JSON API (do this first)

Here is the insight most people miss. The SPA has to get its data from somewhere. It calls an API, gets JSON back, and renders it. If you find that API, you can skip the entire browser and get clean, structured data directly, which is faster and far more reliable than parsing HTML.

How to find it:

1. Open the site in Chrome.
2. Open DevTools, go to the Network tab, filter to Fetch/XHR.
3. Reload the page or trigger the action (scroll, click, search).
4. Watch the requests. Look for ones returning JSON with the data you want.
5. Right-click the request, copy it as cURL, and replicate it in code.

import requests

# The API call the React app makes, found in the Network tab
resp = requests.get(
    "https://some-react-app.com/api/products",
    params={"page": 1, "limit": 50},
    headers={"Accept": "application/json"},
)
products = resp.json()["items"]
for p in products:
    print(p["name"], p["price"])

This is the dream outcome. Clean JSON, pagination via query params, no HTML parsing, no browser. When an SPA has a public JSON API (most do), always prefer it.

A few gotchas:

• Auth headers. The API may need a token or cookie the app sets. Copy those from the request.
• Referer/Origin checks. Some APIs check headers. Copy them from the real request.
• GraphQL. Some apps use a single /graphql endpoint with a POST body. Copy the exact query and variables.

Path 2: Render with a headless browser

When there is no clean API (server-side signed requests, obfuscated endpoints, or data spread across many calls), render the page. A headless browser runs the JavaScript exactly like a real browser, and then you read the finished DOM.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://some-react-app.com/products")
    # Wait for the actual content, not a fixed timer
    page.wait_for_selector(".product-card")
    names = page.locator(".product-card h2").all_inner_texts()
    print(names)
    browser.close()

If you are new to headless browsers, what is a headless browser explains the concept, and scrape JavaScript-rendered websites goes deeper on the tooling.

Wait for content, not for time

The single biggest mistake with SPA scraping is using a fixed sleep. time.sleep(5) is either too short (you miss the data on a slow load) or too long (you waste time on every run). Wait for a signal instead.

Strategy	When to use	Playwright call
Wait for selector	You know an element that appears with the data	page.wait_for_selector(".item")
Wait for network idle	Page makes several calls, then settles	page.goto(url, wait_until="networkidle")
Wait for function	Content depends on some JS condition	page.wait_for_function("...")
Wait for response	You know the API call to expect	page.wait_for_response("**/api/**")

Waiting for a selector that only exists once the data loads is the most reliable. It finishes the instant the content is there and no sooner.

The rendering API alternative

Running headless browsers at scale is real work: memory, crashes, proxies, and bot detection. If you do not want to own that, a rendering API does path 2 for you and hands back clean content.

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://some-react-app.com/products", "format": "markdown"}'

link.sc renders the page (JavaScript and all), then returns markdown or JSON, so an SPA and a static blog look identical to your code. You get the rendered result without maintaining a browser fleet. The fetch endpoint docs cover the options.

Which path should you take?

Factor	Hidden JSON API	Headless browser	Rendering API
Speed	Fastest	Slow	Medium
Reliability	High if API is stable	Medium	High
Data cleanliness	Structured JSON	You parse the DOM	Clean markdown/JSON
Setup effort	Low (once found)	High	Very low
Breaks when	API changes	Selectors change	Rarely

My rule: spend ten minutes in the Network tab looking for the JSON API first. If you find it, you win. If the app hides it well or signs its requests, render the page with a browser or an API. Do not jump straight to a headless browser out of habit, because you often do not need one.

An ethics note

Scrape public data, respect robots.txt, and do not automate your way past a login or paywall. If a site's terms forbid scraping, honor them. Rate-limit yourself so you are not degrading the service for real users. The hidden-API route especially can generate a lot of load fast, so add delays and cache what you can.

Bottom line

SPAs are not harder to scrape than any other site once you know the trick. The content is not in the HTML because a browser builds it from JSON. Find that JSON and you get the cleanest possible result. When you cannot, render the page and wait for the content to appear (never a fixed timer), or let an API render it for you.

---

Scrape React, Vue, and Angular sites without running a browser yourself. Try link.sc free.