You wrote a scraper, pointed it at a page, and got back almost nothing: a bare <div id="root"></div>, a <noscript> tag, and none of the content you can plainly see in your browser. This is one of the most common walls people hit, and it has a specific cause. The page is a Single Page Application (SPA) that renders itself with JavaScript after the initial HTML loads. Here is why that happens and how to get the actual content.
Why Plain HTTP Gets You an Empty Shell
An HTTP client like requests, curl, or fetch does exactly one thing: it downloads the bytes the server sends for that URL. It does not run JavaScript. For a traditional server-rendered page, that is enough, because the server already baked the content into the HTML.
Modern apps built with React, Vue, Angular, Svelte, and friends usually work differently. The server sends a nearly empty HTML document plus a bundle of JavaScript. The browser downloads that bundle, executes it, calls the site's API for the real data, and only then builds the DOM you see. This is client-side rendering. If you never run the JavaScript, you never trigger any of that, so you get the shell and nothing else.
You can confirm this in about ten seconds. Compare what the raw response contains against what the rendered page shows:
curl -s https://example-spa.com/products | grep -c "product-card"
# 0 -> content is rendered client-side, not in the HTML
If the count is zero but you can see dozens of product cards in your browser, JavaScript rendering is your problem.
One thing worth checking before you reach for a browser: open the Network tab in DevTools and watch the XHR/fetch requests. Many SPAs load their data from a clean JSON API (/api/products?page=1). If that endpoint is reachable and not locked down, calling it directly is faster, lighter, and more stable than rendering anything. Always look for the API first. When there isn't one, or it is heavily protected, you need to render.
Rendering With a Headless Browser
The DIY answer is a headless browser: a real browser engine driven by code, no visible window. The main options are Playwright and Puppeteer (both Chromium-based, with Playwright also supporting Firefox and WebKit), and the older Selenium. They launch the browser, load the page, run all the JavaScript, and hand you the finished DOM.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example-spa.com/products")
page.wait_for_selector(".product-card") # wait for content, not just load
html = page.content()
browser.close()
This works, and for a handful of pages it is the right tool. The cost shows up at scale. A headless browser eats hundreds of megabytes of RAM per instance, so concurrency gets expensive fast. You have to install and update browser binaries, keep them patched, and run them in a container with the right system libraries. Sites change their markup, so your selectors break. And many sites detect vanilla headless browsers and block them, which pushes you into stealth plugins, fingerprint patching, and proxy management. None of that is your actual product. It is infrastructure you now own forever.
Getting the Wait Strategy Right
The subtle part of browser scraping is knowing when the page is actually done. Grabbing the HTML too early gives you the same empty shell you were trying to escape. There is no single correct wait, which is why this is where most scrapers quietly break.
load/domcontentloaded: fires when the document and assets finish, but before client-side data arrives. Almost always too early for an SPA.networkidle: waits until network traffic settles. Better, but it hangs on pages with polling, analytics beacons, or open websockets.- Wait for a specific selector: the most reliable approach. Wait for the element that only exists once real data has rendered (
.product-card,[data-loaded], a results count). You are asserting on the thing you actually care about. - Wait for a network response: pause until the specific API call the page depends on returns, then read the DOM.
Tuning these per site, and re-tuning them when the site changes, is the ongoing tax of doing this yourself.
Letting link.sc Handle the Rendering
link.sc exists so you do not have to build and babysit any of that. It is one API to fetch and search anything online, built for LLMs and agents. You pass a URL and it returns clean markdown, structured JSON, raw HTML, or a screenshot. Rendering, proxies, parsing, retries, and wait strategies are handled for you.
Under the hood it escalates cheapest-first: it starts with plain curl_cffi (fast HTTP with real browser TLS fingerprints), and if a page needs JavaScript it moves up to a stealth browser that renders the page with smart waits before extracting content. It also learns which method each domain needs, so repeat requests skip straight to the approach that works.
curl -X POST https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example-spa.com/products",
"format": "markdown"
}'
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_YOUR_KEY"},
json={"url": "https://example-spa.com/products", "format": "markdown"},
)
print(resp.json()["content"])
const res = await fetch("https://api.link.sc/v1/fetch", {
method: "POST",
headers: {
"x-api-key": "lsc_YOUR_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({ url: "https://example-spa.com/products", format: "markdown" }),
});
const { content } = await res.json();
console.log(content);
You get the fully rendered content without launching a browser, picking a wait strategy, or maintaining any of it. Ask for json if you want structured fields, or screenshot if you want the visual. If you would rather run it yourself, link.sc is open core and self-hostable.
Rendering JavaScript is solvable either way. The question is whether you want to own the browser fleet or the feature you are building. If it is the feature, get a free API key with 500 free credits a month, or read the docs to see the full set of formats and options.