Quick answer: Start with a plain HTTP request and read the response like a diagnostic. Full HTML with your content means you're done. A 403/challenge page on the first request usually means TLS fingerprinting, so try a browser-impersonating client (curl_cffi class). A 200 with an empty HTML shell means client-side rendering, so use a headless browser. Challenge pages that survive a normal headless browser, or DataDome/PerimeterX blocks, mean stealth browser plus residential IPs. Never start at the top; each rung costs 10-100x the one below it.
Every site sits somewhere on an escalation ladder, and the whole skill of efficient scraping is figuring out the lowest rung that works. The site tells you, if you know how to read the symptoms.
The escalation ladder, briefly
I went deep on the tooling tradeoffs in curl vs headless vs stealth browser, so here's just the shape:
| Rung | Tooling class | Relative cost per page |
|---|---|---|
| 1. Plain HTTP | curl, requests, httpx | 1x |
| 2. Fingerprint HTTP | curl_cffi, tls-client class | ~1x |
| 3. Headless browser | Playwright, Puppeteer | ~50-100x |
| 4. Stealth browser + residential | patched browsers, per-GB residential IPs | ~100x+ and rising |
The cost gradient is why diagnosis matters. Defaulting everything to rung 3 or 4 "to be safe" is the most common way teams multiply their bill for nothing, a point I labored in how much scraping costs at scale.
Rung 1 diagnostics: what a plain GET tells you
Fire the cheapest possible probe:
curl -sI "https://target.example/page"
curl -s "https://target.example/page" | head -c 2000
Now read the tea leaves:
- 200 with your content visible in the raw HTML: done. Ship rung 1 and be happy. A surprising fraction of the web still lives here.
- 200 but the body is a near-empty shell: a
<div id="root"></div>, a pile of<script>tags, no article text. That's client-side rendering, not blocking. Skip to rung 3 (though check the JSON shortcut below first). - 403 or 503 immediately, especially with
cf-rayorserver: cloudflareheaders: you're being fingerprinted before you've done anything wrong. Go to rung 2. - 429: you're not blocked, you're rate-limited. Slow down, add backoff, respect
Retry-After. This is a politeness problem, not an evasion problem; my 429 post covers it. Escalating tooling to beat a 429 is treating a speed limit as a locked door. - 200 with "Just a moment..." or "Checking your browser" in the body: a Cloudflare JS challenge served with a success code. Log this pattern; it's the classic soft failure. Rung 2 sometimes clears it; otherwise rung 3-4.
Rung 2: when your TLS handshake is the problem
Here's the non-obvious one. Modern anti-bot systems fingerprint the TLS handshake (JA3/JA4) and HTTP/2 settings before a single byte of your request body matters. Python's requests and vanilla curl announce themselves as scripts at the protocol level. Headers can be perfect and you're still burned.
The tell: instant 403s that disappear when you open the URL in a normal browser, on the very first request from a fresh IP. No rate limiting, no gradual degradation, just an immediate wall.
The fix costs almost nothing: clients like curl_cffi impersonate a real Chrome handshake while remaining plain HTTP. Same speed as rung 1, dramatically better pass rates on Cloudflare's lower-tier settings. If rung 2 gives you full HTML, you've dodged the 50-100x browser tax.
One more rung-2 trick: before surrendering to a browser, check whether that empty SPA shell fetches its data from a JSON API. Open devtools, watch the network tab, and hit the API endpoint directly. Cleaner than HTML and cheaper than rendering.
Rung 3: when the content genuinely isn't in the HTML
If the shell is empty and there's no usable JSON endpoint, you need JavaScript execution. Signals you're correctly on rung 3:
- Rung 2 returns 200 reliably, but the content you want never appears in the raw HTML
- Content appears after scrolling or clicking in a real browser
- The page is a React/Vue/Next.js app with client-side data fetching
Plain headless Chrome via Playwright handles this fine for sites that render client-side but don't hunt bots. If you're getting content, stop here. Don't add stealth patches you don't need; they're a maintenance liability.
Rung 4: the signals that you're up against real anti-bot
You know you need stealth plus residential when:
- Vanilla headless gets challenged but your desktop browser doesn't. Cloudflare's higher settings detect headless automation flags (
navigator.webdriver, missing plugins, canvas quirks). - You see DataDome or PerimeterX markers:
datadomecookies, 403 pages referencing "captcha-delivery.com", PerimeterX's_pxcookies. These vendors fingerprint devices and IP reputation aggressively. - Behavior varies by IP class. The same request works from your home connection and fails from your VPS. Datacenter IP ranges are pre-scored as bots; that's what per-GB residential proxies fix. (Cloudflare specifics here.)
- Success decays over time on a setup that used to work. Welcome to the arms race; this rung is the one that never stays solved.
The decision flowchart, in text
- Plain GET. Content present? Stop: rung 1.
- Instant 403/challenge? Try curl_cffi-class client. Content present? Stop: rung 2.
- 200 but empty shell? Look for a JSON API. Found one? Stop: rung 2. Otherwise headless browser. Content present? Stop: rung 3.
- Headless gets challenged, or DataDome/PerimeterX markers? Rung 4: stealth + residential.
- Getting 429s at any rung? That's rate limiting. Slow down; don't escalate.
- Still blocked at rung 4? Decide if this domain is worth the fight, and check the legal and ethical angle while you're at it.
Cache the answer per domain, because it's stable for weeks or months at a time. Then re-probe periodically, since sites add (and occasionally remove) protection.
Or let the ladder run itself
This diagnosis loop is mechanical, which means it's automatable. That's literally what link.sc does: the API keeps a per-domain memory of which rung works, starts every new domain at the cheapest method, escalates automatically on challenge detection, and re-probes as sites change. You make one call:
curl "https://link.sc/v1/fetch?url=https://hard-site.example/page" \
-H "Authorization: Bearer lsc_your_key"
and get markdown back whether the site needed rung 1 or rung 4, priced accordingly. The quickstart has the details, and the free tier's 500 monthly credits are enough to probe your whole domain list and see where each one lands.
Either way, the principle holds: diagnose before you escalate. The cheapest request that works is the right one.
Tired of hand-tuning the ladder per domain? Get a free link.sc key and let the API pick the right method for every site automatically.