Quick answer: a 200 response with blank or nearly empty HTML almost always has one of four causes: the page renders its content with JavaScript, an anti-bot system served you a decoy instead of the real page, a consent or region wall intercepted the request, or the content loads lazily after scroll. Each one leaves a different fingerprint in the response body, so you can diagnose the right one in about two minutes by reading the HTML you actually got back.
The status code will not help you here. As we covered in our HTTP 200 deep dive, 200 only means the server produced a response it considers successful. It says nothing about whether that response contains your data. So stop looking at the status line and start looking at the body.
Step 0: Actually Look at What Came Back
Before any theorizing, save the response and open it. Most people skip this and burn an hour guessing.
import requests
resp = requests.get(url, timeout=15)
print(resp.status_code, len(resp.text))
with open("dump.html", "w") as f:
f.write(resp.text)
Two numbers matter: the byte length, and how it compares to the page in your browser. Open DevTools, load the page, and check the size of the first document request. If your browser gets 180 KB and your scraper gets 4 KB, something replaced or withheld the content. The question is what, and the dump file answers it.
Now walk the tree.
Branch 1: You Got an Empty Application Shell
Fingerprint: the HTML is small but well formed. There is a <head> full of script tags, then a body that is essentially <div id="root"></div> or <div id="app"></div>, maybe a <noscript> warning. No block language, no CAPTCHA, just... nothing where the content should be.
This is client-side rendering. The server sent you the same shell it sends everyone, and in a browser, JavaScript then fetches the data and builds the page. Your HTTP client never runs that JavaScript, so the shell is all you ever see. React, Vue, Angular, and Svelte apps all behave this way by default.
Quick confirmation:
curl -s https://example.com/products | grep -c "product-card"
# 0 means the content is not in the raw HTML
Two fixes, in order of preference. First, check the Network tab for the JSON API the page calls; hitting that directly is faster and more stable than rendering. Second, render the page with a real browser engine. We wrote up both paths, including wait strategies that avoid grabbing the DOM too early, in how to scrape JavaScript-rendered websites.
Branch 2: You Got a Soft Block
Fingerprint: the HTML is not empty, it is wrong. Look for phrases like "Checking your browser", "Verify you are human", "Access denied", "Enable JavaScript and cookies to continue", or a Cloudflare Ray ID at the bottom. Sometimes it is subtler: a legitimate-looking page with every data field blank, or a 200 whose body is a styled error page.
This is an anti-bot system deciding you are automation and serving a challenge or decoy instead of the content. The infuriating part is that it often happens on the second hundred requests, not the first ten, because reputation systems score your IP and TLS fingerprint over time.
Diagnostic questions that separate this branch from the others:
- Does the same URL work from your browser on the same network? If yes, your client fingerprint is the problem.
- Does the first request work and later ones fail? Rate-based blocking.
- Does it fail from your server but work from your laptop? Datacenter IP reputation. In my experience this is the single most common cause for scrapers running on cloud providers.
The fix depends on which detection layer caught you: TLS fingerprinting wants an impersonation client like curl_cffi, JavaScript challenges want a stealth browser, IP reputation wants residential proxies. We walk the full escalation ladder in scraping Cloudflare-protected sites and compare the tooling tiers in curl vs headless vs stealth browsers.
Branch 3: You Hit a Consent or Region Wall
Fingerprint: the HTML is substantial but it is the wrong page. Grep your dump for words like "consent", "cookies", "GDPR", "privacy", or "not available in your region". European IPs commonly receive a cookie consent interstitial that replaces the article entirely. Some US news sites serve EU visitors a stripped "essential news only" page. Paywalled sites serve a teaser shell.
This branch gets misdiagnosed as a block constantly, but the mechanics differ: the site is not refusing you, it is waiting for a decision your scraper never makes.
Fixes are usually mechanical. Many consent walls disappear if you send the cookie a real user would have after clicking accept; load the page once in a browser, click through, and copy the consent cookie into your client. Region walls need an exit IP in an acceptable country. And check whether the site offers the same content at a different path, because AMP pages and print views often skip the interstitial entirely.
Branch 4: The Content Loads Lazily
Fingerprint: the page looks mostly complete. Header, footer, sidebar, maybe the first few items, then placeholders, skeleton divs, or data-src attributes where images and list items should be. This one even bites people using headless browsers, because the page has "loaded" while the part they want has not.
Infinite scroll feeds, image galleries, and comment sections typically fetch content only when it approaches the viewport. A browser that never scrolls never triggers the fetch.
If you are rendering with Playwright or Puppeteer, scroll programmatically and wait for the element you actually care about, not for the load event:
page.goto(url)
for _ in range(5):
page.mouse.wheel(0, 2000)
page.wait_for_timeout(500)
page.wait_for_selector(".review-item")
html = page.content()
Better still, lazy content nearly always arrives via an XHR you can see in the Network tab. Calling that endpoint with the right page parameter beats simulating scroll every time.
The Whole Tree on One Page
| What the body contains | Diagnosis | Fix |
|---|---|---|
| Tiny HTML, empty root div, script tags | JavaScript rendering | Find the JSON API or render with a browser |
| Challenge text, CAPTCHA, Ray ID, decoy page | Soft block | Better fingerprints, residential IPs, stealth browser |
| Cookie banner, GDPR text, region notice | Consent or region wall | Send consent cookies, change exit geography |
| Real page with skeletons and placeholders | Lazy loading | Scroll and wait for selectors, or hit the XHR |
Notice that all four failures return 200. If your pipeline only alerts on status codes, every one of them silently fills your dataset with garbage. Add a content check: minimum length, a required selector, or a keyword that must appear. Treat its absence as a hard failure.
Or Skip the Tree Entirely
Every branch above is diagnosable and fixable by hand, and if you scrape a handful of stable sites, doing it yourself is reasonable. At any real breadth it becomes a permanent maintenance rotation, because sites move between branches without notice.
This is the exact problem link.sc was built to absorb. One API call returns the page as clean markdown or HTML; behind it, the service escalates from fast HTTP with real browser TLS fingerprints up to stealth rendering with smart waits, remembers which method each domain needs, and validates that actual content came back before returning it to you. The decision tree still runs, it just runs on our side.
Tired of debugging blank responses? Get a free link.sc API key and let one endpoint handle rendering, blocks, and waits for you.