How to Scrape a Site Protected by Cloudflare

scrape cloudflare protected sites

You send a clean, correct request to a public page and get back a 403 Forbidden or a page that says "Just a moment..." with a spinning loader. The content never arrives. Cloudflare sits in front of a large slice of the web, and its job is to tell automated traffic apart from human traffic. This post explains how that detection works, how to access public data through it reliably, and how to do so responsibly.

A note up front: this is about collecting public data dependably, not about defeating security controls. Do not use these techniques to bypass authentication, access private data, or ignore a site's stated wishes. Check the terms of service, respect robots.txt, and stop if a site clearly does not want to be scraped.

How Cloudflare Bot Detection Works

Cloudflare layers several signals, and a naive HTTP client fails most of them at once.

TLS / JA3 fingerprinting. Before any HTTP data is exchanged, your client negotiates a TLS handshake. The exact order of cipher suites, extensions, and curves forms a fingerprint (JA3/JA4). A real Chrome build produces a well-known fingerprint; Python's requests, Go's net/http, and curl each produce their own distinct one. Cloudflare can flag "this claims to be Chrome in its User-Agent, but its TLS handshake is Python" before it reads a single header.
HTTP/2 fingerprinting and header order. Real browsers send headers in a consistent order with a specific HTTP/2 settings frame. Libraries often differ, and that difference is measurable.
The JS challenge (the "Just a moment..." interstitial). For suspicious traffic, Cloudflare serves a page that runs JavaScript to probe the environment, checks for browser APIs, and sets a clearance cookie. No JavaScript engine means you sit on the interstitial forever.
Behavioral and reputation signals. The IP address matters. Datacenter IP ranges (AWS, GCP, cheap VPS providers) carry low trust; residential IPs carry more. Request rate, timing, and history all feed the score.

A plain request fails the TLS check immediately, has no JavaScript to solve the challenge, and often comes from a datacenter IP. Three strikes, and you get the 403.

Why Naive Requests Get a 403

Here is the request almost everyone starts with:

import requests
r = requests.get("https://protected-site.com", headers={"User-Agent": "Mozilla/5.0 ..."})
print(r.status_code)   # 403

Spoofing the User-Agent header does nothing, because the User-Agent is not what gave you away. The TLS fingerprint did, and that is set by your HTTP library, not by a header you control. This is why "just add headers" advice rarely works against Cloudflare, and why people burn hours before realizing the problem is a layer below HTTP.

The Escalation Ladder

The reliable approach is to climb only as high as each site forces you to. Higher rungs are slower and costlier, so you do not want to start at the top.

Rung 1 — Real browser TLS impersonation. Use a client that mimics a real browser's TLS and HTTP/2 fingerprint. curl_cffi (Python) and curl-impersonate do exactly this: they are fast HTTP clients that present a genuine Chrome or Safari fingerprint. This alone clears many Cloudflare sites, at nearly the speed of a normal HTTP request and none of the overhead of a browser.

from curl_cffi import requests as cffi
r = cffi.get("https://protected-site.com", impersonate="chrome")
print(r.status_code)   # 200 for many sites

Rung 2 — Stealth browser with JavaScript. When a site serves the JS challenge, you need something that actually runs it. A hardened headless browser (patched to hide automation tells) loads the page, solves the challenge, obtains the clearance cookie, and renders the content. This costs real memory and time per page, which is why it is a second resort, not the default.

Rung 3 — Residential or Tor egress. For the hardest sites, IP reputation is the remaining blocker. Routing through residential proxies or Tor changes where the request appears to originate. This is the slowest and most expensive rung, so it should be reserved for domains that genuinely require it.

Building all three yourself is a real project: TLS impersonation libraries, a fleet of patched browsers, proxy pools, cookie handling, retry logic, and per-domain tuning that drifts every time Cloudflare updates.

Being Polite

Reliable access and responsible access are the same discipline. If you hammer a site, you deserve to be blocked, and you may cause real harm.

Read robots.txt and honor its disallow rules and any Crawl-delay.
Rate limit yourself. A few requests per second, with backoff, is usually plenty. Do not parallelize into the hundreds against one origin.
Cache aggressively. Do not refetch a page you already have. Every request you skip is load you did not impose.
Identify yourself where appropriate, and stop when a site signals it does not want automated access.

How link.sc Handles the Ladder for You

link.sc is one API to fetch and search anything online, built for LLMs and agents. You pass a URL and get back clean markdown, structured JSON, raw HTML, or a screenshot. The escalation ladder above is exactly what it runs internally, so you never touch it.

It goes cheapest-first: plain curl_cffi with real browser TLS fingerprints, then a stealth CloakBrowser with JavaScript rendering for sites that throw a challenge, then a Camoufox browser over Tor for the Cloudflare-hard cases. Crucially, it learns which method each domain needs, so it does not waste a browser render on a site that a fast HTTP request would satisfy, and it goes straight to the working method on repeat requests.

curl -X POST https://api.link.sc/v1/fetch \
  -H "x-api-key: lsc_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://protected-site.com",
    "format": "markdown"
  }'

import requests

resp = requests.post(
    "https://api.link.sc/v1/fetch",
    headers={"x-api-key": "lsc_YOUR_KEY"},
    json={"url": "https://protected-site.com", "format": "markdown"},
)
print(resp.json()["content"])

const res = await fetch("https://api.link.sc/v1/fetch", {
  method: "POST",
  headers: {
    "x-api-key": "lsc_YOUR_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://protected-site.com", format: "markdown" }),
});
const { content } = await res.json();
console.log(content);

One request, and the escalation, retries, and parsing are handled. It is open core and self-hostable if you would rather run the stack yourself.

Getting public data through Cloudflare reliably is a solved problem; the only question is who maintains the solution. If you would rather not, grab a free API key with 500 free credits a month, or read the docs. Scrape responsibly.