Quick answer: curl_cffi is a Python HTTP client built on a patched libcurl that can replay a real browser's TLS handshake. Install it with pip install curl_cffi, then swap import requests for from curl_cffi import requests and add impersonate="chrome" to your calls. That one argument makes your request's TLS and HTTP/2 fingerprint match real Chrome, which clears the fingerprint checks that block plain Python requests before a single HTTP header is read.
from curl_cffi import requests
r = requests.get("https://example.com", impersonate="chrome")
print(r.status_code, len(r.text))
If you have ever set a perfect User-Agent string and still received a 403, this is almost certainly why. Let's walk through what is actually happening, then build up from a single request to sessions, async scraping, and proxies, and finish with the cases where curl_cffi alone is not enough.
Why requests gets blocked and curl_cffi does not
When a client opens an HTTPS connection, it sends a ClientHello message listing its supported cipher suites, extensions, and elliptic curves in a specific order. That combination forms a fingerprint (commonly hashed as JA3 or JA4). Real Chrome has a well-known fingerprint. Python's requests, which speaks TLS through OpenSSL, has a completely different one that identifies it as a script.
Anti-bot services such as Cloudflare and Akamai compare the TLS fingerprint against the claimed User-Agent. If your headers say Chrome but your handshake says OpenSSL, you are flagged during the handshake itself, before your headers are even parsed. No amount of header tweaking in requests or httpx fixes this, because the mismatch lives below the HTTP layer.
curl_cffi fixes it at the right layer. It binds Python to curl-impersonate, a patched build of curl that reproduces the exact handshake of specific browser versions, including TLS extension order and HTTP/2 settings frames. To the fingerprinting service, your script's connection is indistinguishable from a real browser's.
Installation and first request
pip install curl_cffi
Wheels ship for Linux, macOS, and Windows, so there is nothing to compile. The API deliberately mirrors requests:
from curl_cffi import requests
r = requests.get(
"https://tls.browserleaks.com/json",
impersonate="chrome",
)
print(r.json()["ja3_hash"])
That endpoint echoes your TLS fingerprint back to you, which makes it a handy sanity check. Run it once with plain requests and once with curl_cffi, and compare the hashes. The curl_cffi hash will match a real Chrome build.
The impersonate argument accepts specific targets like "chrome131", "safari184", or "firefox133", plus generic aliases like "chrome" that map to the latest supported version. Using the generic alias is usually the right call, since the library updates the mapping as new browser versions are added. One caution: keep your User-Agent header consistent with the browser you impersonate. If you pass impersonate="chrome" and then manually set a Firefox User-Agent, you have recreated the exact mismatch you were trying to eliminate.
Sessions, cookies, and headers
For any real scraping job, use a Session. It reuses connections, persists cookies across requests, and keeps your fingerprint settings in one place:
from curl_cffi import requests
s = requests.Session(impersonate="chrome")
# First request picks up cookies (consent, session, anti-bot tokens)
s.get("https://example.com")
# Later requests send them back automatically
r = s.get("https://example.com/products?page=2")
print(r.status_code)
Cookie persistence matters more than people expect. Many anti-bot systems set a clearance cookie after the first request passes their checks, and requests without that cookie get re-challenged. A session handles the round trip for you.
You can still set headers as usual, and it is worth sending the small set browsers always send:
r = s.get(
"https://example.com/api/items",
headers={
"Accept": "application/json",
"Accept-Language": "en-US,en;q=0.9",
"Referer": "https://example.com/",
},
)
Going async for scale
curl_cffi ships an asyncio-compatible session, so you can fetch pages concurrently without threads:
import asyncio
from curl_cffi.requests import AsyncSession
async def fetch(session, url):
r = await session.get(url, impersonate="chrome")
return url, r.status_code
async def main():
urls = [f"https://example.com/page/{i}" for i in range(1, 21)]
async with AsyncSession() as s:
results = await asyncio.gather(*(fetch(s, u) for u in urls))
for url, status in results:
print(status, url)
asyncio.run(main())
Keep concurrency polite. A correct TLS fingerprint gets you in the door, but a burst of 50 requests per second from one IP is its own bot signal, and you will start seeing 429 responses. If that happens, back off and add jitter between requests; we cover the mechanics in our guide to HTTP 429 errors.
Adding proxies
TLS impersonation and IP reputation are separate problems. If you scrape from a datacenter IP (any cloud VM), some sites will block you on IP reputation alone, with a perfect fingerprint. Routing through residential proxies fixes that, and curl_cffi takes standard proxy syntax:
r = requests.get(
"https://example.com",
impersonate="chrome",
proxies={
"http": "http://user:[email protected]:8000",
"https": "http://user:[email protected]:8000",
},
)
The combination of browser-grade fingerprint plus residential exit IP clears a large majority of blocks you will meet in practice.
Where curl_cffi is not enough
curl_cffi is an HTTP client. It does not execute JavaScript, so two classes of sites remain out of reach:
- Client-rendered content. React or Vue single-page apps that ship an empty HTML shell and load data through XHR calls. You either call the underlying JSON API directly or render the page in a browser. See scraping JavaScript-rendered websites for both approaches.
- Active browser challenges. Cloudflare Turnstile, DataDome, and similar systems run JavaScript proof-of-work and environment checks that only a real browser engine can pass. A correct TLS handshake gets you past passive fingerprinting, not past an interactive challenge. Our Cloudflare scraping guide covers what those walls look like.
For those sites you escalate to a headless or stealth browser, which costs roughly 10 to 100 times more per page in time and memory. The economics of choosing between the tiers are laid out in curl_cffi vs. headless vs. stealth browsers. The short version: use curl_cffi as your default, and reserve browsers for pages that actually need them.
Skipping the tier management entirely
The awkward part of running this in production is that the right tool is per-domain and changes over time. One site works with curl_cffi today and adds Turnstile next month. Maintaining that mapping yourself, plus proxy pools, plus retries, is a standing chore.
That is the problem link.sc exists to remove. Every fetch starts at the cheap tier (curl_cffi with browser impersonation), escalates to a stealth browser only when the response comes back blocked or under-rendered, and remembers what worked per domain so later requests skip straight to the winning method. You send one API call and get clean content back:
import requests
r = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_YOUR_KEY"},
json={"url": "https://example.com"},
)
print(r.json()["content"])
curl_cffi is still doing the work under the hood on most requests. You are just no longer the one deciding when it is the right tool.
Want the curl_cffi-first, escalate-when-needed pipeline without building it? Create a free link.sc account and get 500 free fetches a month through one API call.