Quick answer: Sites detect scrapers by stacking signals: how fast and how regularly you request pages, where your IP comes from, what your TLS and HTTP fingerprints look like, whether you can execute JavaScript, and whether you interact like a human. No single signal convicts you; anti-bot systems score them together and challenge or block above a threshold. Understanding the stack is how you diagnose blocks instead of guessing.
I spend a lot of my time diagnosing why fetches fail, and the single most useful mental model is this: anti-bot detection is a scoring system, not a tripwire. Let me walk through the layers in the order a request meets them.
Layer 1: rate and behavior
The oldest signal is still the most common. Humans read pages; scrapers request them. If one client asks for 500 pages in a minute, at metronome-regular intervals, in perfect alphabetical order, no fingerprinting is needed. Server-side systems track requests per IP, per session, and per URL pattern, and respond with slowdowns, 429s, or blocks.
Behavioral analysis goes further on sites that invest in it: mouse movement, scroll patterns, time-on-page, whether you ever load images. A "user" who navigates 40 product pages in 40 seconds without a single mouse event scores badly even from a pristine IP.
The defensive takeaway is boring and effective: slow down, add jitter, and treat a 429 as an instruction rather than an obstacle. Back off exponentially and cache what you've already fetched.
Layer 2: IP reputation
Before your request body is even interesting, the site knows your IP, and IP intelligence databases classify it: datacenter range, VPN exit, Tor node, residential ISP, mobile carrier. Traffic from AWS or Hetzner ranges starts with a suspicion penalty because almost no human browses from a datacenter.
This is why the same scraper can succeed from a laptop and fail from a server with identical code. In our own diagnosis of failed fetches at link.sc, the majority of hard "blocks" turned out to be IP reputation on datacenter and Tor exits, not anything about the request itself. The fix is usually better-sourced IPs, with all the cost tradeoffs that implies.
Layer 3: protocol fingerprints (TLS and HTTP/2)
This is the layer that surprises people. Before any HTML is exchanged, your client performs a TLS handshake, and the exact combination of ciphers, extensions, and curve preferences it offers forms a fingerprint (JA3 and its successors). Python requests, curl, Go's http client, and Chrome all have recognizably different handshakes. Sending Chrome's User-Agent string with OpenSSL's handshake is an instant contradiction.
HTTP/2 adds another fingerprint: frame ordering, header compression choices, priority settings. Real browsers have consistent, known signatures; most HTTP libraries don't match them.
The important property: no proxy fixes this. A residential IP with a curl handshake is still a curl handshake. This layer is why "I bought good proxies and still get blocked" is such a common complaint. The full mechanics are in TLS fingerprinting and bot detection.
Layer 4: JavaScript challenges, honeypots, and CAPTCHAs
If the scored signals are ambiguous, modern systems make your client prove it's a browser. The page ships a JavaScript challenge: compute this, report your canvas rendering, enumerate your fonts and WebGL renderer, expose your navigator properties. A plain HTTP client can't run it at all. A headless browser can, but default headless environments leak tells (missing plugins, headless-specific properties, impossible screen dimensions) that fingerprinting scripts look for specifically.
This is the layer that forces the jump from HTTP clients to real browsers, and from stock headless to hardened ones.
Two final traps live alongside the challenges:
- Honeypots are elements no human would touch: links hidden with CSS, form fields invisible to the eye. A scraper that follows every
<a>tag or fills every input outs itself instantly, and the honeypot doubles as a high-confidence signal that poisons the session or IP. - CAPTCHAs are the escalation of last resort: the site isn't sure, so it asks. Turnstile and reCAPTCHA increasingly run invisibly, scoring the environment without showing a puzzle unless the score is bad.
The vendor landscape at a glance
Most sites don't build any of this themselves. They buy it, and knowing whose block page you're looking at tells you what you're dealing with.
| Vendor | Where you'll meet it | Signature behavior |
|---|---|---|
| Cloudflare | Everywhere; sits in front of a huge share of the web | Managed challenges, Turnstile, 403s with a Cloudflare Ray ID |
| DataDome | E-commerce, classifieds, travel | Aggressive protocol fingerprinting, fast IP blocking, its own CAPTCHA |
| HUMAN (formerly PerimeterX) | Retail, ticketing, sneaker-adjacent sites | Heavy behavioral scoring, "Press and hold" challenge |
| Akamai Bot Manager | Large enterprises, airlines, banks | Sensor-data collection scripts, long-horizon behavioral profiles |
I keep this table deliberately high level. Each vendor iterates constantly, and any specific bypass detail would be stale in months anyway. The stable insight is that they all combine the same five layers with different weightings. Cloudflare specifically comes up so often that we wrote a dedicated piece: scraping Cloudflare-protected sites.
What this means for legitimate scraping
Here's my honest framing, and it's also the ethics note: anti-bot systems mostly exist to stop credential stuffing, carding, inventory hoarding, and content theft. Legitimate scraping (public pages, polite rates, robots.txt respected, no auth or paywall evasion) is collateral damage in that fight, not the target. Your job is to stop looking like the attacks, not to defeat the defenses. Concretely:
- Be consistent. Your IP type, TLS fingerprint, headers, and behavior should tell one coherent story. Contradictions score worse than any individual signal.
- Match the tool to the layer that's blocking you. Rate blocks need backoff. IP reputation needs better IPs. Fingerprint checks need a real browser fingerprint. JS challenges need a real browser. Escalating further than needed wastes money; escalating less than needed wastes retries.
- Prefer the sanctioned path when one exists. An official API, a data export, a licensing conversation. Sometimes the answer to "how do I scrape this" is "you ask."
- Stay out of gray zones. If access requires a login you don't have, terms you'd be violating, or solving CAPTCHAs at industrial scale, that's not an engineering problem anymore.
Diagnosing which layer blocked you and picking the matching tier is exactly the loop link.sc automates. Every domain gets a per-domain escalation ladder (plain HTTP, browser-fingerprint HTTP, headless, stealth browser over residential), and outcomes feed back so each site gets the cheapest method that works:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_your_key" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/article", "format": "markdown"}'
One request, and the layer-matching happens behind the API instead of in your retry logic.
Bottom line
Sites detect scrapers by scoring rate, IP reputation, protocol fingerprints, JavaScript execution, and behavior together. Blocks are diagnoses waiting to be read: a 429 is rate, a 403 from a datacenter IP is reputation, a challenge page is fingerprinting or JS. Read the signal, respond at the right layer, and keep your scraping on the legitimate side of the line the vendors are actually defending.
Stop guessing which layer blocked you. link.sc diagnoses and escalates per domain automatically, from plain HTTP to stealth. Start with 500 free credits.