Quick answer: To avoid IP bans when scraping, keep your request rate low enough to be indistinguishable from normal traffic (think seconds between requests, not milliseconds), respect 429 and 403 responses with real exponential backoff, keep your TLS and browser fingerprint consistent with your claimed user agent, and understand that many sites block datacenter IP ranges outright regardless of your behavior. If a site blocks datacenter IPs, no amount of politeness from an AWS address fixes it; you need residential egress or a fetch API that provides it.
Most IP bans are self-inflicted. I've caused a few myself, and diagnosed a lot more. Here's why bans actually happen and what prevents them, in order of importance.
Why sites ban IPs in the first place
It helps to think like the defender for a minute. Sites ban for three distinct reasons, and the fix is different for each:
Rate. You sent 50 requests a second from one address. This is the easiest signal to detect and the most justified ban. A human on a browser doesn't do that.
Fingerprint mismatch. Your user agent says Chrome, but your TLS handshake says Python requests. Anti-bot vendors check whether the layers of your request tell a consistent story, and stock HTTP libraries fail that check instantly. I went deeper on this in curl vs headless vs stealth browsers.
IP reputation. Your IP belongs to AWS, Hetzner, or a known proxy range, and the site (or its CDN) pre-emptively distrusts the entire block. You can be banned before your first request based purely on who owns your address. In my experience diagnosing blocked fetches, this category is the biggest one by a wide margin.
Notice that only the first one is about your behavior. You can be a model citizen and still get walled off for the second two.
Start with the respectful-rate baseline
Before any clever infrastructure, get the basics right, because they're free and they solve the most common ban:
- One request at a time per domain, with a delay between requests. I default to 2-5 seconds with jitter.
- Check robots.txt and honor
Crawl-delayif it's set. - Scrape off-peak for the site's timezone when you're doing a big crawl.
- Cache everything. The cheapest request is the one you don't repeat.
import random
import time
import requests
session = requests.Session()
session.headers["User-Agent"] = "yourbot/1.0 ([email protected])"
for url in urls:
resp = session.get(url, timeout=30)
handle(resp)
time.sleep(random.uniform(2.0, 5.0)) # jitter, not a fixed beat
The jitter matters: a request every exactly 3.000 seconds is a metronome, and metronomes are easy to fingerprint.
An honest identifying user agent with contact info is the ethical default for bulk crawling. Site operators do sometimes email instead of banning.
Handle 429 and 403 like you mean it
A 429 is the site telling you your rate is too high while it's still being polite. Ignore it and the polite phase ends: temporary throttling becomes an IP ban.
The correct response is exponential backoff with respect for Retry-After:
def fetch_with_backoff(session, url, max_retries=5):
delay = 2.0
for attempt in range(max_retries):
resp = session.get(url, timeout=30)
if resp.status_code not in (429, 503):
return resp
retry_after = resp.headers.get("Retry-After")
wait = float(retry_after) if retry_after else delay
time.sleep(wait + random.uniform(0, 1))
delay *= 2
raise RuntimeError(f"Still rate limited after {max_retries} tries: {url}")
Also: when you see the first 429, cut your global rate for that domain, don't just retry the one request. I wrote a full breakdown in HTTP 429 explained and how to fix it.
A 403 is a different animal. It usually means an anti-bot verdict or an IP-reputation block, and retrying harder makes it worse. Back off, and treat repeated 403s as a signal to change what you are (fingerprint, IP type), not how fast you are.
Datacenter vs residential IPs
This is the part nobody tells beginners: for a growing set of sites, especially anything behind Cloudflare, Akamai, or DataDome, the decision to block you is made from your IP's ASN before your request is even evaluated.
| Datacenter IPs | Residential IPs | |
|---|---|---|
| Cost | Cheap | Significantly more expensive |
| Speed and reliability | Excellent | Variable |
| Reputation with anti-bot systems | Poor to blocked by default | Generally trusted |
| Good for | APIs, friendly sites, RSS, your own infra | Sites that gate on IP reputation |
The pragmatic strategy is tiered: try the cheap datacenter path first, and escalate to residential only for domains that block it. Escalating everything to residential is burning money; never escalating means a chunk of the web is simply unreachable from your server.
Rotation, done sanely
IP rotation gets treated as the magic fix, and it's often misused. Two rules keep it sane:
Rotate to distribute load, not to evade a ban you earned. If a site banned you for hammering it, rotating to continue hammering it is the behavior anti-bot systems exist to stop, and it will get the whole pool burned.
Keep sessions sticky. If you're logged in or carrying cookies, changing IP mid-session is itself a bot signal. Rotate per session or per task, not per request.
And keep fingerprints consistent with the IP story. A residential IP with a raw-Python TLS handshake is still a mismatch; the layers all have to agree. Consistency beats cleverness here.
Or skip the infrastructure entirely
Everything above (rate control, backoff, fingerprint consistency, tiered IP escalation) is real ongoing engineering, and it's exactly the layer link.sc exists to absorb. A fetch request comes back as clean markdown, and the routing, retries, and escalation across fetch strategies happen behind the API:
curl "https://link.sc/v1/fetch?url=https://example.com/page&format=markdown" \
-H "Authorization: Bearer lsc_your_key"
If your blockers are specifically Cloudflare-shaped, scraping Cloudflare-protected sites covers what's actually going on under that challenge page. The quickstart has runnable examples if you want to test against your problem domain in the next five minutes.
The ethics note (please read this one)
Everything here is framed around legitimate access to public data, and that framing is load-bearing:
- Don't use these techniques to bypass authentication, paywalls, or access controls. That's not scraping, that's intrusion.
- Respect robots.txt and terms of service for anything commercial you're building.
- Don't scrape personal data just because it's technically reachable.
- If a site offers an API, use it, even if scraping seems easier today.
A ban is sometimes a site telling you "no," and the right response to a clear, persistent "no" is to stop, not to escalate. The legal landscape has real edges too; see is web scraping legal before building a business on someone else's data.
Tired of playing whack-a-mole with IP blocks? Create a free link.sc account and let the fetch layer be someone else's problem.