Quick answer: Rotate proxies per request for stateless crawling and use a sticky session (one IP held for minutes) for anything that carries cookies or a login. Health-check your pool so dead or slow IPs get benched, retry a blocked request on a fresh IP with backoff, and pin an IP to a session with a consistent hash so logged-in flows do not break. The strategy you pick depends on whether the target tracks you across requests.
Choosing a good proxy is one problem; choosing when to switch is a different one, and it is the part that actually decides whether you get blocked. This post is about the rotation mechanics: the modes, the health checks, and the retry logic. For which type of proxy to buy in the first place, see best proxies for web scraping.
Two rotation modes
There are really just two families, and everything else is a variation.
Per-request rotation. Every request goes out through a different IP from the pool. This maximizes IP diversity, which is exactly what you want for stateless crawling: hitting a thousand product pages where each request stands alone. No request depends on the last, so a fresh IP each time spreads your footprint thin.
Sticky sessions. One IP is held for a fixed window (say, ten minutes) or for the life of a logical session. You need this whenever the site tracks you across requests: a shopping cart, a login, a multi-step form, or anything that sets a cookie you must keep sending from the same IP. Rotating mid-session looks like your account teleported across the country between clicks, which is a fast way to get flagged.
| Mode | Use for | Breaks if |
|---|---|---|
| Per-request | Stateless crawls, independent pages | Site correlates a session across requests |
| Sticky session | Logins, carts, multi-step flows | IP changes mid-session |
| Hybrid | Mixed workload | You apply the wrong mode per task |
The mistake I see most often is using per-request rotation on a logged-in flow. The site sees a session cookie arriving from twelve different IPs in a minute and blocks the account. Match the mode to whether state is involved.
A rotating pool with health checks
A proxy pool is only as good as its worst IPs. Dead, slow, or already-blocked proxies waste requests and skew your results. Track health per proxy and skip the bad ones.
import time, random, threading
class ProxyPool:
def __init__(self, proxies, cooldown=300):
# each entry: proxy url -> health state
self.state = {p: {"fails": 0, "benched_until": 0} for p in proxies}
self.cooldown = cooldown
self.lock = threading.Lock()
def _healthy(self):
now = time.time()
return [p for p, s in self.state.items() if s["benched_until"] < now]
def get(self):
with self.lock:
healthy = self._healthy()
if not healthy:
# everything benched, take the soonest-free one
healthy = list(self.state.keys())
return random.choice(healthy)
def report(self, proxy, ok):
with self.lock:
s = self.state[proxy]
if ok:
s["fails"] = 0
else:
s["fails"] += 1
if s["fails"] >= 3:
s["benched_until"] = time.time() + self.cooldown
s["fails"] = 0
Three consecutive failures benches a proxy for a cooldown period, then it gets another chance. This keeps a few bad IPs from dragging down the whole run without permanently discarding proxies that were only briefly rate-limited.
Retry on block with a fresh IP
When a request comes back blocked (a 403, a 429, or a CAPTCHA page), the fix is usually a different IP, not a retry on the same one. Combine the pool with exponential backoff and a fresh proxy per attempt.
import requests, time, random
def fetch_rotating(url, pool, headers, max_tries=4):
for attempt in range(max_tries):
proxy = pool.get()
proxies = {"http": proxy, "https": proxy}
try:
r = requests.get(url, headers=headers, proxies=proxies, timeout=30)
except requests.RequestException:
pool.report(proxy, ok=False)
continue
if r.status_code == 200:
pool.report(proxy, ok=True)
return r
if r.status_code in (403, 429, 503):
pool.report(proxy, ok=False) # bench and try a new IP
time.sleep((2 ** attempt) + random.uniform(0, 1))
continue
r.raise_for_status()
raise RuntimeError(f"blocked on all attempts: {url}")
Two details matter here. First, mark the proxy bad on a block so the pool benches it. Second, back off with jitter between attempts. Retrying instantly on a fresh IP just burns your pool faster, and if the block is IP-reputation based, hammering makes it worse. If you keep hitting blocks across many IPs, the problem is probably not rotation at all; see how to avoid IP bans when scraping for the deeper causes.
Session affinity for logged-in flows
When you need sticky behavior across a set of workers, do not pick the IP randomly. Pin each logical session to a proxy with a consistent hash, so every request in that session lands on the same IP even across threads.
import hashlib
def sticky_proxy(session_id, proxies):
h = int(hashlib.sha256(session_id.encode()).hexdigest(), 16)
return proxies[h % len(proxies)]
# every request tagged with the same session_id gets the same IP
p = sticky_proxy("user-cart-8841", proxy_list)
Many commercial residential providers offer sticky sessions natively, where a session token in the proxy username holds the same exit IP for a set duration. If yours does, use it rather than hand-rolling affinity. Either way, the principle is one identity, one IP, for the life of the session.
When to stop rotating and start throttling
Rotation is not a substitute for rate limiting. If you fire requests as fast as the pool allows, you overload targets and burn reputation on every IP you touch. Combine rotation with per-host pacing: rotate the IP to spread your footprint, but still space out requests so no single host sees a flood. The two techniques solve different problems, and you generally want both.
Let the ladder handle it
Rotation mode, pool health, retry-on-block, and session stickiness are a lot of moving parts to maintain, and the right choice changes per target. link.sc runs a per-domain escalation ladder server side: it picks the proxy type, decides per-request versus sticky, retries blocks on fresh IPs, and manages pool health, so a single call returns clean content.
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/product/123", "format": "markdown"}'
You send a URL; the rotation strategy is chosen and applied for you. That is the appeal of not owning this layer: the strategy adapts per domain without you rewriting your pool logic every time a target changes its defenses.
A note on ethics
Rotating IPs is for spreading legitimate load across a public site, not for evading a ban you earned by abusing a service or for getting around authentication. Respect robots.txt and rate limits, use residential IPs sourced with real consent, and do not use rotation to defeat access controls or scrape data you are not permitted to collect. Rotation changes where your requests come from; it does not change what you are entitled to request.
Stop maintaining pool logic and retry loops: link.sc picks and applies the rotation strategy per domain for you. Try it free.