Quick answer: A 5xx error means the server failed, not you. 500 is an internal error, 502 is a bad gateway, 503 is service unavailable, and 504 is a gateway timeout. Most 5xx errors are transient, so retry with exponential backoff and jitter, always respect a Retry-After header, and wrap it in a circuit breaker so you stop hammering a site that is clearly down.
What each 5xx code means
The 5xx family is the server admitting fault. That is the opposite of a 4xx, where the server blames your request. This distinction shapes how you respond: a 4xx usually needs you to change something, while a 5xx often just needs patience.
| Code | Name | Usually means | Retry |
|---|---|---|---|
| 500 | Internal Server Error | The app crashed handling your request | Yes, cautiously |
| 502 | Bad Gateway | A proxy got a bad response from upstream | Yes |
| 503 | Service Unavailable | Overloaded, in maintenance, or a challenge | Yes, respect Retry-After |
| 504 | Gateway Timeout | An upstream server took too long | Yes, with a longer wait |
One nuance for scrapers: some anti-bot systems return 503 with a challenge page instead of a clean block. If you get a steady stream of 503s only from your scraper but the site loads fine in a browser, you may be looking at bot detection wearing a 5xx costume rather than a genuine outage. That is a different problem, covered in how sites detect and block scrapers.
Transient vs persistent
Before you retry anything, decide which kind of failure you are seeing.
Transient errors clear on their own: a momentary overload, a deploy in progress, a brief upstream hiccup. Retrying after a short wait usually succeeds. Most 502s and 504s are transient.
Persistent errors do not clear no matter how many times you retry: a broken deploy, a URL that triggers a server bug every time, or a site that is genuinely down. Retrying these just wastes your time and adds load to a struggling server.
The practical rule: retry a handful of times with growing delays. If it still fails, treat it as persistent, log it, and move on. Do not retry forever.
Retry with backoff and jitter
Exponential backoff means each retry waits longer than the last: 1s, 2s, 4s, 8s. Jitter adds a small random offset so that if many workers fail at once, they do not all retry in lockstep and re-spike the server.
Critically, when the server sends a Retry-After header, obey it. It is the server telling you exactly when to come back, and ignoring it is both rude and counterproductive.
import random
import time
import requests
RETRYABLE = {500, 502, 503, 504}
def fetch_with_backoff(url, max_retries=5, base=1.0, cap=30.0):
for attempt in range(max_retries):
resp = requests.get(url, timeout=20)
if resp.status_code not in RETRYABLE:
return resp # success or a non-retryable error
# Respect an explicit Retry-After if present
retry_after = resp.headers.get("Retry-After")
if retry_after and retry_after.isdigit():
wait = float(retry_after)
else:
wait = min(cap, base * (2 ** attempt))
wait += random.uniform(0, wait * 0.25) # jitter
if attempt < max_retries - 1:
time.sleep(wait)
return resp # exhausted retries, hand back the last response
A few defaults worth keeping: cap the maximum wait so a bad Retry-After cannot stall you for an hour, and cap the retry count so a persistent failure does not loop forever.
Circuit breakers: stop hammering a struggling site
Backoff handles a single request. A circuit breaker handles the whole target. The idea is borrowed from electrical circuits: if failures pile up, you "trip" the breaker and stop sending requests entirely for a cooldown, then test cautiously before resuming.
This protects two parties. It protects the target site, which does not need your scraper piling on while it is already failing. And it protects you, by not burning your quota, proxies, and time on a target that is down.
import time
class CircuitBreaker:
def __init__(self, threshold=5, cooldown=60):
self.threshold = threshold # failures before tripping
self.cooldown = cooldown # seconds to stay open
self.failures = 0
self.opened_at = None
def allow(self):
if self.opened_at is None:
return True
if time.time() - self.opened_at >= self.cooldown:
# half-open: allow one trial request
return True
return False
def record_success(self):
self.failures = 0
self.opened_at = None
def record_failure(self):
self.failures += 1
if self.failures >= self.threshold:
self.opened_at = time.time()
breaker = CircuitBreaker()
def guarded_fetch(url):
if not breaker.allow():
raise RuntimeError("circuit open, skipping request")
resp = fetch_with_backoff(url)
if resp.status_code < 500:
breaker.record_success()
else:
breaker.record_failure()
return resp
When the breaker is open, skip the target for the cooldown window and work on other URLs. After the cooldown, let one trial request through. If it succeeds, close the breaker and resume. If it fails, stay open.
Do not be the reason the site is down
A 503 often means the server is already at its limit. If your scraper responds by retrying aggressively across many workers, you can turn a brief hiccup into a real outage. That is a good way to get your IP range banned and, in bad cases, cause actual harm.
Good hygiene:
- Keep concurrency modest per domain, not just globally.
- Always honor
Retry-Afterand any documented rate limits. - Add jitter so your workers do not synchronize into a thundering herd.
- Trip the circuit breaker early rather than late.
If your retries start turning into outright blocks (429s and 403s), the problem has shifted from server health to your footprint, and avoiding IP bans when scraping is the next thing to read.
An ethics note
Retrying is fine. Retrying without limits against a server that is clearly struggling is not. Treat a 5xx as a request from the server to ease off, not as an obstacle to power through. Scrape public data, respect robots.txt, and keep your load well below what would degrade the site for real users.
Offload the retry logic entirely
If you would rather not build and tune backoff and breakers per target, link.sc handles transient failures and pacing server-side. You send one request and get clean content back, or a clear error if the site is genuinely down.
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/report", "format": "markdown"}'
The retry, backoff, and rate discipline described above run inside the platform. See the link.sc docs for the full behavior.
Do not want to hand-roll backoff and circuit breakers? link.sc handles transient 5xx failures and polite pacing for you. Get started free.