Quick answer: Rate limiting is a server capping how many requests a client can make in a window of time, usually with a token bucket or a fixed/sliding window algorithm, to protect its capacity. Throttling is the response: the server delays or rejects requests over the limit. As a client, you stay under limits by honoring Retry-After, backing off exponentially, and queueing your requests. As a scraper operator, you throttle yourself to be a polite guest.
Rate limiting is the mechanism behind a lot of scraper pain, and it is worth understanding as a concept rather than just a 429 to react to. This post covers how servers implement limits, why they bother, and how to design a client that lives comfortably under them. If you just want to fix a 429 you already hit, read HTTP 429 error explained. This one is about the design.
What rate limiting actually is
A rate limit is a rule like "no more than 100 requests per minute per API key" or "no more than 10 concurrent connections per IP." The server counts your requests and, once you cross the line, it throttles you: it either delays the response or rejects it, usually with a 429 Too Many Requests.
Throttling and rate limiting get used interchangeably, but they are two halves of one system. The limit is the policy. The throttle is the enforcement.
The common algorithms
Servers implement limits with a handful of well-known algorithms. Knowing which one you are up against tells you how bursty you can be.
| Algorithm | How it works | Burst behavior | Feels like |
|---|---|---|---|
| Fixed window | Count resets every N seconds | Allows a burst at window edges | Simple, spiky |
| Sliding window | Rolling count over the last N seconds | Smooths edge bursts | Fairer, steadier |
| Token bucket | Tokens refill at a steady rate; each request spends one | Allows short bursts up to bucket size | Flexible |
| Leaky bucket | Requests drain from a queue at a fixed rate | No bursts, constant output | Very smooth |
| Concurrency limit | Caps simultaneous in-flight requests | Not about rate, about parallelism | Blocks the (N+1)th |
The token bucket is the most common because it is forgiving. The bucket holds, say, 60 tokens and refills at one per second. Each request spends a token. If you have been idle, the bucket is full and you can fire a short burst; if you drain it, you are limited to the refill rate. This is why an API can let you do a quick burst and then slow you to a steady trickle.
Fixed window is the simplest and has a known flaw: if the limit is 100 per minute, a client can send 100 at 0:59 and another 100 at 1:00, hitting 200 in two seconds across the boundary. Sliding window fixes that by counting over a rolling period.
Concurrency limits are a different axis. They do not care about your rate over time, only how many requests are open at once. You can be well under the per-minute limit and still get rejected for opening too many connections in parallel.
Why servers do this
It is easy to read a rate limit as hostility toward scrapers, but the real reasons are mundane and legitimate:
- Capacity protection. A server has finite CPU, memory, and database connections. Limits keep one client from starving everyone else.
- Cost control. Every request costs money to serve. Limits keep costs predictable.
- Abuse prevention. Limits blunt credential-stuffing, scraping floods, and denial-of-service attempts.
- Fairness. On a shared API, limits ensure one heavy user does not degrade the service for everyone.
Reading limits this way changes how you design a client. The server is not your adversary. It is asking you to behave, and it tells you how through headers.
Reading the signals: rate limit headers
Well-behaved APIs advertise their limits so you never have to guess. The common headers:
RateLimit-Limit: 100
RateLimit-Remaining: 12
RateLimit-Reset: 30
Retry-After: 30
RateLimit-Remaining tells you how much budget is left in the current window. RateLimit-Reset says when it refills. When you do get a 429 (or sometimes a 503), Retry-After tells you exactly how long to wait, either in seconds or as an HTTP date. Honoring Retry-After is the single most effective thing you can do, because you are following the server's own instruction instead of guessing.
Staying under the limit as a client
Here is the client-side design that keeps you out of trouble.
1. Honor Retry-After first
If the server tells you when to come back, wait exactly that long. Do not retry sooner.
import time, requests
def fetch(url):
while True:
resp = requests.get(url)
if resp.status_code == 429:
wait = int(resp.headers.get("Retry-After", "5"))
time.sleep(wait) # obey the server
continue
return resp
2. Back off exponentially with jitter
When there is no Retry-After, back off exponentially so you do not hammer a struggling server. Add jitter (a small random offset) so a fleet of workers does not retry in lockstep.
import random, time
def backoff(attempt):
base = 2 ** attempt # 1, 2, 4, 8 seconds
jitter = random.uniform(0, 1)
time.sleep(base + jitter)
Exponential backoff also matters for 5xx responses, which we cover in handle 5xx server errors when scraping.
3. Queue and pace your own requests
Rather than firing requests as fast as your loop runs, put them through a queue that releases at a fixed rate. A simple token-bucket limiter on your side mirrors the server's and keeps you comfortably under the line.
import time
class RateLimiter:
def __init__(self, per_second):
self.interval = 1.0 / per_second
self.next_time = time.monotonic()
def wait(self):
now = time.monotonic()
if now < self.next_time:
time.sleep(self.next_time - now)
self.next_time = max(now, self.next_time) + self.interval
limiter = RateLimiter(per_second=5) # at most 5 requests/sec
for url in urls:
limiter.wait()
fetch(url)
4. Cap your concurrency
Even with a good per-second rate, opening hundreds of connections at once can trip a concurrency limit. Use a semaphore or a worker pool to bound how many requests are in flight.
Throttling your own scraper to be polite
The other side of this coin is self-imposed throttling. Even when a site has no visible rate limit, blasting it is rude and gets you blocked. A polite scraper:
- Adds a delay between requests to the same host (a few hundred milliseconds to a couple of seconds is reasonable for most sites).
- Respects
Crawl-delayinrobots.txtwhere present. - Limits concurrency per domain so you are not one client acting like a hundred.
- Scrapes during off-peak hours for the target when the job is large.
- Sends a real User-Agent so the operator can identify and contact you.
Being polite is not just etiquette. A steady, modest request rate is far less likely to be flagged than a spiky flood, so throttling yourself is also the most reliable way to stay unblocked over time.
How a managed API absorbs this
A big reason to use a managed fetch API is that limits, backoff, and proxy rotation are handled for you. When you call link.sc, the service paces requests against target sites and retries on transient failures, so you send one clean request and get content back.
curl -X POST https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_your_key_here" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "format": "markdown"}'
The API has its own plan-based limits (the free tier is 500 credits a month, with more at link.sc/pricing), but you no longer juggle per-site throttling yourself.
The short version
Rate limiting protects servers, and it is enforced through throttling. Learn the algorithm you are facing, read the RateLimit and Retry-After headers, back off exponentially with jitter, pace your requests through a limiter, and cap concurrency. Then throttle yourself even where no limit is published, because a polite client is a client that keeps working.
Want request pacing, retries, and proxy rotation handled for you? Get a free link.sc key and fetch any URL as clean markdown.