← All posts

How to Build a Polite Scraper That Does Not Overload Sites

GUIDES · JUL 18, 2026 · 5 MIN READ

Quick answer: A polite scraper limits concurrency per host, spaces requests with a small random delay (jitter), honors robots.txt and any Crawl-delay, caches responses so it never refetches the same page, backs off when it sees errors or 429s, and identifies itself with an honest User-Agent and a contact URL. The goal is simple: collect the data you need without making the site slower for anyone else.

Politeness is not just etiquette. A scraper that hammers a server gets blocked, poisons the IP it runs from, and sometimes causes real harm to a small site running on modest hardware. A polite scraper collects more data over time precisely because it does not get itself banned. Let me show you how to build one.

The five habits of a polite scraper

  1. Limit concurrency per host. One slow site should not receive fifty simultaneous connections from you.
  2. Rate-limit per host with jitter. Space requests out, and randomize the spacing so you are not a metronome.
  3. Respect robots.txt and Crawl-delay. If the site tells you what it wants, listen.
  4. Cache aggressively. The politest request is the one you never send because you already have the answer.
  5. Back off on errors. A 429 or 503 means "slow down," so slow down.

Do these five things and you are already more considerate than most crawlers on the web.

Honor robots.txt first

Before fetching anything from a host, read its robots.txt and obey it. Python's standard library has a parser built in.

import urllib.robotparser as robotparser

def make_robots(base_url, ua):
    rp = robotparser.RobotFileParser()
    rp.set_url(base_url.rstrip("/") + "/robots.txt")
    rp.read()
    return rp

rp = make_robots("https://example.com", "PoliteBot")
if rp.can_fetch("PoliteBot", "https://example.com/some/path"):
    ...  # allowed
crawl_delay = rp.crawl_delay("PoliteBot") or 1.0  # seconds, default to 1

robots.txt is a request from the site owner, and honoring it is the baseline of ethical collection. I cover the wider picture in ethical web scraping compliance best practices.

Rate limit per host, with jitter

The key word is per host. A global delay is too blunt: it slows you down across sites that could handle more, while still allowing bursts at a single slow host if you are crawling many at once. Track the last-request time for each host and wait out the remaining delay before the next hit.

import time, random, threading
from urllib.parse import urlparse

class HostRateLimiter:
    def __init__(self, min_delay=1.0, jitter=0.5):
        self.min_delay = min_delay
        self.jitter = jitter
        self._last = {}          # host -> last request timestamp
        self._locks = {}         # host -> lock
        self._guard = threading.Lock()

    def _lock_for(self, host):
        with self._guard:
            return self._locks.setdefault(host, threading.Lock())

    def wait(self, url, crawl_delay=None):
        host = urlparse(url).netloc
        delay = max(self.min_delay, crawl_delay or 0)
        with self._lock_for(host):
            last = self._last.get(host, 0)
            elapsed = time.monotonic() - last
            sleep_for = delay - elapsed + random.uniform(0, self.jitter)
            if sleep_for > 0:
                time.sleep(sleep_for)
            self._last[host] = time.monotonic()

The jitter matters. A perfectly regular request every 1.000 seconds is an obvious machine signature. A request every 1.0 to 1.5 seconds looks far more like organic traffic and is gentler on the server's own rate windows.

Limit concurrency per host

Even with a delay, you want a hard ceiling on simultaneous connections to any one host. A semaphore per host does this cleanly.

from collections import defaultdict
import threading

class HostConcurrency:
    def __init__(self, max_per_host=2):
        self._sems = defaultdict(lambda: threading.Semaphore(max_per_host))

    def slot(self, host):
        return self._sems[host]

# usage
conc = HostConcurrency(max_per_host=2)
host = "example.com"
with conc.slot(host):
    ...  # at most 2 concurrent requests to this host

Two concurrent connections per host is a reasonable default for a stranger's site. Raise it only if you have permission or clear evidence the site can take it.

Back off when the site pushes back

When you get a 429 Too Many Requests or a 503, the right response is to wait, ideally for the duration in the Retry-After header, and retry with exponential backoff plus jitter. Never retry immediately in a tight loop; that turns a temporary throttle into a self-inflicted ban. The full mechanics are in HTTP 429 error explained and how to fix it.

import time, random, requests

def fetch_with_backoff(session, url, headers, max_tries=5):
    for attempt in range(max_tries):
        r = session.get(url, headers=headers, timeout=30)
        if r.status_code == 200:
            return r
        if r.status_code in (429, 503):
            retry_after = r.headers.get("Retry-After")
            wait = float(retry_after) if retry_after and retry_after.isdigit() \
                else (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)
            continue
        r.raise_for_status()
    raise RuntimeError(f"gave up on {url}")

Cache so you never refetch

A response you already fetched should never be requested again until it plausibly changed. Even a simple on-disk cache keyed by URL saves the target from serving the same page twice and saves you time. For volatile pages, use conditional requests with ETag/If-Modified-Since so the server can answer 304 Not Modified cheaply. I wrote a dedicated guide on this: cache web responses to save time and credits.

Identify yourself honestly

A polite scraper does not hide. Send a descriptive User-Agent with a URL where the site owner can learn who you are and how to reach you.

UA = "PoliteBot/1.0 (+https://yoursite.example/botinfo; [email protected])"

This does two good things: it lets a site owner contact you instead of silently blocking you, and it signals good faith. Many sites are happy to let identified, well-behaved bots through.

Putting it together

Here is the shape of the whole polite fetcher.

def polite_get(url, session, limiter, conc, rp, ua):
    if not rp.can_fetch(ua, url):
        return None  # respect robots.txt
    host = urlparse(url).netloc
    with conc.slot(host):
        limiter.wait(url, crawl_delay=rp.crawl_delay(ua))
        return fetch_with_backoff(session, url, {"User-Agent": ua})

Concurrency cap, per-host delay with jitter, robots check, and backoff, all in a handful of lines. Wrap a cache around it and you have a crawler that most site owners would never notice, let alone block.

The lazy way to be polite

Every rule above (per-host limits, backoff, honoring robots.txt, rotation, caching) is work you can hand off. link.sc applies polite defaults server side, so a single call returns clean content without you wiring up rate limiters and retry loops.

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article", "format": "markdown"}'

You still owe the same respect to the sites you collect from, but you are not the one who has to implement the throttle.

A note on ethics

Politeness and legality overlap but are not identical. Scrape only public data, stay within a site's stated limits, do not evade authentication or paywalls, and stop when a site clearly asks you to. Being polite is partly self-interest (you avoid bans) and partly just being a good citizen of a shared web. Both reasons point the same way.

Skip the rate-limiter plumbing: link.sc fetches with polite defaults built in. Get started free.