Quick answer: Web scraping at scale is a distributed systems problem, not a parsing problem. The pattern that holds up is a decoupled pipeline: a fetch layer that returns clean content, a queue that buffers work, stateless workers that pull from it, a dedupe step, durable storage, and a scheduler that decides what to refetch and when. Monitoring, retries, and per-domain rate control sit across all of it. Get the pipeline shape right and the parsing is the easy part.
Scraping ten pages is a script. Scraping ten million is an architecture. The failure modes change completely: a single blocked IP stops mattering, and what matters instead is backpressure, idempotency, and keeping cost per page flat as volume climbs. This is a walkthrough of a reference design that survives that transition, described in text with a diagram you can read.
The Pipeline, In One Diagram
Here is the shape, top to bottom:
[ Seed URLs / Sitemaps ]
|
v
+------------------+ +------------------------+
| Scheduler |------->| URL Frontier (queue) |
| (what & when) | | dedupe + priority |
+------------------+ +-----------+------------+
^ |
| v
+--------+---------+ +-------------------+
| Monitoring / |<---------| Fetch Workers |
| Metrics / Alerts| | (stateless, N) |
+------------------+ +---------+---------+
|
v
+-------------------+
| Fetch Layer |
| (proxies, browser |
| pool, retries) |
+---------+---------+
|
content v
+-------------------+
| Parse + Extract |
+---------+---------+
|
v
+----------------------+----------------------+
| Content-hash dedupe |
+----------------------+----------------------+
|
v
+-------------------+
| Durable Storage |
| (blob + index DB) |
+-------------------+
Every arrow is a place work can back up or fail, which is exactly why the stages are decoupled. No stage calls the next one directly; they hand work across a queue so a slow storage write never stalls a fetch worker.
The Fetch Layer
This is the stage everyone underestimates. At small volume you can requests.get and move on. At scale you are fighting IP bans, TLS fingerprinting, JavaScript-rendered pages, and per-domain rate limits, all at once, across thousands of sites with different defenses.
You have two honest options. Build it: a rotating pool of residential and datacenter proxies, a headless browser fleet for JS-heavy pages, an escalation ladder that tries a cheap HTTP client first and only spins up a browser when it must, and a team to keep all of that current. Or call a fetch API that does that server-side and returns clean content.
import os, requests
def fetch(url: str) -> str:
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": os.environ["LINKSC_KEY"]},
json={"url": url, "format": "markdown"},
timeout=60,
)
resp.raise_for_status()
return resp.json()["content"]
The build-vs-API line moves with volume. Below a few thousand pages a day, a tuned client plus a proxy plan is fine. Above that, the operational cost of the fetch layer usually dwarfs the API bill, which is the whole argument in how much web scraping costs at scale. Either way, treat the fetch layer as a black box behind a function so the rest of the pipeline never knows or cares.
The Queue and the URL Frontier
The queue is what makes the system distributed instead of just parallel. Workers pull URLs from it; they never receive pushed work. That single design choice gives you backpressure for free: if workers fall behind, the queue grows, and nothing crashes.
The frontier is the queue plus the logic on top of it:
- Dedupe on enqueue. Never add a URL already seen this cycle. A Redis set or a Bloom filter keeps this O(1) and cheap.
- Priority. Fresh-critical pages (a pricing page, a news feed) jump ahead of the long tail.
- Per-domain politeness. Group by host so you can rate-limit one site without throttling the whole crawl.
import redis
r = redis.Redis()
def enqueue(url: str, priority: int = 0):
if r.sismember("seen", url): # already queued or done
return
r.sadd("seen", url)
r.zadd("frontier", {url: priority})
Stateless Workers
Workers do one thing: pull a URL, call the fetch layer, parse, hand results to dedupe and storage. Keep them stateless so you can run one or a thousand, and so a crashed worker loses nothing but its current item, which the queue redelivers.
The rule that keeps this sane is idempotency. Fetching the same URL twice must be harmless, because at scale it will happen: redeliveries, retries, overlapping schedules. Key everything by a normalized URL or a content hash and writes stay clean no matter how many times a page flows through.
Dedupe and Storage
Two kinds of dedupe matter. URL dedupe (handled at enqueue) stops you fetching the same address twice. Content dedupe stops you storing the same body under different URLs: mirrors, tracking-param variants, and pagination that loops. Hash the normalized content and skip writes when the hash already exists. The full treatment is in deduplicating pages when crawling.
Storage is usually two tiers: a blob store for raw content (cheap, append-only) and an indexed database for the structured fields you extracted plus metadata (fetch time, content hash, source URL). Keep raw and parsed separate so you can re-parse historical data when your extractor improves, without refetching a thing.
Scheduling and Freshness
At scale you are not crawling once; you are keeping a corpus current. The scheduler decides what to refetch and when, and doing it well is the difference between a flat bill and a runaway one. Refetch a daily-changing page daily and a static archive page monthly. Adaptive scheduling (widen the interval when a page's content hash stops changing, tighten it when it churns) cuts wasted fetches hard. That is its own topic in keeping scraped data fresh.
Retries, Rate Control, and Monitoring
These three cut across every stage.
| Concern | Cheap approach | Scale approach |
|---|---|---|
| Retries | Fixed retry count | Exponential backoff with jitter, dead-letter queue after N |
| Rate control | Global sleep | Per-domain token bucket, respects robots.txt Crawl-delay |
| Monitoring | Print logs | Success-rate per domain, queue depth, alert on drops |
Retries belong in the fetch layer with exponential backoff and jitter so a struggling site does not get hammered by synchronized retries. After a fixed number of failures, park the URL in a dead-letter queue for inspection instead of looping forever.
Rate control is per-domain, not global. One token bucket per host lets you crawl a thousand sites fast while staying polite to each one. Monitoring watches success rate per domain and queue depth: a domain whose success rate falls off a cliff usually means a layout change or a new block, and queue depth climbing means workers cannot keep up. Both should page you before the data goes stale.
Build vs API at Scale
Here is the honest tradeoff, stated plainly:
| Component | Build it | Use link.sc |
|---|---|---|
| Fetch layer (proxies, browsers, fingerprints) | High ongoing cost, full control | Handled server-side, per-request cost |
| Queue, workers, scheduler | You build regardless | You build regardless |
| Dedupe, storage | You build regardless | You build regardless |
Notice what does not move. The fetch API replaces one stage, the hardest and most operationally expensive one, and leaves the rest of the pipeline yours to design. That is usually the right split: own your data model, your scheduling, and your storage, and rent the part that is a full-time adversarial arms race. A free link.sc key is enough to prototype the fetch stage before you commit to building one.
The Defensive View
Scale amplifies your footprint, so the etiquette matters more, not less. Respect robots.txt, honor Crawl-delay, and set per-domain rate limits that a site owner would consider reasonable. Scrape public data, identify your crawler honestly, and do not route around authentication or paywalls. A well-behaved crawler at scale is nearly invisible; a greedy one gets a whole IP range blocked and deserves it.
Get the pipeline shape right first. The parsing, the proxies, and the storage engine are all swappable details once the stages are decoupled and the queue is doing its job.
Skip the hardest stage. Get a free link.sc key and drop a clean fetch layer into your pipeline with one API call.