Quick answer: Async web scraping uses Python's asyncio to fire many HTTP requests concurrently on a single thread instead of waiting for each one to finish before starting the next. Because scraping is IO-bound (your code spends most of its time waiting on the network), overlapping those waits can turn a job that takes minutes into one that takes seconds. The pattern that matters: an async HTTP client like httpx or aiohttp, a semaphore to cap concurrency, and disciplined error handling.
Why async helps IO-bound scraping
Scraping is almost never CPU-bound. When you request a page, your program sends a few bytes and then waits, sometimes for hundreds of milliseconds, for the server to respond. In synchronous code that wait is dead time: nothing else happens.
Say you fetch 100 pages that each take 300 ms. Synchronously that is 30 seconds of mostly waiting. With async, you start all 100 requests, and while each one waits on the network, the others are also waiting. The total time approaches the slowest single request plus overhead, not the sum of all of them.
Async does not make any individual request faster. It lets you wait on many requests at the same time. That is the whole trick, and for network-bound work it is a large win.
A quick map of the options:
| Model | How it scales IO | Best for |
|---|---|---|
| Synchronous loop | One request at a time | Small jobs, simplest code |
| Threads | OS threads share waiting | Moderate concurrency, blocking libraries |
| asyncio | One thread, many awaits | High concurrency, network-bound work |
For a broad tour of scraping techniques beyond concurrency, see our advanced web scraping in Python guide. This post stays focused on doing async well.
The building blocks
You need three things: an event loop, an async HTTP client, and a way to run many tasks.
asyncio gives you the event loop and the async/await keywords. For the HTTP client, httpx and aiohttp are the two mature choices. httpx has a clean API and speaks both sync and async, so I reach for it first. Install it:
pip install httpx
A single async request looks like this:
import asyncio
import httpx
async def fetch(client: httpx.AsyncClient, url: str) -> str:
response = await client.get(url, timeout=10)
response.raise_for_status()
return response.text
async def main():
async with httpx.AsyncClient() as client:
html = await fetch(client, "https://example.com")
print(len(html))
asyncio.run(main())
The await keyword is the point where your function yields control back to the event loop so other tasks can run. Reusing one AsyncClient across requests matters: it pools connections instead of reopening a socket every time.
Running many requests at once
The naive way to scrape many URLs is to create a task per URL and gather them:
async def scrape_all(urls: list[str]) -> list[str]:
async with httpx.AsyncClient() as client:
tasks = [fetch(client, url) for url in urls]
return await asyncio.gather(*tasks)
This works for a handful of URLs. But if urls has 5,000 entries, you just told Python to open 5,000 connections at once. You will exhaust file descriptors, hammer the target server, and probably get rate-limited or blocked. Unbounded concurrency is the most common async scraping mistake.
Bounding concurrency with a semaphore
A semaphore is a counter that only lets N tasks into a section of code at a time. Wrap your fetch in one and you cap how many requests run concurrently, no matter how many URLs you queue.
import asyncio
import httpx
CONCURRENCY = 10
async def fetch(client, url, sem):
async with sem: # at most CONCURRENCY of these run at once
response = await client.get(url, timeout=10)
response.raise_for_status()
return url, response.text
async def scrape_all(urls):
sem = asyncio.Semaphore(CONCURRENCY)
async with httpx.AsyncClient() as client:
tasks = [fetch(client, url, sem) for url in urls]
return await asyncio.gather(*tasks)
Now you can queue 5,000 URLs but only 10 are ever in flight. This is the single most important pattern in async scraping. Start with a low number like 5 to 10, watch the target's response, and raise it only if the server tolerates it. Being polite here is also what keeps you unblocked.
Task groups for cleaner error handling
Python 3.11 added asyncio.TaskGroup, which improves on gather by cancelling sibling tasks when one fails and propagating errors clearly. It is the modern default.
import asyncio
import httpx
async def scrape_all(urls, concurrency=10):
sem = asyncio.Semaphore(concurrency)
results = {}
async def worker(client, url):
async with sem:
try:
r = await client.get(url, timeout=10)
r.raise_for_status()
results[url] = r.text
except httpx.HTTPError as exc:
results[url] = f"ERROR: {exc}"
async with httpx.AsyncClient() as client:
async with asyncio.TaskGroup() as tg:
for url in urls:
tg.create_task(worker(client, url))
return results
Notice that each worker catches its own errors and records them instead of letting one bad URL kill the batch. In scraping you expect failures: dead links, timeouts, the occasional 503. Handling them per-task keeps the crawl moving.
Rate limiting: concurrency is not the same as request rate
A semaphore caps concurrent requests, but it does not cap requests per second. Ten fast requests that each finish in 50 ms means 200 requests per second, which many servers will not accept. If you need a true rate limit, add a small delay or a token-bucket pace.
The simplest approach is a minimum interval between request starts:
import asyncio
import time
class RateLimiter:
def __init__(self, per_second: float):
self._interval = 1.0 / per_second
self._next = 0.0
self._lock = asyncio.Lock()
async def wait(self):
async with self._lock:
now = time.monotonic()
if now < self._next:
await asyncio.sleep(self._next - now)
self._next = max(now, self._next) + self._interval
# usage inside a worker:
# await limiter.wait()
# await client.get(url)
Combine the semaphore (how many at once) with the rate limiter (how fast overall) and you have a crawler that is both quick and well-behaved.
Retries and timeouts
Networks are unreliable, so always set a timeout and retry transient failures with backoff. A minimal retry wrapper:
import asyncio
import httpx
async def fetch_with_retry(client, url, retries=3):
for attempt in range(retries):
try:
r = await client.get(url, timeout=10)
r.raise_for_status()
return r.text
except (httpx.HTTPError, httpx.TimeoutException):
if attempt == retries - 1:
raise
await asyncio.sleep(2 ** attempt) # 1s, 2s, 4s
Retry on timeouts and 5xx responses. Do not retry on a 404 (the page is gone) or a 403 (you are being refused); those are not transient and retrying just wastes requests and looks abusive.
When async stops being the bottleneck
Async solves the concurrency problem. It does not solve the harder problems: pages rendered by JavaScript, IP reputation, or per-domain blocking. If the content is injected client-side, httpx returns HTML without the data no matter how many tasks you run. And if a site starts refusing your datacenter IP, more concurrency just gets you blocked faster. For background on the second issue, see how sites detect and block scrapers and how to avoid IP bans.
When that wall shows up, one option is to keep your async orchestration but offload rendering, proxies, and parsing to a fetch API. link.sc runs a stealth browser pool and proxy ladder server-side, so your async loop just posts URLs and receives clean content:
import asyncio
import httpx
async def link_fetch(client, url, sem):
async with sem:
r = await client.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_your_key"},
json={"url": url, "format": "markdown"},
timeout=60,
)
r.raise_for_status()
return url, r.json()["content"]
async def main(urls):
sem = asyncio.Semaphore(10)
async with httpx.AsyncClient() as client:
async with asyncio.TaskGroup() as tg:
tasks = [tg.create_task(link_fetch(client, u, sem)) for u in urls]
return [t.result() for t in tasks]
The content field comes back parsed and rendered, so your async concurrency handles orchestration while the API handles the parts that break naive scrapers. The quickstart has the full request options.
Ethics note
Concurrency makes it easy to overwhelm a server by accident. Keep concurrency modest, add a real rate limit, honor robots.txt, scrape only public data, and never use async to brute past authentication or protections. Fast and polite are not opposites.
Recap
Async web scraping wins because scraping waits on the network, and asyncio lets you overlap those waits. Use an async client like httpx, cap concurrency with a semaphore, prefer TaskGroup for clean error handling, add a rate limiter so you are polite, and retry transient failures with backoff. When rendering and blocking become the real bottleneck, offload those and keep your fast async loop.
Keep your async crawler and skip the rendering and proxy headaches. Try link.sc free.