Quick answer: Advanced web scraping in Python means moving past a bare requests.get() loop. In practice that's five upgrades: sessions with retries and backoff, async concurrency with httpx, a real browser (Playwright) for JavaScript-heavy pages, curl_cffi when your TLS fingerprint gets you blocked, and deliberate rate limiting so you don't get banned in the first place. Past a certain point, it's cheaper to offload the hard 10% to a fetch API.
Every scraping tutorial on the internet shows you requests plus BeautifulSoup and calls it a day. That works for the first hundred pages. Then you hit timeouts, 429s, JavaScript-rendered content, and mysterious 403s on sites that open fine in your browser. This post is about what you do next.
Sessions, Retries, and Backoff
The single highest-value upgrade is replacing loose requests.get() calls with a Session that retries transient failures automatically. A session reuses TCP connections (faster) and keeps cookies (many sites require them after the first request).
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
def make_session() -> requests.Session:
session = requests.Session()
retry = Retry(
total=4,
backoff_factor=1.5, # 1.5s, 3s, 6s, 12s
status_forcelist=[429, 500, 502, 503, 504],
respect_retry_after_header=True,
)
adapter = HTTPAdapter(max_retries=retry, pool_maxsize=20)
session.mount("https://", adapter)
session.headers.update({
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
})
return session
session = make_session()
resp = session.get("https://example.com/products?page=1", timeout=15)
resp.raise_for_status()
Two details people miss: always pass timeout (requests will happily hang forever without it), and let respect_retry_after_header honor the server's own Retry-After on 429s instead of hammering it on your schedule.
Going Async with httpx
When you have thousands of URLs, sequential fetching is the bottleneck, not parsing. httpx gives you an async client with an API that feels like requests, and asyncio.Semaphore caps concurrency so you're fast without being abusive.
import asyncio
import httpx
async def fetch(client: httpx.AsyncClient, sem: asyncio.Semaphore, url: str):
async with sem:
try:
r = await client.get(url, timeout=15)
r.raise_for_status()
return url, r.text
except httpx.HTTPError as e:
return url, None
async def crawl(urls: list[str], concurrency: int = 8):
sem = asyncio.Semaphore(concurrency)
async with httpx.AsyncClient(follow_redirects=True, http2=True) as client:
return await asyncio.gather(*(fetch(client, sem, u) for u in urls))
results = asyncio.run(crawl([f"https://example.com/item/{i}" for i in range(200)]))
A semaphore of 8 against one host is already aggressive for most sites. Concurrency multiplies your throughput and your ban risk equally, so raise it per-host only when you know the target can take it.
Handling JavaScript with Playwright
If resp.text is a nearly empty <div id="root"></div>, the content is rendered client-side and no amount of HTTP cleverness will help. You need a browser. Playwright is the modern choice in Python (I cover the full comparison in Selenium alternatives for scraping).
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://example.com/spa-listing", wait_until="networkidle")
page.wait_for_selector(".product-card")
html = page.content()
browser.close()
The trap: browsers are 10x to 50x slower and heavier than HTTP requests. Before reaching for Playwright, open DevTools, watch the Network tab, and check whether the page hydrates from a JSON API you can call directly. Half the time it does, and then you don't need a browser at all. More on that pattern in how to scrape JavaScript-rendered websites.
The TLS Fingerprint Problem (and curl_cffi)
Here's the one that confuses people the most. Your headers are perfect, your User-Agent says Chrome, and you still get a 403 while the site loads fine in an actual browser. The tell is usually Cloudflare or Akamai in front of the site.
These services fingerprint the TLS handshake itself (the JA3/JA4 signature). Python's ssl module handshakes like Python, not like Chrome, and no header can hide that. curl_cffi fixes it by impersonating a real browser's TLS stack:
from curl_cffi import requests as creq
resp = creq.get(
"https://protected-site.example.com/",
impersonate="chrome",
timeout=20,
)
print(resp.status_code) # often 200 where requests gets 403
The API mirrors requests closely, so migrating is usually a one-line change. It's the cheapest fix for fingerprint-based blocking, and it stacks with everything above: you can run curl_cffi through a session, through proxies, and with retries. For the deeper theory of what gets detected at each layer, see curl vs headless vs stealth browser.
Rate Limiting Like You Mean It
Retries and backoff are reactive. Rate limiting is proactive, and it's what keeps you from needing the retries. A simple token-bucket per host goes a long way:
import asyncio
import time
from collections import defaultdict
class HostLimiter:
def __init__(self, per_second: float = 1.0):
self.interval = 1.0 / per_second
self.last: dict[str, float] = defaultdict(float)
self.locks: dict[str, asyncio.Lock] = defaultdict(asyncio.Lock)
async def wait(self, host: str):
async with self.locks[host]:
elapsed = time.monotonic() - self.last[host]
if elapsed < self.interval:
await asyncio.sleep(self.interval - elapsed)
self.last[host] = time.monotonic()
Call await limiter.wait(urlparse(url).netloc) before each fetch. One request per second per host is a sane default; robots.txt Crawl-delay, when present, is a stated preference you should honor.
Structured Extraction That Survives Redesigns
Brittle selectors are the silent killer of scrapers. Two habits make extraction dramatically more durable. First, prefer embedded structured data over CSS selectors when it exists:
import json
from bs4 import BeautifulSoup
def extract_jsonld(html: str) -> list[dict]:
soup = BeautifulSoup(html, "lxml")
blocks = []
for tag in soup.select('script[type="application/ld+json"]'):
try:
blocks.append(json.loads(tag.string or ""))
except json.JSONDecodeError:
continue
return blocks
Product pages, articles, and job listings very often ship JSON-LD with price, title, and availability already structured. It changes far less often than the visual markup. Second, validate what you extract with a schema (pydantic works well) so a site redesign fails loudly in your pipeline instead of silently writing garbage for three weeks.
When to Stop Building and Offload
Everything above is worth knowing, and there's a point where it stops being worth operating. Rotating residential proxies, keeping browser fleets patched, and chasing anti-bot updates is a permanent maintenance tax. I've written about that math in the self-hosted web scraping guide.
The pragmatic pattern I recommend: HTTP-first with everything in this post, and route the URLs that still fail to a fetch API that handles rendering and blocking for you. With link.sc that's one call, returning clean markdown instead of raw HTML:
import requests
resp = requests.post(
"https://link.sc/v1/fetch",
headers={"Authorization": "Bearer lsc_your_key"},
json={"url": "https://protected-site.example.com/product/42", "format": "markdown"},
timeout=60,
)
print(resp.json()["markdown"])
The free tier is 500 credits a month, which is plenty to cover the stubborn tail of a mostly self-hosted pipeline. Details are on the pricing page.
My honest take: advanced scraping isn't about one magic library. It's a ladder. Sessions and retries fix 60% of your problems, async fixes throughput, curl_cffi fixes fingerprints, Playwright fixes JavaScript, and an API fixes the remainder you shouldn't spend your life on.
Tired of maintaining the whole ladder yourself? link.sc fetches any URL to clean markdown or JSON, JavaScript and anti-bot handled. Start free with 500 credits.