Quick answer: Cache scraped pages by a normalized URL key, set a time-to-live based on how fast the content changes, and use conditional requests (ETag / If-Modified-Since) so the server can return a cheap 304 Not Modified instead of the full body. Store hot data in Redis, durable data in SQLite or on disk, and invalidate by TTL plus explicit purges. Done right, caching cuts your latency, your bill, and the load you put on other people's servers all at once.
The politest and cheapest request is the one you never make. If your scraper fetches the same page twice, you paid twice, waited twice, and hit someone else's server twice for no new information. A cache fixes all three. Here is how to build one that actually helps.
Why cache at all
Three wins, and they compound:
- Speed. A cache hit returns in microseconds instead of a network round trip.
- Cost. Every fetch you serve from cache is a request (or a credit) you did not spend.
- Politeness. Fewer requests to the origin means less load on the site you are collecting from. This is a real part of building a polite scraper.
For a crawl that revisits index pages, sitemaps, or reference data, the hit rate is often high enough that caching is the single biggest efficiency win available.
Build a good cache key
The cache key is the hard part, and most bugs live here. Two URLs that fetch the same content should produce the same key. Normalize before you hash.
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
import hashlib
# Params that never change the content. Tune per site.
TRACKING = {"utm_source", "utm_medium", "utm_campaign", "utm_term",
"utm_content", "gclid", "fbclid", "ref"}
def normalize_url(url):
parts = urlsplit(url.strip())
scheme = parts.scheme.lower()
host = parts.netloc.lower()
# drop tracking params, sort the rest for a stable order
query = [(k, v) for k, v in parse_qsl(parts.query)
if k.lower() not in TRACKING]
query.sort()
path = parts.path or "/"
return urlunsplit((scheme, host, path, urlencode(query), ""))
def cache_key(url):
return hashlib.sha256(normalize_url(url).encode()).hexdigest()
Normalizing lowercases the host, strips tracking parameters, sorts the query string, and drops the fragment. Now example.com/p?a=1&b=2&utm_source=x and EXAMPLE.com/p?b=2&a=1 map to one key instead of two. Be careful: some sites do use query params meaningfully (pagination, search terms), so keep those. Only strip params you are sure are noise.
Choose a TTL by content volatility
Not all pages age at the same rate. A stock quote is stale in seconds; an archived article barely changes in years. Set the TTL to match.
| Content type | Suggested TTL | Reason |
|---|---|---|
| Live prices, availability | Seconds to minutes | Changes constantly |
| News homepage, feeds | 5 to 15 minutes | Updated frequently |
| Product pages | Hours | Prices and stock shift, but not per second |
| Articles, docs, reference | Days to weeks | Rarely change after publish |
| Archived or historical pages | Months | Effectively immutable |
When in doubt, start conservative (a shorter TTL), measure how often the content actually changed, then lengthen it. A TTL that is too long serves stale data; too short and you lose the benefit. Conditional requests, below, let you cheat this tradeoff.
Use conditional requests
Even when a cached entry expires, the content may not have changed. Conditional requests let the server tell you that cheaply. Store the ETag and Last-Modified from the original response, then send them back as If-None-Match and If-Modified-Since. If nothing changed, the server returns 304 Not Modified with an empty body, and you reuse your cached copy.
import requests
def conditional_get(session, url, cached):
headers = {}
if cached and cached.get("etag"):
headers["If-None-Match"] = cached["etag"]
if cached and cached.get("last_modified"):
headers["If-Modified-Since"] = cached["last_modified"]
r = session.get(url, headers=headers, timeout=30)
if r.status_code == 304:
return cached["body"], cached # unchanged, reuse cache
if r.status_code == 200:
meta = {
"body": r.text,
"etag": r.headers.get("ETag"),
"last_modified": r.headers.get("Last-Modified"),
}
return r.text, meta
r.raise_for_status()
A 304 still costs a request, but it transfers almost no data and confirms freshness, which is often the ideal outcome for a page that changes unpredictably.
Pick a backend: disk, SQLite, or Redis
| Backend | Best for | Tradeoffs |
|---|---|---|
| On-disk files | Simple, large bodies, single machine | No shared access, manual eviction |
| SQLite | Structured metadata, single machine, durable | One writer at a time |
| Redis | Shared across workers, TTL built in, fast | Separate service, memory bound |
For a single-process scraper, SQLite is a sweet spot: one file, durable, and it stores the body plus ETag, Last-Modified, and fetch timestamp in one row.
import sqlite3, time, json
class SqliteCache:
def __init__(self, path="cache.db"):
self.db = sqlite3.connect(path)
self.db.execute("""CREATE TABLE IF NOT EXISTS cache (
key TEXT PRIMARY KEY, meta TEXT, fetched_at REAL)""")
def get(self, key, ttl):
row = self.db.execute(
"SELECT meta, fetched_at FROM cache WHERE key=?", (key,)
).fetchone()
if not row:
return None
meta, fetched_at = json.loads(row[0]), row[1]
meta["_fresh"] = (time.time() - fetched_at) < ttl
return meta
def put(self, key, meta):
self.db.execute(
"REPLACE INTO cache (key, meta, fetched_at) VALUES (?,?,?)",
(key, json.dumps(meta), time.time()))
self.db.commit()
If multiple workers share the cache, move to Redis so the TTL and eviction are handled for you and every worker sees the same entries.
Invalidation
Two ways an entry leaves the cache:
- TTL expiry. The simplest and safest default. When an entry is older than its TTL, treat it as stale and revalidate (ideally with a conditional request).
- Explicit purge. When you know a page changed (you posted it, a webhook fired, a source updated), delete the key so the next read refetches.
Resist the urge to build clever invalidation before you need it. TTL plus conditional revalidation covers the large majority of cases, and "the two hard things in computer science" is a joke about cache invalidation for a reason. Keep it boring.
Caching and a fetch API
If you fetch through link.sc, the same principles apply on your side: cache the clean markdown you get back, keyed by normalized URL, and you avoid spending credits to refetch a page you already have. The response body is already parsed, so your cache stores finished data rather than raw HTML you have to reprocess.
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/article", "format": "markdown"}'
Cache that markdown under cache_key("https://example.com/article") with a TTL that matches how often the article changes, and every repeat read is free. See the pricing page for how credits map to requests so you can size the savings.
A note on ethics
Caching is inherently respectful because it reduces load on the origin, but store responsibly. Do not cache and redistribute content you have no right to republish, honor any Cache-Control: no-store the site sends, and remember that a cached copy of personal data is still personal data under privacy rules. Cache to be efficient and polite, not to hoard content you were only licensed to read.
Cache the clean markdown, not the raw HTML: link.sc returns finished content you can store and reuse. Start free.