Quick answer: Don't recrawl everything on a fixed schedule. Rank pages by how often they actually change, use cheap signals (sitemap lastmod dates, ETag and If-Modified-Since headers, content hashes) to skip pages that haven't changed, and only reprocess the ones that have. A well-tuned incremental recrawl typically touches a small fraction of the pages a full recrawl would, for the same freshness.
The first crawl is the easy part. The uncomfortable truth about any crawled dataset, whether it's a price monitor or the knowledge base behind your RAG bot, is that it starts rotting the moment the crawl finishes. Pages change, pages die, new pages appear. Here's how I keep datasets fresh without paying full-crawl prices every week.
Full Recrawls Don't Scale, and Nightly Ones Are Worse
The naive strategy is a cron job that re-runs the whole crawl every night. It works at 200 pages. At 20,000 pages it's slow, expensive, and rude to the target site, and the punchline is that on a typical docs or content site, the vast majority of pages haven't changed since yesterday. You're paying full price to learn "nothing happened."
The insight that fixes this: freshness is a per-page property, not a per-dataset property. A changelog changes daily. A five-year-old blog post changes never. Treating them the same wastes effort on one and goes stale on the other.
Rank Pages by Change Frequency
The scheduling approach that search engines converged on, and that works just as well at small scale, is adaptive revisit intervals. Track when each page last changed, and adjust its revisit interval based on what you observe: page changed since last visit, revisit sooner; page unchanged, back off.
from datetime import datetime, timedelta, timezone
MIN_INTERVAL = timedelta(hours=6)
MAX_INTERVAL = timedelta(days=60)
def next_check(interval: timedelta, changed: bool) -> timedelta:
if changed:
return max(interval / 2, MIN_INTERVAL)
return min(interval * 2, MAX_INTERVAL)
def schedule(page: dict, changed: bool) -> dict:
interval = next_check(page["interval"], changed)
return {**page,
"interval": interval,
"due": datetime.now(timezone.utc) + interval}
Halve the interval on change, double it on no-change, clamp both ends. It's crude and it works: hot pages settle at your minimum interval, dead pages drift out to the maximum, and your daily recrawl queue shrinks to the pages actually worth visiting.
Seed new pages with a middling interval, say 3 days, and let the feedback loop sort them.
Let the Site Tell You What Changed
Before you fetch anything, check whether the site is already publishing change information. Two mechanisms are common and criminally underused.
Sitemap lastmod. Most sitemaps include a <lastmod> date per URL. Diff today's sitemap against yesterday's and you get a change feed for free, plus new and deleted URLs as a bonus:
import xml.etree.ElementTree as ET
import requests
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
def sitemap_entries(url: str) -> dict[str, str]:
root = ET.fromstring(requests.get(url, timeout=30).content)
out = {}
for u in root.findall("sm:url", NS):
loc = u.findtext("sm:loc", namespaces=NS)
lastmod = u.findtext("sm:lastmod", default="", namespaces=NS)
out[loc] = lastmod
return out
old = sitemap_entries_from_yesterday() # persisted from last run
new = sitemap_entries("https://example.com/sitemap.xml")
changed = [u for u, lm in new.items() if old.get(u) not in (None, lm)]
added = [u for u in new if u not in old]
removed = [u for u in old if u not in new]
One caveat from experience: some CMSes stamp every URL with the deploy date, which makes lastmod useless on those sites. Sanity-check it before trusting it (if every page "changed" at the same second, it's lying).
Conditional GET. HTTP has change detection built in. If the server sent an ETag or Last-Modified header, you can send it back and get a tiny 304 Not Modified response instead of the full page:
import requests
def fetch_if_changed(url: str, etag: str | None, last_modified: str | None):
headers = {}
if etag:
headers["If-None-Match"] = etag
if last_modified:
headers["If-Modified-Since"] = last_modified
r = requests.get(url, headers=headers, timeout=20)
if r.status_code == 304:
return None # unchanged, near-zero bandwidth
return {
"body": r.text,
"etag": r.headers.get("ETag"),
"last_modified": r.headers.get("Last-Modified"),
}
Support varies. Static sites and CDNs handle it beautifully; plenty of dynamic sites ignore it entirely. Costs nothing to send the headers, so send them always and treat 304s as a gift.
When Headers Lie, Hash the Content
Dynamic pages often return a fresh ETag on every request because the HTML contains timestamps, tokens, or rotating "related posts" widgets. The robust fallback is to compare content fingerprints yourself, computed over the extracted article text rather than raw HTML so that cosmetic churn doesn't register as change.
Fetching to markdown makes this almost trivial, because the boilerplate that churns is already gone:
curl "https://link.sc/v1/fetch?url=https://example.com/docs/api&format=markdown" \
-H "Authorization: Bearer lsc_your_key"
Hash the markdown, compare to the stored hash, and you have a change signal that ignores nav menus and ad slots. Store the hash alongside the ETag and interval:
import hashlib
def has_changed(markdown_text: str, stored_hash: str | None) -> tuple[bool, str]:
h = hashlib.sha256(" ".join(markdown_text.split()).encode()).hexdigest()
return (h != stored_hash, h)
If you only care about a handful of high-value pages rather than a whole corpus, you may not need any of this machinery: monitoring specific pages for changes and getting alerts is a simpler shape for that problem.
Keeping a Vector DB in Sync
If the crawl feeds a RAG system, freshness has a second half: the index. The failure mode I see constantly is append-only pipelines, where updated pages get re-embedded and added while the old chunks stay behind. Six months later the bot is answering from three generations of the same page, and the knowledge base you built from the docs site confidently quotes deprecated flags.
The fix is to make the page URL the unit of replacement:
- Give every chunk a deterministic ID like
sha256(url)_chunk3and store the URL and content hash in chunk metadata. - When a page's content hash changes, delete all chunks for that URL, then insert the new ones. Upsert alone is not enough, because a shorter new version leaves orphaned tail chunks.
- When a URL starts returning 404 or 410, or disappears from the sitemap, delete its chunks too. Deletions are changes.
def sync_page(url: str, markdown_text: str, store, embedder):
changed, new_hash = has_changed(markdown_text, store.get_hash(url))
if not changed:
return "skipped"
store.delete_by_url(url) # remove stale chunks first
for i, chunk in enumerate(chunk_text(markdown_text)):
store.insert(
id=f"{hash_url(url)}_c{i}",
embedding=embedder.embed(chunk),
metadata={"url": url, "content_hash": new_hash, "chunk": i},
)
store.set_hash(url, new_hash)
return "updated"
Notice the ordering: skip unchanged pages before you embed anything. Embedding is usually the most expensive step in the pipeline, and the content hash check means you only pay it for pages that actually changed.
The Loop, End to End
Putting it together, my standing recrawl loop looks like this:
- Daily: diff the sitemap. Queue changed, added, and removed URLs immediately.
- Continuously: pop pages whose adaptive revisit time is due.
- Per page: conditional GET first; on 200, extract to markdown and compare content hashes.
- On real change: reprocess, replace the page's chunks in the vector DB, halve the interval.
- On no change: double the interval and move on.
- On 404 or sitemap disappearance: delete from dataset and index.
None of these steps is clever on its own. Together they turn "recrawl 20,000 pages nightly" into "fetch a few hundred, reprocess a few dozen," and your dataset stays fresher than the nightly full crawl ever kept it, because hot pages get visited every few hours instead of every 24.
Building a pipeline that needs fresh web data on tap? link.sc fetches any URL to clean markdown, ready to hash, diff, and embed. Start with 500 free credits a month.