Your RAG pipeline is confidently answering questions from a pricing page that changed three weeks ago. Nobody noticed, because vector databases fail silently: stale embeddings still return high similarity scores, they just return them for content that no longer exists.
Most teams solve the scraping half of this problem and stop. They re-fetch pages on a schedule, land the new HTML in storage, and call it done. But the raw dataset and the vector index are two different systems, and refreshing one does nothing for the other. This post is about the second half: the embedding lifecycle.
Why Vector Staleness Is Sneakier Than Data Staleness
When a scraped dataset goes stale, you can see it. Timestamps are old, diffs are obvious, a spot check catches it.
When a vector index goes stale, everything still works. Queries return results. Scores look healthy. The retrieval step has no concept of "this chunk describes a product you discontinued in May." The old embedding is exactly as valid, geometrically, as the day you created it.
Staleness in a vector DB comes in three flavors, each needing different handling:
- Changed pages: the source updated, your vectors describe the old version.
- Deleted pages: the source 404s, your vectors describe something that no longer exists anywhere.
- Orphaned chunks: the page still exists but got shorter, and chunks 12 through 18 of the old version have no counterpart in the new one.
That third one is the killer. If you re-embed a page but only upsert the new chunks, the leftover tail of the old version stays in the index forever, and it will happily surface in retrieval.
Step 1: Detect Change Before You Re-Embed
Embedding is the expensive step, so gate it behind cheap change detection. The pattern that works:
Fetch, normalize, hash. Pull the page, convert it to clean Markdown, then hash the Markdown, not the raw HTML. Raw HTML churns constantly: CSRF tokens, timestamps in footers, rotating ad markup, session IDs in URLs. Two fetches of an unchanged page can produce different HTML byte-for-byte. Normalized Markdown strips almost all of that noise, so a changed hash actually means changed content.
This is one reason I run fetches through the link.sc fetch API for pipelines like this: it returns Markdown with the boilerplate already stripped, which makes content hashes stable enough to trust.
import hashlib, requests
def fetch_and_hash(url: str, api_key: str):
r = requests.post(
"https://api.link.sc/fetch",
headers={"Authorization": f"Bearer {api_key}"},
json={"url": url, "format": "markdown"},
)
content = r.json()["content"]
return content, hashlib.sha256(content.encode()).hexdigest()
Store the hash alongside the URL in a small state table (SQLite is fine at tens of thousands of URLs). On each refresh cycle, compare. Hash unchanged means skip everything downstream. In my experience this eliminates 80 to 95 percent of embedding work on a typical corpus, because most pages simply do not change between cycles.
HTTP conditional requests (ETag, Last-Modified) can short-circuit even the fetch, but treat them as an optimization, not the source of truth. Plenty of servers send ETags that change on every request, and Last-Modified headers lie in both directions.
Step 2: Diff at the Chunk Level, Not the Page Level
The page hash changed. Now what? The naive move is to re-embed the entire page. For small pages, honestly, just do that. Simplicity wins.
But for long documents, most edits touch one section. A 40-chunk documentation page where someone fixed a typo in the intro does not need 40 new embedding calls. The trick is to hash each chunk individually:
def chunk_states(chunks: list[str]) -> dict[str, str]:
return {
hashlib.sha256(c.encode()).hexdigest(): c
for c in chunks
}
old = set(previous_chunk_hashes) # from your state table
new = chunk_states(chunk(markdown))
to_embed = [new[h] for h in new.keys() - old] # changed or added
to_delete = old - new.keys() # stale
Anything in both sets is unchanged and needs no work. Anything only in the new set gets embedded and upserted. Anything only in the old set gets deleted.
One caveat: this only works if your chunker is deterministic. If chunk boundaries shift every run because the splitter depends on token counts that wobble, every chunk hash changes and you re-embed everything anyway. Fixed, structure-aware splitting (by heading, then by paragraph) keeps boundaries stable across runs. I covered chunking strategy in more depth in the chunking guide.
Step 3: Upsert With IDs You Can Actually Delete By
Here is the design decision that determines whether cleanup is easy or miserable: your vector IDs must encode enough to find every chunk belonging to a URL.
The scheme I use is content-addressed IDs plus a URL in metadata:
- Vector ID:
sha256(url + chunk_text)truncated to something sane. - Metadata:
{"url": ..., "content_hash": ..., "fetched_at": ..., "chunk_index": ...}.
With that in place, the update for a changed page is a three-part transaction:
- Embed the new and changed chunks, upsert them.
- Delete every vector whose hash is in the stale set.
- Update the page hash in your state table, last.
Order matters. Update the state table only after the index writes succeed. If the process dies mid-run, the worst case is that you redo work on the next cycle, which is recoverable. If you update the state table first and then crash, the page reads as "done" while the index still holds the old vectors, and no future cycle will ever fix it.
Avoid positional IDs like url#3. They look tidy until a page shrinks from 20 chunks to 12 and you must remember to delete IDs 12 through 19, or an insertion at the top renumbers every chunk below it and forces a full re-embed of unchanged content.
Step 4: Handle Deletions and Dead Pages
Pages die. If your refresh fetch returns a 404 or 410, delete every vector with that URL in metadata. Most vector databases (Pinecone, Qdrant, Weaviate, pgvector) support metadata-filtered deletes, which is exactly what that URL field is for.
Two edge cases worth deliberate handling:
| Response | What it usually means | What to do |
|---|---|---|
| 404 / 410 | Page is gone | Delete vectors immediately |
| 5xx / timeout | Server is having a bad day | Keep vectors, retry with backoff |
| 200 with near-empty content | Soft 404 or bot wall | Keep vectors, flag for review |
That last row matters. A "page not found" served with a 200 status, or a CAPTCHA page, hashes as "changed content." Re-embed it blindly and you replace real knowledge with the text of an error page. A minimum-content-length check is cheap insurance.
Scheduling: Not Every Page Deserves the Same Cadence
A uniform "re-check everything daily" schedule wastes fetches on pages that change yearly and still misses pages that change hourly. Tier your corpus:
- Hot (pricing, changelogs, news): check every few hours.
- Warm (docs, product pages): daily.
- Cold (about pages, old blog posts): weekly or on demand.
You can bootstrap tiers from observed behavior: track how often each URL's hash actually changes and promote or demote accordingly.
If you are building the full loop from fetch to index, the real-time RAG pipeline guide covers the ingestion side, and once your refresh loop is running you should verify it is actually helping with proper RAG evaluation. A freshness pipeline that quietly degrades retrieval quality is worse than none, because you trust it.
The Whole Loop in One Place
- Fetch the page as normalized Markdown.
- Hash it. Unchanged: stop here.
- Chunk deterministically, hash each chunk.
- Embed only new and changed chunks, upsert with content-addressed IDs.
- Delete stale chunk vectors by hash, and all vectors for dead URLs.
- Commit the new state, after the index writes succeed.
None of this is exotic. It is a couple hundred lines of glue code and a state table. But it is the difference between a vector database that reflects the web as it is and one frozen on launch day.
Need clean, stable Markdown as the input to your embedding pipeline? Get a free link.sc API key and start fetching.