Quick answer: Treat a 404 as data, not a failure. Log it once with the page that linked to it, do not retry it (a 404 is a definitive answer, unlike a 429 or 503), and check for soft-404s where the server returns 200 but the page is actually an error page. The one thing you must never do is let 404 handling raise an exception that stalls the rest of the crawl.
Every crawl of a real website hits 404s. I have never seen a crawl of more than a few hundred pages that didn't. If your pipeline treats them as errors that need fixing, you'll spend your life fixing things that aren't broken. If it ignores them entirely, you'll miss real problems like a deleted docs section or a botched migration. This post is about the middle path.
Why 404s Show Up in Crawls at All
If you're crawling by following links, every 404 you hit means some page on the site links to a URL that doesn't exist. That happens for boring reasons:
- Stale internal links. Someone renamed a page and missed a link in an old blog post.
- Migrations without redirects. The site moved from
/docs/v1/...to/docs/...and nobody set up 301s. - Templated links to optional content. A theme links to
/feed.xmlor/sitemap.xmlon every page whether or not it exists. - Your own URL construction. If you're generating URLs (pagination, ID enumeration), the 404s are often your fault, not the site's.
- Case and trailing-slash mismatches.
/Aboutvs/abouton a case-sensitive server.
The distinction that matters: a 404 from a discovered link is information about the site. A 404 from a URL you constructed is information about your crawler. Log them differently.
Real 404s vs Soft-404s
Here's the part that bites people. Plenty of sites return HTTP 200 with an error page in the body. Google calls these soft-404s, and they are poison for data pipelines because your crawler happily saves "Oops, we couldn't find that page" as if it were content. If you're building an LLM knowledge base from a docs site, one soft-404 indexed into your vector DB means your bot will someday cite "Page not found" as an authoritative source.
Detection is heuristic, but a few signals catch most of them:
| Signal | What to check |
|---|---|
| Title text | "not found", "404", "error", "oops" in <title> |
| Body length | Suspiciously short body compared to the site's median page |
| Canonical/redirect | Page canonicalizes or meta-refreshes to the homepage |
| Fingerprint match | Body hash equals the hash of a known-bad URL |
That last one is the most reliable trick I know: request a URL that definitely doesn't exist, like /definitely-not-a-real-page-xyz123, hash the response body, and flag any crawled page whose body hash matches. Sites that soft-404 usually serve the exact same error page every time.
import hashlib
import requests
def error_fingerprint(base_url: str) -> str:
"""Fetch a guaranteed-missing URL and fingerprint the error page."""
r = requests.get(f"{base_url}/definitely-not-a-real-page-xyz123", timeout=15)
return hashlib.sha256(r.content).hexdigest()
def is_soft_404(body: bytes, fingerprint: str, title: str) -> bool:
if hashlib.sha256(body).hexdigest() == fingerprint:
return True
bad_phrases = ("not found", "404", "no longer exists", "oops")
return any(p in title.lower() for p in bad_phrases) and len(body) < 5000
It's not perfect. A legitimate page titled "Fixing 404 errors" will trip the title check, which is why I combine signals instead of trusting one.
When to Retry and When to Drop
My rule: never retry a plain 404. It's a definitive statement from the server that the resource doesn't exist. Retrying it three times with backoff just wastes requests and politeness budget.
The statuses worth retrying are the transient ones:
| Status | Meaning | Retry? |
|---|---|---|
| 404 | Not found | No. Log and move on |
| 410 | Gone permanently | No. Stronger than 404, drop it |
| 429 | Rate limited | Yes, with backoff (full guide here) |
| 500, 502, 503 | Server trouble | Yes, 2-3 attempts with backoff |
| Timeout / connection reset | Network flake | Yes, 2-3 attempts |
One nuance: some anti-bot setups return 404 to clients they don't like while serving the page fine to browsers. If a URL 404s for your crawler but loads in your browser, that's not a missing page, it's a blocking problem, and you should read up on figuring out which fetching method a site needs instead of retrying blindly.
Log Them Like You Mean It
A 404 log line that just says 404 /pricing-old is nearly useless. The question you'll actually need to answer later is "what links to this?" So record the referrer at discovery time:
import csv
from collections import defaultdict
broken = defaultdict(set) # url -> set of pages linking to it
def record_404(url: str, found_on: str, kind: str = "hard"):
broken[url].add((found_on, kind))
def write_report(path: str = "broken_links.csv"):
with open(path, "w", newline="") as f:
w = csv.writer(f)
w.writerow(["broken_url", "found_on", "kind"])
for url, sources in sorted(broken.items()):
for found_on, kind in sorted(sources):
w.writerow([url, found_on, kind])
That CSV is genuinely valuable output. If you're crawling your own site, it's a broken-link report. If you're crawling someone else's docs for a knowledge base, it tells you which sections are rotting so you can weight your trust accordingly.
Don't Let 404s Stall the Crawl
The most common 404-related bug I see isn't misclassification, it's crawlers that raise on non-200 responses and take the whole run down with them. response.raise_for_status() inside a loop with no try/except has killed more overnight crawls than any anti-bot system.
Structure the loop so every URL gets exactly one verdict: content, broken, or retry-later. Nothing throws past the loop body.
import requests
session = requests.Session()
def crawl_one(url: str, found_on: str, fingerprint: str):
try:
r = session.get(url, timeout=20)
except requests.RequestException:
return ("retry", url)
if r.status_code in (404, 410):
record_404(url, found_on, kind="hard")
return ("dropped", url)
if r.status_code in (429, 500, 502, 503):
return ("retry", url)
if r.ok:
title = extract_title(r.text)
if is_soft_404(r.content, fingerprint, title):
record_404(url, found_on, kind="soft")
return ("dropped", url)
return ("content", r.text)
return ("dropped", url)
If you'd rather not maintain the retry-and-classify machinery yourself, a fetch API does the transient-error handling for you and returns a clean status either way:
curl "https://link.sc/v1/fetch?url=https://example.com/maybe-missing" \
-H "Authorization: Bearer lsc_your_key"
You still check the status in the response, but timeouts, TLS weirdness, and rendering are no longer your problem. The fetch endpoint docs cover the response shape.
Set a Budget, Then Stop Worrying
Last opinion: decide up front what 404 rate is acceptable and alert only when you cross it. On most sites, 1-3% of discovered links being broken is completely normal. I set an alert at around 10% of a crawl returning 404, because that level usually means something structural: a migration, a section deletion, or my own URL construction going wrong. Below that threshold, the CSV report is enough.
404s are weather, not emergencies. Build the crawler that records them accurately and keeps walking.
Crawling at scale and tired of babysitting error handling? link.sc fetches any URL to clean markdown with retries and rendering handled for you. Free tier includes 500 credits a month.