← All posts

How Do I Deduplicate Pages When Crawling?

ENGINEERING · JUL 18, 2026 · 6 MIN READ

Quick answer: Deduplicate twice. First at the URL level, before fetching: normalize case, trailing slashes, and query params so /page?utm_source=x and /page/ don't get crawled as separate pages. Then at the content level, after fetching: hash the extracted text to catch exact duplicates, and use SimHash or MinHash if you also need to catch near-duplicates. URL dedup saves you money; content dedup saves your dataset.

Duplicates are the silent tax on every crawl. You pay to fetch the same page five times, you pay to store it five times, and if the crawl feeds a RAG system, your retrieval results fill up with five copies of the same paragraph. The fix is cheap and mostly mechanical, so let's get into it.

Why the Same Page Has Ten URLs

The web does not agree with itself about what a URL is. All of these can serve identical content:

https://example.com/pricing
https://example.com/pricing/
https://example.com/Pricing
https://www.example.com/pricing
https://example.com/pricing?utm_source=newsletter
https://example.com/pricing?ref=footer
https://example.com/pricing#faq
http://example.com/pricing

A naive crawler that keys its "seen" set on the raw URL string will fetch all eight. On a site with tracking links in every footer and nav, I've seen crawls balloon to 3-4x their real page count this way. If you're planning a full-site crawl, URL normalization is the single highest-leverage thing you can add.

Step One: Canonicalize URLs Before You Fetch

The goal is a function that maps every variant of a page to one canonical string, and to run every discovered link through it before checking your seen-set.

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

TRACKING_PARAMS = {
    "utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content",
    "gclid", "fbclid", "msclkid", "mc_cid", "mc_eid", "ref", "src",
}

def canonicalize(url: str) -> str:
    scheme, netloc, path, query, _fragment = urlsplit(url.strip())

    scheme = "https" if scheme in ("http", "https") else scheme
    netloc = netloc.lower().removeprefix("www.")

    # Trailing slash: pick one convention and stick to it
    if path != "/" and path.endswith("/"):
        path = path.rstrip("/")

    # Drop tracking params, sort the rest for a stable order
    params = [(k, v) for k, v in parse_qsl(query, keep_blank_values=True)
              if k.lower() not in TRACKING_PARAMS]
    query = urlencode(sorted(params))

    return urlunsplit((scheme, netloc, path, query, ""))

A few judgment calls hiding in there, and I'll give you my defaults:

  • Fragments: always drop them. #faq never changes what the server returns.
  • Path case: do not lowercase the path by default. Domains are case-insensitive, paths legally aren't. Lowercase paths only if you've confirmed the site treats them the same.
  • Trailing slash: strip it. If the site actually serves different content at /page and /page/, it has bigger problems than your crawler.
  • Query params: strip known tracking params, keep everything else. ?page=2 and ?id=42 are load-bearing.

And one more source of truth worth respecting: rel=canonical. If a fetched page declares <link rel="canonical" href="...">, record your content under that URL. The site is telling you, explicitly, which variant is the real one. Trust it unless it points somewhere absurd like the homepage (some soft-error pages do exactly that).

Step Two: Hash Content to Catch Exact Duplicates

URL dedup can't catch everything, because sometimes genuinely different URLs serve the same content: a print version, a mirrored path, the same product under two categories. For those you hash the content after fetching.

The critical detail: hash the extracted article text, not the raw HTML. Raw HTML contains timestamps, CSRF tokens, and session IDs that make every response byte-unique. Two identical articles will never hash equal until you strip the boilerplate and hash what's left.

import hashlib

def content_key(markdown_text: str) -> str:
    normalized = " ".join(markdown_text.lower().split())
    return hashlib.sha256(normalized.encode()).hexdigest()

seen_content: dict[str, str] = {}  # hash -> first URL seen

def is_duplicate(url: str, markdown_text: str) -> bool:
    key = content_key(markdown_text)
    if key in seen_content:
        return True
    seen_content[key] = url
    return False

This is where fetching to markdown instead of HTML pays for itself twice: the output is already boilerplate-free, so identical articles produce identical text.

curl "https://link.sc/v1/fetch?url=https://example.com/pricing&format=markdown" \
  -H "Authorization: Bearer lsc_your_key"

Hash what comes back and you have a stable content fingerprint per page. The response format is documented at link.sc/docs.

Step Three: Near-Duplicates with SimHash

Exact hashing misses pages that are 95% identical: the same press release with a different city name, a product page with only the SKU changed. Whether you care depends on the use case, but when you do, the standard tools are MinHash and SimHash.

Both compress a document into a small fingerprint such that similar documents get similar fingerprints. SimHash is the simpler one to run at crawl time: 64 bits per document, and two documents are near-duplicates when their fingerprints differ in only a few bits.

import hashlib

def simhash(text: str, bits: int = 64) -> int:
    vector = [0] * bits
    tokens = text.lower().split()
    # shingles of 3 tokens work better than single words
    for i in range(max(len(tokens) - 2, 1)):
        shingle = " ".join(tokens[i:i + 3])
        h = int.from_bytes(hashlib.md5(shingle.encode()).digest()[:8], "big")
        for b in range(bits):
            vector[b] += 1 if (h >> b) & 1 else -1
    return sum(1 << b for b in range(bits) if vector[b] > 0)

def hamming(a: int, b: int) -> int:
    return (a ^ b).bit_count()

# hamming(simhash(doc1), simhash(doc2)) <= 3 means near-duplicate

For crawls up to the low hundreds of thousands of pages, brute-force comparison against recent fingerprints is honestly fine. Past that, you'll want the standard trick of indexing fingerprints in rotated bands so candidates can be found without comparing against everything. The datasketch library does this for MinHash if you'd rather not build it.

My advice: don't reach for near-dup detection on day one. Start with exact hashing, look at what survives, and add SimHash only if you can see real near-duplicates polluting your output.

URL Dedup or Content Dedup? Both, at Different Times

They solve different problems, and the timing matters:

URL dedup Content dedup
Runs Before fetching After fetching
Catches Same page, different URL string Same content, different real URL
Saves Bandwidth, credits, politeness budget Storage, index quality
Cost of a mistake Missed pages (too aggressive) Wasted fetches (too lax)

The failure modes point in opposite directions. Over-aggressive URL normalization silently drops real pages, so keep it conservative. Content dedup can afford to be stricter because by then you've seen the actual bytes.

What I'd Actually Ship

For a typical docs or content crawl feeding an LLM pipeline:

  1. Canonicalize every discovered link with the function above, conservative settings.
  2. Key the frontier on canonical URLs; never fetch the same canonical URL twice per crawl.
  3. Respect rel=canonical from fetched pages when recording results.
  4. Fetch to markdown, hash the normalized text, drop exact duplicates.
  5. Skip SimHash until you have evidence you need it.

That's maybe sixty lines of code, and it routinely cuts crawl size by a third while making the resulting dataset noticeably cleaner. Few things in scraping have that ratio of effort to payoff.

Want the fetching and cleanup handled so you can focus on the pipeline? link.sc turns any URL into clean, hashable markdown. 500 free credits a month, no card required.