How to Crawl an Entire Website and Extract Every Page

Quick answer: To crawl an entire website, start with its sitemap.xml, which often hands you every URL for free. If there's no usable sitemap, run a breadth-first crawl: fetch the homepage, extract same-domain links, normalize and dedupe them, and repeat with a depth limit, a page cap, and a polite request rate. Then fetch each discovered URL and extract the content you need.

People usually ask this as two questions: "how do I get every page?" and "should I map all the URLs first?" The answer to the second one is yes, almost always. Mapping first means you know the size of the job before you commit to it, and it separates the cheap step (discovering URLs) from the expensive step (fetching and processing every page).

Here's the whole pipeline, in the order I'd actually do it.

Step 1: Check sitemap.xml Before You Write Any Crawler

Most sites that care about SEO publish a sitemap, and it's the cheapest site map you will ever get. Check these, in order:

• https://example.com/sitemap.xml
• https://example.com/sitemap_index.xml
• https://example.com/robots.txt (look for a Sitemap: line)

A sitemap index points to child sitemaps, so you may need one level of recursion:

import requests
from xml.etree import ElementTree

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def sitemap_urls(url):
    root = ElementTree.fromstring(requests.get(url, timeout=15).content)
    if root.tag.endswith("sitemapindex"):
        urls = []
        for loc in root.findall(".//sm:sitemap/sm:loc", NS):
            urls.extend(sitemap_urls(loc.text.strip()))
        return urls
    return [loc.text.strip() for loc in root.findall(".//sm:url/sm:loc", NS)]

urls = sitemap_urls("https://example.com/sitemap.xml")
print(f"{len(urls)} URLs discovered")

If this works, you can skip crawling for discovery entirely and go straight to fetching. Sitemaps do go stale and some omit whole sections, so treat the result as "most of the site" rather than gospel.

Step 2: BFS Crawling When There's No Sitemap

No sitemap, or an obviously incomplete one? Breadth-first search is the standard approach: process all pages at depth 0, then depth 1, and so on. BFS beats depth-first here because the important pages on most sites live within two or three clicks of the homepage.

Two limits are non-negotiable or you will regret it: a max depth (3 to 4 covers most sites) and a max page count (a hard ceiling so a calendar widget or faceted search can't trap you into crawling infinite generated URLs).

Step 3: Normalize and Dedupe URLs

This is where naive crawlers die. https://example.com/about, https://example.com/about/, https://example.com/about?utm_source=x, and https://example.com/about#team are all the same page, and if you don't normalize, you'll crawl it four times and your "10,000 page site" turns out to be 900 pages wearing costumes.

Minimum viable normalization: lowercase the scheme and host, strip fragments, strip tracking parameters, and settle on one trailing-slash policy.

from urllib.parse import urlparse, urlunparse, parse_qsl, urlencode

TRACKING = {"utm_source", "utm_medium", "utm_campaign", "utm_term",
            "utm_content", "gclid", "fbclid", "ref"}

def normalize(url):
    p = urlparse(url)
    query = urlencode([(k, v) for k, v in parse_qsl(p.query)
                       if k not in TRACKING])
    path = p.path.rstrip("/") or "/"
    return urlunparse((p.scheme.lower(), p.netloc.lower(),
                       path, "", query, ""))

Keep a visited set of normalized URLs and check it before queueing anything.

Step 4: Be Polite, on Purpose

Crawling an entire site means sending that site a lot of requests, so the politeness rules stop being optional niceties.

• Rate limit. One request per second per domain is a reasonable default. Faster than that on a small site and you're effectively load-testing someone else's server without asking.
• Respect robots.txt. Python's urllib.robotparser makes this a few lines. Skip disallowed paths and honor Crawl-delay if present.
• Identify yourself. A real User-Agent with a contact URL, so a site owner can reach you instead of just banning you.
• Back off on errors. A 429 or 503 means slow down, not retry harder.

If you're unsure where the ethical lines sit, we covered them properly in ethical web scraping best practices.

Step 5: A Complete Crawl Sketch

Here's the whole thing assembled: BFS, normalization, dedupe, rate limiting, and limits.

import time, requests
from collections import deque
from urllib.parse import urljoin, urlparse
from bs4 import BeautifulSoup

def crawl(start, max_depth=3, max_pages=500, delay=1.0):
    domain = urlparse(start).netloc.lower()
    queue = deque([(normalize(start), 0)])
    visited, results = set(), []

    while queue and len(results) < max_pages:
        url, depth = queue.popleft()
        if url in visited or depth > max_depth:
            continue
        visited.add(url)

        try:
            resp = requests.get(url, timeout=15, headers={
                "User-Agent": "MyCrawler/1.0 (+https://mysite.com/bot)"})
        except requests.RequestException:
            continue
        time.sleep(delay)

        if "text/html" not in resp.headers.get("content-type", ""):
            continue
        results.append(url)

        soup = BeautifulSoup(resp.text, "html.parser")
        for a in soup.find_all("a", href=True):
            link = normalize(urljoin(url, a["href"]))
            if urlparse(link).netloc.lower() == domain and link not in visited:
                queue.append((link, depth + 1))
    return results

pages = crawl("https://example.com")
print(f"Crawled {len(pages)} pages")

Roughly 40 lines, and it will genuinely map most static sites. Note what it doesn't do: render JavaScript, retry failures, run in parallel, or extract content. That's the gap between a sketch and a production crawler.

The JavaScript Problem

If a site renders its navigation client-side (React, Vue, and friends), requests gets HTML with no links in it, and your crawl finds three pages on a three-hundred-page site.

You'll know it's happening when the crawl result is suspiciously tiny. The fix is rendering pages in a headless browser before extracting links, which is slower and heavier; the tradeoffs are covered in scraping JavaScript-rendered websites.

When to Use a Crawl API Instead

Everything above is discovery. You still have to fetch every discovered page, render the JavaScript-heavy ones, get past whatever anti-bot layer exists, and turn raw HTML into something usable.

That second half is where a hosted service earns its keep. With link.sc, you map URLs with the sitemap trick above, then fetch each page as clean markdown with one call per URL:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_your_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/docs/setup", "format": "markdown"}'

Rendering, blocking, and HTML cleanup are handled on the other side of the API. My honest rule of thumb: write your own crawler when crawling is the product, and use an API when the content is the product. Most teams are in the second group, and the quickstart will get you to fetched markdown in a couple of minutes.

One last distinction worth keeping straight: crawling is discovering pages, scraping is extracting data from them. If that split is fuzzy, web crawling vs web scraping untangles it.

---

Want every page of a site as clean markdown without babysitting a crawler? Get a free link.sc API key and start fetching.