← All posts

How to Find All URLs on a Website (Before You Crawl It)

GUIDES · JUL 19, 2026 · 5 MIN READ

Quick answer: Check https://example.com/sitemap.xml first (and robots.txt, which usually points to the sitemap). A sitemap can hand you every URL the site wants indexed in seconds, with no crawling. If there's no sitemap, or it's incomplete, run a link crawl: fetch the homepage, extract internal links, follow them, repeat until nothing new appears. Use Google's site:example.com operator and the Wayback Machine as cross-checks for pages the first two methods missed.

Enumerating a site's URLs is a different job from scraping it. Scraping answers "what does this page say?"; enumeration answers "what pages exist?" Getting the URL list right first is what turns a crawl from an open-ended walk into a bounded, resumable batch job. (For the broader distinction, see web crawling vs. web scraping.)

Here are the four methods, in the order you should try them.

Method 1: The Sitemap (Fastest, Try This First)

Most sites publish an XML sitemap listing the URLs they want search engines to know about. Common locations:

  • https://example.com/sitemap.xml
  • https://example.com/sitemap_index.xml
  • https://example.com/sitemap.xml.gz

Large sites usually serve a sitemap index: a sitemap of sitemaps, each child file holding up to 50,000 URLs. Here's a Python snippet that handles both flat sitemaps and indexes:

import requests
from xml.etree import ElementTree

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def get_urls(sitemap_url, seen=None):
    seen = seen if seen is not None else set()
    root = ElementTree.fromstring(requests.get(sitemap_url).content)
    # Child sitemaps (index file)
    for loc in root.findall("sm:sitemap/sm:loc", NS):
        get_urls(loc.text.strip(), seen)
    # Actual page URLs
    for loc in root.findall("sm:url/sm:loc", NS):
        seen.add(loc.text.strip())
    return seen

urls = get_urls("https://example.com/sitemap.xml")
print(len(urls), "URLs found")

Two caveats. First, sitemaps list what the site wants indexed, which is not always everything that exists: paginated archives, filtered category pages, and old content are often left out. Second, sitemaps can be stale. Treat the sitemap as your primary list, not your only one.

Method 2: robots.txt (The Map to the Map)

https://example.com/robots.txt is worth reading even though it doesn't list pages directly. It typically contains:

Sitemap: https://example.com/sitemap_index.xml
Sitemap: https://example.com/news-sitemap.xml
Disallow: /admin/
Disallow: /cart/

The Sitemap: lines point you to sitemaps you might not have guessed (news sitemaps, image sitemaps, per-language sitemaps). The Disallow: lines tell you which sections the site owner does not want crawled, which you should respect, and they also reveal the site's structure: a Disallow: /internal-search/ line tells you that path pattern exists.

Method 3: A Link Crawl (Complete, but You Do the Work)

When there's no sitemap, or you suspect it's missing sections, discover URLs the way search engines originally did: follow the links. Start at the homepage, collect every same-domain link, visit each new one, and keep going until the frontier is empty.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urldefrag, urlparse
from collections import deque
import time

START = "https://example.com/"
DOMAIN = urlparse(START).netloc

seen, queue = {START}, deque([START])

while queue:
    url = queue.popleft()
    try:
        resp = requests.get(url, timeout=10)
    except requests.RequestException:
        continue
    if "text/html" not in resp.headers.get("content-type", ""):
        continue
    soup = BeautifulSoup(resp.text, "html.parser")
    for a in soup.find_all("a", href=True):
        link = urldefrag(urljoin(url, a["href"]))[0]
        if urlparse(link).netloc == DOMAIN and link not in seen:
            seen.add(link)
            queue.append(link)
    time.sleep(0.5)  # be polite

print(len(seen), "URLs discovered")

The important details, all of which this snippet handles:

  • Normalize before deduplicating. Resolve relative links with urljoin, strip #fragments with urldefrag, and stay on the same domain. Without normalization your "unique" set fills with duplicates like /about, /about#team, and about/.
  • Throttle. One request every half second is a reasonable default. Go faster and you'll meet HTTP 429 rate limits or an outright block.
  • Bound the crawl. Add a max page count and a max URL depth in production. Calendar widgets and faceted filters can generate effectively infinite URL spaces (a "crawler trap"), and an unbounded crawl will happily walk into one.

One real limitation: requests only sees server-rendered HTML. If the site builds its navigation with JavaScript, the anchor tags may not exist in the raw response, and this crawl will come back nearly empty. In that case you need a fetch that renders JavaScript before extracting links.

Method 4: External Indexes (The Cross-Check)

Two outside sources know about URLs that neither the sitemap nor a fresh crawl will surface:

The site: operator. Searching site:example.com in Google or Bing shows pages the engine has indexed, including orphan pages that nothing links to anymore. Narrow it with paths (site:example.com/blog) or exclusions (site:example.com -inurl:tag). It's a manual spot-check, not an export: search engines cap what they'll show you and forbid scraping their results directly, though a SERP API gives you the same data programmatically.

The Wayback Machine CDX API. The Internet Archive will list every URL it has ever captured for a domain, including deleted pages:

https://web.archive.org/cdx/search/cdx?url=example.com/*&output=text&fl=original&collapse=urlkey

This is the best source for historical URLs and a surprisingly good completeness check: if the CDX list contains whole sections your crawl never found, you have a discovery gap.

A Practical Workflow

For a real project, combine the methods rather than picking one:

  1. Pull the sitemap(s) via robots.txt. This is your baseline list, usually 90 percent of the site in under a minute.
  2. Run a bounded link crawl and diff it against the sitemap. Pages in the crawl but not the sitemap are usually pagination and filters; pages in the sitemap but not the crawl are often orphaned or JavaScript-only routes.
  3. Spot-check with site: and the CDX API for anything both methods missed.
  4. Normalize and filter the merged list: lowercase the host, strip fragments and tracking parameters (utm_*), collapse trailing-slash duplicates, and drop paths disallowed by robots.txt.

The output is a clean, deduplicated URL list. From there, fetching every page is a straightforward batch job, and patterns like list crawling cover the extraction side.

The discovery step is usually easy to run from your laptop. The fetch step is where sites push back with rate limits, JavaScript rendering, and bot detection, and that's the part worth offloading to a managed fetch API so your enumeration list actually turns into content.

Have your URL list ready? Get a free link.sc API key and turn every URL into clean, LLM-ready content with one fetch call per page.