Quick answer: A sitemap is an XML file, usually at /sitemap.xml, that lists the URLs a site wants discovered, often with a lastmod date, a change frequency, and a priority. A sitemap index is a sitemap of sitemaps for large sites. To find one, check robots.txt for a Sitemap: line or try /sitemap.xml directly. For anyone crawling a site, the sitemap is the single best place to get a clean, authoritative list of URLs without guessing.
If you are about to crawl a website, do not start by following links blindly. Start with the sitemap. The site already published the list of pages it wants found, complete with hints about which ones changed recently. Ignoring that is like ignoring a map because you would rather wander.
What sitemap.xml Actually Is
A sitemap is an XML document following the sitemaps.org protocol. Each entry is a <url> block. Here is the shape of a minimal one:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/products/widget</loc>
<lastmod>2026-07-10</lastmod>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>
<url>
<loc>https://example.com/products/gadget</loc>
<lastmod>2026-07-15</lastmod>
</url>
</urlset>
The fields that matter to a crawler:
| Field | What it tells you |
|---|---|
<loc> |
The canonical URL of the page (required) |
<lastmod> |
When the page last changed (the useful one for freshness) |
<changefreq> |
A hint at update cadence: daily, weekly, etc. |
<priority> |
The site's own ranking of the URL, 0.0 to 1.0 |
<loc> and <lastmod> are the two you will actually use. Treat changefreq and priority as soft hints; sites set them inconsistently.
Sitemap Index Files
A single sitemap is capped at 50,000 URLs or 50 MB. Large sites split their URLs across many sitemaps and publish a sitemap index that points to all of them:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemap-products.xml</loc>
<lastmod>2026-07-15</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-blog.xml</loc>
<lastmod>2026-07-17</lastmod>
</sitemap>
</sitemapindex>
The tell is the root element: <sitemapindex> instead of <urlset>. When you fetch a sitemap, always check which one you got, because an index needs one more level of fetching to reach the actual URLs. The lastmod on an index entry is a handy shortcut: if a section's sitemap has not changed since your last crawl, you can skip re-fetching all of it.
How to Find a Sitemap
Three reliable methods, in the order I try them.
1. Check robots.txt. By convention, sites declare their sitemaps in robots.txt. Fetch https://example.com/robots.txt and look for lines like:
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml
This is the authoritative source; a site can point to sitemaps anywhere, and robots.txt is where it tells you where.
2. Try the default path. If robots.txt says nothing, guess the conventional locations: /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap.xml.gz. Many sites keep the default.
3. Check search-engine tooling. A site's sitemap is often referenced in its Google Search Console setup, but you will not have access to that for third-party sites, so methods 1 and 2 are your practical options.
Using lastmod for Freshness
The lastmod date is the reason sitemaps are worth parsing on every recrawl, not just the first. Instead of re-downloading a whole site to find what changed, you compare each URL's lastmod against the last time you crawled it and fetch only the pages that moved. On a large site this turns hours of recrawling into minutes.
The workflow is simple: store the lastmod you saw for each URL, and on the next run, queue a URL only if its lastmod is newer than your stored value (or if you have never seen it). This is the cheapest freshness signal on the web, and it is sitting right there in the file.
Parsing a Sitemap to Seed a Crawl
Here is a compact Python example that fetches a sitemap, handles the index case, and yields URLs with their lastmod:
import requests
import xml.etree.ElementTree as ET
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
def parse_sitemap(url):
xml = requests.get(url, timeout=30).content
root = ET.fromstring(xml)
tag = root.tag.split("}")[-1] # strip namespace
if tag == "sitemapindex":
for sm in root.findall("sm:sitemap", NS):
child = sm.find("sm:loc", NS).text
yield from parse_sitemap(child) # recurse into each sitemap
else: # urlset
for u in root.findall("sm:url", NS):
loc = u.find("sm:loc", NS).text
lastmod_el = u.find("sm:lastmod", NS)
lastmod = lastmod_el.text if lastmod_el is not None else None
yield loc, lastmod
for loc, lastmod in parse_sitemap("https://example.com/sitemap.xml"):
print(loc, lastmod)
That gives you a clean, deduplicated seed list of URLs plus freshness dates, which is a far better starting point than crawling link by link. Once you have the list, feeding it into a full crawl is the next step, and I walk through that in how to crawl an entire website.
A Note on Ethics
A sitemap is an invitation: the site is publishing the URLs it wants discovered. That does not override robots.txt disallow rules, which still apply to what you crawl. Use the sitemap to be efficient, not aggressive: it lets you skip unchanged pages and avoid hammering the site with blind discovery, which is exactly the polite behavior most operators want. Respect crawl-delay directives and keep your request rate reasonable even when the map makes it easy to go fast.
Turn Sitemap URLs into Clean Content with link.sc
A sitemap gives you the URLs; you still have to fetch and parse each one, which is where blocks, rendering, and messy HTML slow you down. Pipe your sitemap list into link.sc and each URL comes back as clean markdown or JSON, ready to store or index:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/products/widget", "format": "markdown"}'
Combine that with lastmod filtering and you have an efficient, freshness-aware pipeline: parse the sitemap, fetch only what changed, get clean content back. The free tier is 500 credits a month to build it out.
The Bottom Line
The sitemap is the most underused free resource in crawling. It hands you an authoritative URL list, tells you which pages changed with lastmod, and scales to millions of pages through index files. Find it in robots.txt or at /sitemap.xml, parse it (watching for the index case), and use lastmod to recrawl only what moved. Start every crawl here and you will do less work for better coverage.
Ready to turn a sitemap into a live data feed? link.sc fetches every URL to clean, structured content. Start free.