Quick answer: For small sites (roughly 50-150 pages), you can feed the whole thing directly: crawl every page, convert each one to clean markdown, concatenate, and paste it into the context window. A typical page converts to 1,000-3,000 tokens of markdown, so a 200k-token context fits somewhere between 60 and 150 real-world pages. Beyond that, you need RAG, and if the site publishes an llms.txt file you can skip the crawl entirely.
That's the whole strategy. The rest of this post is the math, the tradeoffs, and the code.
Start With the Token Math
People consistently overestimate how big websites are and underestimate how bloated HTML is. Both mistakes matter here.
Raw HTML is terrible input. A documentation page that reads as 800 words of actual content often ships 150KB of HTML: navigation, scripts, inline SVGs, cookie banners, footer links. Tokenized naively, that one page can eat 30,000+ tokens, most of it garbage.
The same page converted to markdown is usually 4-10KB. Call it 1,000-3,000 tokens depending on how text-dense the page is.
So the budget looks like this for a 200k context window:
| Content format | Tokens per typical page | Pages that fit in 200k |
|---|---|---|
| Raw HTML | 20,000-40,000 | 5-10 |
| Readability-extracted text | 1,500-4,000 | 50-130 |
| Clean markdown | 1,000-3,000 | 60-150 |
Two caveats. First, leave room for your actual prompt, the conversation, and the model's answer; budget maybe 80% of the window for site content. Second, these are estimates for typical docs and marketing pages, not guarantees. A 6,000-word technical deep dive is its own beast.
The practical takeaway: a docs site, a product site, or a small company blog usually fits. A news archive or an e-commerce catalog does not.
I wrote more about squeezing web content into fewer tokens in the token optimization post, and everything there applies double when you're stuffing an entire site in at once.
The Crawl → Markdown → Concatenate Pipeline
For sites that fit, the pipeline is three steps.
1. Enumerate the URLs. Check for /sitemap.xml first; most sites have one and it saves you from writing a crawler. If there's no sitemap, do a shallow BFS from the homepage, staying on-domain.
2. Fetch each page as markdown. This is where most DIY pipelines die, because "fetch a page" hides a pile of problems: JavaScript rendering, redirects, bot walls, boilerplate stripping. I use link.sc for this step since it returns clean markdown from any URL in one call:
curl "https://link.sc/v1/fetch" \
-H "Authorization: Bearer lsc_your_key" \
-H "Content-Type: application/json" \
-d '{"url": "https://docs.example.com/getting-started", "format": "markdown"}'
3. Concatenate with clear delimiters. Models handle multi-document context far better when each document is labeled with its source URL.
Here's the whole thing in Python:
import requests
import xml.etree.ElementTree as ET
API = "https://link.sc/v1/fetch"
HEADERS = {"Authorization": "Bearer lsc_your_key"}
def get_sitemap_urls(domain):
resp = requests.get(f"https://{domain}/sitemap.xml", timeout=15)
root = ET.fromstring(resp.content)
ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
return [loc.text for loc in root.findall(".//sm:loc", ns)]
def fetch_markdown(url):
resp = requests.post(API, headers=HEADERS,
json={"url": url, "format": "markdown"})
return resp.json()["markdown"]
def build_context(domain, max_tokens=160_000):
chunks, used = [], 0
for url in get_sitemap_urls(domain):
md = fetch_markdown(url)
est = len(md) // 4 # rough tokens-per-char estimate
if used + est > max_tokens:
break
chunks.append(f"<page url=\"{url}\">\n{md}\n</page>")
used += est
return "\n\n".join(chunks)
context = build_context("docs.example.com")
prompt = f"{context}\n\nUsing only the pages above, answer: ..."
The len(md) // 4 estimate is crude but fine for budgeting. If you need precision, run a real tokenizer.
Order and Prune Before You Concatenate
Not all pages deserve a seat. Before concatenating, I do two cheap things that noticeably improve answer quality.
Prune the junk. Sitemaps include tag pages, pagination, legal boilerplate, and old announcements. Filter by URL pattern (/tag/, /page/2, /privacy) before fetching. On most sites this cuts 30-50% of URLs at zero cost.
Put the important stuff first and last. Models attend more reliably to the beginning and end of long contexts. Homepage and core docs go first; reference material goes in the middle; the pages most relevant to your expected questions go last, right before the prompt.
When the Site Doesn't Fit: Switch to RAG
Once a site is bigger than your token budget, stop trying to cram. Chunk-level retrieval beats truncated full-site context every time, because a truncated corpus fails silently: the model just doesn't know about the pages you dropped.
The pipeline changes shape: crawl to markdown exactly as above, but instead of concatenating, chunk each page (500-1,000 tokens with overlap), embed the chunks, and retrieve the top handful per query.
I won't rebuild that tutorial here because we already have one: building a RAG pipeline with live web data walks through the full setup, including keeping the index fresh when the site changes.
My rough decision rule:
| Site size | Approach |
|---|---|
| Under ~100 pages | Full-site context, refetch on demand |
| 100-1,000 pages | RAG, re-crawl weekly |
| 1,000+ pages | RAG with incremental crawling and change detection |
The llms.txt Shortcut
Some sites now publish an /llms.txt file: a curated markdown index of their most important pages, written specifically for this use case. A growing number also publish /llms-full.txt, which inlines the actual content of those pages into one big markdown document.
When it exists, it's a gift. The site owner has already done your pruning and ordering for you:
curl -s https://docs.example.com/llms-full.txt | head -50
If llms-full.txt exists and fits your budget, you're done in one request. If only llms.txt exists, use its link list as your crawl frontier instead of the sitemap; it's a better-curated set.
The catch: adoption is spotty and quality varies wildly. Treat it as a fast path to check first, not a plan. Fall back to the sitemap pipeline when it's missing or stale, which is most of the time.
What This Costs
Worth being concrete. Feeding 100 pages into a 200k context means paying for ~150k input tokens every time you ask a question. On a frontier model that's real money per query, and it adds up fast in a chat loop.
Three ways to keep it sane:
- Prompt caching. Most providers discount repeated context heavily. Structure your prompt so the site content is a stable prefix and only the question changes.
- Cheaper models for extraction. Full-site context plus a small model often beats RAG plus a big model for "find the answer in these docs" tasks.
- Fetch once, ask many. Cache the concatenated markdown locally; only refetch pages that change.
On the fetching side, the crawl itself is cheap: link.sc's free tier covers 500 requests a month, which is several small-site crawls before you pay anything.
Where I'd Start
If I were doing this today for a docs site: check for llms-full.txt, fall back to sitemap → markdown → concatenate with the script above, and only reach for RAG once the token estimate blows past ~150k. Don't build the vector database until the math forces you to.
Need clean markdown from every page of a site? Grab a free link.sc API key and crawl your first site on 500 free credits.