Turn Any Docs Site Into an LLM Knowledge Base

Quick answer: To turn a docs site into an LLM knowledge base, discover its pages (sitemap or llms.txt), fetch each one as clean markdown, split the markdown into heading-based chunks, embed the chunks into a vector store, and retrieve the top matches at question time. The pipeline is maybe a hundred lines of code, and the markdown-fetching step is the part most people get wrong.

This is one of the most common real-world RAG projects: your team lives inside some tool, its docs are good, and you want an assistant that answers questions from those docs instead of hallucinating from training data.

Here's the whole pipeline, with working code and the gotchas that docs sites specifically throw at you.

Step 1: Discover the Pages

Docs sites are the friendliest crawl targets on the web. Try these in order:

1. llms.txt. Many docs sites now publish https://docs.example.com/llms.txt, a curated markdown list of their important pages. If it exists, your discovery step is literally "parse a markdown file." More on this shortcut later.
2. sitemap.xml. Nearly every docs platform (Docusaurus, Mintlify, GitBook, ReadTheDocs) generates one automatically.
3. Crawl the nav. Last resort. Docs navigation is usually a single sidebar, so a shallow crawl gets everything. I covered the full technique in how to crawl an entire website.

import requests
from xml.etree import ElementTree

def discover(base):
    root = ElementTree.fromstring(
        requests.get(f"{base}/sitemap.xml", timeout=15).content)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    return [loc.text for loc in root.findall(".//sm:url/sm:loc", ns)
            if "/docs/" in loc.text]

urls = discover("https://docs.example.com")

Filter to actual doc pages. You do not want the changelog archive, the careers page, or 400 auto-generated API stubs unless you actually need them.

Step 2: Fetch Every Page as Markdown

This step decides the quality of everything downstream. Raw HTML from a modern docs site is a nightmare of nav bars, cookie banners, sidebars, and footer links, and if you embed that noise, you retrieve that noise.

You want clean markdown: headings preserved, code blocks fenced, tables intact, chrome stripped. That's exactly what link.sc fetch returns:

import requests, time

def fetch_markdown(url, api_key):
    resp = requests.post(
        "https://link.sc/v1/fetch",
        headers={"Authorization": f"Bearer {api_key}"},
        json={"url": url, "format": "markdown"},
        timeout=60,
    )
    resp.raise_for_status()
    return resp.json()

docs = []
for url in urls:
    data = fetch_markdown(url, "lsc_your_key")
    docs.append({"url": url, "markdown": data["markdown"]})
    time.sleep(0.5)

This also handles the docs sites that render client-side, which is a lot of them these days. If you'd rather build the HTML-to-markdown step yourself, HTML to markdown for LLMs covers why this conversion matters so much and how to do it well.

Step 3: Chunk by Heading, Not by Character Count

Docs have structure, and heading-based chunking preserves it. A chunk that maps to "Configuring webhooks" retrieves far better than 1,000 arbitrary characters that start mid-sentence in one section and end mid-code-block in another.

import re

def chunk_by_heading(markdown, url, max_chars=4000):
    sections = re.split(r"\n(?=#{1,3} )", markdown)
    chunks = []
    for sec in sections:
        heading = sec.splitlines()[0].lstrip("# ") if sec else ""
        if len(sec) <= max_chars:
            chunks.append({"url": url, "heading": heading, "text": sec})
        else:
            for i in range(0, len(sec), max_chars):
                chunks.append({"url": url, "heading": heading,
                               "text": sec[i:i + max_chars]})
    return [c for c in chunks if len(c["text"].strip()) > 50]

Two docs-specific gotchas that generic chunkers get wrong:

• Code blocks. Never split inside a fenced code block. A half-code-sample chunk is worse than useless, because the model will confidently complete it incorrectly. If a section must be split, split at blank lines outside fences.
• Tables. A markdown table split across two chunks loses its header row in the second chunk, turning parameter tables into meaningless pipes. Keep tables whole, or repeat the header row in each piece.

The naive splitter above handles neither edge perfectly; for production, use a markdown-aware splitter or write the fence check yourself. The deeper theory is in what are chunkers.

Also: prepend the page title and heading to each chunk's text before embedding. "Webhooks > Retry policy" as a prefix massively improves retrieval for short sections.

Step 4: Embed and Retrieve

Any embedding model and vector store will do. Here's the shape with Chroma, which runs locally with zero setup:

import chromadb

client = chromadb.Client()
collection = client.create_collection("docs")

for i, c in enumerate(all_chunks):
    collection.add(
        ids=[str(i)],
        documents=[f"{c['heading']}\n\n{c['text']}"],
        metadatas=[{"url": c["url"]}],
    )

results = collection.query(
    query_texts=["how do I rotate an API key?"], n_results=5)

At answer time, stuff the top chunks into your LLM prompt with their source URLs and instruct the model to cite them. Citing URLs matters more than people think: it lets users verify answers, and it tells you which pages your knowledge base actually leans on.

This is a standard RAG setup at heart, and if you want the full architecture beyond docs sites, see building a RAG pipeline with live web data.

Step 5: Keep It Fresh

A knowledge base built once is a knowledge base that's wrong by October. Docs change quietly: new parameters, deprecated endpoints, renamed features.

My recommendation, in increasing order of effort:

Approach	How	Good for
Scheduled recrawl	Cron job re-runs the pipeline weekly or monthly	Most teams, most docs
Diff-based update	Compare sitemap lastmod or content hashes, re-embed only changed pages	Large docs sites, cost-sensitive
Fetch-at-answer-time	Skip the index for volatile pages, fetch live during the query	Pricing, status, changelogs

A weekly full recrawl is honestly fine for most docs sites. A few hundred fetches once a week is cheap, and it's dramatically simpler than change detection.

The llms.txt Shortcut

One more time, because it saves real work: check for llms.txt before building anything. Sites that publish it have already done your discovery and curation for you, listing their most important pages with descriptions. Some also publish llms-full.txt, the entire docs concatenated as one markdown file, which collapses steps 1 and 2 into a single fetch.

It's not universal and it's sometimes stale, so verify against the sitemap. But when it's there and maintained, your pipeline shrinks to: fetch one file, chunk, embed, done.

Total build time for all of this: an afternoon. The link.sc quickstart covers the fetch call in more detail, and the free tier's 500 monthly credits comfortably covers a small docs site including weekly refreshes.

---

Ready to turn a docs site into something your LLM can actually use? Grab a free link.sc API key and build the pipeline this afternoon.