How to Generate an llms.txt File Automatically (Free Method)

Writing an llms.txt file by hand is fine when your site has ten pages. It stops being fine around page fifty, when you are copying URLs, guessing at descriptions, and losing track of which docs you already listed.

So let's automate it. This post walks through generating a starter llms.txt file by crawling your own site, with a script you can run in about a minute. No paid tool required.

What "automatic" actually means here

A good llms.txt is not a dump of every URL you own. That is a sitemap, and your llms.txt should not be a sitemap. The whole point is curation: a short, human-readable index of your most useful pages with a one-line description on each.

So a generator cannot just scrape links and call it done. It needs to do three things:

1. Discover your important pages (usually from your sitemap).
2. Read each page and pull out a real title and summary.
3. Group everything into clean Markdown sections.

Step two is where most homegrown scripts fall apart. Fetching raw HTML gives you a soup of nav bars, cookie banners, and script tags. You need the actual content, converted to something an LLM can summarize. That is the part worth outsourcing.

The building blocks

Here is the approach in plain terms:

• Pull your URL list from https://yoursite.com/sitemap.xml.
• For each URL, fetch a clean Markdown version of the page.
• Take the first heading and first paragraph as the title and description.
• Write it all out as grouped Markdown.

For the fetching step I use the link.sc fetch API, because it returns clean Markdown instead of raw HTML. That means the summary you extract is real prose, not <div class="hero"> garbage. You can read more about why that matters in html to markdown for LLMs, but the short version is: clean input produces clean descriptions.

The script

Here is a working Python version. It reads your sitemap, fetches each page as Markdown, and prints a starter llms.txt to your terminal.

import os
import re
import requests
import xml.etree.ElementTree as ET

SITE = "https://yoursite.com"
API_KEY = os.environ["LINKSC_API_KEY"]

def get_urls(sitemap_url, limit=25):
    xml = requests.get(sitemap_url).text
    root = ET.fromstring(xml)
    ns = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    urls = [loc.text for loc in root.findall(".//s:loc", ns)]
    return urls[:limit]

def fetch_markdown(url):
    r = requests.post(
        "https://api.link.sc/v1/fetch",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"url": url},
    )
    return r.json().get("content", "")

def summarize(markdown):
    lines = [l.strip() for l in markdown.splitlines() if l.strip()]
    title = next((l.lstrip("# ") for l in lines if l.startswith("#")), "Untitled")
    body = next((l for l in lines if not l.startswith(("#", "-", ">", "|"))), "")
    desc = re.sub(r"\s+", " ", body)[:120]
    return title, desc

def build(site, urls):
    out = [f"# {site.split('//')[1]}", "", "> Auto-generated starter file. Edit before publishing.", "", "## Pages"]
    for url in urls:
        md = fetch_markdown(url)
        if not md:
            continue
        title, desc = summarize(md)
        out.append(f"- [{title}]({url}): {desc}")
    return "\n".join(out)

if __name__ == "__main__":
    urls = get_urls(f"{SITE}/sitemap.xml")
    print(build(SITE, urls))

Run it, redirect the output to a file, and you have a first draft:

python generate_llms.py > llms.txt

That is the automatic part. Now comes the part nobody likes to hear.

Automatic gets you 80 percent, not 100

A generated file is a draft, not a finished product. Treat the output as a scaffold you clean up, because a machine cannot make the editorial calls that make llms.txt useful.

Here is what you still need to do by hand:

Trim the list. The script grabs 25 URLs. You probably want your best 15. Cut the thin pages, the tag archives, the duplicate landing pages.

Group into real sections. The script dumps everything under one "Pages" heading. Split it into ## Documentation, ## Product, ## Guides, or whatever fits your site. Grouping is signal.

Fix the descriptions. Auto-extracted summaries are decent but generic. "We help teams work better" tells an AI nothing. "Real-time analytics for SaaS with sub-second latency" tells it a lot. This is the highest-value edit you can make.

Write the blockquote yourself. The one-sentence summary of your whole site under the H1 is too important to auto-generate. Write it like an elevator pitch for a machine.

If you want the full manual checklist, how to create your first llms.txt file covers the spec and the common mistakes in detail.

Why not just scrape the HTML yourself?

You can. But you will spend more time stripping boilerplate than you saved by automating. Marketing pages in particular are hostile to naive extraction: hero sections, testimonial carousels, and floating CTAs all look like content to a dumb parser.

The comparison, roughly:

Approach	Setup time	Description quality	Handles JS pages
Copy-paste by hand	Zero	Best (you wrote it)	Yes
Scrape raw HTML	High	Poor (boilerplate leaks in)	No
Fetch clean Markdown	Low	Good (real prose)	Yes

The middle row is the trap. It feels like progress because you wrote code, but the output needs so much cleanup that the hand method was faster. Fetching clean Markdown is what makes the automated path actually save time.

Does any of this move the needle?

Fair question, and I will not oversell it. Adoption of llms.txt is still early, and the data on whether crawlers respect it is mixed. We dug into that in does llms.txt actually work, and the honest answer is: it depends on who is reading.

But the cost of publishing one is now basically zero. Run the script, spend fifteen minutes editing, ship it. If AI assistants and search tools lean on these files more over the next year, you are already positioned. If they do not, you spent fifteen minutes. That is a good trade.

The short version

Generating llms.txt automatically is not about a magic button. It is about doing the boring 80 percent with a script and reserving your attention for the 20 percent that needs judgment: what to include, how to group it, and how to describe it.

Pull your sitemap, fetch each page as clean Markdown, extract a title and summary, then edit like a human. That is the whole workflow, and you can have a starter file live before your coffee gets cold.

---

Building your own llms.txt generator? Start with the link.sc fetch API and turn any page into clean Markdown in one call.