I spent a weekend reading through dozens of real llms.txt files from companies like Cloudflare, Stripe, Anthropic, Vercel, and Zapier. Some are excellent. Others are clearly auto-generated junk.
Here's what I learned about what separates a useful llms.txt from a wasted file.
Practice 1: Write Descriptions That Actually Describe
This sounds obvious, but most llms.txt files get this wrong. Look at the difference:
Bad:
- [Docs](https://example.com/docs): Documentation
Good:
- [API Reference](https://example.com/docs/api): REST API with 40+ endpoints for user management, billing, and analytics
The first link tells an AI nothing it couldn't infer from the URL. The second gives it real signal. An AI model reading the good version immediately knows the scope, the type (REST), and the domains covered (users, billing, analytics).
Every link description is a chance to convey information. Don't waste it by restating the obvious.
Practice 2: Curate Ruthlessly
The best llms.txt files I found had between 15 and 40 links. The worst had 200+.
This isn't a sitemap. You're not trying to list everything. You're answering the question: "If an AI could only read 30 pages to understand my entire business, which 30 would I pick?"
Cloudflare's llms.txt is a good example of disciplined curation. They have thousands of documentation pages, but their llms.txt focuses on the core developer docs, the most popular guides, and their key product pages. An AI reading it gets a clear, complete picture without drowning in noise.
Practice 3: Group by User Intent, Not Site Architecture
I see this mistake constantly: people organize their llms.txt to mirror their site navigation. But your site nav is designed for humans clicking around. Your llms.txt should be organized for an AI trying to understand your business.
Organized by site structure (less useful):
## /docs
## /blog
## /company
## /legal
Organized by intent (more useful):
## What We Do
## Getting Started
## Technical Reference
## Pricing and Plans
Think about what questions an AI might need to answer about your business, and structure your sections to match those questions.
Practice 4: Keep Your Summary Blockquote Tight and Specific
The blockquote right after your H1 is prime real estate. Some of the best implementations nail it:
Anthropic's approach (paraphrased): Specific about what they build, who it's for, and what makes them different. No fluff.
A weaker approach: "We're a technology company dedicated to innovation and helping businesses succeed." That sentence could describe literally any company on earth.
Your blockquote should pass this test: could a stranger read just this paragraph and accurately explain what your company does to someone else? If not, rewrite it.
Practice 5: Keep Links Fresh
A stale llms.txt with broken links or outdated pages is worse than not having one at all. It actively misinforms AI models about your site.
Set a quarterly reminder to review your file. Specifically check for:
- Broken links: Pages you've removed or restructured
- Outdated content: Blog posts or docs that are no longer accurate
- Missing additions: New important pages you've published since last update
- Description accuracy: Have any pages changed enough that their descriptions are wrong?
Practice 6: Don't Conflict with robots.txt
This one bites people more than you'd expect. If your robots.txt blocks AI crawlers from /docs/, don't link to /docs/ pages in your llms.txt. That's like locking a door and then handing out maps to it.
Audit both files together. Every URL in your llms.txt should be accessible to the bots you're trying to reach.
Practice 7: Include Your Best Content, Not Your Newest
There's a temptation to make llms.txt a "latest content" feed. Resist it. Your llms.txt should contain your evergreen, definitive content — the pages that best represent your expertise regardless of when they were published.
If you wrote the definitive guide to API rate limiting two years ago and it's still accurate, that belongs in your llms.txt more than last week's product update.
Practice 8: Consider Multiple Files
Some companies are experimenting with multiple llms.txt files in subpaths:
/llms.txt— Main company overview/docs/llms.txt— Documentation-specific guide/blog/llms.txt— Blog content guide
This makes sense for large sites with distinct content areas. If your documentation alone has hundreds of pages, a dedicated docs-level llms.txt can provide more focused guidance.
Practice 9: Test with Actual AI Models
Here's a practical testing approach I like: paste your llms.txt content into Claude or ChatGPT and ask:
- "Based on this, what does this company do?"
- "What would you recommend for someone getting started?"
- "What's this company's strongest content area?"
If the answers are wrong or vague, your llms.txt needs work. If the AI nails it, you've done a good job.
Practice 10: Think About What You're Not Including
This is subtle but important. By curating what goes into your llms.txt, you're also implicitly de-prioritizing everything else. That's a feature, not a bug.
If you have legacy documentation, outdated blog posts, or pages that don't represent your current product accurately, keeping them out of llms.txt is a deliberate choice that helps AI models form a more accurate picture of who you are today.
How This Connects to Web Data Workflows
If you're on the other side of this — building AI systems that consume web content — you face the same challenge from a different angle. You need clean, structured content, and you can't rely on every site having a well-crafted llms.txt.
That's exactly why link.sc exists. Whether a site has an llms.txt or not, the link.sc API fetches any URL and returns clean Markdown that's ready for LLM consumption. It's the programmatic version of what llms.txt tries to achieve manually: making web content genuinely useful for AI.
Building AI applications that need web data in a clean, usable format? link.sc converts any webpage to structured Markdown in milliseconds. Start building free.