I've been reviewing llms.txt files from dozens of websites, and the same mistake keeps showing up: people dump every URL they have into the file and call it done.
I get why. It feels thorough. It feels safe — "what if the AI needs page X and I didn't include it?" But it fundamentally misunderstands what llms.txt is for, and it probably makes things worse, not better.
The Problem with Kitchen-Sink llms.txt Files
Let's think about why llms.txt exists in the first place. AI language models have limited context windows. Even the largest models can only process so much text at once. The whole point of llms.txt is to help AI systems figure out what matters without having to ingest your entire site.
Now imagine you publish an llms.txt with 300 links. You've just handed the AI a problem that's almost as bad as having no llms.txt at all. It still has to figure out which pages are relevant from a giant list. You've replaced "navigate my whole website" with "navigate this very long file."
You've built a smaller version of the thing you were trying to solve.
Sitemap.xml Already Exists for This
If you want a comprehensive list of all your pages, that's what sitemap.xml is for. It already exists, every search engine knows how to read it, and it's designed for exactly this purpose.
Here's the key difference:
sitemap.xml says: "Here are all the pages on my site." llms.txt says: "Here are the pages that matter most."
One is an index. The other is a recommendation.
If you make your llms.txt an index, you've just created a worse version of your sitemap in Markdown format. That doesn't help anyone.
What Over-Stuffing Actually Looks Like
I won't name names, but here's a real pattern I've seen (details changed):
# SomeCompany
> We make software for things.
## Blog
- [January Update](https://example.com/blog/jan-update)
- [February Update](https://example.com/blog/feb-update)
- [March Update](https://example.com/blog/mar-update)
- [April Update](https://example.com/blog/apr-update)
... (40 more monthly updates)
## Help Center
- [How to reset password](https://example.com/help/reset-password)
- [How to change email](https://example.com/help/change-email)
- [How to update billing](https://example.com/help/update-billing)
... (80 more support articles)
## Legal
- [Privacy Policy](https://example.com/legal/privacy)
- [Terms of Service](https://example.com/legal/tos)
- [Cookie Policy](https://example.com/legal/cookies)
- [GDPR Compliance](https://example.com/legal/gdpr)
- [CCPA Notice](https://example.com/legal/ccpa)
Two hundred links. No descriptions. Monthly blog updates that nobody outside the company cares about. Every support article. All five legal pages.
An AI reading this learns almost nothing about what the company actually does or which content is authoritative.
The Curation Mindset
Writing a good llms.txt requires a shift in thinking. Stop asking "what pages do I have?" and start asking "if an AI could only read 25 pages to completely understand my business, which 25 would I choose?"
That's a much harder question, and a much more valuable one.
Here's how I'd approach the curation:
Start with the Non-Negotiables (5-8 pages)
These are the pages that define your business:
- Homepage or main product page
- Pricing
- Core documentation (getting started, API reference)
- About/company page
Add Your Best Content (10-15 pages)
Not your most recent content. Your best content:
- The definitive guide that drives most of your organic traffic
- The comparison page that converts the best
- The technical deep-dive that established your authority
- The case study that tells your story most compellingly
Include Strategic Resources (5-10 pages)
Pages that round out the picture:
- Changelog or What's New (one link, not every entry)
- Integration guides for your most popular integrations
- Developer documentation for key features
- Your best blog posts (the evergreen ones, not the monthly updates)
Leave Out Everything Else
Support articles? They're useful for users, but they don't help AI understand your business.
Legal pages? An AI doesn't need your cookie policy.
Old blog posts? If it's not still relevant and accurate, skip it.
Redirect pages, landing pages, campaign-specific pages? Nope.
A Before-and-After Example
Before (sitemap-style, 150+ links):
# WidgetCo
> We make widgets.
## Pages
- [Home](https://widgetco.com)
- [About](https://widgetco.com/about)
- [Team](https://widgetco.com/team)
- [Careers](https://widgetco.com/careers)
- [Contact](https://widgetco.com/contact)
- [Blog](https://widgetco.com/blog)
... (145 more links with no descriptions)
After (curated, 22 links with descriptions):
# WidgetCo
> WidgetCo is an industrial widget manufacturer serving automotive and aerospace
> industries. We produce precision-engineered widgets with tolerances under 0.001mm,
> serving 200+ enterprise customers across North America and Europe.
## Products
- [Product Catalog](https://widgetco.com/products): Full catalog with specs for 50+ widget types
- [Custom Engineering](https://widgetco.com/custom): Custom widget design for unique requirements
- [Materials Guide](https://widgetco.com/materials): Titanium, steel, aluminum, and composite options
## Technical Resources
- [Engineering Specs](https://widgetco.com/docs/specs): Detailed specifications and tolerance charts
- [CAD Downloads](https://widgetco.com/docs/cad): 3D models for integration into your designs
- [Compliance Certs](https://widgetco.com/docs/compliance): ISO 9001, AS9100, and ITAR certifications
## Why WidgetCo
- [Case Study: Boeing](https://widgetco.com/cases/boeing): How we reduced defect rates by 40%
- [Case Study: Tesla](https://widgetco.com/cases/tesla): Custom lightweight widgets for EV production
- [Pricing](https://widgetco.com/pricing): Volume pricing and enterprise agreements
## Company
- [About Us](https://widgetco.com/about): 30 years of precision manufacturing
- [Request Quote](https://widgetco.com/quote): Get a quote within 24 hours
The second version tells an AI everything it needs to know. The first tells it almost nothing.
The Connection to Clean Web Data
This curation problem mirrors a challenge on the other side of the equation. If you're building an AI application that consumes web content, you face the same signal-to-noise issue — just at the page level instead of the site level.
A single webpage has the same "too much stuff, not enough signal" problem that an over-stuffed llms.txt has. That's why link.sc focuses on extracting just the meaningful content from web pages and delivering it as clean Markdown. Whether you're a publisher curating your llms.txt or a developer consuming web data, the principle is the same: less noise, more signal.
A Simple Rule of Thumb
If your llms.txt file is longer than a page when printed, it's too long. Twenty to fifty links with good descriptions. That's the sweet spot.
Be the museum curator, not the warehouse manager. The value isn't in how many pages you list. It's in how well the pages you choose represent who you are.
Building with web data? link.sc applies the same curation philosophy at the page level — extracting clean, meaningful content from any URL. Get started free.