If you've been anywhere near the SEO or AI community in the last year, you've probably heard someone mention llms.txt. Maybe a colleague dropped it in a Slack channel. Maybe you saw a tweet about Cloudflare adding one. Maybe you just Googled "how to make my website AI-friendly" and ended up here.
Whatever brought you, let's talk about what this file actually is — no hype, no jargon walls.
The One-Sentence Version
An llms.txt file is a Markdown document you put in your website's root directory that tells AI language models which pages on your site matter most and what your site is about.
That's it. It's not magic. It's not a silver bullet. It's a curated cheat sheet for machines.
Where Did This Come From?
The idea was proposed at llmstxt.org as a way to solve a real problem: LLMs are terrible at navigating websites. They can't click menus. They struggle with JavaScript-heavy pages. They choke on ads and navigation chrome.
Think about what happens when you ask ChatGPT or Claude about a product. The model either knows about it from training data (which might be months old) or it tries to scrape pages in real time. Either way, it's working with messy, incomplete information.
An llms.txt file says: "Hey, here's what this site is about, and here are the pages you should actually read."
What Does It Look Like?
It's surprisingly simple. Here's a bare-bones example:
# My SaaS Product
> A web API that fetches and searches web content, optimized for feeding data to large language models.
## Docs
- [Getting Started](https://example.com/docs/quickstart): Quick setup guide
- [API Reference](https://example.com/docs/api): Full endpoint documentation
## Blog
- [How We Handle Rate Limiting](https://example.com/blog/rate-limits): Our approach to fair usage
That's a valid llms.txt file. An H1 with your site name, a blockquote summary, and organized links with descriptions. Markdown. Nothing fancy.
Who's Actually Using This?
Some pretty big names, surprisingly:
- Cloudflare uses it to surface their developer docs
- Stripe has one pointing to their API reference
- Anthropic (the company behind Claude) uses it for their documentation
- Vercel, Coinbase, and hundreds of other companies have adopted it
As of late 2025, over 800,000 websites had published an llms.txt file. That number keeps growing.
The Honest Truth About Whether It Works
Here's where I have to be straight with you: the evidence is mixed.
On one hand, no major AI provider has officially confirmed they read llms.txt files during inference. Google's Gary Illyes even compared it to the old keywords meta tag — a standard that websites adopted eagerly but search engines mostly ignored.
On the other hand, the spec is gaining traction. Google included llms.txt in their Agent-to-Agent (A2A) protocol. More AI tools are being built with llms.txt awareness. And the cost of adding one is basically zero.
My take? It's a five-minute investment. If AI systems start reading it tomorrow, you're already there. If they don't, you've lost nothing but five minutes.
Where link.sc Fits In
If you're building applications that need to fetch web content for AI models, you know firsthand how messy raw HTML is. That's the same problem llms.txt tries to solve from the publisher side.
With link.sc's fetch API, you can grab any page as clean Markdown — essentially doing what llms.txt does, but on demand and for any website. It's the other side of the same coin: llms.txt helps AI find your content, and tools like link.sc help AI read everyone else's.
Should You Add One to Your Site?
If you have a website — especially one with documentation, a blog, or product pages — yes. Here's my reasoning:
- It takes five minutes. The effort-to-potential-upside ratio is absurd.
- It forces you to think about your content hierarchy. That's valuable even if no AI ever reads the file.
- Adoption is accelerating. Being early costs nothing; being late might cost visibility.
- It complements your existing SEO. You're not replacing anything, just adding a new signal.
Don't overthink it. Create the file, list your 20-30 most important pages, and move on. You can always refine it later.
Building AI applications that need clean web data? link.sc delivers web content as LLM-ready Markdown — try it free with 500 requests/month.