It's early 2026, and llms.txt sits in an awkward middle ground. Over 800,000 websites have published one. Major tech companies have adopted it. But no AI provider has officially confirmed they use it during inference, and the data on its effectiveness is mixed at best.
So what happens next? I've been thinking about this a lot, and I see a few possible futures. Let me walk through them honestly.
Scenario 1: Quiet Adoption (Most Likely)
In this scenario, AI systems gradually start using llms.txt files without any big announcement. It happens the way most web standards evolve — not with a press release, but with a pull request.
Here's what this might look like:
- Perplexity starts checking for llms.txt when its bot visits a domain for the first time, using it to prioritize which pages to index
- OpenAI's browsing feature starts consulting llms.txt to find the most authoritative pages during real-time search
- Claude's web features reference llms.txt for a quick overview of a domain
- None of these changes are announced explicitly; they just get baked into crawling logic
This is actually how robots.txt evolved. There was no grand announcement from Google saying "we now respect robots.txt." It just gradually became a thing that crawlers did because it was useful.
The signal to watch: AI-related user agents showing up in your llms.txt access logs. When that starts happening consistently, you'll know.
Scenario 2: Official Standard (Possible)
In this scenario, one or more major AI companies formally adopt llms.txt as part of their crawling and retrieval specification. This would look like:
- An official announcement from OpenAI, Anthropic, or Google saying "we read llms.txt"
- Documentation on how to optimize your llms.txt for each platform
- Tool support (like Google Search Console for AI)
- Possible extensions to the spec (more structured metadata, verification, etc.)
The trigger for this scenario would likely be Google. If Google adds llms.txt support to Search Console or officially integrates it into their AI systems, it becomes a de facto standard overnight. They already included it in the Agent-to-Agent (A2A) protocol, which is a promising sign.
Scenario 3: Evolution Into Something Better (Also Possible)
Maybe llms.txt isn't the final answer, but it's the right question. The need for a machine-readable content map is real. But the current spec is pretty basic — it's just Markdown with links. A more structured format might eventually replace it.
What that might look like:
# Hypothetical next-gen spec
version: 2
name: "Acme Corp"
description: "Cloud infrastructure platform"
last_updated: 2026-02-14
contact: [email protected]
content:
- url: https://acme.com/docs/api
type: api_reference
priority: high
freshness: weekly
topics: [authentication, endpoints, rate-limits]
- url: https://acme.com/blog/scaling
type: guide
priority: medium
freshness: monthly
topics: [scaling, performance, architecture]
More structured. Machine-parseable beyond Markdown. Includes metadata like freshness and topic tags. This would be more useful for AI systems, but harder for humans to write — which is the tension the spec will need to navigate.
Scenario 4: Slow Fade (Less Likely But Possible)
In the worst case for llms.txt advocates, it goes the way of the keywords meta tag. Sites keep publishing the file for a while, but without AI provider adoption, interest gradually wanes. The file becomes digital clutter — present but ignored.
Google's Gary Illyes already drew this comparison. And it's not completely unfair. The keywords meta tag was also a well-intentioned attempt to help machines understand content, and it also had enthusiastic early adoption.
The difference, though, is that the keywords meta tag was easily gamed. You could stuff keywords that had nothing to do with your content. llms.txt is harder to abuse because it links to actual pages that can be verified.
What I Think Actually Happens
My honest prediction: a mix of scenarios 1 and 3.
AI systems will start using llms.txt (or checking for it) in limited ways, particularly for inference-time retrieval. Meanwhile, the spec will evolve to become more structured and machine-friendly. Within a year or two, we'll see "llms.txt 2.0" or an equivalent that's more like a structured API than a Markdown file.
The companies driving this evolution will likely be the documentation platforms (Mintlify, GitBook, ReadMe) and the AI tool builders, not the AI model companies themselves. Infrastructure providers have a strong incentive to make their customers' content AI-accessible, and they'll build the tooling that shapes the next version of the standard.
What This Means for You Right Now
Regardless of which scenario plays out, the underlying need doesn't go away. AI systems need to understand your website. The question is just how they'll do it.
Here's what you can do today that pays off in every scenario:
1. Publish an llms.txt Now
Even if the spec evolves, the basic format (Markdown with links) will remain readable by whatever comes next. Five minutes of work that's future-proof.
2. Invest in Clean Content Architecture
Whatever the AI discoverability standard ends up being, it will work better with well-structured content. Clear headings, logical page organization, proper semantic HTML — these matter regardless of llms.txt.
3. Think About Content Accessibility
Can an AI system actually read your pages? If your content is buried in JavaScript, behind auth walls, or drowning in ads, no metadata file will fix that. The trend toward AI-accessible content is accelerating, and tools like link.sc exist specifically because so much of the web isn't AI-friendly yet.
4. Monitor AI Crawler Activity
Start tracking AI-related user agents in your server logs now. When patterns change (and they will), you'll want baseline data to compare against.
5. Follow the Spec
The official spec lives at llmstxt.org. Watch for updates. If you're going to implement the standard, implement it correctly.
The Bigger Trend
Zoom out from llms.txt specifically, and there's a bigger story: the web is being re-architected for AI consumption. Not replacing human access — adding a parallel layer optimized for machines.
We're seeing this across the stack:
- Content layer: llms.txt, structured data, Markdown-first publishing
- API layer: MCP (Model Context Protocol) standardizing how AI connects to data
- Infrastructure layer: Web data APIs like link.sc that bridge the gap between messy HTML and clean, structured content
- Discovery layer: AI-specific search and indexing systems
llms.txt is one piece of this bigger transformation. Its specific format might change. Its role might evolve. But the need it addresses — helping AI systems understand what your website is and what it contains — is only going to grow.
Whether you're publishing content or building AI applications that consume it, investing in this transition now puts you ahead of the curve.
Ready to build AI applications with reliable web data? link.sc delivers clean Markdown from any URL — today's solution for the AI-accessible web. Start free.