Most people think of "AI reading your website" as a single thing. It's actually several very different things, and understanding the difference changes how you think about llms.txt entirely.
Let me break it down.
The Two Types of AI Crawling
There are fundamentally two different reasons an AI system visits your website, and they work in completely different ways.
Type 1: Training Crawlers
These crawlers collect data to train AI models. They work a lot like Google's crawler — they systematically visit pages, download content, and add it to a massive dataset that will eventually be used to train a model.
Common training crawlers:
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google DeepMind)
- CCBot (Common Crawl)
Training crawlers visit your site weeks or months before that data appears in a model. They're building a knowledge base, not answering a question right now.
llms.txt relevance for training crawlers: Low. These crawlers are designed to ingest everything they can find. They don't need a curated map — they're building their own. And they generally don't check for llms.txt files (the log data confirms this).
Type 2: Inference-Time Retrieval
This is when an AI model needs to look something up to answer a question right now. Think of Perplexity searching the web, or ChatGPT with browsing enabled, or an AI agent using a web search API to get current information.
These systems work more like a researcher: they search, fetch specific pages, extract relevant information, and use it to generate a response.
llms.txt relevance for inference-time retrieval: Potentially high. If an inference-time system checks for llms.txt, it could use it to quickly identify the most relevant pages on your site without crawling everything. This is the use case that makes the spec worth watching.
What Actually Happens When a Bot Visits Your Page
Whether it's a training crawler or an inference-time fetcher, here's the typical sequence:
- DNS lookup: Resolve your domain to an IP address
- robots.txt check: Most bots check your robots.txt first. If they're blocked, they stop.
- HTTP request: Fetch the page with a GET request
- HTML download: Receive the raw HTML response
- Content extraction: Strip away navigation, ads, scripts, and chrome
- Text processing: Convert the remaining content to plain text or a structured format
- Storage/processing: Either store for training or use for immediate answering
Steps 5 and 6 are where things get messy. Most websites aren't built for machine readability. A typical page might have:
- 200 lines of
<head>content (meta tags, CSS links, script includes) - A navigation bar with 50+ links
- Cookie consent banners
- The actual article content
- A sidebar with related posts
- Comments sections
- A footer with another 30+ links
- Inline ads and tracking pixels
The crawler has to figure out which part is "the content." This is harder than it sounds, and different crawlers have different success rates. Some use readability algorithms (similar to Firefox's Reader View). Others use custom ML models trained to identify main content areas.
Where llms.txt Saves Time
Here's the thing: even the best content extraction is imperfect. And it requires actually visiting every page.
An llms.txt file offers a shortcut. Instead of crawling 100 pages and trying to figure out which ones matter, a system can:
- Fetch
/llms.txt(one request) - Read a curated list of important pages with descriptions
- Decide which pages are worth fetching in full
- Fetch only those specific pages
For inference-time systems with time constraints (a user is waiting for an answer), this efficiency matters. You'd rather spend your time budget fetching three highly relevant pages than crawling twenty pages hoping to find the right ones.
The Content Quality Gap
Even when AI systems find the right pages, the content quality problem remains. Here's what a raw HTML page looks like to a bot versus what it actually needs:
What the bot downloads (simplified):
<!DOCTYPE html>
<html>
<head>
<!-- 50 lines of meta tags, CSS, JS -->
</head>
<body>
<nav><!-- 40 links --></nav>
<div class="cookie-banner"><!-- consent form --></div>
<main>
<article>
<h1>How to Use Our API</h1>
<p>Here's how to authenticate...</p>
<!-- THE ACTUAL CONTENT: maybe 2KB -->
</article>
</main>
<aside><!-- sidebar: related posts, ads --></aside>
<footer><!-- another 30 links --></footer>
<!-- 20 script tags for analytics, ads, etc. -->
</body>
</html>
What the bot actually needs:
# How to Use Our API
Here's how to authenticate...
The useful content might be 5-10% of the downloaded payload. The rest is noise. This is a fundamental reason why tools like link.sc exist — the fetch API does this extraction automatically, returning just the clean Markdown content from any URL.
AI Crawlers You Should Know About
Here's a quick reference of the major AI crawlers, what they do, and how to identify them in your logs:
| Crawler | Company | Purpose | User Agent String |
|---|---|---|---|
| GPTBot | OpenAI | Training + retrieval | GPTBot/1.0 |
| ChatGPT-User | OpenAI | Real-time browsing | ChatGPT-User |
| ClaudeBot | Anthropic | Training | ClaudeBot/1.0 |
| Google-Extended | AI training | Google-Extended |
|
| PerplexityBot | Perplexity | Search + answers | PerplexityBot |
| Bytespider | ByteDance | Training | Bytespider |
| CCBot | Common Crawl | Open dataset | CCBot/2.0 |
You can control access for each of these in your robots.txt:
User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /admin/
User-agent: ClaudeBot
Allow: /
How to Check if AI Crawlers Are Visiting Your Site
If you're curious whether any of this matters for your site specifically, check your server logs:
# Look for AI crawler user agents in nginx logs
grep -E "(GPTBot|ClaudeBot|PerplexityBot|ChatGPT-User|Google-Extended)" /var/log/nginx/access.log
You might be surprised. Some sites get significant AI crawler traffic. Others get virtually none. The volume depends on your site's authority, content type, and how often your domain appears in AI queries.
The Practical Takeaway
Understanding how AI crawlers work changes the calculus on llms.txt:
- For training crawlers: llms.txt doesn't do much. They're going to crawl what they crawl.
- For inference-time retrieval: llms.txt could be genuinely useful, but adoption by AI systems is still unconfirmed.
- For all AI systems: Clean, extractable content matters more than any metadata file. If your pages are JavaScript-heavy SPAs with content loaded dynamically, neither llms.txt nor the crawlers themselves can help you.
The best strategy is layered: publish an llms.txt for discoverability, make your content technically crawlable, and ensure your pages deliver clean content that AI systems can actually extract value from.
And if you're building on the consumption side — creating AI applications that need to read web content — don't rely on llms.txt being available. Build your pipeline to handle any website, ideally using a tool like link.sc that handles the content extraction automatically.
link.sc handles the hard part of web data for AI — converting messy HTML into clean, structured Markdown. Your LLM gets the content without the noise. Try it free.