Quick answer: An AI crawler is a bot that fetches web pages to feed AI systems instead of a search index. Some collect training data for future models (GPTBot, ClaudeBot), some fetch pages live to answer a user's question (ChatGPT-User, Perplexity-User), and some build retrieval indexes for AI search. They've become a major share of bot traffic on most sites, and you control them the same way you control Googlebot: robots.txt, plus newer conventions like llms.txt.
If you run a website, these bots are already in your logs. If you build AI products, you may need to become one. Both sides are worth understanding.
The Bots You'll Actually See
The names change as companies rebrand, but the recurring cast in server logs looks like this:
| Bot | Operator | What it feeds |
|---|---|---|
| GPTBot | OpenAI | Model training corpora |
| ChatGPT-User | OpenAI | Live fetches during ChatGPT sessions |
| OAI-SearchBot | OpenAI | ChatGPT search index |
| ClaudeBot | Anthropic | Model training corpora |
| Claude-User | Anthropic | Live fetches during Claude sessions |
| PerplexityBot | Perplexity | AI search index |
| Google-Extended | Gemini training (a robots.txt token, not a separate bot) | |
| CCBot | Common Crawl | Open dataset many labs train on |
| Bytespider | ByteDance | Model training |
The three-way split in that table matters more than the individual names: training crawlers, live retrieval fetchers, and AI search indexers. Blocking a training bot keeps you out of future model weights. Blocking a live fetcher means an assistant can't read your page when a user pastes your URL. Different trade-offs, and blanket-blocking everything AI usually costs more visibility than people expect.
How AI Crawlers Differ from Googlebot
Superficially they're the same thing: a bot that fetches pages. In practice there are real differences.
No reciprocity by default. Googlebot's deal was always explicit: let me crawl, I send you traffic. A training crawler offers no link back. AI search bots like PerplexityBot cite sources, which restores some of the bargain. This asymmetry is why AI crawlers are controversial in a way Googlebot never was.
Most don't render JavaScript. Googlebot runs a full Chromium rendering pipeline. Most AI crawlers grab raw HTML and move on. If your content only exists after client-side rendering, you're invisible to much of the AI ecosystem.
They want prose, not signals. Google indexes for retrieval and ranking. Training crawlers want clean, substantive text. Boilerplate, nav, and thin pages are noise to them.
Crawl behavior is younger and rougher. Google spent 25 years tuning politeness. Some AI crawlers have earned reputations for aggressive request rates, which is a big reason CDNs now ship one-click AI bot blocking.
Controlling AI Crawlers on Your Site
The major players' bots respect robots.txt, so per-bot policy is straightforward:
# Allow AI search and live fetching, opt out of training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
Three honest caveats. First, robots.txt is a request, not a lock; well-run companies honor it, scrapers pretending to be browsers don't. Second, it's not retroactive: content already in a training set stays there. Third, enforcement beyond robots.txt means CDN-level bot management or your own detection: see ethical web scraping and compliance best practices for how that landscape looks from both sides.
There's also llms.txt, a proposed file that tells AI systems which of your pages matter most. Adoption by the AI companies themselves remains unproven, and I've written up how AI crawlers read your website and how llms.txt fits in if you want the full picture before spending time on it.
The Flip Side: Building Your Own AI Crawler
Every RAG pipeline, AI research agent, and LLM app with web access contains a small AI crawler. If you're building one, the requirements differ from a classic scraper in three ways.
Output format is the whole game. An LLM doesn't want HTML. It wants clean markdown with nav, ads, and cookie banners stripped, because every token of boilerplate is context window you paid for and reasoning quality you lost.
Freshness beats scale. You rarely need a billion pages. You need the fifty pages relevant to this query, fetched now.
You inherit the bot problem. Your crawler is an unknown agent, so sites that welcome Googlebot will CAPTCHA you.
This stack is precisely what link.sc is: fetching, rendering, unblocking, and HTML-to-markdown cleanup behind one endpoint, with an MCP server at mcp.link.sc if your agent speaks MCP. One call gets you LLM-ready content:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_your_key" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/changelog", "format": "markdown"}'
And when you're crawling other people's sites, follow the same rules you'd want AI crawlers to follow on yours: honor robots.txt, rate limit, identify yourself. Symmetry is a decent ethical compass here.
Should You Block AI Crawlers? My Take
It depends on what your content earns you.
If your business sells the content itself (paywalled journalism, proprietary research), blocking training bots is rational; you're protecting the asset. If your business benefits from being known (SaaS docs, ecommerce, most blogs), being absent from AI answers is a real cost, because a growing slice of your audience asks ChatGPT instead of Google.
My default recommendation for most sites: allow AI search and live fetchers, decide on training bots case by case, and revisit quarterly. This ecosystem is moving fast enough that any permanent policy written today will look wrong within a year.
Building an AI agent that needs to read the web? link.sc fetches any URL as clean markdown, with 500 free credits a month to start.