GUIDES

llms.txt: Complete Guide

Everything you need to know about the llms.txt standard — specification, implementation, best practices, and how it relates to building AI-powered applications with link.sc.

llms.txt: Complete Guide

This guide covers the llms.txt standard — a proposed specification for helping AI language models understand your website's content. Whether you're a publisher creating an llms.txt for your own site or a developer building AI applications that consume web content, this guide has you covered.

What is llms.txt?

llms.txt is a Markdown file placed at the root of a website (example.com/llms.txt) that provides AI language models with a curated map of the site's most important content. It was proposed at llmstxt.org as a way to solve the challenge LLMs face when trying to understand websites.

Think of it as a companion to robots.txt and sitemap.xml:

File Purpose Audience
robots.txt Controls crawl access Search engine bots
sitemap.xml Lists all indexable pages Search engines
llms.txt Curates important content AI language models

The Specification

An llms.txt file follows a specific Markdown structure:

Required

  • H1 heading: The name of your project or site
  • Blockquote: A concise summary of your site immediately after the H1
  • H2 sections: Organize links into logical categories
  • Link lists: URLs with optional descriptions

Format

# Site Name

> A one-to-three sentence summary that explains what this site
> or project is, who it's for, and what makes it notable.

## Section Name
- [Page Title](https://example.com/page): Brief description of what this page contains
- [Another Page](https://example.com/another): Another description

## Another Section
- [More Content](https://example.com/more): Description here

Rules

  • The file must be valid Markdown
  • Only the H1 heading is technically required
  • Links follow the format: - [Title](URL): Description
  • The description after the colon is optional but strongly recommended
  • The file should be accessible at your site's root path: /llms.txt

Implementation Guide

Step 1: Create the File

Create a new file named llms.txt in your site's root directory.

Step 2: Write the Header

# Your Site Name

> Your site in one to three sentences. Be specific about what you do,
> who you serve, and what makes you different.

Select 20-50 of your most important pages. Organize them under H2 headings:

## Documentation
- [Getting Started](https://yoursite.com/docs/start): Setup and first steps
- [API Reference](https://yoursite.com/docs/api): Complete API documentation

## Resources
- [Blog](https://yoursite.com/blog): Technical articles and updates
- [Pricing](https://yoursite.com/pricing): Plans and pricing information

Step 4: Deploy

Place the file where it will be served at https://yoursite.com/llms.txt:

Platform Location
Next.js public/llms.txt
Astro public/llms.txt
WordPress Site root via FTP, or use the Yoast SEO plugin
Netlify / Vercel public/llms.txt in your project
Static hosting Upload to document root

Step 5: Verify

Visit https://yoursite.com/llms.txt in your browser. You should see raw Markdown content.

Best Practices

Content Curation

  • Include 20-50 links — this is not a sitemap
  • Choose evergreen content over recent posts
  • Prioritize pages that define your business: product pages, core docs, authoritative guides
  • Write descriptions for every link — they provide context without requiring page visits

Structure

  • Group by user intent, not site architecture (e.g., "Getting Started" vs "/docs")
  • Put the most important sections first
  • Keep the summary blockquote under 3 sentences

Maintenance

  • Review quarterly or after major content changes
  • Check for broken links when updating
  • Ensure consistency with robots.txt — don't link to pages that are blocked from AI crawlers

Common Mistakes

Mistake Why It's Bad Fix
Listing every page Defeats the purpose of curation Limit to 20-50 links
No descriptions on links Wastes context opportunity Add : description after each URL
HTML instead of Markdown Doesn't follow the spec Use plain Markdown only
Conflicting with robots.txt Sends mixed signals Audit both files together
Never updating Links go stale Review quarterly

llms.txt and AI Crawlers

Current Adoption

As of early 2026:

  • Over 800,000 websites have published llms.txt files
  • Notable adopters include Cloudflare, Stripe, Anthropic, Vercel, and Coinbase
  • No major AI provider has officially confirmed they read llms.txt during inference
  • Studies show very low AI crawler access rates for the file

AI Crawler Reference

Crawler Company Purpose
GPTBot OpenAI Training data
ChatGPT-User OpenAI Real-time browsing
ClaudeBot Anthropic Training data
Google-Extended Google AI training
PerplexityBot Perplexity Search + answers

Monitoring

Check your server logs for AI crawler access:

grep "llms.txt" /var/log/nginx/access.log | grep -E "(GPTBot|ClaudeBot|PerplexityBot)"

Optional: llms-full.txt

Some sites also publish an llms-full.txt that includes full page content inline rather than just links. This is useful for:

  • API documentation (endpoints, parameters, examples)
  • Critical reference material
  • Content you want AI to have immediate access to without additional fetches

Keep the total size manageable — under 100KB or approximately 50,000 tokens.

llms.txt for API Products

API documentation is a particularly strong use case for llms.txt. Developers increasingly use AI assistants to write integration code, and accurate API information prevents broken implementations.

Recommended structure for API products:

# Your API

> Brief description of the API, its main capabilities, auth method,
> and base URL.

## Authentication
- [Auth Guide](https://yoursite.com/docs/auth): Setup and key management

## Core Endpoints
- [Endpoint A](https://yoursite.com/docs/api/a): What this endpoint does
- [Endpoint B](https://yoursite.com/docs/api/b): What this endpoint does

## SDKs
- [Python](https://yoursite.com/docs/sdk/python): pip install yourpackage
- [Node.js](https://yoursite.com/docs/sdk/node): npm install yourpackage

## Guides
- [Quickstart](https://yoursite.com/docs/quickstart): First API call
- [Examples](https://yoursite.com/docs/examples): Common integration patterns

llms.txt addresses the content discoverability problem from the publisher's side — helping AI find your content. link.sc solves the same underlying problem from the developer's side — making any web content AI-readable.

For Publishers

If you publish an llms.txt, your content becomes easier for AI systems to discover and prioritize. But the content on those pages still needs to be extractable. Pages with heavy JavaScript rendering, complex layouts, or content behind interaction walls remain difficult for AI to process.

For Developers

When building AI applications that consume web content, you can't rely on every website having a well-crafted llms.txt. The link.sc API provides a universal solution:

import linksc

client = linksc.Client(api_key="lsc_...")

# Fetch any URL as clean Markdown — no llms.txt required
page = client.fetch(
    url="https://example.com/docs/api",
    format="markdown"
)

# Search the web and get LLM-ready results
results = client.search(
    q="llms.txt best practices",
    format="markdown",
    num_results=5
)

Whether a site has an llms.txt or not, link.sc delivers clean, structured content that's ready for LLM consumption.

The Complementary Approach

The ideal setup combines both:

  1. Publish llms.txt on your own site for AI discoverability
  2. Use link.sc in your AI applications to consume web content reliably

This covers both sides — making your content findable and making the web's content usable.

Further Reading

link.sc is a web fetch and search API optimized for LLMs. Convert any URL to clean Markdown and search the web with AI-ready results. Get started free with 500 requests per month.

PREVIOUS← Research Agent