Quick answer: Crawling is how a search engine discovers pages: a bot follows links across the web and downloads what it finds. Indexing is what happens next: the engine parses each page, extracts its content, and stores it in a giant lookup structure (an inverted index) so it can answer queries in milliseconds. A page must be crawled before it can be indexed, and indexed before it can rank. Two separate steps, two separate ways for your content to go missing.
That last sentence is the part most people get wrong, so let me unpack the whole pipeline.
Step One: Crawling Is Discovery
Googlebot doesn't know your new page exists until something points to it. It finds URLs three ways: links from pages it already knows, your sitemap.xml, and direct submission through Search Console.
Every discovered URL goes into a queue (the crawl frontier), gets prioritized, and eventually gets fetched. Modern Googlebot also renders JavaScript, though rendering is a second, delayed pass, which is why JS-heavy sites often index slower than server-rendered ones.
Crawling produces raw material: a pile of downloaded pages. On its own that pile answers no queries. That's the index's job.
Step Two: Indexing Is Organization
Here's the mental model that made indexing click for me: a book's index. You don't read a whole book to find mentions of "crawl budget." You flip to the index, which maps every term to page numbers.
An inverted index is exactly that, at web scale. Instead of storing "page X contains words A, B, C," it stores "word A appears on pages X, Y, Z." When you search "crawl budget," the engine doesn't scan billions of pages. It looks up two term lists, intersects them, and ranks the survivors.
Indexing also involves judgment calls. The engine picks a canonical version when several URLs have the same content, decides whether the page is worth storing at all, and extracts structured data. This is why "crawled, currently not indexed" is a real status in Search Console: Google saw your page and declined to file it. Crawling succeeded, indexing said no thanks.
Where Pages Fall Through the Cracks
| Stage | Failure | Common cause |
|---|---|---|
| Discovery | Never crawled | No internal links, missing from sitemap |
| Crawl | Fetch fails | Blocked in robots.txt, server errors, slow responses |
| Render | Content invisible | Critical content only appears after JS the bot didn't run |
| Index | Crawled, not indexed | Thin or duplicate content, weak site signals |
| Rank | Indexed, invisible | Indexed fine, just outcompeted |
Diagnosing "why isn't my page on Google" means finding which row you're stuck on. People jump straight to ranking tweaks when the actual problem is two stages earlier.
Crawl Budget: Why Google Won't Fetch Everything
Google allocates each site a rough amount of crawling attention based on server capacity and how much it wants your content. Small sites never notice. Sites with hundreds of thousands of URLs absolutely do.
The classic crawl budget killers are infinite URL spaces: faceted navigation that generates ?color=red&size=m&sort=price in every combination, calendar pages that paginate forever, session IDs in URLs. The bot burns its allocation on junk permutations and your actual new content waits days for a visit.
The fixes are boring and effective: block junk parameters in robots.txt, keep the sitemap tight and current, fix redirect chains, and return fast responses. A snappy server literally gets crawled more.
This Now Matters for AI, Not Just Google
The same crawl-then-index pipeline is running behind every AI assistant. GPTBot, ClaudeBot, and PerplexityBot crawl the web; the content lands in training corpora and retrieval indexes; and when someone asks a question, the model draws on what got indexed.
Same rules, new audience. If your content is unreachable, blocked, or trapped behind JavaScript, AI systems can't cite you any more than Google can rank you. The main difference is that most AI crawlers still don't render JavaScript at all, so the render-stage failures in the table above hit harder. I went deeper on this in how AI crawlers read your website.
Building Your Own Crawler vs Being Googlebot
Flip the perspective: sometimes you're the one crawling and indexing, say to build a search feature or a RAG knowledge base over a set of sites. Your job is genuinely easier than Google's in some ways and harder in others.
Easier: you crawl targeted sites, not the whole web, and you don't need PageRank. A vector database or even SQLite full-text search replaces the planetary-scale inverted index.
Harder: Googlebot gets welcomed; your bot gets blocked. Sites that roll out the red carpet for search engines serve CAPTCHAs to unknown crawlers, which is the single biggest practical difference.
That fetch-and-clean layer is exactly what I'd outsource. With link.sc, the crawl step becomes one API call per page and you get markdown ready for chunking and embedding:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_your_key" \
-H "Content-Type: application/json" \
-d '{"url": "https://docs.example.com/api/auth", "format": "markdown"}'
Then your "indexing" step is just embedding the markdown into your vector store. The quickstart covers the response format, and how to crawl an entire website covers doing this across a full domain instead of one page.
The Practical Checklist
For your own site, in priority order:
- Every important page reachable by internal links within three clicks.
- sitemap.xml current and submitted, containing only pages you want indexed.
- robots.txt not accidentally blocking things (this happens constantly after site migrations).
- Critical content in server-rendered HTML, not JS-only.
- Junk URL parameters blocked so crawl budget goes to real pages.
- Search Console's coverage report checked monthly for "crawled, not indexed" trends.
None of this is glamorous. All of it beats another round of keyword tweaks, because crawling and indexing are the gate everything else waits behind.
Building a search or RAG pipeline over web content? link.sc handles the crawling and cleanup so you can focus on the index. Free tier includes 500 credits a month.