Quick answer: Search engines work in three stages. First they crawl the web with bots that follow links and fetch pages. Then they index those pages, parsing the content and storing it in a giant database. Finally they rank results, deciding which stored pages to show for a given query and in what order. Everything you do for SEO is really about influencing one of those three stages.
I have watched a lot of people treat search as a black box. It is not. It is a pipeline with three well-defined stages, and once you see them clearly, most SEO advice stops sounding like superstition and starts sounding like plumbing.
Let me walk through each stage, then show where AI Overviews bolted a fourth step onto the end.
Stage 1: Crawling
Crawling is discovery. A search engine runs bots (Googlebot is the famous one) that fetch a page, read the HTML, extract every link on it, and add those links to a queue of URLs to fetch next. Repeat that a few trillion times and you have a rolling snapshot of the public web.
A crawler cannot index a page it never fetched, so this is the stage where pages quietly disappear. Common reasons a crawler never reaches your content:
- No internal links point to the page, and it is not in your sitemap, so nothing tells the bot it exists.
robots.txtdisallows the path.- The server is slow or throwing errors when the bot arrives, so the crawler backs off.
- The content only appears after heavy JavaScript execution the crawler did not run.
Crawlers also budget their effort. Big, healthy sites get crawled often. New or thin sites get crawled sparingly. That "crawl budget" is why a fresh domain can publish a page and wait days before anything happens.
If you want the full mechanics of this stage and the indexing that follows, I wrote a deeper piece on what crawling and indexing actually mean. This post is the map; that one is the terrain.
Stage 2: Indexing
Once a page is fetched, the engine has to understand it. Indexing is the parsing-and-storing stage.
The engine renders the page (increasingly running JavaScript to see the final DOM), extracts the visible text, notes the title, headings, links, images, and structured data, and works out what the page is about. It then writes all of that into an inverted index: a data structure that maps every word to the list of pages containing it. That is what makes retrieval fast. When you search "how search engines work," the engine does not scan the web live; it looks up those words in the index and pulls the matching pages instantly.
Indexing is also a filter. Not every crawled page gets indexed. Duplicate pages, thin pages, and pages the engine judges as low value can be crawled and then dropped. In Google Search Console you will see these as "Crawled - currently not indexed," which is the engine's polite way of saying "we looked and decided not to bother."
Things that help a page make it into the index:
| Factor | Why it helps indexing |
|---|---|
| Unique, substantial content | Gives the engine a reason to store the page separately |
| Clean HTML and fast rendering | The renderer sees your content without timing out |
| A canonical tag | Tells the engine which version of near-duplicate URLs to keep |
| Internal links from indexed pages | Passes signals that the page is part of the site, not an orphan |
| Structured data | Makes the page's meaning explicit rather than inferred |
Stage 3: Ranking
Ranking is the stage everyone obsesses over, and it is the one you control least directly. When a query comes in, the engine retrieves every indexed page that could match, then scores them against hundreds of signals and returns an ordered list.
The signals fall into rough buckets:
- Relevance: does the page actually answer this query? Term matching is the crude version; modern engines use language models to understand meaning and intent.
- Quality and authority: is this a trustworthy source? Links from other reputable sites, brand signals, and a track record all feed in.
- User context: location, language, device, and sometimes search history reshape results.
- Freshness: for time-sensitive queries, newer wins; for evergreen ones, it barely matters.
The important mental shift: ranking is per-query, not per-page. A page does not have "a rank." It has a rank for each query, and that rank moves as the competition, the query intent, and the engine's models change.
The New Last Mile: AI Overviews
For most of search history, the pipeline ended at stage three: retrieve, rank, show a list of blue links. AI Overviews (and the AI answer boxes in Bing, Perplexity, and others) added a synthesis step on top.
Now the engine can take the top-ranked pages, feed them to a language model, and generate a written answer that cites a handful of sources. The ranking still happens underneath; the model is mostly summarizing what already ranked well. But the presentation changed the game. Instead of ten links competing for a click, you get one synthesized answer citing two or three sources, and being one of those cited sources is the new prize.
This is why the SEO conversation has shifted toward answer-engine optimization. The stages that get you cited in an AI Overview are the same crawling and indexing you always needed, plus content structured so a model can lift a clean, quotable answer out of it. If you want to go deeper on that shift, how AI search engines rank content covers the synthesis layer in detail.
What You Actually Control
Map your effort back to the three stages and the priorities get obvious:
| Stage | What you control | Concrete action |
|---|---|---|
| Crawling | Discoverability | Sitemap, internal links, sane robots.txt, fast server |
| Indexing | Understandability | Unique content, canonicals, structured data, clean HTML |
| Ranking | Competitiveness | Genuinely better answers, authority, matching search intent |
You do not control the ranking algorithm. You control whether your page is even eligible to be ranked, and how clearly it communicates once it is. Most sites lose at stages one and two, then blame stage three.
Reading the Web Like an Engine Does
If you build tools or agents that consume search results, you end up rebuilding a small version of this pipeline yourself: fetch pages, parse them into something usable, and pull out the answer. That is exactly the problem link.sc handles. Our fetch endpoint turns any URL into clean markdown, and our search endpoint returns full page content instead of the thin snippets a normal results page gives you, so your code can read a page the way an indexer would rather than scraping brittle HTML.
curl -X POST https://api.link.sc/v1/search \
-H "x-api-key: lsc_your_key_here" \
-H "Content-Type: application/json" \
-d '{"q": "how search engines work", "engine": "google"}'
You get structured results back, ready to parse, without running your own crawler fleet. See the quickstart if you want to wire it into an application.
The Takeaway
Search engines crawl to discover, index to understand, and rank to decide. AI Overviews added a synthesis step that rewards the same fundamentals plus clean, quotable structure. When something is not showing up in search, do not start at ranking. Start at the top of the pipeline and ask: was it crawled, was it indexed, and only then, is it competitive?
Building something that reads the web the way a search engine does? link.sc gives you clean fetch and full-content search from one API. Start free.