Quick answer: Crawl budget is the number of pages a crawler will fetch from your site in a given window. Google frames it as two forces multiplied together: crawl capacity (how much traffic your server can take without slowing down) and crawl demand (how much the crawler actually wants your pages, based on popularity and freshness). You optimize it by spending that budget on pages that matter: fix errors, kill duplicates, keep your server fast, and point crawlers at the URLs worth having.
The term comes from SEO, where it describes Googlebot's behavior on your site. But the same idea governs any crawler, including the one you write to collect data. Understanding both sides makes you better at getting crawled and better at crawling politely.
The Two Halves of a Crawl Budget
Google splits crawl budget into two components, and they pull in opposite directions.
Crawl capacity limit is the ceiling your server sets. Googlebot watches how fast your pages respond and how often requests fail. Fast, healthy responses raise the limit; slow responses and 5xx errors lower it. The crawler is deliberately conservative because it does not want to be the reason your site falls over. If your server starts timing out, the crawler backs off, exactly the same instinct a well-behaved scraper should have when it sees an HTTP 429.
Crawl demand is how much the crawler wants your content in the first place. Popular URLs with lots of inbound links get crawled more. Pages that change often get revisited more. Pages nobody links to and nothing points at get crawled rarely, if at all. Stale URLs that have not changed in months drift to the back of the queue.
Your effective crawl budget is roughly capacity multiplied by demand. A fast server with boring, static, unlinked pages still will not get crawled much, because demand is low. A popular, frequently updated site on a slow server gets throttled by capacity. You want both levers up.
Who Actually Needs to Care
Most sites do not. If you have a few thousand URLs and they get indexed within a day or two, crawl budget is not your problem, and chasing it is wasted effort. Google has said as much: crawl budget is a concern for large sites, not small ones.
It starts to matter when:
- You have hundreds of thousands or millions of URLs.
- You publish or update content faster than it gets indexed.
- A large share of your URLs are auto-generated (faceted navigation, filters, search results, session parameters).
- Your logs show Googlebot spending most of its visits on pages you do not care about.
That last point is the crux. The problem is almost never that the crawler visits too few pages. It is that it visits the wrong ones, and the good pages wait behind a pile of junk.
How to Optimize Crawl Budget (Site Owner Side)
The goal is simple to state: make every crawl request land on a page worth indexing. Everything below serves that.
Cut duplicate and near-duplicate URLs. This is the single biggest lever on most sites. Faceted navigation that produces ?color=red&size=large&sort=price in every combination can turn a 10,000-page catalog into millions of crawlable URLs, all thin variants of each other. Use canonical tags, consolidate parameters, and block the combinatorial explosions you do not want indexed.
Fix your error pages. Long chains of redirects, soft 404s, and pages returning 5xx all consume budget and teach the crawler your site is unreliable, which lowers your capacity limit. A redirect chain of four hops spends four requests to deliver one page.
Keep the server fast. Response time directly feeds the capacity limit. Faster pages mean the crawler can fetch more of them in the same window without risking your stability. Caching, a CDN, and lean HTML all pay off here.
Use the tools that steer crawlers. A clean XML sitemap tells the crawler which URLs you consider canonical and when they last changed, so it does not have to discover them by chance. A well-written robots.txt keeps crawlers out of infinite spaces like calendars and internal search. These two files are the main levers you have, and they deserve their own attention. Note that robots.txt controls crawling, not indexing, and it is a different tool from the AI-focused files we cover in llms.txt vs robots.txt.
Prune dead weight. Expired products, thin tag pages, and abandoned content all sit in the crawl queue. Removing or consolidating them frees budget for pages that earn it.
Read your logs. Server logs are the ground truth for what the crawler actually does. If Googlebot spends 60 percent of its requests on parameter URLs you never wanted indexed, no amount of guessing beats seeing it in the logs and fixing the specific pattern.
The Same Idea, From the Crawler's Side
Here is where crawl budget stops being an SEO term and becomes a design principle. When you write a crawler, you are spending someone else's crawl budget. Every request you make competes with real traffic and with Googlebot for that site's server capacity. The polite-crawler playbook is the mirror image of the site-owner playbook.
Rate-limit yourself. Pick a concurrency you can defend, watch response times, and back off the moment the server slows or returns 429s. If a site's capacity limit is real for Googlebot, it is real for you. This is the same discipline that separates a scraper that runs for years from one that gets blocked in an hour, and it is central to ethical crawling practice.
Deduplicate aggressively. The site owner's duplicate-URL problem is your wasted-request problem. Normalize URLs before you queue them: strip tracking parameters, canonicalize trailing slashes, collapse ?sort= variants that return identical content, and keep a seen-set so you never fetch the same page twice. A crawler without dedup spends most of its budget re-fetching the same thin variants that bloat the site owner's budget too. If you are enumerating listing pages, the list crawling pattern is where dedup pays off most.
Respect the map you are given. Read the sitemap and robots.txt before you start. The sitemap hands you the canonical URLs directly, so you skip the discovery crawl entirely. The robots file tells you which infinite spaces to avoid, which saves your budget as much as it protects the host.
Prioritize by demand. You have a crawl demand too. Crawl the pages you actually need, in priority order, rather than blindly following every link. This is the practical difference between crawling and scraping: discovery should be targeted, not exhaustive.
Where a Fetch API Fits
Most of the crawler-side work above (pacing, retries, backoff, IP rotation, rendering) is undifferentiated plumbing. A managed fetch layer like link.sc handles the politeness mechanics so your crawl budget goes to logic that matters: which URLs to queue, how to dedup them, and what to extract. You send one API call per URL and get clean Markdown back, with rate handling and retries built in.
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "YOUR_API_KEY"},
json={"url": "https://example.com/product/123", "format": "markdown"},
)
page = resp.json()["content"]
The Bottom Line
Crawl budget is capacity times demand: how much a server can take, times how much a crawler wants the pages. As a site owner, you optimize it by spending every request on a page worth indexing: cut duplicates, fix errors, stay fast, and steer crawlers with your sitemap and robots file. As a crawler, you spend budget you do not own, so the same rules run in reverse: pace yourself, dedup ruthlessly, and read the map before you walk it. The two sides are the same discipline seen from opposite ends.
Building a crawler and want the politeness handled for you? Create a free link.sc account: clean web data from one API call, pacing and retries included.