Quick answer: You scope a crawl with three levers, and you should usually set all three. A depth limit caps how many link hops the crawler follows from your seed URL (depth 2 or 3 covers most site sections). A page count cap is your hard safety net (say, 500 pages, then stop no matter what). And URL patterns (include/exclude rules like "only URLs starting with /docs/") are the most precise lever, because they define the section by shape instead of by distance.
Almost everything written about "crawl depth" online is SEO advice about how Googlebot budgets its visits to your site. This post is the other side: you're the one running the crawler, you want the docs section and not the other 40,000 pages, and you'd like the crawl to finish before lunch.
Why an Unscoped Crawl Blows Up
Crawls grow geometrically. If the average page links to 30 new pages, depth 1 is 30 pages, depth 2 is around 900, and depth 3 is pushing 27,000. Real sites dedupe a lot of those links, but the shape of the curve holds: each extra depth level multiplies your crawl, your bandwidth, and your politeness problem with the target server.
Unscoped crawls also wander. One footer link to /careers and your "documentation crawl" is now indexing job listings. One link to Twitter and, if you forgot to stay on-domain, you're crawling Twitter.
So the goal isn't just "fewer pages." It's only the right pages. That's what separates scoping from simply stopping early.
Lever 1: Depth
Depth means link hops from the seed, not slashes in the URL. If you start at example.com/docs/ and it links to /docs/api/auth, that page is depth 1 even though its path is three segments deep. Crawlers count the graph, not the path.
Depth is the bluntest of the three levers, and here's the catch most people hit: on heavily interlinked sites, depth barely means anything. If every page carries a sidebar linking to every other page in the section, the entire section is reachable at depth 1, and everything outside the section may be too. Depth limits work best on hierarchical sites (listing pages, category trees, paginated archives) and worst on modern docs sites with global navigation.
Practical starting points:
| Depth | What you typically get |
|---|---|
| 0 | The seed page only |
| 1 | Seed plus everything it links to (a hub page and its children) |
| 2 | A full site section on most hierarchical sites |
| 3 | Deep archives; also where crawl size starts to explode |
If you find yourself wanting depth 4 or more, you almost certainly want a URL pattern instead.
Lever 2: Page Count Caps
A page cap is not a scoping tool. It's a circuit breaker. It exists because your depth and pattern rules will occasionally be wrong, and "the crawl stopped at 500 pages" is a much better failure than "the crawl ran all weekend."
Set it to roughly double what you expect the section to contain. If the docs section should be about 200 pages and the cap trips at 500, that's a signal your filters are leaking, not a reason to raise the cap.
Caps also keep you polite. A bounded crawl is one you can throttle sensibly, and staying under a site's tolerance is how you avoid a wall of 429 responses halfway through.
Lever 3: URL Patterns
This is the lever that actually expresses "only the section I need." Two rules cover most jobs:
- Include: only enqueue URLs matching a prefix or pattern, like
^https://example\.com/docs/. - Exclude: drop known junk even inside that prefix:
logout,?sort=,?page=,/print/, calendar URLs.
The excludes matter more than people expect. Faceted navigation and query strings are the classic crawler traps: /products?color=red&sort=price&page=7 can generate effectively infinite unique URLs for the same content. A crawl scoped to /products/ but without query-string excludes can still run forever.
Pattern scoping also survives the interlinking problem that breaks depth limits. It doesn't matter that the global sidebar links everywhere; anything outside /docs/ never enters the queue.
Putting It Together: wget
For quick one-off jobs, wget has all three levers built in:
wget --recursive \
--level=3 \
--quota=100m \
--no-parent \
--include-directories=/docs \
--reject-regex '.*\?(sort|page|ref)=.*' \
--wait=1 \
https://example.com/docs/
--level is depth, --quota is a byte-based cap (wget lacks a page-count cap, so bytes stand in), and --no-parent plus --include-directories is the include pattern. --no-parent alone is the single most useful flag here: it means "never climb above the seed's directory," which is 90 percent of section scoping in one switch.
Putting It Together: Scrapy
For anything recurring, Scrapy makes the three levers explicit settings:
class DocsSpider(CrawlSpider):
name = "docs"
start_urls = ["https://example.com/docs/"]
allowed_domains = ["example.com"]
custom_settings = {
"DEPTH_LIMIT": 3, # lever 1
"CLOSESPIDER_PAGECOUNT": 500, # lever 2
"DOWNLOAD_DELAY": 1.0,
}
rules = [
Rule(
LinkExtractor(
allow=r"/docs/", # lever 3: include
deny=[r"\?sort=", r"\?page=", r"/print/"], # lever 3: exclude
),
callback="parse_page",
follow=True,
),
]
allowed_domains deserves a mention as lever zero: stay on the domain you came for. It's the default assumption in most frameworks, but wget will happily span hosts if you tell it to, and half of all runaway crawls I've seen started with an accidental off-site follow.
The Order That Actually Works
In my experience the productive workflow is patterns first, depth second, cap always:
- Define the include pattern from the section's URL structure. Browse the section for two minutes first; URL structure is usually obvious.
- Add excludes for query strings and utility pages.
- Set depth to 3 as a backstop, not as the primary filter.
- Set a page cap at about 2x your estimate.
- Dry-run and read the URL list before you fetch content. Crawling is discovery, and scraping is extraction; you can validate the discovery step almost for free by logging URLs without downloading bodies.
That dry run is the step everyone skips and everyone should not. Thirty seconds of scanning the queued URLs catches a leaking filter before it costs you an hour of fetching.
One more structural shortcut: if the section you want is a paginated listing, you may not need general crawling at all. Walking ?page=1 through ?page=50 with a URL template is list crawling, and it's cheaper and more predictable than link-following.
Scoping Doesn't Fix Fetching
Everything above controls which URLs you visit. It does nothing about whether those URLs actually return content, because the modern web's real crawl problems are JavaScript rendering and bot detection, not queue management. A tightly scoped 200-page crawl that gets 200 Cloudflare challenge pages is still a failed crawl.
The clean division of labor: scope the crawl yourself with the three levers, then hand each URL to a fetch layer that reliably turns it into content. That's exactly the split link.sc's Fetch API is built for: your crawler owns the frontier and the filters, and each POST /v1/fetch call returns rendered, LLM-ready markdown for one URL, with retries and anti-bot handling done for you.
Scope tight, cap always, and read the URL list before you fetch. Those three habits turn crawling from an overnight gamble into a five-minute job.
Crawling a site section for your app or your RAG pipeline? Get a free link.sc API key and let the Fetch API turn every URL your crawler finds into clean markdown.