You want to crawl a page. Before you fire off a request, one question matters: is this URL allowed by the site's robots.txt, for the user-agent you plan to send?
Most people guess. They open robots.txt in a browser, squint at the directives, and hope they read the wildcard rules correctly. That guessing is where crawlers get themselves into trouble.
This page is a tester. Give it a site, a path, and a user-agent, and work out the real answer using the same matching rules that Googlebot and every well-behaved crawler follow.
The 30-Second Test
Here is the fastest way to check a single URL. Fetch the site's robots.txt, then read the rules for your user-agent top to bottom.
curl -s https://example.com/robots.txt
Say you want to know whether Googlebot may crawl /products/shoes. You find this block:
User-agent: Googlebot
Disallow: /products/
Allow: /products/shoes
User-agent: *
Disallow: /
The answer is allowed. Googlebot gets its own group, so the * group is ignored entirely. Within Googlebot's group, /products/ is disallowed, but the more specific Allow: /products/shoes wins. That last part trips people up constantly, so let's slow down on the rules.
How robots.txt Matching Actually Works
The Robots Exclusion Protocol became a formal standard (RFC 9309) in 2022. It is not a suggestion box. It has precise rules, and testing a URL means applying them in order.
1. Only one group applies to your bot
A crawler picks the single most specific User-agent group that matches its name. If Bingbot is named explicitly, Bingbot uses that group and nothing else. It does not fall back to the * group. The * group only applies to bots that have no group of their own.
2. The longest matching rule wins
Within your group, the crawler compares every Allow and Disallow rule against the path. The rule with the longest matching pattern decides the outcome. Length is measured by the number of characters in the path portion of the rule.
Disallow: /folder/
Allow: /folder/public/
For /folder/public/report.pdf, the Allow rule matches 15 characters versus the Disallow rule's 8. Allow wins. The URL is crawlable.
3. Ties go to Allow
When an Allow and a Disallow rule match a path with equal length, the crawler treats the URL as allowed. This is the tiebreaker that keeps overly broad blocks from locking out pages a publisher meant to permit.
4. Wildcards and anchors
Two special characters change the matching:
*matches any sequence of characters.$anchors the rule to the end of the URL.
Disallow: /*.pdf$
That blocks every URL ending in .pdf but leaves /report.pdf?download=1 allowed, because the query string sits after where the $ anchor expects the string to end.
5. No robots.txt means fully allowed
If the file returns a 404, or the server returns a 4xx status other than 429, the whole site is treated as crawlable. A 5xx response is different: a well-behaved crawler treats a persistent server error as "disallow everything" until the file comes back.
A Quick Reference Table
| Scenario | Result |
|---|---|
| No robots.txt (404) | Everything allowed |
Disallow: with empty value |
Everything allowed |
Disallow: / |
Everything blocked for that group |
| Explicit group for your bot | The * group is ignored |
Allow and Disallow both match, same length |
Allowed |
Longer Disallow than Allow |
Blocked |
| Server returns persistent 5xx | Treat as fully blocked |
Testing It Programmatically
For a single path, Python's standard library already ships a parser. No dependencies, no install.
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
path = "/products/shoes"
agent = "Googlebot"
print(rp.can_fetch(agent, path)) # True or False
This is fine for spot checks. It falls short in two spots worth knowing about. The parser fetches robots.txt from your own IP, so a site that blocks datacenter ranges may hand you a different file (or a challenge page) than a real browser sees. And urllib follows an older interpretation of the longest-match rule that occasionally disagrees with Google's current parser on edge cases.
If you are testing at scale, or the target site is picky about who fetches its robots.txt, you want the file retrieved the way a normal client would see it. That is where a fetch API earns its keep.
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"Authorization": "Bearer YOUR_KEY"},
json={"url": "https://example.com/robots.txt"},
)
print(resp.json()["content"])
You get the actual robots.txt as the site serves it to a well-behaved client, then run your matching logic against clean text. The link.sc fetch endpoint handles the retrieval so a hostile origin does not quietly feed you the wrong rules.
Common Mistakes This Tester Catches
Assuming the * group applies to your named bot. It does not. If you send User-agent: MyScraper and the site has a MyScraper group, only that group counts.
Reading Disallow as case-insensitive. Paths in robots.txt are case-sensitive. /Admin/ and /admin/ are different rules.
Forgetting that robots.txt is per host and scheme. The file at https://example.com/robots.txt does not govern https://blog.example.com/ or http://example.com/. Each host and scheme has its own file.
Confusing crawl rules with legal permission. A URL being allowed by robots.txt is a technical answer, not a legal one. Robots.txt says nothing about copyright, terms of service, or personal data. For that side of the question, read our take on whether web scraping is legal and the compliance checklist for ethical scraping.
robots.txt Is Not the Whole Picture
Robots.txt controls crawl access. It does not tell an AI system which pages are worth reading, and it is a separate file from the newer standard aimed at language models. If that distinction matters to you, we broke it down in llms.txt vs robots.txt.
For most crawling work, the workflow is simple. Fetch robots.txt, find the group for your user-agent, apply longest-match with the Allow tiebreaker, and respect the answer. Do that on every host before you crawl it, and you stay on the right side of the protocol.
Want robots.txt fetched exactly the way a real client sees it, then any allowed page returned as clean Markdown? Start free with link.sc.