Quick answer: In most jurisdictions, robots.txt is not a law, and ignoring it is not automatically illegal. But it can be used as evidence against you in legal disputes, ignoring it is widely considered bad practice, and sites that catch you doing it will block you. So the practical answer is: you should obey it, and the cases where deliberately ignoring it makes sense are rarer than people want them to be.
This question gets asked two ways. Sometimes it's "am I legally required to?" and sometimes it's "can I get away with not?" Those are different questions with different answers, so let's take them apart properly.
I find three lenses cover it: legal, ethical/practical, and platform. And this is where I say the obvious: I'm not a lawyer, and this isn't legal advice. It's an engineer's honest map of the terrain.
First, What robots.txt Actually Is
robots.txt is a plain text file at the root of a domain that tells automated clients which paths the site owner would prefer they not crawl:
User-agent: *
Disallow: /admin/
Disallow: /search
Crawl-delay: 2
User-agent: GPTBot
Disallow: /
The key word is "prefer." It's a request, not a mechanism. Nothing enforces it. Your HTTP client will happily fetch a disallowed path, because robots.txt is a convention from 1994 that survives entirely on voluntary compliance.
That voluntary nature is exactly why the "do I have to" question is interesting.
Lens 1: Legal
Here's the part that surprises people: in the US and most other jurisdictions, there is no statute that says "thou shalt obey robots.txt." It's a technical convention, not legislation.
But "not a law by itself" is doing a lot of work in that sentence, because robots.txt shows up in court as evidence:
- Evidence of authorization. In disputes under laws like the CFAA, a central question is whether access was authorized. A robots.txt disallow is one signal (among many) that the site owner said no. It rarely decides a case alone, but crawling past an explicit disallow makes "I had no reason to think they minded" much harder to argue.
- Terms of service interplay. Many ToS documents explicitly require compliance with robots.txt. Ignore the file and you may be breaching a contract, which is its own legal theory independent of any statute.
- The good-faith narrative. In the major web scraping cases, courts have paid attention to how scrapers behaved overall. Respecting robots.txt is part of the picture that says "careful, good-faith actor" rather than "someone who bypasses whatever is in their way."
The broader legal landscape (public vs gated data, personal data, copyright) is bigger than one file, and we've mapped it in is web scraping legal. The one-line summary: robots.txt compliance won't save an otherwise-illegal scrape, and violating it won't necessarily doom an otherwise-legal one, but it colors everything.
Lens 2: Ethical and Practical
Set the law aside. robots.txt is the clearest signal a site owner can send about what they're okay with, short of emailing you personally. Ignoring it means knowingly doing something the owner asked you not to do, on their infrastructure, at their cost.
There's also a self-interested version of this argument. Disallowed paths are often disallowed for reasons that benefit you:
- Crawler traps. Infinite calendar pages, faceted search combinations, session-URL mazes. Owners disallow them partly to save crawlers from wasting their own budget.
- Garbage content. Print views, internal search results, staging paths. You don't want that in your dataset anyway.
- Load-sensitive endpoints. Expensive search or export routes that fall over under crawl traffic, which then gets everyone blocked, including you.
And Crawl-delay, where present, is the owner telling you exactly how fast is too fast. Following it is the cheapest anti-blocking measure that exists. The full etiquette list lives in ethical web scraping best practices.
Lens 3: Platform Reality
Even if you don't care about lens 1 or 2, sites can see what you're doing.
A classic bot-detection trick is watching for clients that fetch disallowed paths; some sites even list honeypot URLs in robots.txt specifically to catch crawlers that read the file and do the opposite. Hit one and you're fingerprinted and blocked. Ignoring robots.txt doesn't just risk a courtroom someday, it risks your crawler not working next Tuesday.
There's a compounding effect too. Blocked crawlers escalate to proxies and evasion, evasion looks like intent to circumvent, and intent to circumvent is exactly what turns a gray-zone civil dispute into something scarier. The practical path and the safe legal path point the same direction: don't start down the escalation ladder.
What robots.txt Does Not Control
Worth clearing up, because the misconceptions run in both directions:
| People think robots.txt... | Reality |
|---|---|
| Makes pages private | It's public and unenforced; anyone can fetch disallowed paths |
| Controls what appears in search results | Only crawling; blocked pages can still be indexed from external links |
| Applies to humans in browsers | It only addresses automated clients |
| Governs how fetched data may be used | It says nothing about use; that's ToS and law |
| Is legally binding on its own | It's a convention; it matters as evidence and via ToS |
The flip side matters too: a path allowed by robots.txt isn't thereby fair game for anything. Personal data, copyrighted content, and rate limits are all still your problem.
So When Is Ignoring It Defensible?
Honestly, in narrow cases. Fetching a single public page on explicit user request (the way a browser or a link-preview does) is generally treated differently from bulk crawling, and archival and research crawlers have long-standing debates I won't relitigate here. Your own site, obviously, is yours to crawl.
But "my competitor blocked everyone and I want their data" is not one of the defensible cases, and that's usually what's really being asked.
At link.sc we sit with the boring consensus: politeness by default, sane rates, no pretending to be something we're not. If you want web data without personally hand-rolling these decisions, the quickstart takes about two minutes.
My bottom line: nobody will arrest you for ignoring robots.txt, and anyone who tells you it's flatly illegal is wrong. But it's the single cheapest signal of good faith you can give, it keeps you out of traps and blocks, and every scraping-adjacent legal fight goes better for the party that respected it. Obey it.
Want web data without becoming the person who ignores robots.txt at 3am? Try link.sc free and let us handle the polite crawling.