Short version: it depends, and anyone who gives you a confident yes-or-no is selling something. Web scraping sits at the intersection of contract law, copyright, privacy regulation, and computer-access statutes, and the answer shifts based on what you scrape, how you get to it, and where you and the site operator are.
This post is general information, not legal advice. I am an engineer writing for engineers. For anything that matters to your business, talk to a lawyer who knows your jurisdiction and your specific facts. With that said, here is the lay of the land as it is generally understood.
Public Data vs. Data Behind a Gate
The single most important distinction is whether the data is publicly available or sitting behind a login, paywall, or other access control.
Scraping data that anyone can view without signing in — a public product page, a public profile, a news article — is generally treated very differently from data you can only reach after authenticating. In the United States, the leading case here is hiQ Labs v. LinkedIn. At a high level, the courts were skeptical that scraping publicly accessible data violated the Computer Fraud and Abuse Act (CFAA), which is the federal anti-hacking law. The reasoning was roughly: if a page is open to the public, accessing it with a bot is not "unauthorized access" in the way breaking into a password-protected system would be. (The case had a long, winding history and later turned on other issues like breach of contract, which is exactly why the "it depends" caveat matters.)
The flip side: the moment you create an account, click "I agree," and then scrape, you have likely entered into a contract and you are accessing a gated system. Bypassing authentication, sharing credentials, or defeating technical access controls moves you toward genuine CFAA exposure and clear terms-of-service violations. That is a different and riskier category, full stop.
The Considerations That Actually Move the Needle
No single factor decides legality. These are the ones worth weighing:
- Terms of Service. Many sites prohibit automated access in their ToS. Whether a "browsewrap" ToS you never clicked is enforceable varies, but a ToS you affirmatively accepted (by registering) generally is. Violating it is a contract problem even when it is not a CFAA problem.
- Copyright. Facts are not copyrightable, but creative expression is. Copying and republishing substantial portions of articles, photos, or databases can infringe. Extracting facts (prices, specs, scores) is on much safer ground than wholesale republishing of prose.
- Personal data. This is where a lot of teams get surprised. If you scrape data about identifiable people, privacy law applies regardless of whether the data was "public." The EU's GDPR and California's CCPA/CPRA impose obligations around lawful basis, purpose limitation, and individuals' rights. "It was on a public web page" is not, by itself, a lawful basis under GDPR.
- The CFAA and its state analogues. As above, this centers on "unauthorized access." Public data leans one way; circumventing access controls leans hard the other way.
- Jurisdiction. You, the site, and the site's users may all be in different countries with different rules. Where you can be sued and which law applies is its own thorny question.
The practical takeaway: scraping public, factual data at a polite rate for internal analysis is the low-risk end of the spectrum. Scraping personal data, bypassing logins, or republishing copyrighted content at scale is the high-risk end.
What robots.txt Is and Isn't
robots.txt is a file at the root of a site that tells automated clients which paths the operator would prefer they not crawl:
User-agent: *
Disallow: /private/
Disallow: /search
Crawl-delay: 10
Two things are true at once. First, robots.txt is a convention, not a law. It is part of the Robots Exclusion Protocol, and there is no statute that says "thou shalt obey robots.txt." Ignoring it is not automatically illegal.
Second — and this is the part people miss — courts and regulators increasingly treat robots.txt compliance as evidence of good faith, and ignoring it as evidence of bad faith. It is also a clear, machine-readable statement of the operator's wishes, which is exactly the kind of signal that matters when someone is deciding whether your access was "authorized" or whether you acted reasonably. Respecting it is cheap insurance and, frankly, the neighborly thing to do.
So: you are not legally compelled to obey robots.txt in most cases, but obeying it is almost always the right call, both ethically and strategically.
Ethical Best Practices That Also Reduce Risk
Good manners and good legal posture overlap almost entirely here. If you do nothing else:
- Rate limit. Do not hammer a server. Add delays between requests, respect
Crawl-delayandRetry-Afterheaders, and cap concurrency per domain. Degrading someone's service for real users is where "annoying" becomes "actionable." - Identify your bot. Use an honest User-Agent with a contact URL:
MyBot/1.0 (+https://example.com/bot; [email protected]). Hiding behind a fake browser string looks like exactly what it is. - Cache and minimize. Do not re-fetch what you already have. Take only the fields you need. Less data collected is less liability and less load.
- Honor robots.txt and opt-outs. Check it, respect it, and provide a way for people to ask you to stop.
- Prefer official APIs when they exist. They are the sanctioned path and usually more stable anyway.
None of this makes scraping legal on its own, but it keeps you on the reasonable, good-faith side of every judgment call — which is where you want to be.
How link.sc Handles This
link.sc is built to make the responsible defaults the easy ones. Its crawler checks and respects robots.txt, and it rate-limits politely so you are not the reason a small site falls over. Requests go out with sane throttling rather than a flood. That means the boring compliance work — the crawl delays, the robots checks, the backoff — is handled for you, so you can focus on what you do with the data rather than how you fetched it.
That is a starting point, not a substitute for judgment. You are still responsible for what you scrape and what you do with it, especially where personal data or copyrighted content is involved. When in doubt, ask a lawyer.
If you want a fetching layer that behaves well by default, link.sc gives you 500 free credits a month. Sign up at link.sc/register or read the docs.