Here is the mistake I see most often: a team scrapes public profile pages, someone asks about GDPR, and the answer is "it's public data, so we're fine." That answer is wrong. Under GDPR, "publicly available" is not a lawful basis, not an exemption, and not a defense.
The good news is that GDPR does not ban scraping. Plenty of scraping is perfectly compliant, and a lot of it never touches GDPR at all. The trick is knowing which side of the line you are on and doing the (fairly mechanical) compliance work when you cross it.
The usual disclaimer applies: I am an engineer, not a lawyer, and this is general information rather than legal advice. For the broader legal picture beyond the EU, see our post on whether web scraping is legal.
When GDPR Applies to Your Scraper (and When It Doesn't)
GDPR regulates the processing of personal data, meaning any information relating to an identified or identifiable person. Scraping is processing. Collecting, storing, parsing, and analyzing all count.
So the first question is not "is the data public?" but "does the data relate to people?"
| What you scrape | GDPR relevant? |
|---|---|
| Product prices, specs, stock levels | No, no personal data involved |
| News articles (author bylines stripped) | Generally no |
| Company pages with named contacts | Yes |
| Public social or professional profiles | Yes, squarely |
| Reviews with usernames | Yes, usernames can identify people |
| Forum posts, comments | Yes, and watch for special categories |
Two things surprise people here. First, territorial scope: GDPR applies if you are established in the EU, or if you are anywhere else and monitoring the behavior of people in the EU. A US company scraping EU residents' profiles is in scope. Second, identifiability is broad. A username plus a profile photo plus a city is personal data even without a legal name.
If your pipeline only ever touches prices, documentation, or content with people fields stripped at ingestion, you can mostly stop reading. If not, keep going.
Lawful Basis: Legitimate Interests Is Your Only Realistic Option
Every processing of personal data needs one of six lawful bases under Article 6. For scraping, the field narrows fast.
Consent is out. You have no relationship with the people whose data you are collecting, so you cannot ask them first. Contract, legal obligation, vital interests, and public task rarely fit a commercial scraper.
That leaves legitimate interests, Article 6(1)(f). It is workable, but it is a test you have to actually run and document, not a box you tick. Regulators expect a three-part assessment:
- Purpose. Is your interest legitimate and specific? "Market research on hiring trends" is specific. "Building a big dataset in case it's useful" is not.
- Necessity. Is scraping this data necessary for that purpose, and is there a less intrusive way? If aggregated or anonymized data would do the job, you are expected to use it.
- Balancing. Do your interests override the individuals' rights and reasonable expectations? A person posting a professional profile reasonably expects recruiters to read it. They do not reasonably expect it to be cloned into a face-recognition database.
That last example is not hypothetical. Clearview AI scraped billions of public photos and was fined roughly 20 million euros each by regulators in Italy, France, and Greece, precisely on the grounds that public availability did not create a lawful basis. The EDPB has since repeated the point in its guidance on scraping for AI training: legitimate interests can work, but only with a real assessment behind it.
Write the assessment down (it is called an LIA). It is a one-page document, and it is the first thing a regulator asks for.
Data Minimization: The Cheapest Compliance Win
Article 5 requires that you collect only the data you need. For scraping pipelines, this is genuinely the highest-leverage move, because every field you drop reduces legal exposure, storage, and breach impact at the same time.
Concretely:
- Filter at extraction, not at analysis. If you need review sentiment, extract the review text and rating and discard usernames and avatars before anything hits disk.
- Aggregate early. If the deliverable is "average salary by role," individual rows are a liability the moment the aggregate exists.
- Never touch special category data. Health, religion, political opinions, sexual orientation, and biometric data are near-impossible to justify under Article 9 for scraped datasets. Forum and social content is full of this. Filter it out.
- Set retention limits. Scraped personal data should have a TTL. "We keep everything forever" fails the storage limitation principle on its face.
A structured extraction step helps here. If you pull pages as clean Markdown or targeted fields through a fetch API rather than warehousing raw HTML, you decide at ingestion exactly which fields exist downstream, and the PII you never stored is PII you never have to defend, delete, or disclose.
The Part Everyone Forgets: Transparency and DSARs
Here is the obligation that catches scraping teams off guard. Article 14 says that when you collect personal data from a source other than the person, you must inform them, normally within a month. For scraping, that usually means a public privacy notice describing what you collect, why, your lawful basis, and how to object. Where notifying millions of individuals directly is genuinely impossible, the disproportionate-effort exemption can apply, but you still owe the public notice.
Then come data subject rights. People whose data you scraped can send you:
- Access requests (DSARs): "Show me everything you hold about me." You need to be able to search your scraped datasets by person and export the results.
- Erasure requests: "Delete it." And it has to stay deleted, which is awkward if your scraper will happily re-collect the same profile next week. You need a suppression list your pipeline checks.
- Objections: Under legitimate interests, people can object, and for anything resembling direct marketing the objection is absolute.
This is the real cost of scraping personal data. Not the scraping, the aftermath. If you cannot answer "how would we find and delete one person from this dataset within 30 days?", you are not ready to hold it. The Polish DPA fined a company in 2019 not for scraping public business registries, but for deciding that emailing millions of notices was too expensive. The obligation was the notice, and skipping it was the violation.
A Practical Compliance Checklist
If you scrape anything involving people in the EU, the minimum viable setup looks like this:
- Map what personal data your pipeline touches, field by field.
- Write a legitimate interests assessment for each purpose.
- Minimize at ingestion: drop identifier fields you do not need, block special category content, set retention TTLs.
- Publish an Article 14 privacy notice covering your scraping.
- Build a DSAR path: per-person search, export, deletion, and a re-scrape suppression list.
- Respect robots.txt and rate limits anyway; good-faith collection practices matter in the balancing test, and they overlap heavily with ethical scraping practices you should follow regardless.
None of this is exotic engineering. Most of it is a filter step, a document, and a delete endpoint. Teams get in trouble not because compliance is hard, but because they assumed "public" meant "free to keep forever."
Need a fetching layer that behaves politely by default while you handle the data side? Sign up at link.sc/register for 500 free credits a month.