Quick answer: A compliant lead-generation pipeline scrapes only public pages, extracts firmographic and business-contact signals (company name, industry, size, role-based emails, public listings), enriches and scores them, and respects privacy law at every step. The scraping is the easy part. The part that keeps you out of trouble is treating personal data lawfully under GDPR, following CAN-SPAM for outreach, and honoring do-not-contact requests. Build the compliance in from the start, not after your first complaint.
Web scraping can feed a sales pipeline faster than any manual research, but lead gen is also where scraping bumps directly into privacy law, because leads are people. This post walks the full pipeline (find, extract, enrich, score, comply) with the compliance woven through, because bolting it on later does not work.
If your specific need is just pulling emails and phone numbers from a set of sites, the focused how-to is extract contact info from company websites. This post is the bigger picture: the end-to-end workflow and, above all, the rules.
The pipeline at a glance
| Stage | Goal | Source type |
|---|---|---|
| Source discovery | Find relevant companies | Directories, listings, search |
| Extraction | Pull firmographic and contact signals | Public company pages |
| Enrichment | Add context to each record | Multiple public sources |
| Scoring | Rank by fit and intent | Your own criteria |
| Compliance | Keep every record lawful | Applies to all stages |
Stage 1: Find the source pages
Good lead gen starts with the right sources, not the most sources. Decide who a good customer is (industry, size, region, technology) and find the public places those companies appear: industry directories, association member lists, event exhibitor pages, public company sites, and search results for your niche.
A web search API is the fastest way to build a source list, because each result comes back as structured data (title, description, and a targetUrl) that you can fetch in full to judge relevance:
curl https://api.link.sc/v1/search \
-H "x-api-key: lsc_..." \
-H "Content-Type: application/json" \
-d '{"q": "mid-size logistics companies in the Netherlands", "engine": "google"}'
From serpData.results in the response you collect each targetUrl as a candidate to process in the next stage.
Stage 2: Extract firmographic and contact signals
For each source page, pull the signals that make a lead useful. Firmographics describe the company (name, industry, size, location, tech stack), and business-contact signals are the role-based ways to reach it (a sales@ or info@ address, a contact form, a main phone line).
Fetching each page as clean markdown means you extract from content, not from brittle HTML. With link.sc you fetch the page once, then pull fields like company name, industry, headquarters, and contact email from the markdown:
curl https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example-company.com/about", "format": "markdown"}'
The structured-extraction pattern is covered in depth in extract structured JSON from any webpage. The key discipline for lead gen: prefer role-based and company-level contacts over individuals' personal details, which raises both your data quality and your compliance footing.
Stage 3: Enrich
A raw record ("Acme Logistics, logistics, Rotterdam") becomes a lead when you add context: recent news, headcount range, whether they use a technology your product complements, funding signals. Enrichment means visiting a few more public sources per company and merging what you find.
import requests
def fetch(url):
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_..."},
json={"url": url, "format": "markdown"},
timeout=30,
)
return resp.json()["content"]
def enrich(company):
company["about"] = fetch(company["website"] + "/about")
company["careers_open"] = "careers" in fetch(company["website"]).lower()
return company
Keep a provenance field on every record noting where each piece of data came from and when. You will need it if anyone ever asks why you hold their information.
Stage 4: Score
Not every extracted company is a lead worth contacting. Score each record against your fit criteria (right industry, right size, right region) and any public intent signals (hiring for a relevant role, recently expanded). Rank, then hand only the top slice to sales. This is where scraping stops being a data dump and becomes a pipeline.
Stage 5: Compliance is not optional
This is the stage that decides whether your pipeline is an asset or a liability. Read it twice.
GDPR and personal data
If any of your leads are people in the EU or UK, or you are established there, the GDPR applies, and business contacts are still personal data when they identify an individual. The obligations that bite hardest in lead gen:
- Lawful basis. For B2B prospecting, companies often rely on legitimate interest, which requires a documented balancing test weighing your interest against the person's rights. "It was public" is not, by itself, a lawful basis.
- Transparency. People have a right to know you hold their data. Have a privacy notice ready and be prepared to point to it.
- Data subject rights. You must be able to find, correct, and delete an individual's data on request. That is why provenance and a clean data model matter.
- Data minimization. Collect only what you need for the stated purpose. Hoovering up every field "just in case" works against you.
Public availability reduces some friction but does not switch off these duties. Prefer company-level and role-based data, which is less likely to be personal data at all.
CAN-SPAM and outreach
When you email the leads, US CAN-SPAM rules apply to commercial email: no deceptive headers or subject lines, a clear indication the message is a solicitation, a valid physical postal address, and a working unsubscribe that you honor promptly. Other regions layer on stricter consent regimes (several require opt-in before you email at all), so check the rules for where your recipients are.
Do-not-contact and suppression
Maintain a suppression list from day one. When someone asks not to be contacted, unsubscribes, or exercises a deletion right, they go on it and stay off every future campaign. Check every send against it. This is both a legal requirement in most regimes and the difference between a sender reputation that survives and one that gets you blocklisted.
Respect the sites too
Everything from our ethical web scraping guide applies: honor robots.txt, keep request rates polite, cache so you do not refetch the same page, and stay inside each site's terms of service. A lead pipeline that hammers a directory into an outage has already failed.
Putting it together
A defensible lead-gen pipeline is a loop: search for sources, fetch and extract public signals, enrich from more public pages, score for fit, and run every record through a compliance gate before it reaches outreach. link.sc handles the search-and-fetch layer so you can focus on the scoring and the compliance, which are the parts that actually differentiate a good pipeline. See pricing to size it to your volume.
The teams that win at scraped lead gen are not the ones who collect the most. They are the ones whose data is accurate, well-sourced, and lawful, because those leads convert and those pipelines do not blow up.
Build a lead pipeline on clean, structured public web data. Try link.sc free.