Most threat intelligence starts as unstructured text scattered across the public web: a new CVE on a vendor advisory, a company domain showing up in a paste, a phishing kit template posted to a forum. The signal is out there. The hard part is collecting it fast enough to matter.
That collection layer is web scraping. Not the scary kind, the boring plumbing kind that keeps a SOC informed.
Let me walk through where scraping actually earns its keep in security work, and how to build a fetch pipeline that does not get you blocked, sued, or paged at 3am.
What threat intel teams actually collect
Threat intelligence is only as good as its inputs. Here are the categories that lean hardest on scraping, and what each one feeds.
| Source type | What you pull | Feeds |
|---|---|---|
| Vendor advisories | New CVEs, patch notes, EOL notices | Vuln management, patch prioritization |
| Paste sites and dumps | Leaked creds, config files, tokens | Breach detection, account takeover alerts |
| Forums and marketplaces | Kits, exploit chatter, access sales | Actor tracking, early warning |
| News and blogs | Campaign writeups, IOC lists | Detection engineering, hunting |
| Certificate transparency and DNS | New subdomains, lookalike domains | Attack surface, phishing takedown |
None of these arrive in a clean JSON feed. Some have APIs, most do not, and the ones that do rate-limit you into uselessness the moment you scale.
OSINT: the reconnaissance layer
Open-source intelligence is where scraping and security overlap most obviously. Analysts pull public data about a target (yours, defensively, or an adversary's) and turn it into a picture.
A defensive OSINT workflow usually looks like this:
- Enumerate your own external footprint: subdomains, exposed panels, forgotten S3 buckets referenced in old pages.
- Monitor mentions of your brand and domains across news, social, and paste sites.
- Watch for lookalike domains registered to phish your users.
The scraping job here is monitoring, not a one-time crawl. You want the same set of pages fetched on a schedule, diffed against last time, and alerted on change. A newly registered yourcompany-login.com that resolves to a live page is worth a same-day alert.
Breach and paste monitoring
Credential leaks show up in predictable places before they show up in your logs. Paste sites, breach-index pages, and dump aggregators post plaintext that mentions email domains, internal hostnames, or API key prefixes.
The pattern is simple: fetch the page, extract the text, run it against a watchlist of your identifiers.
import requests
WATCH = ["@acmecorp.com", "acme-internal", "AKIA"] # domains, hostnames, key prefixes
def scan(url):
r = requests.get(
"https://api.link.sc/v1/fetch",
params={"url": url},
headers={"Authorization": "Bearer YOUR_KEY"},
)
text = r.json()["content"]
hits = [w for w in WATCH if w in text]
if hits:
alert(url, hits)
The Fetch API returns cleaned text instead of raw HTML, which matters here because paste sites wrap content in enough markup that naive grep on the raw response produces false positives on nav and ad code. You want the body, not the chrome.
One honest caveat: a lot of the interesting material lives behind logins or on sites that actively fight scrapers. Public paste monitoring catches the careless leaks, which is still most of them. It will not catch a closed Telegram channel. Set expectations accordingly.
CVE and advisory feeds
Vulnerability data is the most structured of the bunch, and it is still a mess. NVD has an API. Vendor advisories mostly do not, or they publish to NVD days late. If your patch window is measured in hours, days late is a breach.
So teams scrape the vendor pages directly: Cisco, Microsoft, Fortinet, whoever is in their stack. Fetch the advisory index, diff for new entries, parse severity and affected versions, and route the high-severity ones straight to the patch queue.
The trick that saves you here is normalization. Every vendor formats severity, product names, and version ranges differently. Pull the raw advisory text, then run it through an extraction step (regex for the predictable fields, an LLM for the messy prose) to land everything in one schema. Feeding clean markdown to a model instead of raw HTML cuts token cost and hallucination both, which is the same argument I made in token optimization for feeding web data to LLMs.
A live-fetch pipeline that holds up
Here is the architecture I would actually run. Four stages, each independently retryable.
watchlist ──> scheduler ──> fetch ──> extract ──> match ──> alert
(cron) (link.sc) (regex+LLM) (rules) (slack/SIEM)
The stages:
- Scheduler. A cron job or queue that emits URLs to check. Advisories hourly, paste feeds every few minutes, brand OSINT daily.
- Fetch. This is the stage that breaks in-house. Target sites rotate IPs into blocklists, throw Cloudflare challenges, and rate-limit datacenter ranges hard. Offloading fetch to a managed API that handles residential routing and JS rendering means your pipeline logic never has to know a site is protected.
- Extract. Convert to text or markdown, pull structured fields.
- Match and alert. Run against watchlists, dedupe against yesterday, push only new hits to your SIEM or chat.
Why not just run headless Chrome yourself? You can, and for a handful of static pages it is fine. At scale you inherit a proxy-management problem, a browser-fleet problem, and a maintenance treadmill every time a target changes its anti-bot setup. That decision tree is worth its own read: curl vs headless vs stealth browser.
Keep it defensive and legal
This is a security blog post, so I will be blunt about the guardrails.
Scrape public data only. Public means no login, no bypassing access controls, no credential stuffing to reach a page. The moment you authenticate into something you are not authorized for, you have left threat intelligence and entered the thing you are supposed to be defending against.
Respect the same norms you would in any collection job: honor robots directives where they apply, rate-limit yourself, identify your bot, and keep provenance records for every artifact so an audit can trace where a given IOC came from. The ethical scraping guide covers the compliance side in depth, and it applies cleanly to security use cases.
Two more security-specific rules:
- Sandbox what you fetch. Threat-intel pages serve malicious content by definition sometimes. Do not render untrusted pages in a browser sitting on your production network. Fetch through an isolated service, treat the response as hostile data, and never execute anything it contains.
- Store minimally. Leaked credentials you collect for monitoring are still leaked credentials. Hash them, alert on them, and do not build a second copy of the breach in your own datastore.
Where this goes
The volume of security-relevant text on the public web grows faster than any team can read. The teams that win are the ones who treat collection as infrastructure: reliable, scheduled, normalized, and boring, so the analysts can spend their attention on judgment instead of copy-paste.
Get the fetch layer right and the rest of the pipeline is just rules and routing. Get it wrong and you spend your quarter fighting blocklists instead of adversaries.
Build a threat-intel fetch pipeline that does not get blocked: start free with link.sc.