Quick answer: The best job aggregators source from company career pages and official feeds, not from scraping the big boards, whose terms mostly prohibit it. The pipeline is: discover postings via career pages, sitemaps, and ATS feeds; extract structured fields (title, location, salary, description); dedupe aggressively, because the same job appears everywhere; and expire stale postings fast. Dedupe and freshness are where aggregators earn or lose their reputation.
I've watched a few job-aggregation projects go from prototype to product, and the pattern is consistent: sourcing decisions made in week one determine whether the whole thing is viable in month six. So let's start there.
Where to Get Postings (and Where Not To)
| Source | Allowed? | Quality | Notes |
|---|---|---|---|
| Company career pages | Generally yes (public, check robots.txt) | High | The canonical source |
| ATS-hosted boards (Greenhouse, Lever, Ashby, Workable) | Often yes, several expose public JSON APIs | High | Structured, stable, wonderful |
| Google Jobs structured data (JobPosting JSON-LD) | Yes, it's on the employer's own page | High | Machine-readable by design |
| Aggregator boards (Indeed, LinkedIn, Glassdoor) | Mostly no, ToS prohibit scraping | Mixed | Don't build on this |
| Official partner feeds/APIs from boards | Yes, with agreement | High | The legitimate path to board data |
| Government and university job sites | Usually yes | High | Often refreshingly scrape-friendly |
The honest version of the legal note: the major boards explicitly prohibit scraping in their terms, several have litigated over it, and an aggregator built on scraping aggregators is legally shaky and also just worse data, since you're copying copies. Company career pages are the source of truth, they're public, and employers actively want those postings seen. Build there. For the broader legal picture, see is web scraping legal.
Discovery: Finding the Postings
Three discovery mechanisms cover most of the market:
ATS APIs. A huge share of tech and mid-market companies host jobs on a handful of ATS platforms, and several expose public JSON endpoints per company (Greenhouse and Lever are the classic examples). Once you know a company's ATS slug, discovery is a clean API call, no scraping involved.
Sitemaps. Career sites usually list job URLs in their sitemap, often with lastmod dates that tell you what changed since yesterday. This is the cheapest incremental discovery you'll find. The technique generalizes; I covered it in how to crawl an entire website.
Search. For filling gaps, a web search API can find career pages you don't know about yet:
curl https://api.link.sc/v1/search \
-H "x-api-key: lsc_your_key" \
-H "Content-Type: application/json" \
-d '{"q": "site:boards.greenhouse.io \"data engineer\" remote", "engine": "google"}'
Extraction: JSON-LD First, LLM Second
Because Google Jobs rewards it, a large fraction of postings embed schema.org JobPosting markup: title, location, salary range, employment type, and posting date, all machine-readable. Parse that first. For pages without it, fetch the page as clean markdown and hand it to an LLM; that combination handles the long tail of weird career-page layouts:
import requests
FIELDS = {
"title": "job title",
"company": "hiring company name",
"location": "location, or 'Remote' if remote",
"remote": "true if fully remote",
"salary_min": "minimum salary as a number, null if absent",
"salary_max": "maximum salary as a number, null if absent",
"currency": "salary currency code, null if absent",
"employment_type": "full_time, part_time, contract, or internship",
"posted_date": "date posted, ISO format if determinable",
"description": "job description as markdown",
}
def fetch_job_page(url: str) -> str:
r = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_your_key"},
json={"url": url, "format": "markdown"},
timeout=60,
)
r.raise_for_status()
return r.json()["content"]
def extract_job(url: str) -> dict:
content = fetch_job_page(url)
# Hand the markdown plus FIELDS to your LLM of choice
# and ask for a JSON object matching those keys.
return llm_extract(content, FIELDS)
Two extraction rules I'd insist on: store salary_min and salary_max as nullable numbers rather than parsing "competitive salary" into fiction, and keep the raw description alongside your structured fields so you can re-extract when your schema evolves. The pattern is the same one from extract structured JSON from any webpage.
Dedupe: The Same Job Is Everywhere
One real vacancy shows up on the company site, two ATS mirrors, and a syndicated copy or three. If you don't dedupe, your users see six listings for one job and conclude your aggregator is spam.
Matching signals, in order:
- Canonical URL or ATS job ID. When two sources point at the same application URL, they're the same job. This catches a surprising majority.
- Company + normalized title + location. Normalize titles ("Sr." to "Senior", strip "(Remote)") before comparing.
- Description similarity. Embed descriptions and compare; near-identical text at the same company is the same role. Use this to merge, never to auto-delete, and keep the version from the most authoritative source (the company's own page wins).
def job_key(job: dict) -> str:
title = normalize_title(job["title"])
loc = normalize_location(job["location"])
return f'{job["company"].lower()}|{title}|{loc}'
Exact-key matching plus embedding-based merge review gets you clean enough for launch. Perfect dedupe doesn't exist; visible dupes on page one is what you're preventing.
Freshness: Expire Ruthlessly
Nothing kills a job aggregator like dead listings. People click, hit a 404 or a "position filled" page, and never come back. Freshness policy matters more than corpus size:
- Re-check every active posting's URL on a rolling schedule (daily for hot categories, every 2 to 3 days otherwise). A 404, a redirect to the careers home, or a "no longer accepting applications" string means expire it.
- Trust sitemap
lastmodand ATS API presence as cheap liveness signals before spending a full fetch. - Show posting age in your UI and be honest about it. "Posted 34 days ago" builds more trust than hiding the date.
A smaller board where every listing is real beats a huge board that wastes people's application effort. That's the entire competitive dynamic in this space.
Compliance Note
Beyond ToS: job postings contain contact details sometimes, and applicant flows always involve personal data. Aggregate the postings, link out to the original application page, and don't proxy applications through your system unless you're ready to handle personal data properly (GDPR, CCPA, retention policies). Respect robots.txt on every career site, identify your crawler honestly, and keep per-site request rates trivially low; a career site with 40 openings needs 40 fetches every couple of days, not a firehose.
The Bottom Line
Source from career pages, ATS APIs, and official feeds; skip the big boards unless you have a partner agreement. Extract JSON-LD first and use LLM extraction for the messy remainder. Dedupe on canonical URLs and normalized keys, expire listings ruthlessly, and link people to the real application page. Do those things and you'll have the rarest thing in this category: a job board people trust.
Building a job data pipeline? link.sc fetches and structures career pages with one API call, search included. Start free with 500 credits a month.