Quick answer: Do not scrape Indeed's job pages. Indeed's Terms of Service prohibit automated data collection, and the site actively blocks scrapers. To build a jobs product, source postings from where they originate: employer Applicant Tracking Systems (Greenhouse, Lever, Workable, and similar), aggregator partner programs, and company career pages that publish structured feeds. That data is cleaner, fresher, and legal to use.
If you want "Indeed data," what you actually want is job postings: title, company, location, salary, description, apply link. Indeed did not create those. Employers did, and they publish them in many places. Let me show you how to get them at the source instead of fighting a bot wall.
The Indeed Terms of Service Reality
Indeed's Terms of Service prohibit accessing the site through automated means, scraping, and reproducing its content. Indeed has historically enforced this, and the site uses bot detection that makes brute-force scraping fragile and short-lived.
There is also a data-provenance issue. A single job frequently appears on Indeed, LinkedIn, the company's own site, and three aggregators at once. Scraping Indeed gets you a downstream copy, often stale, sometimes deduplicated or reformatted by Indeed. Going to the source gets you the canonical posting.
So the goal is not to defeat Indeed's defenses. It is to find where each posting is published in a machine-readable, permitted form. That is almost always the ATS.
Where Job Postings Actually Come From
Most employers post jobs through an Applicant Tracking System. Many of the major ATS platforms expose public job-board APIs or endpoints specifically so that partners and job boards can ingest openings. That is the front door, and it is wide open.
| Source | Access | Data quality | Notes |
|---|---|---|---|
| Scraping Indeed HTML | Prohibited by ToS | Downstream, fragile | Blocked aggressively |
| ATS public job APIs | Documented, often keyless | Canonical, structured | Per-employer coverage |
| Aggregator partner APIs | Approved partner access | Broad | Terms and quotas apply |
| Company career pages | Public, sometimes structured | Canonical | Coverage varies |
Common ATS platforms with public job endpoints include Greenhouse, Lever, Workable, SmartRecruiters, and Ashby. Each exposes a company's open roles by a board token or slug, usually returning clean JSON. No scraping required.
Official Option 1: Aggregator Partner Programs
If you need breadth rather than specific employers, several job aggregators and networks offer partner or publisher programs with sanctioned API access. These come with terms, attribution requirements, and quotas, but they give you a legitimate firehose. Indeed itself has historically offered publisher and partner arrangements; check their current partner program terms rather than assuming the old ones still apply.
The tradeoff is straightforward: you accept the partner terms and get reliable, legal access instead of a scraper that breaks weekly.
Official Option 2: ATS Job Board APIs
This is my favorite path for a focused product. If you want jobs from a defined set of companies (say, all funded startups in a sector), you can hit each company's ATS board directly.
For example, Greenhouse and Lever expose a company's public postings by board token. You get title, location, department, and full description as structured data, published by the employer for exactly this purpose. Build a registry of the companies you care about, map each to its ATS, and poll politely.
When Fetching Public Pages Fits
Some employers, especially smaller ones, publish jobs only on their own career page with no ATS feed and no API. Those pages are public and intended to be read. Pulling them at a respectful rate to normalize into your schema is a reasonable, narrow use of fetching.
That is different from bulk-scraping Indeed. Respect robots.txt, throttle, cache, and never bypass authentication or anti-bot challenges. For the boundaries, see our guides on whether web scraping is legal and ethical web scraping best practices.
For those career-page gaps, link.sc can fetch a public URL and return clean markdown or structured JSON, so you extract the posting without running a browser farm. Use it on pages you are permitted to read.
A Code Sketch: ATS First, Fetch as Fallback
The primary source is an ATS job API. The fallback fetch handles individual public career pages with no feed.
import requests
# Primary: an ATS public job board API (Greenhouse-style example).
def fetch_greenhouse_jobs(board_token):
url = f"https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs"
r = requests.get(url, params={"content": "true"}, timeout=30)
r.raise_for_status()
jobs = r.json()["jobs"]
return [
{
"title": j["title"],
"location": j.get("location", {}).get("name"),
"url": j["absolute_url"],
"updated_at": j["updated_at"],
}
for j in jobs
]
# Some ATS and partner APIs use Authorization: Bearer tokens. That is fine.
# link.sc authenticates with the x-api-key header instead.
And the permitted public-page fallback with link.sc:
import requests
def fetch_career_page(url):
r = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_your_key_here"},
json={"url": url, "format": "markdown"},
timeout=60,
)
r.raise_for_status()
return r.json()["content"]
Note the two auth styles. ATS and partner APIs commonly use Authorization: Bearer. link.sc uses x-api-key. Keep them straight.
Building the Job Board
Once you have ATS feeds and a handful of permitted career pages, the real engineering is deduplication (the same role appears many places), normalization, and freshness. We covered that architecture in aggregate job postings, including how to merge sources and dedupe across boards.
Legal and Ethics Note
Not legal advice. But the picture is consistent: Indeed's ToS prohibits scraping, the postings are downstream copies of employer data, and enforcement is real. The compliant sources (ATS APIs, partner programs, direct career pages) give you canonical, fresher data with clear terms. Prefer them. When a posting exists only behind a login or a bot wall, that is a signal to stop, not to route around it.
The short version: jobs originate in the ATS. Go there, respect each source's terms, and reserve fetching for public career pages that have no feed.
Want to pull clean job data from the public web without babysitting scrapers? Start free with link.sc.