You want company reviews and salary ranges from Glassdoor for recruiting research, an employer-brand audit, or a compensation benchmark. You write a quick script, run it, and get back a login wall, a 403, or an endless "verifying you are human" spinner. Glassdoor is one of the most aggressively defended sites on the public web, and it does not fall to a spoofed User-Agent.
Before any code: this is about collecting the data Glassdoor already shows the public, responsibly. It is not about scraping private profiles, defeating a login, or republishing scraped reviews as your own. Read the terms, respect what is gated, and think hard about the personal data you are touching. More on that below, because on Glassdoor the legal and ethical framing matters more than the scraping mechanics.
Why Glassdoor Is Harder Than a Normal Site
Glassdoor stacks nearly every defense a modern site can run, and a plain HTTP request trips most of them at once.
- TLS and HTTP/2 fingerprinting. Your client's TLS handshake gives away that it is Python or Go long before the server reads a header. Spoofing the User-Agent changes nothing, because the User-Agent was never what flagged you.
- A JavaScript-rendered app. Review text, ratings, and salary distributions load through client-side calls, not in the first HTML payload. Fetch the raw page and you get a shell.
- Soft login and metering walls. Glassdoor loves to show you one review, then demand an account or a "contribute to keep reading" gate. That wall is a product decision, not a bug you should route around with stolen credentials.
- IP reputation scoring. Datacenter IP ranges (AWS, GCP, cheap VPS hosts) carry low trust. Hit the site a few times from one and you get challenged or throttled.
I wrote a deeper breakdown of the fingerprinting layer in how to scrape a Cloudflare-protected site, and the pattern is the same here: the block usually lives a layer below the HTTP request you are staring at.
Start With the ToS and PII Question, Not the Code
This is the part most scraping tutorials skip, and on Glassdoor it is the part that can actually hurt you.
Glassdoor's terms of service explicitly prohibit automated access and scraping. That does not automatically make scraping public data illegal (US case law like hiQ v. LinkedIn leans toward public data being fair game), but a ToS breach is real and can get your accounts and IPs banned. I covered the nuance in is web scraping legal; the short version is that "public" and "permitted" are not the same word.
The bigger issue is personal data. Salary submissions and reviews are written by identifiable employees, and even "anonymous" reviews can deanonymize someone at a small company. If you operate in the EU or UK, that is GDPR-regulated personal data the moment you store it. In California, CCPA applies.
So a few rules I hold to:
- Aggregate, do not archive individuals. Store the salary distribution for a role, not "this person at this company said X."
- Never scrape behind the login. If it requires an account, it is gated, and gated content is off the table.
- Do not republish review text verbatim. Summarize trends. Reproducing copyrighted reviews is a separate problem from collecting them.
- Skip anything that identifies a real person unless you have a lawful basis to process it.
Get this framing right and the rest is just engineering. Get it wrong and no amount of clever proxying saves you.
The Escalation Ladder for Glassdoor
The reliable approach climbs only as high as the site forces you to, because higher rungs are slower and pricier.
| Rung | Method | Clears | Cost |
|---|---|---|---|
| 1 | Browser-TLS HTTP client (curl_cffi) |
Basic fingerprint checks | Cheap, fast |
| 2 | Stealth headless browser | JS rendering plus interstitials | Memory and time per page |
| 3 | Residential IP egress | IP reputation blocks | Slowest, priciest |
Rung 1 is a client like curl_cffi that presents a genuine Chrome TLS and HTTP/2 fingerprint. It clears the fingerprint layer at nearly the speed of a normal request.
from curl_cffi import requests as cffi
r = cffi.get("https://www.glassdoor.com/Reviews/index.htm", impersonate="chrome")
print(r.status_code)
For Glassdoor specifically you will usually need Rung 2, because the review and salary data render client-side. That means a hardened headless browser that runs the JavaScript, waits for the data calls to resolve, and reads the rendered DOM. See scraping JavaScript-rendered websites for the mechanics.
Rung 3 is the one people miss. The live ladder that pairs a stealth browser with a clean residential IP is exactly what gets you past Glassdoor's reputation scoring. A stealth browser on a datacenter IP still looks like a datacenter. A stealth browser reached through a residential exit looks like a person on a home connection, which is the combination Glassdoor is not tuned to reject. Fetch-through-residential is the missing tier for sites this defensive.
Rate Limiting and Politeness
Being reliable and being polite are the same discipline here.
- Crawl slowly. A handful of pages per minute, with jitter and backoff, not a burst. Glassdoor's throttling triggers on rhythm as much as volume.
- Cache everything. Do not refetch a review page you already have. Salary data barely moves week to week.
- Back off on a
429. If you get rate-limited, honor it and slow down rather than rotating IPs to punch through. The 429 error guide has the details. - Stop when the signal is clear. Repeated hard blocks mean the site does not want you there. Respect that.
Letting the Ladder Run Itself
Building all three rungs is a real project: TLS impersonation, a fleet of patched browsers, a residential proxy pool, cookie handling, retries, and per-domain tuning that drifts every time the site updates.
link.sc is one API that runs that ladder for you. You pass a URL and get back clean markdown, JSON, or raw HTML, and the escalation happens internally, cheapest-first, learning which method a domain needs so it does not waste a browser render where a fast request would do.
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_YOUR_KEY"},
json={"url": "https://www.glassdoor.com/Reviews/company-reviews.htm", "format": "markdown"},
)
print(resp.json()["content"])
One request, and the fingerprinting, JS rendering, residential egress, and parsing are handled. It is open core and self-hostable if you would rather run the stack yourself. You still own the ToS and PII judgment calls; the API only handles reliable access to what is already public.
Collecting public review and salary data reliably is a solved problem: grab a free API key with 500 credits a month and let the escalation ladder run itself.