You are building a reputation dashboard. Maybe you track your own Trustpilot score against three competitors, maybe you feed review text into a sentiment model, maybe you just want to know when a one-star review lands before a customer tweets about it. Either way you point a script at a Trustpilot business page, and instead of reviews you get a challenge page, a 403, or HTML with no review text in it at all.
Before any code, the honest framing: this is about collecting the reviews Trustpilot already shows the public, for legitimate reputation monitoring. It is not about faking reviews, scraping private account data, or republishing someone else's reviews as your own content. On a review platform the terms and the personal-data rules matter as much as the request mechanics, so start there.
Check the Official API First
Unlike most scraping targets, Trustpilot has a real API. The Business API and the public Consumer/Product Reviews API expose ratings, review text, and aggregate scores through documented endpoints with an API key.
If you have a Trustpilot business account for your own domain, this is the correct path. It is sanctioned, stable, and it will not break when the site ships a redesign. Scraping the HTML to read your own reviews when an API hands them to you is effort spent in the wrong place.
Scraping earns its keep in the cases the API does not cover well: monitoring competitors you do not own, pulling a broad cross-industry sample, or backfilling historical reviews the API rate limits make painful. That is the scope this guide is about.
Get the ToS and PII Question Right
This is the part most tutorials skip, and on a review site it is the part that can actually hurt you.
Trustpilot's terms prohibit automated collection. That does not automatically make scraping public pages illegal, since US case law such as hiQ v. LinkedIn leans toward public data being fair game, but a terms breach is real and gets your IPs and accounts banned. I dug into the nuance in is web scraping legal. "Public" and "permitted" are not the same word.
The heavier issue is personal data. Every review is written by an identifiable person, with a display name, sometimes a country, sometimes a review history that deanonymizes them. The moment you store that, you are processing personal data, which means GDPR in the EU and UK and CCPA in California.
A few rules I hold to:
- Aggregate, do not archive individuals. Store the star distribution and sentiment trend for a business, not "this named person said X on this date."
- Do not republish review text verbatim. Summarize themes. Reproducing reviews wholesale is a copyright and attribution problem separate from collecting them.
- Never touch gated or account data. If it needs a login, it is off the table.
- Have a lawful basis before you keep anything that identifies a real reviewer.
Get this framing right and the rest is engineering. Get it wrong and no clever proxying saves you. The same discipline applies to any review target, which is why the Glassdoor reviews walkthrough leads with the identical warning.
Why Trustpilot Is Harder Than a Static Blog
Trustpilot is a modern app with real defenses, and a plain HTTP request trips several of them at once.
- TLS and HTTP/2 fingerprinting. Your client's handshake reveals it is Python or Go before the server reads a single header. Spoofing the User-Agent changes nothing, because the User-Agent was never what flagged you.
- Client-side rendering. Review cards and pagination hydrate through JavaScript, so the raw HTML you fetch is often a shell. See scraping JavaScript-rendered websites for the mechanics.
- Bot-management challenges. Interstitials and rate challenges kick in when traffic looks automated, the same family of defenses covered in scraping Cloudflare-protected sites.
- IP reputation scoring. Datacenter ranges from AWS, GCP, and cheap VPS hosts carry low trust. A few hits from one and you get challenged.
The block usually lives a layer below the request you are staring at.
The One Trick That Simplifies Everything: Read the Schema
Here is the insight that saves the most work. Trustpilot embeds structured schema.org Review and AggregateRating data in the page as JSON-LD, the same markup that lets Google show star ratings in search results.
That means you often do not need to fight the rendered DOM at all. Parse the <script type="application/ld+json"> block and you get ratings, review counts, and review bodies as clean JSON, already labeled. It is far more stable than CSS selectors, which shift with every redesign.
import json, re
def extract_reviews(html: str):
blocks = re.findall(
r'<script type="application/ld\+json">(.*?)</script>',
html, re.DOTALL,
)
for raw in blocks:
data = json.loads(raw)
agg = data.get("aggregateRating")
if agg:
return {
"rating": agg.get("ratingValue"),
"count": agg.get("reviewCount"),
"reviews": data.get("review", []),
}
return None
Treat the JSON-LD as your primary source and the visible DOM as a fallback. This is structured data extraction done the easy way: let the site hand you the schema it already built for search engines.
The Escalation Ladder
Climb only as high as the site forces you to, because higher rungs are slower and pricier.
| Rung | Method | Clears | Cost |
|---|---|---|---|
| 1 | Browser-TLS HTTP client (curl_cffi) |
Fingerprint checks | Cheap, fast |
| 2 | Stealth headless browser | JS rendering, interstitials | Memory and time per page |
| 3 | Residential IP egress | IP reputation blocks | Slowest, priciest |
Rung 1 is a client that presents a genuine Chrome TLS and HTTP/2 fingerprint, clearing the fingerprint layer at near-normal speed.
from curl_cffi import requests as cffi
r = cffi.get("https://www.trustpilot.com/review/example.com", impersonate="chrome")
print(r.status_code)
Because the JSON-LD often ships in the initial HTML, Rung 1 alone gets you further on Trustpilot than on a fully client-rendered site. When pagination or a challenge blocks you, add Rung 2 to render and wait for the data calls. When reputation scoring throttles you, add Rung 3: a stealth browser reached through a residential exit looks like a person on a home connection, which is the combination the site is not tuned to reject.
Rate Limiting and Politeness
Being reliable and being polite are the same discipline.
- Crawl slowly. A handful of pages per minute with jitter, not a burst. Throttling triggers on rhythm as much as volume.
- Cache aggressively. Scores barely move hour to hour. A daily or hourly poll is plenty for reputation and competitor monitoring.
- Back off on a
429. Honor the limit rather than rotating IPs to punch through it. The 429 guide covers the details. - Stop when the signal is clear. Repeated hard blocks mean the site does not want you there.
Letting the Ladder Run Itself
Building all three rungs is a real project: TLS impersonation, a fleet of patched browsers, a residential proxy pool, cookie handling, retries, and per-domain tuning that drifts on every redesign.
link.sc runs that ladder for you. You pass a URL and get back clean markdown, JSON, or raw HTML, and the escalation happens internally, cheapest-first, learning which method a domain needs so it never wastes a browser render where a fast request would do.
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_YOUR_KEY"},
json={"url": "https://www.trustpilot.com/review/example.com", "format": "markdown"},
)
print(resp.json()["content"])
One request, and the fingerprinting, rendering, residential egress, and parsing are handled. It is open core and self-hostable if you would rather run the stack yourself. You still own the ToS and PII judgment calls; the API only handles reliable access to what is already public.
Turn public reviews into a reputation feed without babysitting proxies: grab a free API key with 500 credits a month and let the escalation ladder run itself.