Quick answer: Google Scholar has no official API, and it aggressively rate-limits and blocks automated access, so scraping it directly is fragile and against its terms. For citation counts, paper metadata, abstracts, and reference graphs, the right answer is a purpose-built scholarly API: Semantic Scholar, OpenAlex, Crossref, or the arXiv API. They are free, documented, and built for programmatic access. Reserve careful public-page fetching for the rare gap where no API covers what you need.
Why Google Scholar resists scraping
Google Scholar is a search interface over academic literature. It is genuinely useful, and its citation counts are widely cited in turn. But Google never shipped a public API for it, and the Google Scholar terms prohibit automated querying. In practice, Scholar detects bursts of requests quickly and responds with CAPTCHAs or IP blocks.
That combination (no API plus active blocking) means any scraper you build is a maintenance treadmill. Selectors change, CAPTCHAs appear, and your IPs get flagged. Before you invest a day in that, check whether an open scholarly API already has the data. Usually it does.
The APIs that answer most "Google Scholar" questions
Most people who want to scrape Google Scholar actually want one of a few things: paper metadata, abstracts, citation counts, or the citation graph (who cites whom). Four open APIs cover almost all of it.
| Source | What it is best for | Auth | Cost |
|---|---|---|---|
| Semantic Scholar | Abstracts, citation graph, influential-citation flags, embeddings | API key (free) | Free |
| OpenAlex | Broad metadata: works, authors, venues, institutions, concepts | None (email in header polite) | Free |
| Crossref | Authoritative DOI metadata, references, funder data | None (polite pool with email) | Free |
| arXiv API | Preprints in physics, math, CS, and related fields | None | Free |
These are the actual data sources many tools sit on top of. They give you clean JSON, stable schemas, and terms that welcome programmatic use. Confirm the current rate limits on each provider's own docs, since they do change.
Semantic Scholar
Semantic Scholar (from AI2) is the closest thing to a "Google Scholar API." It exposes abstracts, references, citations, and a paper's influential citations. Request a free key for higher limits.
import requests
# Semantic Scholar Academic Graph API
resp = requests.get(
"https://api.semanticscholar.org/graph/v1/paper/search",
params={
"query": "retrieval augmented generation",
"fields": "title,year,citationCount,abstract,authors",
"limit": 5,
},
headers={"x-api-key": "YOUR_S2_KEY"}, # optional but raises limits
)
for paper in resp.json()["data"]:
print(paper["citationCount"], paper["title"])
Note the x-api-key header here belongs to Semantic Scholar, not link.sc. Different services, different keys.
OpenAlex
OpenAlex is an open catalog of the global research graph: works, authors, institutions, venues, and concepts. No key required. Including your email in the User-Agent or a mailto param gets you into the faster "polite pool."
import requests
resp = requests.get(
"https://api.openalex.org/works",
params={
"search": "graph neural networks",
"per-page": 5,
"mailto": "[email protected]",
},
)
for work in resp.json()["results"]:
print(work["cited_by_count"], work["display_name"])
Crossref and arXiv
Crossref is the system of record for DOIs. If you have a DOI and want authoritative bibliographic metadata or the reference list, Crossref is your source. The arXiv API is the right call for preprints; it returns Atom XML you can parse with any feed library.
When careful public-page fetching actually fits
There is a narrow, legitimate case for fetching public pages instead of an API: the data is public, no API exposes it, and you are respectful about it. An example is a specific author's public profile page on a university site, or a public dataset landing page that no scholarly API indexes.
If you hit that gap, link.sc can fetch a public URL and return clean markdown so you skip the HTML parsing. Note the auth header is x-api-key, and the base is https://api.link.sc/v1.
curl -X POST https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_your_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.edu/faculty/jane-doe/publications",
"format": "markdown"
}'
The response is { "content": "..." }. From there you can pass the markdown to an LLM to pull out structured fields, which is far more robust than brittle CSS selectors.
You can also run a search that returns full page content rather than snippets:
curl -X POST https://api.link.sc/v1/search \
-H "x-api-key: lsc_your_key" \
-H "Content-Type: application/json" \
-d '{
"q": "open access citation dataset licensing",
"engine": "google"
}'
The search field is q, and the response includes serpData.
What not to do
Do not point a scraper at scholar.google.com and hammer it. You will get blocked, you will violate the terms, and the data you scrape (citation counts) is available more reliably from OpenAlex or Semantic Scholar anyway. Rotating proxies to defeat Scholar's blocks is exactly the kind of anti-bot evasion that turns a data project into a terms-of-service problem.
A short legal and ethics note
Scraping is a legal gray area that depends on jurisdiction, the site's terms, whether you bypass technical access controls, and what you do with the data. Public factual data (like a citation count) is generally treated differently from copyrighted full text or personal data, but "generally" is not legal advice. Two rules keep you out of trouble: prefer the official API when one exists, and respect the technical signals a site sends (robots.txt, rate limits, terms). For the bigger picture, see our guide on whether web scraping is legal and our ethical scraping and compliance best practices.
The workflow I would actually build
- Start with OpenAlex or Semantic Scholar for the paper and citation graph. It covers the vast majority of "Scholar" use cases.
- Use Crossref to resolve DOIs and pull authoritative reference lists.
- Use the arXiv API for preprints and full-text links where the license allows.
- Only when a specific public page has data no API exposes, fetch that single page with link.sc and extract fields with an LLM.
That order gets you clean data, stays inside terms, and does not put you on a treadmill of chasing CAPTCHAs. The APIs are the right answer. Fetching is the fallback for the gap, not the default.
Building a research or citation pipeline? link.sc turns any public page into clean, structured content for the cases an API does not cover. Start free.