← All posts

How to Scrape GitHub Data With the Official API

GUIDES · JUL 18, 2026 · 5 MIN READ

Quick answer: Do not scrape GitHub's HTML pages. GitHub ships a full REST API and a GraphQL API that return repos, issues, pull requests, users, and code search results as clean JSON, with generous rate limits once you authenticate with a token. Use those. Fetching a public page only makes sense for the rare thing the API does not expose, and GitHub's terms explicitly discourage scraping when an API exists.

If you want data out of GitHub, you almost never need a scraper. GitHub is a developer platform, and it treats its API as a first-class product. That changes the calculus completely: the sanctioned path is also the easier and faster one.

Start With the REST API

The REST API lives at https://api.github.com and covers nearly everything you see in the UI. Fetching a repository's issues looks like this:

import requests

token = "ghp_your_token_here"  # a fine-grained or classic PAT
headers = {
    "Authorization": f"Bearer {token}",
    "Accept": "application/vnd.github+json",
    "X-GitHub-Api-Version": "2022-11-28",
}

resp = requests.get(
    "https://api.github.com/repos/getlinksc/linksc-go/issues",
    headers=headers,
    params={"state": "open", "per_page": 100},
    timeout=30,
)
for issue in resp.json():
    print(issue["number"], issue["title"])

Note that yes, GitHub's own REST API uses Authorization: Bearer for its tokens. That is GitHub's contract, not a general rule. (link.sc, by contrast, uses x-api-key, which we will get to.)

The endpoints you will reach for most:

You want Endpoint
A repo's metadata GET /repos/{owner}/{repo}
Issues / PRs GET /repos/{owner}/{repo}/issues
Commits GET /repos/{owner}/{repo}/commits
A user or org GET /users/{username}
Search repos GET /search/repositories?q=...
Search code GET /search/code?q=...
List releases GET /repos/{owner}/{repo}/releases

Results are paginated. Follow the Link header (it contains rel="next") rather than guessing page numbers, and set per_page=100 to cut the number of round trips.

Use GraphQL When You Need Precise Shapes

The REST API is great for simple pulls, but if you need many related objects at once (a repo plus its last 20 issues plus each issue's labels and author) REST makes you fire a request per object. GraphQL at https://api.github.com/graphql lets you ask for exactly that shape in one call:

import requests

query = """
query($owner: String!, $name: String!) {
  repository(owner: $owner, name: $name) {
    stargazerCount
    issues(last: 20, states: OPEN) {
      nodes { number title author { login } }
    }
  }
}
"""

resp = requests.post(
    "https://api.github.com/graphql",
    headers={"Authorization": "Bearer ghp_your_token_here"},
    json={"query": query, "variables": {"owner": "getlinksc", "name": "linksc-go"}},
    timeout=30,
)
print(resp.json()["data"]["repository"])

GraphQL is the right tool when you would otherwise make N+1 REST calls. For a single object type, REST is simpler.

Rate Limits and Tokens

This is the part people get wrong, and it is the whole reason to authenticate.

  • Unauthenticated requests are limited to 60 per hour per IP. That is nearly useless for real work.
  • Authenticated with a personal access token, the primary REST limit jumps to 5,000 requests per hour. GitHub Apps and enterprise accounts get more.
  • Search endpoints have their own tighter limit (a small number of requests per minute), so treat search as a scarce resource and cache results.
  • GraphQL uses a point-based budget rather than a flat request count; complex queries cost more points.

Always read the response headers. X-RateLimit-Remaining tells you how much budget is left, and X-RateLimit-Reset is the Unix time when it refills. When you hit zero, back off until the reset rather than retrying in a tight loop.

Create tokens with the narrowest scopes you need. A read-only fine-grained token limited to public repositories is enough for most data collection, and it cannot do damage if it leaks.

When to Fetch a Public Page Instead

The API covers so much that the honest answer is: rarely. But there are edge cases where fetching a rendered public page is reasonable:

  • A rendered README or GitHub Pages site where you want the final HTML converted to markdown.
  • A public profile or repo view where the exact rendered layout matters and no API field maps to it.
  • A one-off pull where standing up API auth is more work than it is worth.

For those, a fetch API returns clean markdown in one call. With link.sc:

curl -X POST https://api.link.sc/v1/fetch \
  -H "x-api-key: lsc_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://github.com/getlinksc/linksc-go",
    "format": "markdown"
  }'

The response is { "content": "..." }. That is the right shape when you want the page as a reader sees it. But if what you actually want is issue data or commit history, the API is faster, cleaner, and sanctioned: reach for the fetch only when the API genuinely does not fit.

The Terms of Service Reality

GitHub's Acceptable Use Policies address scraping directly. The short version:

  • Scraping for personal or research use of public information is generally tolerated.
  • Scraping to build a competing service, or scraping personal user information for spam or resale, is prohibited.
  • GitHub asks that you use the API rather than scraping when an API is available, which for GitHub is almost always.

Because the API exists and is generous, "I scraped the HTML" is hard to justify to anyone, including GitHub's abuse systems. The API is the compliant path and the practical one at the same time.

If you are weighing the broader legal picture, our post on whether web scraping is legal covers the terrain, and the general principles in ethical web scraping and compliance best practices apply here too.

A Short Ethics Note

Even within the API, be a good citizen. Do not vacuum up personal user data (email addresses, follower graphs) to build contact lists or train on people without regard for how they would feel about it. Respect the rate limits as a floor, not a target to max out. Cache what you pull so you are not re-requesting the same data every run. The API gives you a lot of power cheaply; use it in proportion to what you actually need.

Bottom Line

For GitHub, the "scraping" question mostly dissolves. The REST API handles simple pulls, GraphQL handles complex related fetches, and a token unlocks a 5,000-per-hour limit that covers serious projects. Save HTML fetching for the rendered-page edge cases, authenticate everything, watch your rate-limit headers, and stay on the right side of the terms by simply using the tools GitHub built for exactly this.

Building an agent or pipeline that reads GitHub and the wider web? link.sc turns any URL into clean markdown and runs live search in one API. Start free with 500 credits a month.