How to Scrape Hacker News (You Probably Do Not Need To)

Quick answer: You almost never need to scrape Hacker News. There are two free official APIs: the HN Firebase API for live items (stories, comments, users) and the Algolia HN Search API for full-text search and filtering. Both return clean JSON, require no key, and cover nearly every use case. Fetching HTML is only worth it for the linked article behind a story, not for HN's own data.

Hacker News is a favorite data source for trend analysis, and it is also one of the easiest sites to get data from correctly. The community built and maintains two APIs that make scraping the site itself pointless. Let me walk through both.

The Official HN Firebase API

Y Combinator publishes the official API through Firebase at https://hacker-news.firebaseio.com/v0/. No key, no auth, no rate-limit paperwork. Everything is an "item" (story, comment, job, poll) addressed by ID.

import requests

BASE = "https://hacker-news.firebaseio.com/v0"

# Grab the current top story IDs
top_ids = requests.get(f"{BASE}/topstories.json", timeout=30).json()

# Fetch the first 10 stories
for item_id in top_ids[:10]:
    item = requests.get(f"{BASE}/item/{item_id}.json", timeout=30).json()
    print(item["score"], item["title"], item.get("url", "[self post]"))

What the Firebase API gives you:

Endpoint	Returns
/topstories.json	IDs of the current front page
/newstories.json	IDs of the newest submissions
/beststories.json	IDs of the highest-scoring recent stories
/item/{id}.json	A single story, comment, job, or poll
/user/{id}.json	A user's karma, about, and submitted IDs
/maxitem.json	The largest item ID (useful for firehose crawls)
/updates.json	Recently changed items and profiles

The one quirk: comments are nested by reference. An item's kids field is a list of child IDs, so building a full comment tree means fetching each child. That is a lot of small requests, so batch and cache them. There is no full-text search here either, which is where the second API comes in.

The Algolia HN Search API

Algolia runs the free search backend for HN at https://hn.algolia.com/api/v1/. This is the one to use whenever you want to filter, search, or pull by time range instead of walking item IDs.

import requests

# Full-text search for stories mentioning "rust", most recent first
resp = requests.get(
    "https://hn.algolia.com/api/v1/search_by_date",
    params={
        "query": "rust",
        "tags": "story",
        "hitsPerPage": 50,
    },
    timeout=30,
)
for hit in resp.json()["hits"]:
    print(hit["points"], hit["title"], hit["url"])

Useful parameters:

• tags filters by type: story, comment, poll, show_hn, ask_hn, front_page, or author_pg_username.
• numericFilters filters by points, num_comments, or created_at_i (for example, points>100).
• search ranks by relevance, search_by_date sorts newest first.
• Results are paginated with page and hitsPerPage.

For most analytics work (find every Show HN with more than 50 points last month, or every comment mentioning a company) the Algolia API answers it in one request. It is genuinely the better tool than the Firebase API for anything search-shaped.

Firebase vs Algolia: Which One

Need	Use
Live front page / newest items	Firebase (topstories, newstories)
A specific item or comment tree	Firebase (/item/{id})
A user's profile and submissions	Firebase (/user/{id})
Full-text search	Algolia (/search)
Filter by points, date, or type	Algolia (numericFilters, tags)
Historical bulk pulls by date	Algolia (search_by_date)

Between the two, you have complete coverage of Hacker News data without ever touching an HTML page.

When Fetching Actually Helps

Here is the honest part: scraping HN itself is unnecessary. But there is one thing the APIs do not give you, and it is often the thing you actually care about: the content of the linked article.

A story item hands you a url, not the article behind it. If you are doing sentiment analysis, summarization, or building a reading digest, you need that article's text. That is a fetch job, and the target site may be heavy HTML, paywalled soft-walls, or JavaScript-rendered. A fetch API handles it in one call. With link.sc:

import requests

def article_markdown(url):
    resp = requests.post(
        "https://api.link.sc/v1/fetch",
        headers={"x-api-key": "lsc_your_key_here"},
        json={"url": url, "format": "markdown"},
        timeout=60,
    )
    return resp.json()["content"]

# Combine the HN API with a fetch for the linked article
story = requests.get(
    "https://hacker-news.firebaseio.com/v0/item/8863.json", timeout=30
).json()
if story.get("url"):
    print(article_markdown(story["url"]))

That combination (official HN API for the metadata, a fetch API for the linked article) is the pattern that actually gets used in production. If your project needs many articles and you want to summarize or classify them, our guide on extracting structured JSON from any webpage shows how to go past raw markdown.

The Rare Case for HTML

Are there times you might fetch an HN page directly? A few:

• You want the page exactly as rendered, including relative comment timestamps or vote arrows the API does not model.
• You are debugging a discrepancy between what the API returns and what the site shows.

Even then, be gentle. HN is run by a small team, so keep request rates low and cache everything. Honestly, in years of working with HN data I have never needed to parse its HTML: the two APIs have always been enough.

A Quick Ethics and Rate-Limit Note

The APIs are free and open, which makes it easy to forget they cost someone money to run.

• Cache aggressively; item data changes slowly and search results even more so.
• Batch comment-tree fetches instead of firing thousands of tiny requests in a tight loop.
• Do not rebuild and rehost the full HN dataset as a competing front page; use the data, do not clone the site.
• If you republish comments, remember they were written by real people; attribute and link back.

The broader principles in ethical web scraping and compliance best practices apply to APIs too, not just scrapers.

Bottom Line

Hacker News is the easy case. The Firebase API gives you live items and users, the Algolia API gives you search and filtering, and neither needs a key. The only thing worth fetching is the article behind each story, and a fetch API handles that in one line. Skip the scraper entirely, cache what you pull, and treat the free infrastructure with the respect it deserves.

---

Want the article behind every Hacker News link as clean markdown, plus live web search in one API? link.sc does exactly that. Start free with 500 credits a month.