← All posts

How to Scrape Google News (Compliant Ways That Work)

GUIDES · JUL 18, 2026 · 5 MIN READ

Quick answer: Do not scrape Google News results pages directly. Google's terms prohibit automated access to its services, and the pages are engineered to resist it. Instead, use the free Google News RSS feeds for headlines and links, pull the actual article text from each publisher's page, and use a proper search API for coverage across sources. Then dedupe, because the same story runs on dozens of sites.

Google News is an aggregator, which is exactly why scraping its results pages is both the hardest and the least useful approach. The value is not in Google's ranking, it is in the underlying articles, and there are compliant ways to get those. Here is how the collection actually works.

The ToS Reality First

Let me be direct about the boundary. Google's Terms of Service prohibit sending automated queries to its services without permission, and Google News results pages sit behind anti-automation systems for that reason. Scraping the results page (news.google.com/search?q=...) with a headless browser is against the terms, technically fragile, and unnecessary given the alternatives below.

So this post is not about defeating Google's protections. It is about the sanctioned inputs that give you the same information: RSS feeds, publisher pages, and a search API. Those are public, stable, and do not put you at odds with anyone's terms.

Google News RSS Feeds

Google News still publishes free RSS feeds, and they are the correct entry point. They give you headlines, source names, publish times, and links, all as clean XML with no key required.

import feedparser
import urllib.parse

def google_news_rss(query, lang="en-US", country="US"):
    q = urllib.parse.quote(query)
    url = (
        f"https://news.google.com/rss/search"
        f"?q={q}&hl={lang}&gl={country}&ceid={country}:en"
    )
    return feedparser.parse(url)

feed = google_news_rss("renewable energy")
for entry in feed.entries[:10]:
    print(entry.published, entry.source.title, entry.title)
    print(entry.link)

Feed variants worth knowing:

Feed URL pattern
Search by keyword /rss/search?q=QUERY
Top headlines /rss
A topic (tech, business, etc.) /rss/headlines/section/topic/TECHNOLOGY
A specific publisher's Google feed /rss/search?q=site:example.com

RSS gets you the map: what stories exist, who published them, and when. What it does not give you is the article body, and the link field often points to a Google redirect rather than the publisher directly.

Get the Article Text From the Publisher

Once RSS hands you a link, the real content lives on the publisher's own page, which is public and yours to read. That is a fetch job. Publisher pages are messy (ads, newsletter modals, JavaScript rendering, soft paywalls), so a fetch API that returns clean markdown saves a lot of parsing. With link.sc:

import requests

def fetch_article(url):
    resp = requests.post(
        "https://api.link.sc/v1/fetch",
        headers={"x-api-key": "lsc_your_key_here"},
        json={"url": url, "format": "markdown", "render_js": True},
        timeout=60,
    )
    return resp.json()["content"]

for entry in feed.entries[:10]:
    text = fetch_article(entry.link)
    store(entry.title, entry.source.title, text)

This two-step pattern (RSS for discovery, fetch for the body) is the compliant backbone of most news pipelines. You are reading each publisher's public article one at a time, at a polite rate, exactly as a human reader would.

Use a Search API for Broader Coverage

RSS is good but shallow: it returns a limited window and leans toward recent items. When you need real coverage of a topic across many sources, or historical results, a search API is the better tool. link.sc's search returns full page content rather than just snippets, and the search field is q:

curl -X POST https://api.link.sc/v1/search \
  -H "x-api-key: lsc_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "q": "renewable energy policy 2026",
    "engine": "google"
  }'

The response is { "serpData": {...}, "searchEngine": "..." }. For building a news monitoring system specifically, we wrote a dedicated walkthrough: the news search API guide covers query design, freshness, and source filtering in more depth. And if you are wiring live results into an LLM, real-time web search for LLMs is the companion piece.

Dedupe Across Syndication

Here is the problem everyone hits: the same wire story (from Reuters, AP, or a press release) runs on 40 sites with near-identical text and slightly different headlines. If you do not dedupe, your dataset is 80% noise.

A practical layered approach:

import hashlib
import re

def canonical_key(title, body):
    # 1) Normalize the title
    t = re.sub(r"[^a-z0-9 ]", "", title.lower()).strip()
    # 2) Hash the first ~500 chars of the body to catch wire copies
    body_sig = hashlib.md5(body[:500].encode("utf-8")).hexdigest()
    return (t, body_sig)

seen = {}
for article in articles:
    key = canonical_key(article["title"], article["body"])
    if key not in seen:
        seen[key] = article    # keep the first (or the highest-authority source)

Refinements that help in practice:

  • Prefer the canonical URL. Many pages expose <link rel="canonical">; syndicated copies often point back to the origin. Group by that when present.
  • Use fuzzy title matching (token overlap or a similarity ratio) rather than exact string equality, since headlines get rewritten.
  • Keep the highest-authority source per cluster, but record the others so you can show "also reported by."
  • Cluster by content, not just URL, because the same story appears under many URLs.

Dedupe first, then analyze. Otherwise sentiment and volume metrics get skewed by whichever wire story happened to syndicate widest.

A Legal and Ethics Note

Collecting public news for analysis, monitoring, or internal research is common and generally defensible. A few guardrails keep it clean:

  • Do not republish full article text. Headlines, short snippets, links, and your own analysis are the safe zone; wholesale copying of a publisher's article is a copyright problem.
  • Respect each publisher's robots.txt and rate limits. Fetch one article at a time at a human pace, not a flood.
  • Do not circumvent paywalls or auth. If an article is gated, that is a signal to stop, not a puzzle to solve.
  • Attribute and link back to the original publisher whenever you surface a story.

For the wider framing, our post on whether web scraping is legal lays out the terrain, and ethical web scraping and compliance best practices covers the operational side.

Bottom Line

Skip the Google News results page entirely. Use the free RSS feeds to discover stories, fetch each article from the publisher's own public page, and lean on a search API when you need real coverage or history. Then dedupe hard, because syndication will otherwise drown you. That pipeline is compliant, durable, and gives you better data than any results-page scraper could.

Building a news monitor or research agent? link.sc pairs live web search with clean article fetches in one API. Start free with 500 credits a month.