Quick answer: A news aggregator has four moving parts: discover articles (search plus RSS), fetch the full article body (not the truncated RSS snippet), dedupe the same story that ran on ten syndicated sites, and categorize and summarize with an LLM. The hard parts are getting full content instead of teasers and collapsing syndicated duplicates, so an API that fetches clean article text and a search that returns full pages do most of the heavy lifting.
RSS gets you 90% of the way to a bad news aggregator. The feeds give you headlines and 40-word summaries, wire stories show up under a dozen bylines, and half the "articles" are the same AP report. Fixing those two things (full content and dedupe) is what separates a real product from a feed reader.
The four stages
| Stage | Job | Main tool |
|---|---|---|
| Discover | Find candidate articles | Search + RSS feeds |
| Fetch | Get full clean article body | Fetch API |
| Dedupe | Collapse syndicated copies | Fingerprint + similarity |
| Enrich | Categorize + summarize | LLM |
Discover: RSS is a start, search is the reach
RSS feeds are great for sources you already know: point at a publication's feed and get new items as they post. But feeds only cover outlets you have subscribed to, and they miss stories that break somewhere you are not watching.
Search fills that gap. A single query surfaces coverage of a topic across the whole web, not just your feed list.
curl https://api.link.sc/v1/search \
-H "x-api-key: lsc_..." \
-H "Content-Type: application/json" \
-d '{
"q": "semiconductor export policy 2026",
"engine": "google"
}'
The response lists results under serpData.results, each with a targetUrl you hand to the fetch stage below.
Run RSS on a tight loop for known sources and search on a slower loop for topic discovery. Together they give you both depth on the outlets you trust and breadth across the rest.
Fetch: get the full article, not the teaser
This is the step people underestimate. An RSS <description> is usually a headline and a sentence, sometimes the first paragraph. You cannot summarize, categorize, or dedupe reliably from that. You need the article body.
Fetching the article URL to clean markdown gives you the real text without the nav, related-links rail, newsletter modal, and comment section.
curl https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_..." \
-H "Content-Type: application/json" \
-d '{
"url": "https://news.example.com/2026/07/chip-policy",
"format": "markdown"
}'
Clean full text is what every downstream stage depends on. Feed a summarizer the teaser and it summarizes the teaser; feed it the article and it summarizes the article. For the broader case of search returning content agents can act on, see real-time web search for LLMs.
Dedupe: the syndication problem
Wire services mean one story runs verbatim (or near-verbatim) across dozens of sites. A naive aggregator shows the same AP report ten times. Users hate that more than almost anything.
Two layers of dedupe handle it. First, an exact fingerprint catches identical reprints cheaply. Second, a similarity check on the body catches near-duplicates that were lightly edited or re-headlined.
import hashlib
def title_url_key(article: dict) -> str:
# Normalize the URL host away so the same path on syndication
# partners does not defeat exact matching.
norm = article["title"].lower().strip()
return hashlib.sha256(norm.encode()).hexdigest()[:16]
# Near-duplicate pass: compare embeddings of the article bodies and
# cluster anything above a similarity threshold into one story.
def is_near_duplicate(vec_a, vec_b, threshold=0.9) -> bool:
return cosine_similarity(vec_a, vec_b) >= threshold
When a cluster forms, pick one canonical article (earliest timestamp, or the original source if you can identify it) and keep the rest as "also covered by" links. That turns ten duplicate rows into one story with a source list, which is what a reader actually wants.
Enrich: categorize and summarize with an LLM
With full clean text and duplicates collapsed, the LLM stage is simple. One call per story does both jobs at once: assign a category from a fixed list and write a short neutral summary.
import anthropic, json
client = anthropic.Anthropic()
CATEGORIES = ["politics", "business", "tech", "science", "sports", "world"]
prompt = f"""Categorize this article into exactly one of:
{CATEGORIES}. Then write a neutral 2-sentence summary.
Return JSON: {{"category": "...", "summary": "..."}}.
Do not add opinion or information not in the text.
Article:
{article_body}
"""
resp = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=512,
messages=[{"role": "user", "content": prompt}],
)
enriched = json.loads(resp.content[0].text)
Give it a fixed category list rather than an open field, or you will end up with 200 near-synonym categories. And tell it to stay neutral and not add information; a news summary that invents a detail is worse than no summary.
Keep it fresh
News is the most time-sensitive content there is. A day-old aggregator is a stale one.
- Poll known RSS feeds every few minutes; they are cheap.
- Run topic search every 15 to 30 minutes for breadth.
- Fetch full content on ingest, once per unique article, then cache. Never re-fetch the same URL.
- Expire aggressively. Age stories off the front page on a schedule that matches your audience's attention.
A schema to store it
Storing the enriched story is the last step. Keep the canonical article plus the cluster of sources.
{
"story_id": "…",
"canonical_url": "https://…",
"title": "…",
"category": "tech",
"summary": "…",
"published_at": "2026-07-18T09:12:00Z",
"sources": ["https://apnews…", "https://reuters…"],
"fingerprint": "…"
}
Respect the sources
An aggregator sits on top of other people's reporting, so the ethics are real. Pull public article pages, honor each site's robots.txt and terms, keep your request rate polite, and do not republish full articles as if they were yours. Link back to the original, show a short summary rather than the whole body, and credit the source. An aggregator that drives traffic to publishers is a partner; one that strips their content is a problem. Keep yourself in the first category.
Build it in that order (discover, fetch full text, dedupe, enrich) and you get an aggregator that shows one clean summary per story with sources attached, which is the whole reason to build one instead of reading ten feeds yourself.
Building a news aggregator? link.sc gives you web search and clean-markdown article fetch in one API, so dedupe and summarization work on real text. Start free.