Quick answer: An AI newsletter generator is a pipeline with five stages: gather sources with web search and fetch, deduplicate near-identical stories, summarize each with an LLM, format the results into a newsletter template, and run the whole thing on a schedule. The hard part is not the LLM, it is getting clean, current source material, which a search-and-fetch API like link.sc handles in two calls.
Newsletters live or die on their sourcing. A great summarizer fed stale or duplicate links produces a bad newsletter. So most of this build is about the pipeline around the model, not the prompt. Here is the whole thing, stage by stage.
The Pipeline at a Glance
| Stage | Job | Tool |
|---|---|---|
| 1. Gather | Find and fetch current stories | Search + fetch API |
| 2. Dedupe | Drop repeats of the same story | Hashing + similarity |
| 3. Summarize | Turn each story into 2-3 sentences | LLM (Claude) |
| 4. Format | Assemble a clean newsletter | Template |
| 5. Schedule | Run it automatically | Cron / Worker |
This is a close cousin of a news aggregator. If you want the collection layer in more depth, our guide to building a news aggregator with an API covers source selection and freshness in detail; this post takes that output and turns it into a finished newsletter.
Stage 1: Gather Sources
Start with search to find the current stories on your topics, then fetch the full content of each result. Snippets are not enough to summarize well; you want the actual article text. The link.sc search API uses q for the query and returns full results:
import requests
SEARCH = "https://api.link.sc/v1/search"
FETCH = "https://api.link.sc/v1/fetch"
HEADERS = {"x-api-key": "lsc_your_key", "Content-Type": "application/json"}
def gather(topics, per_topic=5):
stories = []
for topic in topics:
r = requests.post(SEARCH, headers=HEADERS, json={
"q": f"{topic} news",
"engine": "google"
})
results = r.json()["serpData"].get("results", [])[:per_topic]
for res in results:
stories.append({"topic": topic, "title": res["title"], "url": res["targetUrl"]})
return stories
def fetch_content(url):
r = requests.post(FETCH, headers=HEADERS, json={
"url": url,
"format": "markdown",
"render_js": True
})
r.raise_for_status()
return r.json()["content"]
Stage 2: Deduplicate
Search across multiple topics returns the same big story from several outlets. Dedupe before you summarize, or your newsletter repeats itself and you pay for LLM calls you do not need.
Two layers of dedup:
- Exact: normalize URLs and drop identical links.
- Near-duplicate: compare titles for high similarity so three write-ups of the same announcement collapse to one.
from difflib import SequenceMatcher
def dedupe(stories):
seen_urls, kept = set(), []
for s in stories:
url = s["url"].split("?")[0].rstrip("/")
if url in seen_urls:
continue
# near-duplicate title check against what we've kept
if any(SequenceMatcher(None, s["title"].lower(),
k["title"].lower()).ratio() > 0.8 for k in kept):
continue
seen_urls.add(url)
kept.append(s)
return kept
For a larger volume, swap the title comparison for embeddings and a similarity threshold. The difflib approach is fine for a newsletter pulling a few dozen stories.
Stage 3: Summarize With an LLM
Now the model earns its keep. Feed each story's fetched content to Claude and ask for a tight, factual summary. Keep the prompt strict so summaries stay consistent in length and tone.
import anthropic
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY
def summarize(title, content):
msg = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=200,
messages=[{
"role": "user",
"content": (
"Summarize this article in 2 to 3 factual sentences for a "
"newsletter. No hype, no opinion, no first person.\n\n"
f"Title: {title}\n\n{content[:6000]}"
)
}]
)
return msg.content[0].text.strip()
Truncate the content you send (here to 6000 characters) so you control token cost, and pin the model version so output stays stable across runs. Check the latest model ids and pricing before you ship, since they change.
A note on accuracy: the summary is only as good as the source. Because you fetched the full article rather than a snippet, the model has real material to work from, which is the single biggest lever on summary quality.
Stage 4: Format the Newsletter
Assemble the summaries into a template. Group by topic, link every item back to the source, and keep it scannable.
from datetime import date
def render(sections):
out = [f"# The Weekly Digest ({date.today():%B %d, %Y})\n"]
for topic, items in sections.items():
out.append(f"## {topic}\n")
for it in items:
out.append(f"**[{it['title']}]({it['url']})**\n\n{it['summary']}\n")
out.append("---\n_Generated automatically. Reply with feedback._")
return "\n".join(out)
Markdown renders cleanly in most email tools and converts to HTML easily. Always link back to the original source for every item; a newsletter that summarizes without crediting is both bad manners and a legal risk.
Stage 5: Schedule It
The last stage is what makes it a product instead of a script you run by hand. Put the pipeline behind a scheduler:
- A cron job on any server.
- A Cloudflare Worker on a cron trigger, if you want it serverless.
- A GitHub Action on a schedule, if the output goes to a repo or a send API.
def run_pipeline(topics):
stories = dedupe(gather(topics))
sections = {}
for s in stories:
try:
s["summary"] = summarize(s["title"], fetch_content(s["url"]))
except requests.RequestException:
continue
sections.setdefault(s["topic"], []).append(s)
return render(sections)
if __name__ == "__main__":
print(run_pipeline(["AI regulation", "open source models", "GPU supply"]))
Wire the output to your send provider and you have a newsletter that writes its own first draft every week.
Cost and Scale
Two meters run here: fetch/search credits and LLM tokens. Keep both sane by deduping before summarizing (you already do) and by limiting stories per topic. link.sc's free tier of 500 credits a month at link.sc/pricing is enough to prototype a weekly digest, and the paid tiers scale the gathering step without touching the code.
Keep a human in the loop for the first several issues. Read what the pipeline produces, tighten the summarizer prompt, adjust topics, and fix any source that keeps returning junk. Once it reliably produces a draft you would send, then let it run on the schedule.
Want the source-gathering layer of your newsletter to just work? Start free on link.sc and get clean article content from search and fetch in one pipeline.