Quick answer: An article scraper is a tool that takes the URL of a news story or blog post and extracts just the article: title, author, publish date, and body text, with the ads, navigation, cookie banners, related-story widgets, and comment sections stripped away. The classic open-source approaches are Mozilla's Readability and the Python libraries trafilatura and newspaper4k; the API approach is a fetch service that returns the page as clean Markdown.
The interesting part of article scraping isn't fetching the page. It's the separation problem: on a typical news page, the article is maybe 20 percent of the HTML, and finding that 20 percent reliably across a million differently-designed sites is the whole game.
Why Article Extraction Is Its Own Problem
Generic scraping targets structured pages: product grids, tables, listings, where you write a selector per field. Articles are the opposite: unstructured prose wrapped in wildly inconsistent markup. There is no universal .article-body selector; every publisher builds differently, and no one writes selectors per news site when they're processing hundreds of sources.
So article scrapers use content-detection heuristics instead of selectors: they score DOM regions by text density, link density (navigation is link-heavy, prose isn't), paragraph structure, and semantic hints (<article> tags, itemprop="articleBody", Open Graph and JSON-LD metadata). The highest-scoring region wins and everything else is discarded. This is exactly what your browser's Reader Mode does, and Mozilla's Readability library, which powers Firefox's version, is the most battle-tested implementation.
The Open-Source Toolbox
trafilatura (Python) is the current accuracy leader in most published benchmarks and my default recommendation for pipelines:
import trafilatura
downloaded = trafilatura.fetch_url("https://example.com/news/some-story")
result = trafilatura.extract(downloaded, output_format="json", with_metadata=True)
# {"title": ..., "author": ..., "date": ..., "text": ...}
newspaper4k (Python) adds convenience features like multi-article crawling of a whole news site and keyword extraction.
@mozilla/readability (JavaScript) is the right pick in Node.js, paired with jsdom:
import { Readability } from "@mozilla/readability";
import { JSDOM } from "jsdom";
const dom = new JSDOM(html, { url });
const article = new Readability(dom.window.document).parse();
// article.title, article.byline, article.textContent
All three are excellent on static pages. Their shared weakness is everything around extraction: JavaScript-rendered articles come back empty, paywalled and bot-protected publishers block plain HTTP fetches, and none of them handle retries or rate limits. The extractor is the easy 80 percent; the fetch layer is the hard 20.
The API Approach: URL to Markdown
The alternative is to let a fetch API do both halves. link.sc's Fetch API renders the page (JavaScript included), runs content extraction, and returns Markdown:
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "YOUR_API_KEY"},
json={"url": "https://example.com/news/some-story",
"render_js": True, "format": "markdown"},
)
print(resp.json()["content"]) # clean article as Markdown
Markdown output matters more than it used to, because the biggest consumer of article scrapers in 2026 is LLM pipelines: news summarization, RAG over current events, media monitoring agents. Markdown preserves the structure LLMs read well (headings, lists, emphasis) at a fraction of HTML's token count; we measured the difference in HTML to Markdown for LLMs.
What Article Scrapers Are Used For
Concretely: media monitoring (every mention of your company across the press, summarized each morning), finance (news sentiment feeding trading models), research corpora, read-later apps, and AI agents that need to actually read the pages a search step surfaces. That last pipeline (search, fetch each result as an article, feed to an LLM) has become the standard architecture for research assistants.
The Legal and Polite Bits
Articles are copyrighted expression, which makes this category legally sharper than scraping prices. Extracting facts and producing summaries sits on firm ground; republishing full article text does not. Respect robots.txt and paywalls, keep volume reasonable, and for the fuller picture read our guide on whether web scraping is legal.
The Bottom Line
An article scraper solves the separation problem: prose out, boilerplate gone. Use trafilatura or Readability if your sources are static and friendly and you want free; use a fetch API when your sources render with JavaScript, block bots, or feed an LLM that wants Markdown. Either way, the extractor you pick matters less than making the fetch layer reliable, because that's where article pipelines actually fail.
Turn any article URL into clean Markdown with one call: get a free link.sc API key, 500 requests a month included.