Quick answer: To remove boilerplate from scraped pages, use a content-extraction library like trafilatura (Python) or Mozilla Readability (JavaScript) instead of writing per-site rules. These tools score DOM nodes by text density and link density to find the main content block, the same trick browser Reader modes use. They work well on articles and blogs, fail on docs sites and forums, and the best output format for LLM pipelines is markdown, not stripped HTML.
A typical webpage is maybe 20% content and 80% chrome: navigation, footers, cookie banners, "related articles" widgets, newsletter popups. If you're feeding pages to an LLM, that 80% isn't just noise, it's money. Here's how to strip it properly.
How readability algorithms actually work
Every Reader mode and content extractor descends from the same core insight: the main content of a page looks statistically different from its boilerplate.
The classic signals:
Text density. Content blocks contain long runs of text with few tags. A paragraph is 400 characters inside one <p>. A nav menu is 400 characters spread across 30 <li> and <a> tags.
Link density. In body text, maybe 5% of characters are inside links. In a footer or sidebar, it's closer to 90%. High link density is the single strongest "this is boilerplate" signal.
DOM heuristics. Class names and IDs like content, article, main score up; sidebar, footer, promo, cookie score down. Semantic tags (<article>, <main>) are strong hints. Comment sections get special-cased.
Block coherence. Content tends to be one contiguous region. Extractors find the highest-scoring node, then greedily absorb adjacent siblings that also look content-like.
That's essentially the algorithm behind Arc90's original Readability script from 2010, and everything since is refinement on it.
The tools worth knowing
Mozilla Readability is the engine behind Firefox Reader View, maintained as a standalone JS library. Battle-tested on news and articles.
import { Readability } from "@mozilla/readability";
import { JSDOM } from "jsdom";
const dom = new JSDOM(html, { url: "https://example.com/article" });
const article = new Readability(dom.window.document).parse();
console.log(article.title, article.textContent.length);
trafilatura is my pick in Python. It combines several extraction strategies with fallbacks, handles dates and metadata, and consistently scores at or near the top of extraction benchmarks. It can also emit markdown directly.
import trafilatura
downloaded = trafilatura.fetch_url("https://example.com/article")
text = trafilatura.extract(
downloaded,
output_format="markdown",
include_tables=True,
include_links=False,
)
print(text)
Others exist (readability-lxml, goose3, boilerpy3, newspaper4k), but honestly, if trafilatura or Readability doesn't solve your problem, the next tool in the list probably won't either. The failures are structural, not implementation details.
Where these tools fail (and why)
Readability algorithms were built for one page shape: an article with a single dominant text block. Plenty of pages aren't that.
| Page type | What goes wrong |
|---|---|
| News articles, blogs | Almost nothing. This is the happy path. |
| Documentation sites | Sidebars full of links get stripped correctly, but so do code blocks, API tables, and parameter lists that "look like" boilerplate |
| Forums and Reddit-style threads | Many small posts, none dominant. Extractors keep one post and discard the thread |
| E-commerce product pages | Specs tables and price blocks have low text density and get dropped |
| Landing pages | Content is fragmented across hero sections and feature grids; extractors return a paragraph or nothing |
| Paywalled or JS-rendered pages | The content isn't in the HTML at all, so there's nothing to extract |
The docs-site failure is the one that bites LLM builders hardest, because docs are exactly what you want to feed a model. A heuristic that penalizes link-dense, tag-dense blocks will happily delete a perfectly good API reference table.
The fix is context-dependent extraction: detect the page type first, then apply article-mode extraction only where it fits, and structure-preserving conversion elsewhere.
Why markdown output matters for LLM costs
Once you've isolated the main content, the output format is not a detail. Raw HTML burns tokens on attributes, wrapper divs, and inline styles that carry zero meaning for a model. Markdown keeps the structure that matters (headings, lists, tables, links) and drops everything else.
The practical effect is that the same content costs a fraction of the tokens, which means cheaper calls, more room in the context window, and usually better answers because the model isn't wading through markup. I went deep on the mechanics in HTML to markdown for LLMs, but the short version: boilerplate removal and markdown conversion together are the highest-leverage preprocessing you can do for a web-data pipeline.
If you're counting tokens seriously, this breakdown of token optimization covers the rest of the stack.
The DIY pipeline vs. letting an API do it
Rolling your own looks like this: fetch the page (with a headless browser for JS-heavy sites), run trafilatura or Readability, handle the failure cases per page type, convert to markdown, and monitor quality over time as sites change.
That's a real amount of ongoing work, and the failure-case handling is where most of it lives.
The alternative is an API that does the whole chain. link.sc applies content extraction and markdown conversion automatically on every fetch: one request in, clean main-content markdown out, with rendering handled for JS-heavy pages.
curl "https://link.sc/v1/fetch?url=https://example.com/article&format=markdown" \
-H "Authorization: Bearer lsc_your_key"
The response is the article body as markdown, with the nav, footer, cookie banner, and newsletter popup already gone. The quickstart has examples in Python and Node if curl isn't your thing.
My recommendation
For a weekend project on article-shaped sites: trafilatura, markdown output, done in ten lines.
For anything production-grade that touches diverse page types: either budget real time for the failure cases above, or use a fetch API that has already paid that cost. Either way, insist on markdown as your pipeline's interchange format. Your token bill will thank you, and so will your model.
Want clean, boilerplate-free markdown from any URL in one API call? Try link.sc free with 500 credits a month.