How to Extract Text from a Webpage (Without the Junk)

Quick answer: To extract text from a webpage, don't just strip the HTML tags. Use a content-extraction library like trafilatura or readability-lxml that identifies the main content first, then pulls the text from that. Tag-stripping approaches like BeautifulSoup's .get_text() return everything on the page, including menus, footers, and cookie banners, which is rarely what you want. For arbitrary URLs at scale, a fetch API handles the extraction (plus rendering and blocking) in one call.

"URL to text" sounds like the simplest possible scraping task. It's actually two different problems wearing one trench coat: getting the HTML, and deciding which text inside it counts as content. Most tutorials only solve the first one.

The naive approach, and why it disappoints

Every scraping tutorial starts here:

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com/article").text
soup = BeautifulSoup(html, "html.parser")
text = soup.get_text(separator="\n", strip=True)

Run this on a real news article and count what you get. On a typical page, the article itself is maybe 30 to 40 percent of the extracted text. The rest is the navigation menu (every link, as text), the footer (every legal link, as text), "Subscribe to our newsletter," the cookie consent wording, image captions from the sidebar, and the headlines of twelve related articles.

There are two problems with shipping that downstream. First, if a human reads it, it's noise. Second, if an LLM reads it, you're paying tokens for noise, and the noise actively degrades answers: models happily quote the related-articles sidebar as if it were the article. I've gone deeper on that failure mode in removing boilerplate from scraped pages.

A slightly better naive version targets semantic tags:

main = soup.find("article") or soup.find("main") or soup.body
text = main.get_text(separator="\n", strip=True)

This helps on well-built sites and does nothing on the large fraction of the web that wraps everything in anonymous <div>s.

Readability-style extraction: get the article, not the page

The real fix is content extraction: algorithms that score parts of the page by text density, link density, and structural signals, then keep only the winner. In Python, trafilatura is my current pick. It benchmarks well and handles more page types than the older readability ports.

import trafilatura

downloaded = trafilatura.fetch_url("https://example.com/article")
text = trafilatura.extract(
    downloaded,
    include_comments=False,
    include_tables=True,
    favor_precision=True,
)
print(text)

favor_precision=True biases toward dropping doubtful content rather than keeping it. Flip it to favor_recall=True when missing a paragraph is worse than catching some junk, for example when you're indexing for search rather than quoting verbatim.

The output is the article body as plain text: no menus, no footer, no consent banner. On clean editorial pages, extraction quality is genuinely good. On forums, docs sites, and product listings, expect misses, because these algorithms were tuned on articles.

Text, markdown, or HTML: pick the right output

Before you standardize on plain text, decide what you actually need. This trips up a lot of pipelines.

Output	Keeps structure?	Token cost	Best for
Plain text	No	Lowest	Search indexing, classification, embeddings, dedup
Markdown	Yes (headings, lists, tables, links)	Low	LLM prompts, RAG chunks, summaries, anything a model reasons over
Raw HTML	Yes, plus all the junk	5 to 20x higher	Only when you need attributes, forms, or exact DOM structure

The one I want to flag: plain text is often the wrong choice for LLM input. When you flatten a page to text, tables collapse into undifferentiated word lists, heading hierarchy vanishes, and list items merge into run-on lines. The model loses the signals that say "these five cells belong to the same row." Markdown preserves that structure for a small token premium over text, and both are dramatically cheaper than HTML. I benchmarked the differences in HTML to markdown for LLMs.

So my rule of thumb: text for machines that only need words (embeddings, keyword search, language detection), markdown for anything an LLM will read and reason about.

Doing it with one API call

The DIY route above still leaves you fetching the page yourself, which means JavaScript-rendered sites return empty shells and bot-protected sites return challenge pages. A fetch API rolls fetching, rendering, and extraction into one request. With link.sc:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "format": "markdown"
  }'

You get back the main content as clean markdown. If you truly need plain text, stripping markdown syntax is a one-liner (or just leave it: markdown is close enough to text that most text-only consumers don't care):

import re

def markdown_to_text(md: str) -> str:
    md = re.sub(r"```.*?```", "", md, flags=re.S)      # drop code blocks
    md = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", md)   # links -> anchor text
    md = re.sub(r"[#*_`>|-]+", " ", md)                # remaining syntax
    return re.sub(r"\s+", " ", md).strip()

The docs list the available formats and options if you want the response shaped differently.

Edge cases that will bite you

Whatever route you take, these are the failure modes I see most often in real pipelines:

JavaScript rendering. If requests returns 40 lines of HTML and a bundle script, the content doesn't exist until a browser runs it. trafilatura can't extract what was never fetched. You need headless rendering or an API that does it for you.

Encoding. Pages still lie about their charset in 2026. If you see mojibake, trust the bytes plus a detector (charset_normalizer) over the declared header.

Paywalls and consent walls. Extraction can't recover text the server never sent. If the "article" you extracted is two teaser paragraphs and "Subscribe to continue," that's all the HTML contained.

Infinite scroll and pagination. One URL does not always mean one document. Comment threads and listings load incrementally; a single fetch captures only the first slice.

PDFs pretending to be webpages. Check the content type before parsing. HTML extractors fed a PDF byte stream produce nothing useful.

None of these are exotic. On a random sample of user-submitted URLs you'll hit all five within the first hundred fetches, which is the honest argument for not owning this layer yourself once volume matters.

---

Want the readable text from any URL without building the extraction stack? link.sc returns clean, LLM-ready content in one call. Try it free.