URL to Markdown: Turn Any Webpage into Clean Markdown

Quick answer: Converting a URL to markdown takes three steps: fetch the HTML, isolate the main content, and convert what's left to markdown. You can wire that up yourself with readability-lxml and markdownify in Python (or Readability and Turndown in Node), or you can make a single API call to a fetch API like link.sc that does all three, plus JavaScript rendering and anti-bot handling. DIY works fine for simple, static sites. The API call wins the moment real-world pages get involved.

I've written before about why markdown is the right format for LLMs conceptually. This post is the hands-on version: you have a URL, you want markdown, here's exactly how to get it.

The three-step pipeline

Every URL-to-markdown tool, hosted or homemade, does the same three things:

1. Fetch. Download the HTML. Sounds trivial, is frequently not (more on that below).
2. Extract. Strip navigation, sidebars, footers, cookie banners, and ads so only the main content remains.
3. Convert. Translate the surviving HTML into markdown: headings to #, links to [text](url), lists to -, tables to pipes.

Skipping step 2 is the most common mistake. If you convert the whole page, your markdown contains the cookie banner and the "related articles" widget, faithfully formatted. That wastes tokens and confuses whatever reads it downstream. I covered the cost side of this in token optimization for web data.

DIY in Python: readability plus markdownify

Here's a working pipeline in about 20 lines:

import requests
from readability import Document
from markdownify import markdownify as md

def url_to_markdown(url: str) -> str:
    resp = requests.get(url, headers={
        "User-Agent": "Mozilla/5.0 (compatible; MyBot/1.0)"
    }, timeout=15)
    resp.raise_for_status()

    doc = Document(resp.text)
    title = doc.short_title()
    main_html = doc.summary()

    body = md(main_html, heading_style="ATX", bullets="-")
    return f"# {title}\n\n{body}"

print(url_to_markdown("https://example.com/some-article"))

Install with pip install requests readability-lxml markdownify. The Document class is a port of the algorithm behind Firefox's Reader View: it scores DOM nodes by text density and link ratio, then keeps the highest-scoring subtree. markdownify converts the result. heading_style="ATX" gives you # headings instead of underlined ones, which is what you want for LLM input.

DIY in Node: Readability plus Turndown

The same pipeline in JavaScript uses Mozilla's actual Readability library:

import { Readability } from "@mozilla/readability";
import { JSDOM } from "jsdom";
import TurndownService from "turndown";

async function urlToMarkdown(url) {
  const res = await fetch(url, {
    headers: { "User-Agent": "Mozilla/5.0 (compatible; MyBot/1.0)" },
  });
  const html = await res.text();

  const dom = new JSDOM(html, { url });
  const article = new Readability(dom.window.document).parse();

  const turndown = new TurndownService({
    headingStyle: "atx",
    codeBlockStyle: "fenced",
  });
  return `# ${article.title}\n\n${turndown.turndown(article.content)}`;
}

Set codeBlockStyle: "fenced" or Turndown will emit indented code blocks, which lose their language tags and copy badly. If you scrape technical content, also add the turndown-plugin-gfm package so tables convert to pipe tables instead of collapsing into paragraph soup.

Where the DIY pipeline breaks

I want to be honest about this, because the 20-line version demos beautifully and then fails on the third real URL you throw at it.

JavaScript rendering. requests.get() returns the initial HTML. If the site is React, Vue, or anything that hydrates client-side, that HTML is often an empty shell. Now you need Playwright or Puppeteer, a headless browser per fetch, and the memory bill that comes with it.

Blocking. Datacenter IPs get challenged or refused outright by Cloudflare, DataDome, and friends. A plain requests call from a cloud server fails on a meaningful share of commercial sites regardless of how polite your user agent string is.

Extraction misses. Readability was tuned for news articles. On documentation sites, forums, and product pages it regularly drops content it should keep or keeps chrome it should drop. If you're seeing that, I wrote up strategies in removing boilerplate from scraped pages.

Tables and code blocks. These are the two structures that punish a naive converter. Nested tables, colspan cells, and syntax-highlighted code (which arrives as hundreds of <span> tags) all need special handling that the default settings of markdownify and Turndown don't fully provide.

One API call instead

The alternative is to let a service own the pipeline. With link.sc it's one request:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/some-article",
    "format": "markdown"
  }'

The response is clean markdown for the main content: fetched, rendered if the page needs a browser, extracted, and converted, with tables and code blocks handled. In Python:

import requests

resp = requests.post(
    "https://link.sc/v1/fetch",
    headers={"Authorization": "Bearer lsc_your_api_key"},
    json={"url": "https://example.com/some-article", "format": "markdown"},
)
markdown = resp.json()["markdown"]

That's the whole integration. The docs cover the other output formats and options.

Which approach should you use?

Situation	DIY (readability + converter)	Fetch API
A handful of static pages you control	Good fit	Overkill
Learning how extraction works	Best way to learn	Hides the details
Mixed, arbitrary URLs from users	Fails often	Good fit
JavaScript-heavy sites	Needs headless browser	Handled
Sites behind anti-bot walls	Mostly fails	Mostly works
Production pipeline with SLAs	You own every failure	Vendor owns them

My honest take: build the DIY version once. It takes an afternoon and teaches you what extraction actually involves, which makes you a better judge of any tool you buy later. Then, for anything production-shaped where the input is "arbitrary URLs from the internet," use an API. The failure modes (rendering, blocking, weird markup) are exactly the ones that don't show up until you're at scale, and they're the ones a dedicated service exists to absorb. The free tier is 500 credits a month, which is plenty to find out whether the output quality holds up on your URLs.

---

Need clean markdown from any URL? link.sc turns webpages into LLM-ready markdown with one API call. Start free with 500 credits a month.