← All posts

URL Extractor Guide: Extract Every Link From a Web Page

GUIDES · JUL 18, 2026 · 6 MIN READ

Quick answer: To extract URLs from a page, parse the HTML and read every anchor tag's href attribute, then resolve each one against the page's base URL so relative paths become absolute. In Python use requests plus BeautifulSoup with urljoin; in Node use fetch plus Cheerio with new URL(href, base). For links that only appear after JavaScript runs, you need a rendered DOM or an API that renders for you.

A URL extractor sounds trivial until you actually build one. The anchor tags are the easy part. The hard parts are the relative paths, the junk links (javascript:void(0), mailto:, #anchor), the duplicates, and the links that do not exist in the raw HTML at all because a framework injected them client-side.

I have written this same extractor a dozen times. Here is the version that handles the edge cases you will actually hit.

What "extracting URLs" really means

There are more link sources on a page than most people expect. A thorough extractor looks at all of them, not just <a> tags.

Source HTML What it is
Anchor links <a href> Navigation, the main target
Images <img src>, srcset Media you might also want
Scripts/styles <script src>, <link href> Assets, usually filtered out
Iframes <iframe src> Embedded pages
Sitemaps sitemap.xml A pre-built list of every URL
Canonical/alternate <link rel> SEO hints, other language versions

For most jobs you want anchor href values. But if you are building a crawler or an asset downloader, know that the other sources exist.

Step 1: Get all anchor hrefs

Start with the raw HTML. Here is the Python version.

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com", timeout=15)
soup = BeautifulSoup(resp.text, "html.parser")

hrefs = [a.get("href") for a in soup.find_all("a") if a.get("href")]
print(hrefs)

And the Node version with Cheerio.

import * as cheerio from "cheerio";

const html = await fetch("https://example.com").then((r) => r.text());
const $ = cheerio.load(html);

const hrefs = $("a[href]")
  .map((_, el) => $(el).attr("href"))
  .get();

console.log(hrefs);

At this point you have raw href strings. Half of them are probably useless. Now the real work starts.

Step 2: Resolve relative URLs

A page can link to /about, ../pricing, contact.html, or //cdn.example.com/x. None of those are usable on their own. You have to resolve each against the page's base URL.

Do not do this with string concatenation. Both languages ship a correct resolver.

from urllib.parse import urljoin, urldefrag

base = "https://example.com/docs/guide"
absolute = [urldefrag(urljoin(base, h))[0] for h in hrefs]

const base = "https://example.com/docs/guide";
const absolute = hrefs.map((h) => {
  try {
    const u = new URL(h, base);
    u.hash = ""; // drop #fragments
    return u.href;
  } catch {
    return null; // skips mailto:, javascript:, tel:
  }
});

Wrapping new URL() in a try/catch is the cheap trick that filters out mailto:, tel:, and javascript: links, because those throw or resolve to a non-http scheme. Watch for that.

Step 3: Filter out the noise

Now decide what you actually want. Common filters:

  • Drop non-HTTP schemes (mailto:, tel:, javascript:).
  • Drop fragment-only links (#section), which point back to the same page.
  • Separate internal (same host) from external links.
  • Match a pattern, for example only /blog/ URLs.
from urllib.parse import urlparse

base_host = urlparse(base).netloc

def keep(url):
    if not url:
        return False
    p = urlparse(url)
    return p.scheme in ("http", "https")

clean = [u for u in absolute if keep(u)]
internal = [u for u in clean if urlparse(u).netloc == base_host]
external = [u for u in clean if urlparse(u).netloc != base_host]

Splitting internal from external is the single most common filter. Crawlers stay internal. Link auditors care about external. Pattern filtering (a simple if "/blog/" in u) layers on top.

Step 4: Deduplicate

Pages repeat links constantly: a header, a footer, and three in-body mentions of the same page. Dedupe before you do anything expensive with the list.

unique = sorted(set(clean))

const unique = [...new Set(absolute.filter(Boolean))];

One subtlety: https://example.com/page and https://example.com/page/ are different strings but usually the same page. If duplicates matter, normalize trailing slashes and lowercase the host before you dedupe. Do not normalize the path case, since many servers are case-sensitive.

Step 5: Handle JavaScript-rendered links

Here is where a lot of extractors quietly fail. If the page is a React, Vue, or Angular app, requests.get() returns a near-empty shell and your href list comes back tiny or empty. The links get injected into the DOM after the JavaScript runs.

You have two options. Render the page yourself with a headless browser, or call an API that renders it for you. For the browser route, see how to scrape JavaScript-rendered websites, which walks through Playwright and Puppeteer in depth.

The API route skips the browser entirely. link.sc renders the page and returns clean content, so the anchors are all present:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "markdown"}'

Markdown output keeps links as [text](url), so a quick regex like \[.*?\]\((https?://[^)]+)\) pulls every URL from the response with no DOM parsing at all. If you would rather have structured output, the fetch endpoint can return links as a JSON array.

Step 6: Skip the page and read the sitemap

Sometimes the fastest URL extractor is not an extractor. If a site has a sitemap.xml, it already lists every URL the owner wants indexed, complete and deduplicated.

import requests
from bs4 import BeautifulSoup

xml = requests.get("https://example.com/sitemap.xml", timeout=15).text
soup = BeautifulSoup(xml, "xml")
urls = [loc.text for loc in soup.find_all("loc")]

Check /sitemap.xml and /robots.txt (which often points to the sitemap) before you crawl anything. It saves you the crawl entirely. If you do need to walk the whole site, our guide on how to crawl an entire website covers the queue-and-visited-set pattern.

Choosing your approach

Situation Best tool
Static HTML, small job requests + BeautifulSoup
Static HTML, Node project fetch + Cheerio
Links appear after JS Headless browser or a rendering API
Full site inventory sitemap.xml first
No infra, just want the links curl to a fetch API

A note on ethics and rate limits

Extract links from public pages, not from behind logins. Respect robots.txt, and if you are extracting from many pages, add a delay between requests so you do not hammer the server. A link extractor that fires hundreds of requests a second is indistinguishable from a small attack, and it will get your IP blocked. Slow and polite beats fast and banned.

Putting it together

A production URL extractor is five small steps: read the hrefs, resolve them to absolute, filter, dedupe, and handle the JS case. The code is short. The value is in getting the edge cases right, especially relative resolution and the client-rendered links that never show up in raw HTML.

If you want to skip the browser overhead for that last case, an API that renders and returns clean, link-preserving output does the extraction for you in one request.

Want every link from any page, rendered and clean, in one API call? Try link.sc free.