← All posts

JavaScript Crawling: How to Build a Web Crawler in Node.js

GUIDES · JUL 14, 2026 · 4 MIN READ

Quick answer: To crawl websites in JavaScript, you fetch pages with the built-in fetch, parse them with Cheerio, extract the links, and repeat with a URL queue and a visited set. That's a complete crawler in about 40 lines of Node.js; working code below. For pages that render their content with client-side JavaScript, you swap the fetch step for a headless browser (Playwright) or a fetch API with JS rendering.

"JavaScript crawling" means two different things to people, and this guide covers both: crawling with JavaScript (Node.js as your language), and crawling of JavaScript (sites that render client-side).

A Complete Crawler in Node.js

No framework, two dependencies (npm install cheerio p-limit):

import * as cheerio from "cheerio";
import pLimit from "p-limit";

const START = "https://books.toscrape.com/";
const visited = new Set();
const queue = [START];
const limit = pLimit(3); // max 3 concurrent requests
const results = [];

async function crawlPage(url) {
  const res = await fetch(url);
  if (!res.ok) return;

  const $ = cheerio.load(await res.text());

  // Extract what you care about
  $("article.product_pod").each((_, el) => {
    results.push({
      title: $(el).find("h3 a").attr("title"),
      price: $(el).find(".price_color").text(),
    });
  });

  // Discover new URLs, stay on-domain
  $("a[href]").each((_, el) => {
    const next = new URL($(el).attr("href"), url).href;
    if (next.startsWith(START) && !visited.has(next)) {
      visited.add(next);
      queue.push(next);
    }
  });
}

while (queue.length > 0) {
  const batch = queue.splice(0, 20);
  await Promise.all(batch.map((u) => limit(() => crawlPage(u))));
}

console.log(`Crawled ${visited.size} pages, extracted ${results.length} items`);

Every crawler ever written is a variation on these four pieces: a queue of URLs to visit, a visited set so you never fetch twice, an extractor for the data, and a link discoverer that feeds the queue. The p-limit wrapper is the politeness valve: three concurrent requests is assertive but reasonable; fifty is how you get IP-banned.

Node is genuinely good at this. Crawling is I/O-bound (you're mostly waiting on the network), and Node's async model handles hundreds of in-flight requests without threads. That's the honest answer to "Python or JavaScript for crawling?": Python has more scraping libraries; Node has a concurrency model that fits the problem. Pick whichever you already write.

The Problem: Sites That Render with JavaScript

Run the crawler above against a React or Next.js site and you'll often get this:

<body><div id="root"></div><script src="/app.js"></script></body>

The HTML is an empty shell; the content arrives only after a browser executes the JavaScript. Your crawler isn't broken; it's just reading the pre-render document. You have three escape hatches, in order of preference:

1. Find the hidden API. Open DevTools → Network → filter by XHR while browsing the site. Most JS-rendered sites fetch their content from a JSON endpoint you can call directly. When it works, this is strictly better: structured data, no parsing, faster.

2. Render with a headless browser. Playwright drives a real Chromium:

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: "networkidle" });
const html = await page.content(); // post-render HTML for Cheerio
await browser.close();

It works on everything, but it's heavy: each page costs a browser session's worth of CPU and memory, and at crawl scale you're now managing a browser farm.

3. Use a fetch API with rendering built in. One HTTP call, rendering happens server-side:

const res = await fetch("https://api.link.sc/v1/fetch", {
  method: "POST",
  headers: { "x-api-key": process.env.LINKSC_KEY, "Content-Type": "application/json" },
  body: JSON.stringify({ url, render_js: true, format: "markdown" }),
});
const { content } = await res.json();

Your crawler code stays exactly as it was (queue, visited set, extractor), and only the fetch line changes. This also quietly solves the other scale problem: Cloudflare challenges and bot detection, which plain Node requests and vanilla Playwright both fail against on protected sites. We covered that arms race in curl vs. headless vs. stealth browsers.

Crawling Manners

Three rules keep your crawler welcome:

  • Respect robots.txt. Parse it (the robots-parser package) and skip disallowed paths.
  • Back off on 429s. A 429 response means slow down. Honor Retry-After instead of retrying instantly.
  • Identify yourself. A User-Agent with contact info (MyCrawler/1.0 ([email protected])) turns "block this bot" into "email this person" when a site notices you.

When to Reach for a Framework

Hand-rolled crawlers are perfect for one site or a few thousand pages. When you need persistent queues, automatic retries, proxy rotation, and crash recovery, Crawlee is the mature Node.js framework: it wraps the same queue/extract/discover loop with production plumbing. The concepts you just learned transfer directly.

Crawling JS-heavy or protected sites? Grab a free link.sc API key and get rendered, Markdown-ready pages from a single API call.