← All posts

JavaScript Web Scraper: A Practical Node.js Scraping Guide

GUIDES · JUL 18, 2026 · 5 MIN READ

Quick answer: To build a JavaScript web scraper in Node.js, use the built-in fetch plus Cheerio to parse static HTML, and switch to Playwright when the page needs JavaScript to render. Fetch and Cheerio are fast and lightweight; Playwright runs a real browser for dynamic sites. Pick the lightest tool that returns the data, and reach for an API when you want to skip proxy and rendering upkeep.

Node is a great fit for scraping. You get real async I/O, so hundreds of requests overlap cleanly, and the parsing libraries are excellent. This guide builds a JavaScript scraper from a single static page up to a paginated dynamic site, with the code you can actually run.

This is about scraping individual pages and extracting data. If you want to walk an entire site following links, our JavaScript crawling with Node.js guide covers the crawler side.

The two modes of a JS scraper

Every scraping job is one of two shapes, and picking the wrong one wastes time.

Mode Page type Tool Cost
Static HTML has the data fetch + Cheerio Cheap, fast
Dynamic JS builds the data Playwright Heavy, slower

Always try static first. Fetch the page, check if your data is in the raw HTML, and only escalate to a browser if it is not. A lot of scrapers reach for Playwright reflexively when a two-line Cheerio call would do.

Static pages: fetch plus Cheerio

Node ships fetch natively now, so you only need one dependency for parsing.

npm install cheerio

Cheerio gives you jQuery-style selectors over an HTML string. It does not run JavaScript, which is exactly why it is fast.

import * as cheerio from "cheerio";

const res = await fetch("https://news.ycombinator.com", {
  headers: { "User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)" },
});
const html = await res.text();
const $ = cheerio.load(html);

const stories = [];
$(".athing").each((_, el) => {
  const title = $(el).find(".titleline a").first().text();
  const link = $(el).find(".titleline a").first().attr("href");
  stories.push({ title, link });
});

console.log(stories.slice(0, 5));

Setting a User-Agent matters. Many servers reject requests with no user agent or the default one. Send something honest and identifiable.

Check whether you even need a browser

Before writing any Playwright code, run this test. If your target text is in the fetched HTML, stay on the light path.

const html = await fetch(url).then((r) => r.text());
console.log(html.includes("the text you want")); // true = use Cheerio

If it comes back false, the page is likely rendered client-side and you need a browser. Our comparison of JavaScript vs Python web scraping has more on when each ecosystem shines, but the static-vs-dynamic decision is the same in both.

Dynamic pages: Playwright

When the data is not in the raw HTML, run a real browser. Playwright is the modern choice for Node.

npm install playwright
npx playwright install chromium

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto("https://some-spa.com/products", {
  waitUntil: "networkidle",
});

// Wait for the real content, not a timer
await page.waitForSelector(".product-card");

const products = await page.$$eval(".product-card", (cards) =>
  cards.map((c) => ({
    name: c.querySelector("h2")?.textContent?.trim(),
    price: c.querySelector(".price")?.textContent?.trim(),
  })),
);

console.log(products);
await browser.close();

$$eval runs your callback in the browser context against all matching elements and returns plain data. Wait for a selector that only exists once the content loads rather than a fixed sleep. For the mechanics of getting the post-JavaScript DOM, see get rendered HTML after JavaScript runs.

Handling pagination

Most real jobs span many pages. The pattern is a loop that stops when a page returns nothing.

import * as cheerio from "cheerio";

async function scrapePage(pageNum) {
  const url = `https://example.com/products?page=${pageNum}`;
  const html = await fetch(url).then((r) => r.text());
  const $ = cheerio.load(html);
  return $(".product")
    .map((_, el) => $(el).find("h2").text())
    .get();
}

const all = [];
for (let n = 1; n <= 50; n++) {
  const items = await scrapePage(n);
  if (items.length === 0) break; // empty page = we are done
  all.push(...items);
  await new Promise((r) => setTimeout(r, 800)); // be polite
}
console.log(`Scraped ${all.length} items`);

The setTimeout delay between requests is not optional. It keeps you from hammering the server, which is both courteous and the best way to avoid getting blocked.

A small end-to-end scraper

Putting the pieces together: fetch, parse, paginate, and save. This one auto-detects whether it needs a browser.

import * as cheerio from "cheerio";
import { writeFile } from "node:fs/promises";

async function scrape(url) {
  const html = await fetch(url, {
    headers: { "User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)" },
  }).then((r) => r.text());

  const $ = cheerio.load(html);
  const items = $("article")
    .map((_, el) => ({
      title: $(el).find("h2").text().trim(),
      url: new URL($(el).find("a").attr("href"), url).href,
    }))
    .get();

  if (items.length === 0) {
    console.warn("No items found. Page may be JS-rendered; use Playwright.");
  }
  return items;
}

const data = await scrape("https://example.com/blog");
await writeFile("out.json", JSON.stringify(data, null, 2));
console.log(`Saved ${data.length} items`);

Note the new URL(href, url) call, which resolves relative links to absolute ones. That is the same resolution step every link extractor needs.

When to reach for an API instead

Self-hosted scraping has a long tail of maintenance: rotating proxies when you get blocked, keeping Chromium updated, handling bot detection, and fixing selectors after redesigns. When that upkeep outweighs the work, a fetch API renders and parses for you.

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://some-spa.com/products", "format": "markdown"}'

link.sc handles rendering, proxies, and parsing, and returns clean markdown or JSON, so static and dynamic pages look identical to your code. From Node it is a plain fetch:

const res = await fetch("https://link.sc/v1/fetch", {
  method: "POST",
  headers: {
    Authorization: "Bearer lsc_...",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://some-spa.com", format: "markdown" }),
});
const { content } = await res.json();

The quickstart docs cover the parameters. It is the right call when you would otherwise spend more time babysitting infrastructure than parsing data.

An ethics note

Scrape public data, send an honest User-Agent, respect robots.txt, and rate-limit yourself. Do not scrape behind logins or paywalls, and honor a site's terms of service. A polite scraper that adds delays and caches results is one that keeps working, because it does not get blocked and does not degrade the sites it depends on.

Bottom line

A solid JavaScript web scraper is really two tools with one decision between them. Use fetch plus Cheerio for static HTML, escalate to Playwright only when the data is JS-rendered, and always test which one you need before writing browser code. Add pagination loops with delays, resolve your URLs, and when the infrastructure upkeep gets heavy, let an API carry the rendering and proxy load.

Build your JavaScript scraper without owning a browser fleet or a proxy pool. Try link.sc free.