JavaScript & Node.js Web Scraping: The Complete Guide

JavaScript is the language the web renders in, so scraping with it has a logic Python can’t match: when you drive a headless browser, the page’s own scripts and your scraper speak the same language. This guide is the full path for Node.js. Which tool does what, a fetch-plus-Cheerio walkthrough you can paste and run, how to handle JavaScript-rendered pages with Playwright, how to paginate, and how to stay unblocked at scale. I ran every snippet on Node v22.18.0 in June 2026, against books.toscrape.com and quotes.toscrape.com, two sandboxes built for practice, and pasted the real output.

Is Node.js good for web scraping?

Yes, Node.js is a strong choice for web scraping, and it has one structural advantage over every other language. JavaScript runs the browser, so when a page renders its content with client-side scripts and you reach for a headless browser, your scraper and the page run the same language end to end. There is no context switch between the runtime you scrape with and the runtime the page was written in.

Three things make Node pull its weight here:

• The browser tools are native. Playwright and Puppeteer are JavaScript projects first. The newest browser-automation features tend to land in the Node API before any port.
• Async is built in. The event loop and async/await let one process fire hundreds of concurrent requests without threads or extra libraries.
• fetch ships in the runtime. Since Node 18 there is a global fetch, the same API you already use in the browser, so a basic request needs zero dependencies.

Python has more scraping tutorials and a deeper bench of data libraries, which I cover in the Python guide. Node wins when the targets are JavaScript-heavy or your stack is already JavaScript.

What are the main JavaScript web scraping libraries?

Four tools cover the whole field, and the right one depends on whether the page needs JavaScript to render. Static HTML needs only a request and a parser. JavaScript-rendered pages need a real browser engine.

A few notes from using all four:

• fetch is the global HTTP client built into Node 18+. For most GET requests it is all you need, and it removes the old node-fetch dependency entirely.
• Axios is still worth installing when you want request/response interceptors, automatic JSON parsing, timeouts, and retry logic configured once and reused. For a one-file script, fetch wins on zero-dependency simplicity.
• Cheerio parses an HTML string and gives you the jQuery $ selector API server-side. It is fast because it does no rendering, and it pairs with either HTTP client.
• Playwright and Puppeteer launch a real Chromium, so they render anything a user sees. You pay in speed and memory, so use them only when the data is not in the initial HTML.

How do you set up a Node.js scraper?

Make a project, then install only what the page needs. The HTTP client is built in, so a static scrape needs just a parser:

mkdir my-scraper && cd my-scraper
npm init -y
npm install cheerio

For JavaScript-rendered pages, add a browser. Playwright downloads its own matching Chromium, so there is no separate driver to manage:

npm install playwright
npx playwright install chromium

Use ES modules so top-level await works (the snippets below rely on it). Either set "type": "module" in package.json or name your files .mjs. Here is the exact toolchain I tested on:

How do you scrape a website with Node.js step by step?

The loop is two steps: fetch the page into an HTML string, then pull fields with Cheerio selectors. This script fetches the first books.toscrape.com catalogue page and prints the title and price for the first three books:

import * as cheerio from "cheerio";

const url = "https://books.toscrape.com/catalogue/page-1.html";

const res = await fetch(url, {
  headers: { "User-Agent": "Mozilla/5.0 (compatible; BookScraper/1.0)" },
});
console.log("status:", res.status);

const html = await res.text();
const $ = cheerio.load(html);

const books = $("article.product_pod");
console.log("books on page:", books.length);

books.slice(0, 3).each((i, el) => {
  const title = $(el).find("h3 a").attr("title");
  const price = $(el).find(".price_color").text();
  console.log(`${price}  ${title}`);
});

Running it:

status: 200
books on page: 20
£51.77  A Light in the Attic
£53.74  Tipping the Velvet
£50.10  Soumission

Each part maps to one job:

• fetch(url, { headers }) makes the HTTP request. Setting a browser-like User-Agent matters: many sites block default or empty agent strings, so set it on every real scrape.
• await res.text() reads the response body as a string. For a JSON API you would call res.json() instead and skip the parser.
• cheerio.load(html) parses that string and returns $, the same selector function you know from jQuery.
• $("article.product_pod") returns every match; .find(...) scopes a sub-query to one element; .text() reads visible text and .attr("title") reads an attribute, which is where this page stores the full title.

This is the pattern I ship for any static page. The selectors are the only part that changes per site.

How do you follow links and handle pagination?

Follow the “next” link until it disappears. Most paginated lists put a next-page anchor at the bottom of each page, so you scrape a page, read that link, and repeat until there is no next link. This crawler walks the entire books.toscrape.com catalogue:

import * as cheerio from "cheerio";

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const base = "https://books.toscrape.com/catalogue/";

let next = "page-1.html";
let pages = 0;
let total = 0;

while (next) {
  const res = await fetch(base + next, {
    headers: { "User-Agent": "Mozilla/5.0 (compatible; CatalogueCrawler/1.0)" },
  });
  const $ = cheerio.load(await res.text());

  total += $("article.product_pod").length;
  pages += 1;

  const href = $("li.next a").attr("href");
  next = href || null;

  await sleep(300); // be polite
}

console.log("pages crawled:", pages);
console.log("total books: ", total);

Running it:

pages crawled: 50
total books:  1000

Two details make this robust on a real target. The li.next a lookup returns undefined on the last page, which is the natural stop signal for the while loop. And sleep(300) paces the crawl so you are not hammering the server; on a production job, also wrap the fetch in a try/catch so one failed page does not kill the run, and read the site’s robots.txt and terms first. The same “find the next anchor, repeat” loop handles any next-button pagination.

How do you scrape JavaScript-rendered pages in Node.js?

Use a real browser, because Cheerio cannot run JavaScript. When a page loads its data with client-side scripts after the initial response, that data is absent from the HTML fetch downloads, so your Cheerio selectors find nothing. Playwright launches a real Chromium, runs those scripts, and hands you the finished DOM.

Here is the proof. quotes.toscrape.com has a JavaScript build at /js/ where the quotes are injected by script after load. I scraped it two ways in one file: once with fetch + Cheerio (no browser), once with Playwright (real browser):

import * as cheerio from "cheerio";
import { chromium } from "playwright";

const url = "https://quotes.toscrape.com/js/";

// 1) Plain fetch: JavaScript has not run
const html = await fetch(url, {
  headers: { "User-Agent": "Mozilla/5.0" },
}).then((r) => r.text());
const $ = cheerio.load(html);
console.log("fetch + cheerio: .quote in raw HTML:", $(".quote").length);

// 2) Playwright: the browser runs the JavaScript
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: "networkidle" });
const count = await page.locator(".quote").count();
console.log("playwright: .quote in rendered DOM:", count);

const first = await page.locator(".quote .text").first().textContent();
console.log("first quote:", first);

await browser.close();

Running it:

fetch + cheerio: .quote in raw HTML: 0
playwright: .quote in rendered DOM: 10
first quote: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”

fetch sees nothing because the quotes do not exist until JavaScript runs. Playwright sees all 10 because Chromium executed that script and built the DOM. The key line is waitUntil: "networkidle", which holds until network activity settles so the injected content is present before you read it. When fetch returns a shell with no data, this is the gap a browser fills. For the browser-automation deep dive in Python, see the Selenium tutorial; the same idea applies in Node with Playwright.

How do you paginate with a headless browser?

Click the “Next” link and re-read the DOM each time. With a real browser you drive the page the way a user would: scrape, click next, wait for the new content, repeat. This Playwright script walks all 10 pages of the JavaScript quotes build and collects every quote:

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto("https://quotes.toscrape.com/js/", { waitUntil: "networkidle" });

const rows = [];
let pages = 0;

while (true) {
  await page.waitForSelector(".quote");
  const onPage = await page.locator(".quote").evaluateAll((nodes) =>
    nodes.map((n) => ({
      text: n.querySelector(".text").innerText,
      author: n.querySelector(".author").innerText,
    }))
  );
  rows.push(...onPage);
  pages += 1;

  const next = page.locator("li.next > a");
  if ((await next.count()) === 0) break;
  await next.click();
}

await browser.close();

console.log("pages crawled:", pages);
console.log("quotes collected:", rows.length);
console.log("unique authors:", new Set(rows.map((r) => r.author)).size);
console.log("last author:", rows[rows.length - 1].author);

Running it:

pages crawled: 10
quotes collected: 100
unique authors: 50
last author: George R.R. Martin

page.waitForSelector(".quote") is the habit that keeps browser scraping reliable: it pauses until the element exists rather than guessing with a fixed sleep. evaluateAll runs in the page context and extracts every quote in one round trip, which is faster than reading nodes one at a time. The empty li.next count on the last page ends the loop cleanly.

When should you use Axios instead of fetch?

Reach for Axios when you want batteries that fetch leaves out: automatic JSON parsing, interceptors, per-request timeouts, and retry logic configured once. For one-off GETs the built-in fetch is enough, but on a larger codebase a shared Axios client removes boilerplate. This version fetches the same page, builds an array with Cheerio’s .map(), and writes it to JSON:

import axios from "axios";
import * as cheerio from "cheerio";
import { writeFileSync } from "node:fs";

const { data, status } = await axios.get(
  "https://books.toscrape.com/catalogue/page-1.html",
  { headers: { "User-Agent": "Mozilla/5.0 (compatible; BookScraper/1.0)" } }
);
console.log("axios status:", status);

const $ = cheerio.load(data);
const books = $("article.product_pod")
  .map((i, el) => ({
    title: $(el).find("h3 a").attr("title"),
    price: $(el).find(".price_color").text(),
  }))
  .get();

writeFileSync("books.json", JSON.stringify(books, null, 2));
console.log("rows:", books.length);
console.log("first:", JSON.stringify(books[0]));

Running it:

axios status: 200
rows: 20
first: {"title":"A Light in the Attic","price":"£51.77"}

Two ergonomics show up here. Axios parses the response for you and puts the body on .data, so there is no await res.text() step. And Cheerio’s .map(...).get() turns matched elements straight into a plain array of objects, which JSON.stringify then writes to disk. Swapping writeFileSync for a CSV builder is a few more lines.

How do you scrape at scale without getting blocked?

Look like a real browser, slow down, and spread requests across IP addresses. A site flags scrapers by three signals: requests that arrive too fast, headers that look automated, and too much traffic from one IP. Each has a countermeasure that applies whether you use fetch, Axios, or Playwright.

These measures carry a small project a long way. The trouble starts at scale: rotating a large IP pool, solving CAPTCHAs, masking headless fingerprints, and handling per-site quirks becomes its own engineering job, and it pulls focus from the data you actually want. That is where a scraper API like ChocoData fits the category, with a universal endpoint plus 453 dedicated endpoints that return the rendered page so your Node code keeps working unchanged while the blocking is handled upstream. You point fetch at the API instead of the target, parse the response with Cheerio the same way, and skip the proxy and CAPTCHA plumbing. I cover that full tradeoff, and the legal side, in the web scraping guide and in how to scrape without getting blocked.

For the scraping itself, the stack in this guide will carry you far. Take the Cheerio crawler for static pages and the Playwright crawler for JavaScript pages, point them at a site you care about, adjust the selectors until the fields come out clean, and add the anti-block measures as the target pushes back. The quick test for which path you need stays the same: fetch a page and search the result for a value you can see in the browser. When it is missing, the page is JavaScript-rendered and you move to a browser.