Web Scraping with Puppeteer: Complete Tutorial

Puppeteer is the tool I reach for when a Node.js scraper needs a real browser to see the data. It drives Chromium (and Firefox) through the DevTools Protocol, runs the page’s JavaScript, and lets me read the rendered DOM the same way a user’s browser does. This tutorial covers the whole loop: install it, scrape a page, select elements, walk pagination, handle JavaScript-rendered content, and take a screenshot. Every snippet here ran against quotes.toscrape.com, a sandbox built for practice, in June 2026 on Puppeteer 25.1.0 and Node v22.18.0.

What is Puppeteer?

Puppeteer is a Node.js library from the Chrome team that launches and controls a browser from your code. You tell it to open a URL, click, type, and read elements, and it does that inside a real Chromium while running every script on the page. Because a genuine browser renders the page, the HTML you query already contains the JavaScript-built content.

It started as a browser-automation and testing tool, which is why a few scraping-friendly traits come built in:

Puppeteer is Node-only (JavaScript and TypeScript). If you want the same idea in Python, the Playwright tutorial covers the closest equivalent, and the JavaScript scraping guide puts Puppeteer in context with the rest of the Node ecosystem.

How do you install Puppeteer?

One command installs the library and downloads a browser to drive:

npm i puppeteer

That pulls the puppeteer package and a pinned Chromium build into your project. If you want to manage the browser yourself or skip the download, install puppeteer-core instead and point it at an existing Chrome with executablePath. For a fresh scraper, plain puppeteer is the right default.

To confirm what you installed, ask the package directly. This is the real output on the machine I tested with:

node -e "console.log(require('puppeteer/package.json').version)"

25.1.0

How do you scrape a page with Puppeteer?

Launch a browser, open a page, then query elements and read their text. This script pulls the text, author, and tags for the first quote on the front page:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto('https://quotes.toscrape.com/', { waitUntil: 'domcontentloaded' });

  const quotes = await page.$$('div.quote');
  console.log(quotes.length, 'quotes on page 1');

  const first = await page.$eval('div.quote', el => ({
    text: el.querySelector('span.text').innerText,
    author: el.querySelector('small.author').innerText,
    tags: [...el.querySelectorAll('a.tag')].map(t => t.innerText),
  }));
  console.log('text:', first.text);
  console.log('author:', first.author);
  console.log('tags:', JSON.stringify(first.tags));

  await browser.close();
})();

When I ran it, it found all 10 quotes and read the first one cleanly:

10 quotes on page 1
text: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
author: Albert Einstein
tags: ["change","deep-thoughts","thinking","world"]

Three things from that code are worth noting:

• Everything is async. launch, goto, and the query methods all return Promises, so await each one.
• page.$$(selector) returns every matching element handle; page.$(selector) returns the first.
• page.$eval(selector, fn) runs fn inside the browser on the first match and returns a serializable result, which keeps the DOM work on the page side.
• await browser.close() shuts Chromium down. Skip it and the process leaves a browser running.

How do you select and extract many elements at once?

Use page.$$eval to map over every match in a single in-browser pass. page.$$ gives you element handles you query one at a time, which means a round trip per read. page.$$eval runs your callback inside the page across all matches and returns the finished array, so a whole page of cards comes back in one call:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto('https://quotes.toscrape.com/', { waitUntil: 'domcontentloaded' });

  const data = await page.$$eval('div.quote', cards =>
    cards.map(c => ({
      text: c.querySelector('span.text').innerText,
      author: c.querySelector('small.author').innerText,
    }))
  );

  console.log('extracted:', data.length, 'quotes');
  console.log('first:', data[0].author);
  console.log('sample text:', data[0].text);

  await browser.close();
})();

Real output:

extracted: 10 quotes
first: Albert Einstein
sample text: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”

Here is how the four selection methods divide up:

For list extraction I default to $$eval. The callback runs in the browser with normal querySelector access, and one call returns the full dataset instead of one handle at a time.

How do you handle pagination?

Click the “next” link until it disappears. Most paginated sites put a next-page anchor at the bottom of the list, so you scrape a page, look for that link, click it, and repeat. When the link is gone you are on the last page:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto('https://quotes.toscrape.com/', { waitUntil: 'domcontentloaded' });

  const all = [];
  while (true) {
    const batch = await page.$$eval('div.quote', cards =>
      cards.map(c => ({
        text: c.querySelector('span.text').innerText,
        author: c.querySelector('small.author').innerText,
      }))
    );
    all.push(...batch);

    const next = await page.$('li.next a');
    if (!next) break;
    await Promise.all([
      page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
      next.click(),
    ]);
  }

  await browser.close();
  console.log('total quotes:', all.length);
  console.log('first author:', all[0].author);
  console.log('last author:', all[all.length - 1].author);
})();

This walked all 10 pages and collected every quote:

total quotes: 100
first author: Albert Einstein
last author: George R.R. Martin

The Promise.all around waitForNavigation and click is the part to get right. A click that loads a new page is async, so you start waiting for navigation and trigger the click together, then await both. Click first and await second and you can miss the navigation event. On real sites add a short await new Promise(r => setTimeout(r, 1000)) inside the loop to avoid hammering the server.

How does Puppeteer handle JavaScript-rendered pages?

It runs the JavaScript before you read the DOM, which is the whole reason to use a browser instead of a plain HTTP request. quotes.toscrape.com has a /js version that ships an empty shell and builds the quotes client-side. This script fetches that page two ways and counts the quote blocks in each:

const https = require('https');
const puppeteer = require('puppeteer');

const URL = 'https://quotes.toscrape.com/js/';

function rawGet(url) {
  return new Promise((resolve, reject) => {
    https.get(url, { headers: { 'User-Agent': 'Mozilla/5.0' } }, res => {
      let body = '';
      res.on('data', c => (body += c));
      res.on('end', () => resolve(body));
    }).on('error', reject);
  });
}

(async () => {
  // 1) Raw HTTP request: no JavaScript engine
  const html = await rawGet(URL);
  const rawCount = (html.match(/class="quote"/g) || []).length;
  console.log('raw HTTP   -> div.quote in HTML:', rawCount);

  // 2) Puppeteer: runs the page's JavaScript first
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto(URL, { waitUntil: 'domcontentloaded' });
  await page.waitForSelector('div.quote');
  const rendered = (await page.$$('div.quote')).length;
  console.log('puppeteer  -> div.quote rendered:', rendered);
  await browser.close();
})();

The contrast is the point of the whole tutorial. Real output:

raw HTTP   -> div.quote in HTML: 0
puppeteer  -> div.quote rendered: 10

The raw HTML had zero quotes because the data arrives through a script the HTTP client never executes. Puppeteer ran that script, so by the time waitForSelector returned, all 10 quotes were in the DOM. waitForSelector blocks until the element exists, which is the safe way to handle content that loads a moment after navigation. For static HTML, a parser like Cheerio or Python’s BeautifulSoup is lighter; the moment data depends on JavaScript, a browser earns its weight.

How do you take a screenshot with Puppeteer?

Call page.screenshot() with a path. This is handy for debugging a scraper, because you see exactly what the browser saw at that moment:

const puppeteer = require('puppeteer');
const fs = require('fs');

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();
  await page.setViewport({ width: 1280, height: 800 });
  await page.goto('https://quotes.toscrape.com/', { waitUntil: 'networkidle2' });
  await page.screenshot({ path: 'quotes.png', fullPage: true });
  await browser.close();
  console.log('saved:', fs.statSync('quotes.png').size, 'bytes');
})();

It wrote a real PNG to disk:

saved: 137215 bytes

fullPage: true captures the entire scrollable page, not just the viewport. Drop it to capture only what fits in the 1280x800 window I set. waitUntil: 'networkidle2' holds until the network has been quiet, so late-loading images land in the shot.

Puppeteer vs Playwright vs Selenium

All three drive a real browser and render JavaScript. They differ in language reach, browser support, and how the browser gets installed:

If your stack is Node and your targets are Chrome-shaped, Puppeteer is the lean pick: one install, an API maintained alongside Chrome itself. Need WebKit coverage or a Python API, and Playwright fits better. Teams already deep in Selenium have less reason to switch. I weigh the broader options in the Python scraping guide.

Scraping at scale and getting past blocks

A browser solves rendering, and it adds two costs on real targets. Each Puppeteer instance launches a full Chromium, so memory and CPU climb fast when you run hundreds in parallel, and you manage that with separate pages, batching, and queues. The harder problem is blocking, and a default headless Puppeteer exposes exactly the signals anti-bot vendors fingerprint. I confirmed both on my own run with page.evaluate:

navigator.webdriver: true
user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/149.0.0.0 Safari/537.36

The navigator.webdriver flag returns true, and the user agent announces HeadlessChrome. Either one is enough for a protected site to flag the request, which is why a scraper that breezes through quotes.toscrape.com can hit a challenge wall on a production target.

You can soften the footprint with a realistic user agent, real headers, a non-headless run, and stealth plugins, and that buys some headroom. Past a certain level of protection, the reliable route is a scraper API that handles proxy rotation, browser fingerprinting, and challenge solving for you. ChocoData is one such service: it exposes a universal endpoint plus 453 dedicated endpoints, returns the rendered HTML, and lets your Puppeteer parsing code stay the same while it deals with staying unblocked. I walk through that tradeoff in the guide to scraping without getting blocked and the main web scraping guide.

For the rendering and extraction itself, Puppeteer will carry you a long way. Take the pagination script above, point it at a JavaScript-heavy site you care about, and adjust the selectors until the fields come out clean.