Web Scraping with Playwright: Complete Tutorial

Playwright is the tool I reach for when a page needs a real browser to show its data. It controls Chromium, Firefox, and WebKit through one API, runs the page’s JavaScript, and lets me read the rendered DOM the same way a user’s browser sees it. This tutorial covers the whole loop: install it, scrape a page, find elements, walk pagination, handle JavaScript-rendered content, and take a screenshot. Every snippet here ran against quotes.toscrape.com, a sandbox built for practice, in June 2026 on Playwright 1.56.0.

What is Playwright?

Playwright is a browser automation library from Microsoft that launches and drives a real browser from your code. You tell it to open a URL, click, type, and read elements, and it does that inside Chromium, Firefox, or WebKit while running all the page’s scripts. Because a genuine browser renders the page, the HTML you query already has the JavaScript-built content in it.

It started as an end-to-end testing tool, which is why two scraping-friendly traits come built in:

Playwright covers Python, Node.js, Java, and .NET. This tutorial uses Python because the examples stay short, and the same method names carry over to the other languages.

How do you install Playwright?

Install the package, then download a browser binary. Those are two separate steps, and skipping the second is the most common first-run error:

pip install playwright
python -m playwright install chromium

The first command installs the Python library. The second downloads the actual Chromium build Playwright drives. You can run python -m playwright install with no browser name to fetch all three engines, or name just chromium to keep the download small. For Node.js the equivalent is npm i playwright followed by npx playwright install chromium.

To confirm the version you have, ask the CLI directly. This is the real output on the machine I tested with:

python -m playwright --version

Version 1.56.0

How do you scrape a page with Playwright?

Launch a browser, open a page, then query elements and read their text. This script pulls the text, author, and tags for the first quote on the front page:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/")

    quotes = page.query_selector_all("div.quote")
    print(len(quotes), "quotes on page 1")

    first = quotes[0]
    print("text:", first.query_selector("span.text").inner_text())
    print("author:", first.query_selector("small.author").inner_text())
    print("tags:", [t.inner_text() for t in first.query_selector_all("a.tag")])

    browser.close()

When I ran it, it found all 10 quotes and read the first one cleanly:

10 quotes on page 1
text: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
author: Albert Einstein
tags: ['change', 'deep-thoughts', 'thinking', 'world']

Three things from that code are worth noting:

• sync_playwright() is a context manager, so the engine shuts down cleanly when the with block ends.
• headless=True runs the browser with no visible window, which is what you want on a server. Set it to False while debugging to watch the page.
• query_selector_all takes a CSS selector and returns every match; query_selector returns the first.

How do you find elements? Locators vs query_selector

Playwright gives you two ways to target elements, and the team recommends locators for most work. A locator is a lazy reference to an element that re-finds it when you act, which sidesteps stale-element errors and adds auto-waiting:

Here is the same first-quote read written with locators:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/")

    q = page.locator("div.quote").first
    print("text:", q.locator("span.text").inner_text())
    print("author:", q.locator("small.author").inner_text())

    browser.close()

Real output:

text: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
author: Albert Einstein

I default to locators. They auto-wait before reading, so a slow-loading element does not throw before it appears, and the same handle stays valid even if the page re-renders.

How do you handle pagination?

Click the “next” link until it disappears. Most paginated sites put a next-page anchor at the bottom of the list, so you scrape a page, check for that link, click it, and repeat. The count() check tells you when you have reached the last page:

from playwright.sync_api import sync_playwright

all_quotes = []
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/")

    while True:
        for q in page.locator("div.quote").all():
            all_quotes.append({
                "text": q.locator("span.text").inner_text(),
                "author": q.locator("small.author").inner_text(),
            })
        nxt = page.locator("li.next a")
        if nxt.count() == 0:
            break
        nxt.click()
        page.wait_for_load_state("networkidle")

    browser.close()

print("total quotes:", len(all_quotes))
print("first author:", all_quotes[0]["author"])
print("last author:", all_quotes[-1]["author"])

This walked all 10 pages and collected every quote:

total quotes: 100
first author: Albert Einstein
last author: George R.R. Martin

After a click() that loads new content, call page.wait_for_load_state("networkidle") so the next iteration reads the new page, not the old one. On real sites add a short page.wait_for_timeout(1000) inside the loop to avoid hammering the server.

How does Playwright handle JavaScript-rendered pages?

It runs the JavaScript before you read the DOM, which is the whole reason to use a browser instead of a plain HTTP request. quotes.toscrape.com has a /js version that ships an empty shell and builds the quotes client-side. This script fetches that page two ways and counts the quote blocks in each:

import urllib.request
from playwright.sync_api import sync_playwright

URL = "https://quotes.toscrape.com/js/"

# 1) Raw HTTP request: no JavaScript engine
req = urllib.request.Request(URL, headers={"User-Agent": "Mozilla/5.0"})
html = urllib.request.urlopen(req).read().decode("utf-8")
print("raw HTTP  -> div.quote in HTML:", html.count('class="quote"'))

# 2) Playwright: runs the page's JavaScript first
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(URL)
    page.wait_for_selector("div.quote")
    print("playwright -> div.quote rendered:", page.locator("div.quote").count())
    browser.close()

The contrast is the point of the whole tutorial. Real output:

raw HTTP  -> div.quote in HTML: 0
playwright -> div.quote rendered: 10

The raw HTML had zero quotes because the data arrives through a script the HTTP client never executes. Playwright ran that script, so by the time wait_for_selector returned, all 10 quotes were in the DOM. wait_for_selector blocks until the element exists, which is the safe way to handle content that loads a moment after navigation. For static HTML, a library like BeautifulSoup is lighter; the moment data depends on JavaScript, a browser earns its weight.

How do you take a screenshot with Playwright?

Call page.screenshot() with a path. This is handy for debugging a scraper, because you see exactly what the browser saw at that moment:

from playwright.sync_api import sync_playwright
import os

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1280, "height": 800})
    page.goto("https://quotes.toscrape.com/")
    page.screenshot(path="quotes.png", full_page=True)
    browser.close()

print("saved:", os.path.getsize("quotes.png"), "bytes")

It wrote a real PNG to disk:

saved: 126043 bytes

full_page=True captures the entire scrollable page, not just the viewport. Drop it to capture only what fits in the 1280x800 window I set.

When should you use the async API?

Use it when you want many pages running at once in a single process. The sync API runs one action at a time, which is fine for a single crawl. The async API lets you fire several goto calls concurrently and await them together, which raises throughput on I/O-bound scraping. Here is the front-page count rewritten with the async API:

import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto("https://quotes.toscrape.com/")
        count = await page.locator("div.quote").count()
        print("async found:", count, "quotes")
        await browser.close()

asyncio.run(main())

Real output:

async found: 10 quotes

Every call gains await and methods come from async_playwright, but the method names match the sync API one to one. Start with sync for a single scraper, and reach for async once concurrency is the bottleneck.

Playwright vs Selenium vs Puppeteer

All three drive a real browser and render JavaScript. They differ in setup, language reach, and API ergonomics:

If you are starting fresh and want one library that bundles its browsers and auto-waits, Playwright is the cleanest pick. Teams already deep in Selenium have less reason to switch, and Puppeteer fits Node-only Chrome work. I cover the Python-specific tradeoffs in the Selenium tutorial and the broader Python scraping guide.

Scraping at scale and getting past blocks

A browser solves rendering, and it introduces two costs on real targets. Each Playwright instance spins up a full browser, so memory and CPU climb fast when you run hundreds in parallel, and you manage that with browser contexts, batching, and queues. The harder problem is blocking: a default headless Playwright browser exposes the navigator.webdriver flag and a headless user agent, and anti-bot vendors fingerprint exactly those signals, so a scraper that works on quotes.toscrape.com can hit a challenge wall on a protected production site.

You can soften the footprint with realistic headers, a non-headless context, and stealth patches, and that buys you some headroom. Past a certain level of protection, the reliable route is a scraper API that handles the proxy rotation, browser fingerprinting, and challenge solving for you. ChocoData is one such service: it exposes a universal endpoint plus 453 dedicated endpoints, returns the rendered HTML, and lets your Playwright parsing code stay the same while it deals with staying unblocked. I walk through that whole tradeoff in the guide to scraping without getting blocked and the main web scraping guide.

For the rendering and extraction itself, Playwright will carry you a long way. Take the pagination script above, point it at a JavaScript-heavy site you care about, and adjust the selectors until the fields come out clean.