Web Scraping with Selenium: Complete Tutorial

Selenium drives a real Chrome or Firefox the same way a person would: it loads the page, runs the JavaScript, clicks the buttons. That makes it the tool I reach for when a page builds its content in the browser instead of shipping it in the HTML. This tutorial walks the whole loop in Python: install Selenium, scrape a page, wait for elements properly, paginate, and handle a JavaScript-rendered site. Every snippet here ran against quotes.toscrape.com, a sandbox built for practice, in June 2026 on Selenium 4.44.

What is Selenium and when should you use it?

Selenium is a browser automation framework that controls a real browser through a driver. You write Python (or Java, C#, JavaScript, Ruby), Selenium sends commands to a browser like Chrome, and the browser does the work: navigating, rendering, clicking, typing. It was built for testing web apps, and that same control makes it useful for scraping pages that a plain HTTP request can’t read.

The reason to pay the cost of a full browser is JavaScript. A lot of modern pages send a near-empty HTML shell, then fetch and render the real content with JavaScript after load. Requests and BeautifulSoup see the shell. Selenium sees the finished page.

Here’s how the common Python tools line up:

If the data is already in the HTML, use Requests and BeautifulSoup or Scrapy. Reach for Selenium when the page needs a browser to come alive.

How do you install Selenium?

Install Selenium with pip, and on version 4.6 or newer you’re done:

pip install selenium

Selenium 4.6 (released late 2022) bundled Selenium Manager, which removes the old chore of downloading a driver binary and matching it to your browser version by hand. On first run, Selenium detects your installed browser, downloads the matching driver, and caches it. You need a browser installed (Chrome, Edge, or Firefox); the driver is handled for you.

I ran this tutorial on:

If you’re on a locked-down machine with no internet at runtime, you can still point Selenium at a driver you downloaded yourself with a Service object. For most setups, skip that and let Selenium Manager do it.

How do you scrape a page with Selenium?

Launch a headless browser, load the URL, then find elements with the By locators. This script opens quotes.toscrape.com and pulls the text, author, and tags of the first quote:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

options = Options()
options.add_argument("--headless=new")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")

driver = webdriver.Chrome(options=options)
driver.get("https://quotes.toscrape.com/")
print("title:", driver.title)

quotes = driver.find_elements(By.CLASS_NAME, "quote")
print("quotes on page 1:", len(quotes))

first = quotes[0]
print("text:", first.find_element(By.CLASS_NAME, "text").text)
print("author:", first.find_element(By.CLASS_NAME, "author").text)
tags = [t.text for t in first.find_elements(By.CLASS_NAME, "tag")]
print("tags:", tags)

driver.quit()

Running it:

title: Quotes to Scrape
quotes on page 1: 10
text: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
author: Albert Einstein
tags: ['change', 'deep-thoughts', 'thinking', 'world']
driver quit ok

Three things to note. --headless=new runs Chrome without a visible window, which is what you want on a server. find_element returns the first match and throws if nothing is found; find_elements (plural) returns a list and gives you an empty list instead of an error. And always call driver.quit() at the end, or you leak browser processes that pile up fast.

Which locator should you use to find elements?

Selenium gives you eight locator strategies through the By class, and CSS selectors or XPath cover almost everything. Pick the locator that reads clearly and won’t break when the page changes slightly.

CSS selectors are my default because they’re short and fast. I switch to XPath when I need to select by visible text or walk up to a parent, things CSS can’t do. For a deeper reference, see XPath and CSS selectors for scraping.

How do you wait for elements to load?

Use an explicit wait with WebDriverWait and expected_conditions, which pauses until a condition is true or a timeout hits. This is the single most important habit in Selenium scraping. A browser is asynchronous: the page keeps loading after get() returns, so grabbing an element immediately often fails because it isn’t there yet.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "quote"))
)

This waits up to 10 seconds for at least one .quote element to appear, then continues the instant it does. Avoid time.sleep() for this. A fixed sleep either wastes time when the page is fast or fails when it’s slow. Explicit waits adapt to the page.

Common conditions worth knowing:

How do you scrape multiple pages?

Loop: scrape the current page, find the “Next” link, click it, repeat until there’s no next link. quotes.toscrape.com paginates with a li.next link at the bottom, and its absence on the last page is the natural stop signal. This script walks all 10 pages and collects every quote:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException

options = Options()
options.add_argument("--headless=new")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")

driver = webdriver.Chrome(options=options)
driver.get("https://quotes.toscrape.com/")

all_rows = []
page = 1
while True:
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "quote"))
    )
    for q in driver.find_elements(By.CLASS_NAME, "quote"):
        all_rows.append({
            "text": q.find_element(By.CLASS_NAME, "text").text,
            "author": q.find_element(By.CLASS_NAME, "author").text,
            "tags": ", ".join(t.text for t in q.find_elements(By.CLASS_NAME, "tag")),
        })
    try:
        nxt = driver.find_element(By.CSS_SELECTOR, "li.next > a")
    except NoSuchElementException:
        break
    nxt.click()
    page += 1

driver.quit()
print("pages crawled:", page)
print("quotes collected:", len(all_rows))
print("unique authors:", len(set(r["author"] for r in all_rows)))
print("last quote author:", all_rows[-1]["author"])

Running it:

pages crawled: 10
quotes collected: 100
unique authors: 50
last quote author: George R.R. Martin

Catching NoSuchElementException on the next-page lookup is what ends the loop cleanly. From here, writing all_rows to CSV is a few lines with Python’s csv module, the same export shown in the BeautifulSoup tutorial.

How does Selenium handle JavaScript-rendered pages?

This is the reason Selenium exists for scrapers, so here’s the proof. quotes.toscrape.com has a JavaScript build at /js/ where the quotes are injected by JavaScript after the page loads instead of being in the HTML. The HTML that arrives over the wire contains zero quote markup. I scraped that page two ways: once with Requests (no browser), once with Selenium (real browser).

import requests
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

URL = "https://quotes.toscrape.com/js/"

# 1) Plain HTTP request: JavaScript has not run
html = requests.get(URL, headers={"User-Agent": "Mozilla/5.0"}).text
print("requests: quote markers in raw HTML:", html.count('class="quote"'))

# 2) Selenium: the browser runs the JavaScript
options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
driver.get(URL)
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "quote"))
)
print("selenium: quotes in rendered DOM:", len(driver.find_elements(By.CLASS_NAME, "quote")))
driver.save_screenshot("quotes_js.png")
driver.quit()

import os
print("screenshot bytes:", os.path.getsize("quotes_js.png"))

Running it:

requests: quote markers in raw HTML: 0
selenium: quotes in rendered DOM: 10
screenshot bytes: 26545

Requests sees nothing because the quotes don’t exist until JavaScript runs. Selenium sees all 10 because the browser executed that JavaScript and built the DOM. The save_screenshot call also captured the rendered page to a 26 KB PNG, which is useful for debugging what the browser actually saw. When you hit a page where requests.get() returns a shell, this is the gap Selenium fills.

How do you scrape at scale without getting blocked?

A headless browser still announces itself, so defended sites detect and block it. Selenium controls Chrome through automation hooks, and large targets fingerprint those: the navigator.webdriver flag, the headless User-Agent, missing browser plugins, and the request rate. Run a hundred tabs from one IP and you’ll see CAPTCHAs and 403s quickly.

The basic defenses help on lighter sites:

These slow the problem rather than solve it. Browser fingerprinting and IP-based rate limits are a separate fight from rendering the page, and on heavily protected targets you spend more time patching the browser than scraping. That’s where a scraper API earns its keep: tools like ChocoData handle the rendering, IP rotation, and block-solving behind one endpoint, so you send a URL and get back the finished HTML without running a browser fleet yourself. My full playbook on detection lives in scraping without getting blocked.

Selenium vs Playwright: which should you pick?

Both drive real browsers and both run JavaScript, so for scraping the choice comes down to ergonomics and ecosystem. Playwright auto-waits for elements before acting, which removes most of the explicit-wait code you write in Selenium, and its async API suits high-concurrency scraping. Selenium has a longer history, more language bindings, and the W3C WebDriver standard behind it, which matters in test automation.

For a brand-new scraping project I lean Playwright for the auto-waiting alone. If your team already runs Selenium in its test suite, the shared knowledge is worth more than the syntax savings. Either way, the surrounding strategy is the same as the rest of the Python scraping guide.

Wrapping up

Selenium is the right tool the moment a page needs a browser to render. The workflow is consistent: launch headless Chrome, wait for elements with WebDriverWait, locate with CSS or XPath, click through pagination, and quit the driver when you’re done. The JavaScript test above is the clearest signal for when to use it: if requests.get() returns a shell with no data, render it in a browser. When the page is static, drop down to lighter tools. And when blocks become the bottleneck, hand the fetching to a scraper API and keep your code focused on the data. For the bigger picture, start with the web scraping guide.