Python Web Scraping: The Complete Guide

I have written Python scrapers for years to feed price trackers and market-research tools, and Python is still the first thing I open for a new job. This guide is the map I wish I had on day one: whether Python is the right pick, which library does what, a full requests + BeautifulSoup walkthrough you can run, how to handle JavaScript-heavy pages, and how to keep a scraper alive once a site starts pushing back. Every code sample here ran against books.toscrape.com, a sandbox built for practice, in June 2026.

Is Python good for web scraping?

Yes, Python is the most practical language for web scraping, and it is the one I recommend to almost everyone. The reason is the ecosystem: requests, BeautifulSoup, Scrapy, lxml, and Playwright cover every stage of a scrape, from fetching a page to rendering JavaScript, and they are all mature and well documented. When something breaks at 2am, the answer is usually already on Stack Overflow.

Three things make Python win here:

• Readable code. A working scraper fits in 15 lines, so you spend your time on the target, not the syntax.
• A library for every layer. Fetching, parsing, crawling, and browser automation each have a dedicated, battle-tested tool.
• A huge community. More tutorials, more answered questions, and more maintained packages than any rival stack.

Python is slower than Go or Rust on raw CPU work, but scraping is almost always bound by network and by how fast a site lets you request, not by your language. That speed gap rarely matters in practice.

What are the main Python web scraping libraries?

The four tools below cover the whole field, and the right one depends on the site. Static HTML needs only a fetcher and a parser. Large crawls need a framework. JavaScript-rendered pages need a real browser.

A few notes from using all four:

• requests + BeautifulSoup is the default stack and where most people start. I cover it in depth in the BeautifulSoup guide.
• Scrapy is a framework, not a library you sprinkle in. It brings its own crawl loop, concurrency, and export pipelines. See the Scrapy guide.
• Playwright has largely replaced Selenium for new projects: faster, a cleaner API, and built-in waiting for elements.
• lxml is the engine under a lot of other tools. Reach for it directly when you need raw parsing speed or XPath, which I walk through in the XPath and CSS selectors guide.

For everything that follows I use requests + BeautifulSoup, because it is the foundation the rest build on.

How do you scrape a website with Python step by step?

The loop is three steps: install the libraries, fetch the page with requests, then parse the HTML with BeautifulSoup and pull the fields you want. Here is each step against a live page.

Step 1: Install the libraries

Install both with pip in one command:

pip install requests beautifulsoup4

beautifulsoup4 is the package name. The import in your code is bs4. You do not need a separate parser because html.parser ships with Python.

Step 2: Fetch and parse one page

This script fetches the first catalogue page, then pulls the title and price for every book on it:

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/catalogue/page-1.html"
resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "html.parser")

books = []
for card in soup.select("article.product_pod"):
    books.append({
        "title": card.h3.a["title"],
        "price": card.select_one(".price_color").get_text(strip=True),
    })

print(f"{len(books)} books on this page")
for b in books[:5]:
    print(f"{b['price']:>8}  {b['title']}")

When I ran it, it found all 20 books on the page and printed the first five:

20 books on this page
 £51.77  A Light in the Attic
 £53.74  Tipping the Velvet
 £50.10  Soumission
 £47.82  Sharp Objects
 £54.23  Sapiens: A Brief History of Humankind

Three things from that code are worth keeping:

• resp.raise_for_status() throws on a 404 or 500 so a bad fetch fails loudly instead of feeding empty HTML to the parser.
• soup.select(...) takes a CSS selector and returns every match. select_one(...) returns the first.
• The title lives in the link’s title attribute, so card.h3.a["title"] reads the attribute rather than the visible text.

Step 3: Set a User-Agent

Notice the headers={"User-Agent": "Mozilla/5.0"} in the request. By default requests announces itself as python-requests, which many sites block on sight. Sending a normal browser User-Agent is the single cheapest step to look less like a bot, and you should do it on every real scrape.

How do you handle pagination in Python?

Follow the “next” link until it disappears. Most paginated sites put a next-page anchor at the bottom of each list, so you scrape a page, look for that link, build the next URL, and repeat until there is no next link.

This script walks the entire books.toscrape.com catalogue and counts both pages and books:

import requests
from bs4 import BeautifulSoup

BASE = "https://books.toscrape.com/catalogue/"
url = BASE + "page-1.html"
pages = 0
total = 0

while url:
    soup = BeautifulSoup(requests.get(url).text, "html.parser")
    total += len(soup.select("article.product_pod"))
    pages += 1
    nxt = soup.select_one("li.next a")
    url = BASE + nxt["href"] if nxt else None

print(f"pages crawled: {pages}")
print(f"total books:   {total}")

It crawled the full catalogue and returned every book:

pages crawled: 50
total books:   1000

On a real target, add time.sleep(1) inside the loop so you do not hammer the server, and wrap the request in a try/except so one failed page does not kill the whole run. The pattern itself scales to any “next button” pagination.

How do you scrape JavaScript-heavy sites in Python?

Use a browser automation tool like Playwright, because requests cannot run JavaScript. When a page loads its data with JavaScript after the initial response, that data is not in the HTML requests downloads, so BeautifulSoup sees an empty shell. The fix is to render the page in a real browser first, then read the rendered HTML.

Here is the shape of a Playwright scrape. Install it with pip install playwright followed by playwright install chromium:

from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com")
    page.wait_for_selector(".product")   # wait for JS to inject content
    html = page.content()
    browser.close()

soup = BeautifulSoup(html, "html.parser")

The pattern stays the same as before: get HTML, hand it to BeautifulSoup, select your fields. Playwright just produces the HTML after JavaScript has run. The cost is speed and resources, since you are driving a full browser, so use it only when a site genuinely needs it. The quick test is to fetch a page with requests and search the response for a value you can see in the browser. When it is missing, the page is JavaScript-rendered and you reach for Playwright.

How do you avoid getting blocked when scraping in Python?

Look like a normal browser, slow down, and spread requests across IP addresses. A site flags scrapers by three signals: requests that arrive too fast, headers that look automated, and too much traffic from one IP. Each has a countermeasure.

These measures carry a small project a long way. The trouble starts at scale: rotating a large IP pool, solving CAPTCHAs, and handling per-site quirks becomes its own engineering job. That is where a scraper API like ChocoData fits the category, with a universal endpoint plus dedicated endpoints that return the page so your Python code keeps working unchanged while the blocking is handled for you. I cover that full tradeoff, and the legal side, in the web scraping guide.

For the scraping itself, the stack in this guide will carry you a long way. Take the pagination script, point it at a site you care about, adjust the selectors until the fields come out clean, and add the anti-block measures as the target pushes back.