How to Scrape Any Website: A Method Decision Tree

“How do I scrape this site?” is the wrong first question. The right one is “which of three buckets does this site fall into?” because the answer picks your tool for you. Static HTML goes to requests and BeautifulSoup. JavaScript-rendered pages go to Playwright. Sites that fight back go to a scraper API. This guide is the decision tree I run on every new target, with two methods tested live against a real page in June 2026.

Can you legally scrape any website?

Mostly yes for public data, with three hard limits. Scraping publicly visible pages is broadly lawful in the United States: in hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping public profiles does not violate the Computer Fraud and Abuse Act because public data is not “without authorization” access. That is the anchor case people cite.

The word “any” hides three places where scraping gets risky:

So “can I scrape any website” splits into two questions: is the data public (usually fine) and what will I do with it (where the real exposure lives). Read the target’s Terms of Service and robots.txt first. For the full breakdown including the post-hiQ rulings, see is web scraping legal.

The decision tree: which method does this site need?

Run three checks in order, and stop at the first one that fails. Each failure points to the next method.

The single most useful habit here is check 1, and most beginners skip it. They write a requests scraper, get an empty list, and assume the selector is wrong. The selector is fine. The data was never in the HTML their code received. The next section proves it with real output.

How do I tell static HTML from a JavaScript site?

View the page source and search for a value you can see on screen. If it’s in the source, the server sent it and requests will get it. If it’s missing, the browser built it with JavaScript after load, and requests receives an empty shell.

I tested both cases against the same markup. books.toscrape.com serves static HTML. quotes.toscrape.com/js/ uses identical-looking pages where JavaScript injects the content. Here is the exact script I ran:

import requests
from bs4 import BeautifulSoup

UA = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) demo/1.0"}

# Static site: server-rendered HTML
r = requests.get("https://books.toscrape.com/", headers=UA, timeout=30)
soup = BeautifulSoup(r.text, "html.parser")
books = soup.select("article.product_pod")
print("books.toscrape status:", r.status_code)
print("book cards found:", len(books))

# JavaScript-rendered page
r2 = requests.get("https://quotes.toscrape.com/js/", headers=UA, timeout=30)
soup2 = BeautifulSoup(r2.text, "html.parser")
quotes = soup2.select("div.quote")
print("quotes.toscrape/js status:", r2.status_code)
print("quote elements in raw HTML:", len(quotes))

Real output:

books.toscrape status: 200
book cards found: 20
quotes.toscrape/js status: 200
quote elements in raw HTML: 0

Both requests returned 200. The static page handed back 20 parseable cards. The JavaScript page handed back 200 and zero data rows, because the quotes are not in the HTML my code received, they are written in by a script that runs in a browser. A 200 status code tells you the server answered. It says nothing about whether your data is in the response. That gap is the whole reason this decision tree exists.

Method 1: scrape a static site with requests + BeautifulSoup

For static HTML, requests fetches the page and BeautifulSoup parses it, and that is the entire stack. This is the right tool whenever check 1 passes. Extending the test above, here is a complete scraper that pulls the fields off each product card:

import requests
from bs4 import BeautifulSoup

UA = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) demo/1.0"}

r = requests.get("https://books.toscrape.com/", headers=UA, timeout=30)
r.raise_for_status()
soup = BeautifulSoup(r.text, "html.parser")

for card in soup.select("article.product_pod")[:3]:
    title = card.h3.a["title"]
    price = card.select_one("p.price_color").get_text(strip=True)
    stock = card.select_one("p.instock.availability").get_text(strip=True)
    print(f"{title} | {price} | {stock}")

The real output from that run:

A Light in the Attic | £51.77 | In stock
Tipping the Velvet | £53.74 | In stock
Soumission | £50.10 | In stock

Three things make this work and keep it polite:

• A real User-Agent. The default python-requests/2.x agent is an instant tell. A browser string lowers your odds of a quick block.
• raise_for_status(). It turns a silent 403 or 404 into an exception you’ll actually notice, instead of feeding error-page HTML into your parser.
• CSS selectors over brittle paths. article.product_pod and p.price_color survive small layout edits better than deep positional chains.

For the deeper version of this stack, see web scraping with BeautifulSoup and the broader Python web scraping guide. When you need to crawl thousands of pages with retries and concurrency, graduate to Scrapy.

Method 2: render JavaScript pages with Playwright

When check 1 fails, the data lives behind JavaScript, so you need a tool that runs a real browser. Playwright launches Chromium, lets the page’s scripts execute, and then hands you the rendered HTML. The same div.quote selector that found 0 elements in the raw fetch finds all of them once the page renders:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

    quotes = page.query_selector_all("div.quote")
    print("quotes after render:", len(quotes))
    for q in quotes[:3]:
        text = q.query_selector("span.text").inner_text()
        author = q.query_selector("small.author").inner_text()
        print(f"{author}: {text[:50]}")

    browser.close()

Install it with pip install playwright then playwright install chromium. I did not run this snippet in the test environment (Playwright’s browser was not installed there), so I’m not pasting fabricated output. The behavior is well documented: after goto with wait_until="networkidle", the scripts have run and query_selector_all sees the same elements your browser shows.

Before reaching for a full browser, check one shortcut. Many JS sites fetch their data from a backend JSON endpoint that you can call directly. Open DevTools, go to the Network tab, filter by Fetch/XHR, and reload. If you see a clean JSON response, hit that URL with requests and skip the browser entirely. It’s faster, lighter, and less fragile. Playwright is the answer when the data only exists after client-side rendering and there’s no exposed API.

Method 3: scrape defended sites with a scraper API

When you hit 403s, CAPTCHAs, or a Cloudflare wall, the bottleneck moved from parsing to access, and a scraper API solves access. Large sites run bot detection that fingerprints your IP, TLS handshake, headers, and request cadence. Beating that yourself means renting rotating residential proxies, running a headless-browser fleet, and solving CAPTCHAs, which is a full-time infrastructure job. A scraper API rents you that infrastructure per request.

The pattern: you send the target URL to one endpoint, the API handles proxies, browser rendering, and retries, and returns the HTML. Your parsing code stays exactly the same, so only the fetch line changes.

import requests

API_KEY = "YOUR_CHOCODATA_KEY"

resp = requests.get(
    "https://api.chocodata.com/v1/universal",
    params={
        "api_key": API_KEY,
        "url": "https://example.com/protected-page",
        "render_js": "true",       # run the page in a real browser
        "country": "us",           # geotarget the proxy exit
    },
    timeout=90,
)
print(resp.status_code)
html = resp.text   # feed this straight into BeautifulSoup

ChocoData exposes a universal endpoint that takes any URL plus a few flags, alongside 453 site-specific endpoints for common targets that return parsed JSON instead of raw HTML. Two details that matter for cost: JavaScript rendering is an opt-in flag rather than always-on, and only successful (2xx) responses are billed, so failed fetches don’t burn credits. The free tier is 1,000 requests with no card required, which is enough to confirm a defended target actually returns data before you commit.

When does the API earn its price over DIY? This is the trade-off:

If your target is static and friendly, Method 1 is free and fine. The moment you’re spending more time dodging blocks than parsing data, the API is the cheaper path. For the full anti-blocking playbook, see scraping without getting blocked.

What’s the honest limitation of a “scrape anything” approach?

No tool scrapes literally any website, and the failure modes are predictable. Three walls stop everyone:

• Authentication. Data behind a login is a different legal and technical problem. Logging in to scrape usually breaches the Terms you accepted, and that’s where lawsuits start.
• Hard CAPTCHAs and behavioral checks. Some sites gate on mouse movement, session history, or device attestation that even good APIs miss. A residential-proxy success rate is high, not 100%.
• Per-site structure. A universal endpoint solves transport, not parsing. The HTML differs on every site, so your selectors are always custom work. There is no parser that reads every site’s layout for you.

The decision tree handles the transport question well: static, rendered, or defended, each has a clear tool. The parsing question is always yours. Start at check 1 on your next target, and let the first failure tell you which method to reach for. The full web scraping guide covers the rest of the pipeline once you’ve got the HTML in hand.