How to Scrape Cloudflare-Protected Sites

Cloudflare sits in front of a large share of the web, and its bot management is the single most common reason a scraper that worked yesterday returns a wall of HTML you cannot parse today. This guide explains what Cloudflare actually checks, shows the real 403 I got hitting two protected sites with plain Python, and walks through the two methods that get through: a stealth browser for small jobs, and a scraper API for anything at scale. Where I ran code, the output below is pasted verbatim.

Can you legally scrape a Cloudflare-protected site?

Yes, with the same limits that apply to any scraping. Cloudflare is an access-control layer in front of a site; it does not change who owns the data or what the law says about reading public pages. In the US, the 2022 Ninth Circuit ruling in hiQ Labs v. LinkedIn held that scraping publicly available data does not violate the Computer Fraud and Abuse Act, because public pages are not access “without authorization.” Clearing a bot challenge to reach a public page falls under that same logic.

The risk lives in the surrounding facts, not the challenge itself:

The practical rule I follow: scrape only public pages, read the target’s Terms of Service and robots.txt first, stay off anything behind a login, and never collect personal data without a lawful basis. For the full breakdown see my guide on whether web scraping is legal. None of the techniques below change the legal picture; they only change whether the request technically succeeds.

How does Cloudflare detect and block scrapers?

Cloudflare runs a layered check on every request and decides in milliseconds whether to serve the page, challenge the client, or block it. No single trick defeats all the layers, which is why a swapped user agent alone never works. The checks stack like this:

When the verdict is “challenge,” Cloudflare returns an HTTP 403 with a cf-mitigated: challenge response header and a small HTML page that runs the JavaScript check. Cloudflare documents three challenge flavors: a non-interactive challenge (a silent JS proof-of-work, usually under five seconds), a managed challenge (Cloudflare picks the type dynamically per request), and an interactive challenge (the visitor clicks a Turnstile widget). A plain HTTP client receives the challenge HTML, never runs the JavaScript, and so never gets the clearance token that unlocks the real page.

What happens when you scrape Cloudflare with plain requests?

You get a 403 and the challenge page instead of content. I tested this directly. The script below sends a GET request with a realistic Chrome user agent to Indeed (jobs) and G2 (software reviews), both behind Cloudflare, and prints the status, the Server header, and the cf-mitigated header:

import requests, re

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36")

for url in ["https://www.indeed.com", "https://www.g2.com/products/notion/reviews"]:
    r = requests.get(url, headers={"User-Agent": UA}, timeout=20)
    title = re.search(r"<title[^>]*>(.*?)</title>", r.text, re.I | re.S)
    print("URL:        ", url)
    print("STATUS:     ", r.status_code)
    print("Server:     ", r.headers.get("Server"))
    print("cf-mitigated:", r.headers.get("cf-mitigated"))
    print("TITLE:      ", title.group(1).strip()[:60] if title else "(none)")
    print("-" * 40)

That returned a clean 403 from Cloudflare on both, with Indeed handing back the telltale cf-mitigated: challenge header and a “Security Check” title instead of job listings:

URL:         https://www.indeed.com
STATUS:      403
Server:      cloudflare
cf-mitigated: challenge
TITLE:       Security Check - Indeed.com
----------------------------------------
URL:         https://www.g2.com/products/notion/reviews
STATUS:      403
Server:      cloudflare
cf-mitigated: None
TITLE:       g2.com
----------------------------------------

The body of the G2 response was just the string “Please enable JS and disable any ad blocker.” Both responses are the challenge interstitial, not the page I asked for. Stripping the HTML, adding more headers, or retrying the same way changes nothing, because the wall is the unsolved JavaScript challenge, not a missing header.

One honest caveat from the same test run: not every Cloudflare site challenges a plain request. The same script against crunchbase.com and nowsecure.nl returned 200 with Server: cloudflare, because those zones run a looser security level for that path. Cloudflare protection is a dial the site owner sets, so always test your specific target before assuming you need heavy machinery.

Method 1: Bypass Cloudflare with a stealth browser (DIY)

Use a real browser engine with anti-detection patches so the JavaScript challenge actually executes and passes. This is the right DIY route because the challenge is a browser proof-of-work; a tool that runs a genuine browser can solve it the same way a human visitor’s browser does. Three options, from lightest to heaviest:

For most JS challenges, a stealthed Playwright run is the cleanest starting point. Install it with pip install playwright then playwright install chromium. The script below opens a real Chromium, navigates to the protected page, waits for the challenge to clear, and reads the title:

from playwright.sync_api import sync_playwright

URL = "https://www.indeed.com/jobs?q=python+developer&l=Remote"

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)  # headful clears more challenges
    page = browser.new_page(user_agent=(
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"))
    page.goto(URL, wait_until="domcontentloaded")
    page.wait_for_timeout(8000)  # give the JS challenge time to run
    print("TITLE:", page.title())
    # Once past the challenge, parse normally:
    cards = page.query_selector_all("div.job_seen_beacon")
    print("cards:", len(cards))
    browser.close()

I did not paste output for this script because a reliable Cloudflare clearance depends on the runner’s real IP reputation, and my test environment’s datacenter IP is exactly the kind Cloudflare pre-scores as risky, so a result from it would not generalize to your machine. The mechanics are sound and widely used: a headful real browser executes the proof-of-work and receives the cf_clearance cookie. The honest limits of this method:

• It is slow. Each page spins up a browser. Tens of pages are fine; tens of thousands are not.
• Your IP still matters. A flagged datacenter IP can get challenged or blocked even with a perfect browser. You will need residential proxies to scale, which you wire in yourself.
• It breaks. Cloudflare ships detection updates, and stealth patches play catch-up. Expect to maintain it.
• Headless is weaker. headless=True is faster but trips more challenges; headful passes more but needs a display or xvfb.

For a deeper Playwright walkthrough see my Playwright scraping guide. For the broader anti-block playbook (proxies, headers, pacing) see scraping without getting blocked.

Method 2: Bypass Cloudflare with a scraper API

Send the target URL to a scraper API and let it solve the challenge, rotate residential IPs, and return the finished HTML. This is the method I reach for at any real volume, because it moves the three hard parts (TLS fingerprint, JS challenge, IP reputation) off your machine and onto infrastructure built to keep them current. You make one HTTP call and parse the result with the same BeautifulSoup selectors you would use anyway.

ChocoData is the service I use for this category. It exposes a universal endpoint (one URL for any site) plus 453 dedicated endpoints for common targets. A Cloudflare-protected page with no dedicated endpoint goes through the universal endpoint with the full target URL, which is what that endpoint is built for. Here is the request shape, swapping the failing requests.get from Method 1 for the API:

import requests
from bs4 import BeautifulSoup

API_KEY = "cd_live_YOUR_KEY"  # free tier: 1,000 requests/month, no card
target = "https://www.indeed.com/jobs?q=python+developer&l=Remote"

endpoint = "https://api.chocodata.com/api/v1/universal/get"
params = {
    "api_key": API_KEY,
    "url": target,
    "render": "true",   # run a real browser to clear the JS challenge
    "country": "us",    # US residential IP so US results resolve
}

resp = requests.get(endpoint, params=params, timeout=90)
print("STATUS:", resp.status_code)

# The rendered HTML comes back in the payload; parse the same selectors
# as a normal page, now with content instead of a 403.
soup = BeautifulSoup(resp.text, "html.parser")
print("cards:", len(soup.select("div.job_seen_beacon")))

What changes behind that one call: ChocoData fetches through a country-matched residential IP that rotates automatically and retries through a fresh IP when a challenge appears, and render=true runs a real browser so the Cloudflare interstitial that gave me a 403 gets cleared instead of returned. Per the ChocoData docs the service handles CAPTCHAs and challenge pages, headless-browser detection, and IP blocking, and billing applies only to successful (2xx) responses, so a blocked attempt does not drain your quota.

I did not paste live API output here, because that requires a paid key and I will not fake numbers. The endpoint path, parameters, free-tier limit, and success-only billing above come from the ChocoData docs. The honest claim is narrow: the universal endpoint takes the URL and returns the rendered page through a residential IP, which removes the exact failure (HTTP 403 with cf-mitigated: challenge) I reproduced in Method 1.

Should you bypass Cloudflare yourself or use an API?

Match the method to your scale and tolerance for maintenance. The decision is mostly about volume and how much breakage you are willing to debug:

My rule of thumb: prototype with Playwright to confirm the data is there and the page parses, then move to a scraper API the moment you need volume, reliability, or you find yourself maintaining stealth patches instead of writing scraping logic. The break-even comes fast once you factor in residential proxy costs and engineering time, both of which a per-request API folds into one price.

Key takeaways

Cloudflare blocks scrapers with a layered check (IP reputation, TLS fingerprint, and a JavaScript challenge), and it returns HTTP 403 with cf-mitigated: challenge when it does. I confirmed that live: plain Python got a real 403 from both Indeed and G2. Plain requests cannot pass the challenge because it never runs the JavaScript. For small jobs, a stealth browser like Playwright executes the proof-of-work and gets through, with the caveats that it is slow, IP-sensitive, and needs upkeep. For volume, a scraper API such as ChocoData’s universal endpoint rotates residential IPs and clears the challenge in one call. Whichever you pick, scrape only public pages, respect the target’s terms, and stay off personal data. Start from the web scraping pillar guide for the full toolkit.