Web Scraping Without Getting Blocked: 9 Techniques That Work

Getting blocked is the most common reason a scraper that worked yesterday returns nothing today. The site noticed the traffic was automated and started serving 403s, 429s, or CAPTCHA pages. This guide is the playbook I use to avoid that: nine techniques in the order I apply them, from a one-line header fix to a full scraper API. Every Python snippet here ran against httpbin.org, a request-inspection sandbox, in June 2026, so the output is real.

A quick honesty note before the techniques: the most reliable way to stay unblocked is to scrape politely. Pull public data, keep your rate low, and check the site’s terms and API first. The methods below help you look like a normal client. They do not make you welcome on a site that has told you to leave.

Why do websites block scrapers in the first place?

Websites block scrapers when traffic looks automated rather than human. Anti-bot systems score every request on a handful of signals and challenge or reject the ones that score as a bot. Knowing the signals tells you exactly what to fix.

The signals that get you blocked, roughly in order of how often they bite:

Work down this list. The first five fixes are cheap and handle most sites. The last two are for sites that invest in blocking.

How do realistic request headers stop blocks?

Realistic headers stop blocks because anti-bot systems compare your headers against what a real browser sends, and a thin or inconsistent header set is an immediate tell. A browser sends a dozen headers on every request; a bare scraper often sends three. Matching the browser closes that gap.

The headers worth setting, and why each one matters:

One rule ties these together: keep them consistent. A Chrome User-Agent paired with headers no Chrome build ever sends is worse than a plain request, because the mismatch itself is a signal. Pick a real browser and copy its full header set from your own DevTools Network tab.

What is a real User-Agent string I can use?

A real User-Agent is the exact string a current browser sends, copied verbatim. The default python-requests/2.34.2 announces a bot on every request, so replacing it is the single highest-value change you can make. Here is a current desktop Chrome string you can drop in:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

To prove the server actually receives it, I sent a request through a requests.Session with custom headers and a retry adapter, then asked httpbin to echo back what it saw:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " \
     "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"

session = requests.Session()
session.headers.update({
    "User-Agent": UA,
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "https://www.google.com/",
})

retry = Retry(
    total=5,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
    respect_retry_after_header=True,
)
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))

r = session.get("https://httpbin.org/headers", timeout=30)
print(r.status_code)
data = r.json()["headers"]
print(data["User-Agent"])
print(data["Accept-Language"])

The server echoed back the exact User-Agent and Accept-Language I set, confirming the headers arrive intact:

200
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
en-US,en;q=0.9

The first time I ran this, the public httpbin instance was throwing 503s. The retry adapter mounted on the session absorbed them and the request still came back 200, which is the next technique in action.

Why use a Session with cookies instead of plain requests?

A requests.Session keeps cookies and headers across requests, so the site sees one continuous visitor instead of a stream of strangers. Real users carry a session cookie from page to page. A scraper that drops cookies and starts fresh on every call looks nothing like a browser, and many sites use that pattern to spot bots.

The session also saves you work. Set your headers once with session.headers.update(...) and every request reuses them. It pools and reuses the underlying TCP connection, which is faster and, again, closer to how a browser behaves. For any multi-page scrape I default to a session and never look back.

import requests

session = requests.Session()
session.headers.update({"User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"})

# A login or first page sets a cookie; the session carries it forward.
session.get("https://httpbin.org/cookies/set/sessionid/abc123")
print(session.cookies.get_dict())

Cookies set on the first call are sent automatically on every later call from the same session, which is what keeps you logged in and consistent.

How do rate limiting and random delays help?

Rate limiting and random delays help because request timing is one of the loudest bot signals: humans pause between clicks, scripts do not. A loop firing requests back to back at machine speed produces a traffic pattern no person creates, and rate limiters key on exactly that. Spacing requests out, with some randomness, smooths the pattern.

Two things to get right. First, the base rate: start around one request every 2 to 5 seconds per site and only speed up if you see no 429s. Second, the jitter: a fixed sleep(2) is itself a detectable signature because the intervals are too perfect. Randomize it.

import time
import random

def polite_delay(base=2.0, jitter=1.5):
    time.sleep(base + random.uniform(0, jitter))

for page in range(1, 4):
    # fetch the page here
    polite_delay()  # waits 2.0 to 3.5 seconds between requests

If you scrape with concurrency, count total requests across all workers, not per worker. Ten threads each waiting 2 seconds still hit the site five times a second.

How do retries with backoff keep a scraper alive?

Retries with exponential backoff keep a scraper alive by treating transient failures as temporary, waiting longer after each one instead of hammering a struggling server. A 429 or 503 often clears in a few seconds. The right response is to pause and try again, with the pause growing each attempt so you ease off rather than pile on.

I tested this against httpbin’s /status/429 endpoint, which always returns 429, then made a normal request that succeeds. The function logs each attempt and the delay before the next:

import time
import random
import requests

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " \
     "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"

session = requests.Session()
session.headers.update({"User-Agent": UA, "Accept-Language": "en-US,en;q=0.9"})

def get_with_backoff(url, max_tries=4, base=1.0):
    for attempt in range(1, max_tries + 1):
        resp = session.get(url, timeout=30)
        print(f"attempt {attempt}: HTTP {resp.status_code}")
        if resp.status_code != 429:
            return resp
        if attempt < max_tries:
            delay = base * (2 ** (attempt - 1)) + random.uniform(0, 0.5)
            print(f"  429 received, sleeping {delay:.2f}s")
            time.sleep(delay)
    return resp

# First: an endpoint that returns 429 to exercise the backoff
get_with_backoff("https://httpbin.org/status/429", max_tries=3)

# Then: the real request, which succeeds
ok = session.get("https://httpbin.org/get", timeout=30)
print(f"final request: HTTP {ok.status_code}")
print("url seen by server:", ok.json()["url"])

The real output shows the 429s, the delays doubling between attempts, and the final 200:

attempt 1: HTTP 429
  429 received, sleeping 1.42s
attempt 2: HTTP 429
  429 received, sleeping 2.28s
attempt 3: HTTP 429
final request: HTTP 200
url seen by server: https://httpbin.org/get

For most code I let urllib3 do this automatically with the Retry adapter shown in the User-Agent section: set backoff_factor and a status_forcelist, mount it on the session, and every request retries on its own. Setting respect_retry_after_header=True makes it honor a Retry-After header when the server sends one, which is the polite and effective move on a 429.

When do you need rotating IPs or proxies?

You need rotating IPs once a single address makes enough requests to trip a per-IP limit, no matter how clean your headers are. Sites count requests per IP over time. Past some threshold the IP gets throttled or blocked outright, and the only fix is to spread requests across many addresses. That is what proxy rotation does.

Signs you have hit an IP limit rather than a header problem:

• A request that works from a fresh network returns 403 or 429 from your scraping host.
• The block follows the IP: switching networks clears it immediately.
• You were fine for the first few hundred requests, then everything started failing.

Proxies come in a few flavors (datacenter, residential, mobile) at different price and trust levels. Choosing and configuring a proxy provider is its own topic and out of scope here. The point for blocking is the principle: when volume per IP is the problem, distributing that volume across IPs is the solution, and a managed scraper API (below) bundles rotation so you do not run pools yourself.

How do headless browsers get past JavaScript blocks?

Headless browsers get past JavaScript blocks by running the page’s JavaScript the way a real browser does, so content and anti-bot checks that depend on JS actually execute. A plain HTTP request downloads the initial HTML and stops. If the data loads after that through JavaScript, or the site runs a JS challenge before serving content, a raw request sees an empty shell or a block page. A headless browser renders the full page.

Tools like Playwright and Selenium drive a real Chromium or Firefox. They execute scripts, fire events, and return the rendered DOM. A minimal Playwright fetch:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/126.0.0.0 Safari/537.36"
    )
    page.goto("https://example.com")
    html = page.content()  # fully rendered HTML
    browser.close()

The cost is speed and resources: a browser is far heavier than an HTTP request, so I only reach for one when the page genuinely needs JavaScript. Set a real User-Agent on the browser context too, since headless defaults can leak that you are automated. For sites that fingerprint the browser itself, stealth plugins and managed browser endpoints help, but at that point a scraper API is usually the simpler call.

Should you respect robots.txt and rate limits?

Yes, respect robots.txt and posted rate limits, both because it is the right default and because it keeps you unblocked. robots.txt is a file at the site root that lists which paths automated clients should avoid. Honoring it keeps you out of the areas most likely to be watched and trapped, and Python reads it for you:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://en.wikipedia.org/robots.txt")
rp.read()

print(rp.can_fetch("MyScraper", "https://en.wikipedia.org/wiki/Web_scraping"))
print(rp.crawl_delay("MyScraper"))

can_fetch tells you whether a path is allowed for your User-Agent, and crawl_delay returns the delay the site asks you to keep between requests when it specifies one. Treating those as instructions rather than suggestions does two things at once: it lowers your odds of a block and it keeps your scraping defensible. For a worked example on a real target, see my Wikipedia scraping guide.

When should you use a scraper API instead?

Use a scraper API when a site blocks hard enough that managing headers, proxy pools, browsers, and retries yourself stops being worth it. At that point you are maintaining anti-blocking infrastructure instead of collecting data. A scraper API folds proxy rotation, browser rendering, and retry logic behind one HTTP call, so you send a URL and get back HTML.

This is the right tool for three situations: pages behind serious anti-bot systems, jobs that need rotating residential IPs at scale, and JavaScript-heavy sites where you would otherwise run a browser farm. ChocoData is one example of the category, a web scraper API with a universal endpoint plus 453 site-specific endpoints, so a request that keeps failing on your own stack at scale becomes a single call that returns the rendered page. The tradeoff is cost per request against the engineering time of running the pipeline yourself, and the web scraping guide lays out where each side wins.

The shorter version of this whole playbook: look like a normal client, move at a human pace, spread your traffic, and respect the site’s rules. Most blocks come from violating one of those four. Fix the cheap signals first, the headers and the timing, and reach for proxies, browsers, or an API only when a specific site forces the upgrade.