User Agents for Web Scraping: What They Are and How to Rotate Them

A User-Agent is the first thing I change when a scraper that worked yesterday starts getting blocked. It is a short HTTP header that names the client making the request, and many sites read it to sort browsers from scripts. This guide explains what the header is, why the defaults from python-requests and cURL get filtered, gives you a list of current strings, and shows a tested way to rotate them. For the wider workflow, see the web scraping guide.

What is a User-Agent in web scraping?

A User-Agent is an HTTP request header that identifies the client software sending the request. Every browser sends one. When you load a page in Chrome, the request carries a line like User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) ..., and the server uses it to log traffic, serve the right layout, and judge whether the visitor looks human.

The string packs several facts into one line: the browser engine, the browser version, the operating system, and sometimes the device. A server reads it as a quick declaration of who is knocking. That makes it the cheapest signal a site can check, and the cheapest one you can control.

Why do default python-requests and cURL user agents get blocked?

Default agents get blocked because they announce a script instead of a browser. When you call requests.get(url) with no headers, the library sends User-Agent: python-requests/2.34.2. cURL sends curl/8.x. Both name the tool directly, so a site can filter automated traffic with a single substring match.

I confirmed the default by sending a plain request to an echo endpoint. Here is what the server received:

default -> python-requests/2.34.2

That line is a flag to any site that cares. Blocking it is one rule on a server, and it costs the site nothing because no real visitor sends it. Swapping in a browser string is the first fix, and it often turns a 403 back into a 200 on lightly defended sites. The same logic applies to the command line, where the cURL guide covers setting the agent with the -A flag.

Which browser user agents look realistic in 2026?

A realistic User-Agent is a current browser version on a common operating system. The table below lists strings I pulled from live browsers in June 2026. Version numbers move every few weeks, so treat these as a starting point and refresh them from your own browser’s DevTools.

Two habits keep this list useful. Use strings from browsers that real people run, so Chrome and Safari carry more cover than a niche browser. Keep the version numbers honest, because a Chrome string claiming version 9 or an impossible OS pairing stands out more than the plain library default.

How do I set and rotate the User-Agent in Python?

Set the User-Agent by passing a headers dictionary to your request, and rotate it by picking a different string from a pool on each call. The script below holds four browser strings, sends one plain request to show the default, then sends four requests that each choose a random agent.

I ran this against https://httpbingo.org/user-agent, an echo service that returns the User-Agent it received as JSON. The article originally targeted httpbin.org/user-agent, but that host returned 503 Service Unavailable during testing, so I used httpbingo, the maintained reimplementation with the identical response shape.

import random
import requests

# A small pool of current, realistic browser User-Agent strings.
USER_AGENTS = [
    # Chrome 126 on Windows 11
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    # Firefox 127 on Windows 11
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) "
    "Gecko/20100101 Firefox/127.0",
    # Safari 17 on macOS Sonoma
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
    "(KHTML, like Gecko) Version/17.5 Safari/605.1.15",
    # Chrome 126 on Android
    "Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/126.0.0.0 Mobile Safari/537.36",
]

ECHO = "https://httpbingo.org/user-agent"

# 1. Default requests UA (what the server sees if you set nothing).
default = requests.get(ECHO, timeout=15)
print("default ->", default.json()["user-agent"])

# 2. Send each request with a different UA picked from the pool.
session = requests.Session()
for i in range(4):
    ua = random.choice(USER_AGENTS)
    resp = session.get(ECHO, headers={"User-Agent": ua}, timeout=15)
    seen = resp.json()["user-agent"]
    print(f"request {i} -> {seen}")

The output confirms the server saw a different agent than the library default, and that the rotation varied across requests:

default -> python-requests/2.34.2
request 0 -> Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0
request 1 -> Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
request 2 -> Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
request 3 -> Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

random.choice picks independently each time, so you will see repeats like the three Chrome hits above. That is fine for breaking up a uniform fingerprint. If you need every agent used before any repeats, shuffle the list and cycle through it instead. The same headers argument works whether you parse the result with BeautifulSoup or feed it into a Scrapy spider.

Why is rotating the User-Agent alone not enough?

Rotating the User-Agent alone falls short because modern anti-bot systems read far more than one header. A request with a perfect Chrome agent still looks automated if the rest of the headers are wrong, the IP is flagged, or the traffic pattern is mechanical. The UA is one of several signals, and serious sites cross-check them.

Here is what else a site weighs alongside the agent:

Match your headers to your agent. If the User-Agent claims Chrome on Windows, the Accept-Language and Sec-CH-UA headers should match a real Chrome request, because a Chrome agent paired with a Python TLS handshake is a contradiction a fingerprinter catches.

At scale, or once a target starts blocking despite clean headers and IPs, a scraper API handles the rotation for you. Tools in that category, such as ChocoData, expose a universal endpoint plus dedicated endpoints and manage user agents, headers, IPs, and JavaScript rendering behind one request. That moves the fingerprinting problem off your code when a site has defeated your own pool.

How often should I update my User-Agent list?

Update your list every two to three months, and sooner if block rates climb. Browser versions ship on a fast cycle, so a Chrome 126 string that looked current in mid-2026 reads as stale once most real traffic has moved several versions ahead. An agent that is many versions behind is itself a weak signal.

Pull fresh strings from a source you trust: your own browser’s navigator.userAgent in the console, or the Network tab in DevTools. Those give the exact strings real visitors send right now. A short list of four to six genuine, recent agents does more for you than a long list of synthetic ones, because every entry in it matches traffic the site already sees.