Web Scraping Glossary: Every Term Explained

I get the same question from people new to scraping every week: what does half this vocabulary actually mean? Scraper, crawler, parser, proxy, headless, rate limit, 429. The words get used loosely, and loose words lead to broken scrapers. This glossary defines every term I lean on, grouped so each one builds on the last. Start at the top or jump to the section you need. The pillar web scraping guide ties it all together end to end.

What is the meaning of “scraper”?

A scraper is a program that downloads a web page and extracts specific pieces of data from it. You point it at a URL, it pulls the content, and it returns structured values (prices, titles, reviews, listings) instead of a wall of HTML. The output usually lands in a CSV, a JSON file, or a database.

The word describes the extraction step. A scraper “scrapes” the data you want off a page the same way you would copy a price into a spreadsheet by hand, except it does it for thousands of pages without stopping. The term covers the whole tool in casual use, but precisely it means the part that reads content and recovers fields.

Scraping is one branch of the wider family called data scraping, which covers extracting data from any machine output. Reading a web page’s HTML is web scraping. Reading a terminal or desktop screen built for human eyes is screen scraping. Same goal, different source.

What is the difference between a crawler and a scraper?

A crawler discovers and follows links across a site; a scraper extracts data from the pages a crawler finds. They do two different jobs, and most real projects run both at once. The crawler answers “which pages exist?” and the scraper answers “what data is on this page?”

A search engine is the clearest example. Google’s crawler (Googlebot) walks the web following links to find pages. A separate process then indexes the content. In your own projects, a crawler might start on a category page, collect every product URL, and hand that list to a scraper that visits each URL and pulls the price.

People say “spider” to mean crawler; the terms are interchangeable. Scrapy, a popular Python framework, calls its core class a Spider because it does both jobs in one object. See the Scrapy guide for how that combined model works.

What are fetching, parsing, and extracting?

These are the three sequential jobs inside every scraper: fetch the page, parse the HTML into a tree, then extract the fields you want. Get the vocabulary for these three straight and most scraping tutorials suddenly read clearly.

• Fetch (request): download the raw page. Your code sends an HTTP request to the server and gets back the response: status code, headers, and body. In Python this is the requests library or httpx. The body at this stage is raw HTML text, not yet usable data. See the requests guide.
• Parse: turn that raw HTML string into a structured tree your code can navigate. A parser reads the tags and builds a model of the document (the DOM). BeautifulSoup and lxml are the common Python parsers. Covered in the BeautifulSoup guide.
• Extract: pull the specific values out of the parsed tree using selectors. You tell the parser “give me the text inside this element” and it returns the price or the title.

A selector is the address you use to point at an element. The two selector languages are CSS selectors and XPath. div.price is a CSS selector for a div with class price. //div[@class="price"]/text() is the XPath for the same thing.

What is the DOM, and why does rendering matter?

The DOM (Document Object Model) is the in-memory tree a browser builds from a page’s HTML and scripts. Every element (heading, link, image, div) becomes a node in that tree, and JavaScript on the page can add, remove, or change nodes after the initial HTML loads. The DOM is what you actually see rendered, which can differ from the HTML the server first sent.

Rendering is the process of running that JavaScript and building the final DOM. This is the single biggest gotcha for beginners. When you fetch a page with requests, you get the raw HTML the server sent, before any scripts run. If the site loads its data with JavaScript after the page arrives (most modern sites do), that raw HTML is nearly empty and your scraper finds nothing.

A headless browser solves rendering. It is a real browser engine (Chromium, Firefox) running with no visible window, driven by code. It loads the page, runs the JavaScript, and builds the full DOM, then you scrape the rendered result. Selenium and Playwright are the common tools. The cost is speed: a headless browser is far heavier than a plain HTTP request.

What is an API, and how does it differ from scraping?

An API (Application Programming Interface) is a structured endpoint a site offers on purpose for programs to pull data, usually as clean JSON. When a site has a public API, you request data through it and skip HTML entirely. The server hands you exactly the fields you asked for, already structured.

Scraping and APIs sit at opposite ends of the same goal. An API is the front door the site built for machines. Scraping reads the same data from the page the site built for humans. Use the API when one exists and its terms allow your use; scrape when there is no API or it omits the data you need.

There is a middle path worth knowing. Many “static” sites quietly call their own internal JSON API in the background to load data (you can spot these in your browser’s Network tab). Hitting that internal endpoint directly is often faster and cleaner than rendering the whole page, though it is undocumented and can change without warning.

A scraper API is a third category and the one I get asked about most. It is a paid service you send a target URL to, and it handles fetching, rendering, proxies, and anti-bot evasion, then returns the HTML or parsed data. ChocoData is one: it offers a universal endpoint plus 453 dedicated endpoints for specific sites, so you call one URL and skip building the blocking-evasion stack yourself.

What is a proxy, and what types exist?

A proxy is an intermediary server that forwards your request so the target site sees the proxy’s IP address instead of yours. You route your scraper’s traffic through it. The site logs the proxy’s IP, which lets you spread requests across many addresses and avoid one IP getting blocked for sending too much traffic.

Proxies are the backbone of scraping at scale, because sites rate-limit and ban by IP. The type you choose changes how the target perceives your traffic.

Two more proxy terms come up constantly. A rotating proxy automatically swaps to a new IP on a schedule or on every request, so no single address sends enough traffic to get flagged. A sticky session keeps the same IP for a set window (often 10 to 30 minutes), which you need when a site ties a login or cart to one IP. For the full anti-blocking picture, see scraping without getting blocked.

What is a User-Agent, and why do scrapers set one?

A User-Agent is an HTTP request header, a string that identifies the application, operating system, and version making the request (per MDN). Every browser sends one. A real Chrome request announces itself as Chrome on a specific OS; a default Python requests call announces itself as python-requests/2.x, which is an obvious tell that you are a bot.

Scrapers set a realistic User-Agent so requests look like they come from a normal browser. Sending the default library string is one of the fastest ways to get blocked, because anti-bot systems filter known automation signatures. Setting a current browser’s User-Agent is the cheapest single improvement you can make to a scraper’s success rate. See user agents for web scraping for current strings and rotation.

A header is any key-value pair sent with an HTTP request or response, carrying metadata like the User-Agent, the content type, cookies, and the referring page. Real browsers send a consistent set of headers together; a scraper that sends only a User-Agent and nothing else still looks suspicious because the rest of the header fingerprint is wrong.

What do robots.txt and rate limiting mean?

Robots.txt is a file at the root of a site that tells automated clients which paths they may and may not request. It lives at /robots.txt (for example example.com/robots.txt) and is defined by RFC 9309, the Robots Exclusion Protocol. It lists rules per crawler: which URL paths are disallowed, and sometimes a crawl delay. It is a published request, not a technical lock; compliance is voluntary, and respecting it is part of scraping responsibly. Whether ignoring it carries legal weight is covered in is web scraping legal.

Rate limiting is a server controlling how many requests a client may send in a given window. Exceed the limit and the server starts refusing requests, usually with a 429 status code (more below). Sites rate-limit to protect their infrastructure and to catch bots, because a scraper hammering 50 requests a second looks nothing like a human clicking a few pages a minute.

Throttling is the scraper-side answer to rate limiting: you deliberately add delays between requests to stay under the limit. A few seconds between requests, randomized, keeps your traffic from spiking and reads more like a person. It is slower, and it is what keeps you served instead of banned.

What is a CAPTCHA, and what is fingerprinting?

A CAPTCHA is a challenge a site shows to prove a visitor is human, like clicking traffic-light images or solving a puzzle. The name stands for Completely Automated Public Turing test to tell Computers and Humans Apart. When an anti-bot system suspects automation, it serves a CAPTCHA; passing it requires either a human or a paid solving service, and hitting one mid-scrape usually means your earlier behavior already looked like a bot.

Browser fingerprinting is a site building a unique signature of your client from dozens of signals: the headers you send, your screen size, installed fonts, time zone, how your browser renders a hidden canvas, and more. Even with a fresh IP and a good User-Agent, an inconsistent fingerprint (a “Chrome” request that lacks the properties real Chrome has) gives away automation. This is why a plain HTTP client with a spoofed User-Agent still gets caught: the deeper fingerprint does not match the browser it claims to be.

Two related defenses you will read about:

• TLS fingerprinting: the server inspects the low-level handshake your client makes when it opens an HTTPS connection. Real browsers and scripting libraries handshake differently, so the connection itself can flag you before a single page loads.
• Behavioral analysis: the site watches patterns over time (request timing, mouse movement, navigation order) and scores how human the session looks.

Which HTTP status codes do scrapers hit most?

A status code is a three-digit number the server returns with every response to say what happened. For scraping, a handful of them are the daily diagnosis of why a request worked or failed. Reading them correctly tells you whether to retry, slow down, swap proxies, or change approach.

Three of these carry specific scraping meaning. A 403 Forbidden means the server understood your request and refused it on purpose, per MDN, which during scraping almost always signals a block on your IP or fingerprint rather than a real permission error. A 429 Too Many Requests is the direct rate-limit signal, defined in RFC 6585; the server often includes a Retry-After header naming how many seconds to wait, and honoring it is the polite and effective fix. A 503 Service Unavailable can be a genuinely overloaded server, but on protected sites it is frequently the anti-bot layer turning you away, so treat a sudden run of 503s as a soft block and back off.

A redirect (301 or 302) is worth one extra note: the server is sending you to a different URL. Most HTTP libraries follow redirects automatically, but a redirect to a login page or a CAPTCHA page is itself a block signal dressed up as a 302.

Quick-reference glossary

Every term above in one place, for when you just need the one-line definition:

If you want the next step after the vocabulary, the Python scraping guide puts fetch, parse, and extract together in working code, and the pillar guide covers the full workflow from first request to data at scale.