Rust Web Scraping: The Complete Guide

I have written scrapers in several languages, and Rust is the one I reach for when the scraper has to run as a service: a long-lived crawler, a high-volume job on a small box, or a single binary I can drop on a server with nothing else to install. This guide is the map for that path. Which crates do what, a full reqwest-plus-scraper walkthrough you can compile, how to parse with CSS selectors, how to fetch pages concurrently with async, how to handle JavaScript-rendered sites, and how to keep a scraper alive once a site pushes back. The code uses the documented API for each crate at its current 2026 version. I point the examples at books.toscrape.com, a sandbox built for scraping practice.

Is Rust a good language for web scraping?

Yes, Rust is a strong choice when throughput, memory use, or single-binary deployment matter more than how fast you write the first draft. The stack is small: reqwest for HTTP, scraper for parsing, tokio for async concurrency. A static scrape is around 30 lines, the compiler rejects code that mishandles a missing element, and the result is one native executable with no interpreter or virtual environment to ship alongside it.

Three things make Rust pull its weight here:

• One binary, no runtime. cargo build --release produces a single executable. You copy it to a server and run it. There is no Python version to match or node_modules to install.
• Async concurrency is built in. reqwest on tokio fetches hundreds of URLs at once from one thread pool, so a crawler saturates the network instead of waiting on one request at a time.
• The type system catches parsing bugs early. A selector that finds nothing returns an Option you have to handle, so the “it worked on page 1, panicked on page 40” class of bug surfaces at compile time.

Python has far more scraping tutorials and a faster path to a one-off script, which I cover in the Python guide. Rust wins when the scraper is a service or a high-volume crawler rather than a quick analysis script.

What are the main Rust web scraping crates?

Four crates cover the whole field, and the right set depends on whether the page needs JavaScript and how much request control you want. Static HTML needs only reqwest and scraper. JavaScript-rendered pages need a browser engine, which means fantoccini.

A few notes from using all four:

• reqwest is the HTTP workhorse and the crate you reach for first. It fetches pages, sets headers and a User-Agent, follows redirects, and handles cookies. It is async by default and runs on tokio.
• scraper is the parsing half. It takes the HTML string reqwest returns, builds a document, and lets you query it with the same CSS selectors you use in the browser. It is built on the html5ever and selectors crates that also power parts of Firefox.
• tokio is the async runtime. You rarely call it directly beyond the #[tokio::main] attribute, but reqwest’s futures need a runtime to drive them, and tokio is the standard one.
• fantoccini drives a real browser through WebDriver (chromedriver or geckodriver). When a page renders its content with JavaScript, fantoccini loads it in the browser, runs the scripts, and hands you the rendered HTML to parse with scraper.

How do you set up a Rust scraping project?

Create a project with Cargo and add the three core crates, two of which need feature flags. cargo new scaffolds the project, then cargo add writes the dependencies into Cargo.toml. reqwest’s text-fetching path is fine with defaults, and tokio needs its full feature to enable the runtime macro.

cargo new book-scraper
cd book-scraper
cargo add reqwest --features json
cargo add scraper
cargo add tokio --features full

That produces a Cargo.toml dependency block like this:

[dependencies]
reqwest = { version = "0.13", features = ["json"] }
scraper = "0.27"
tokio = { version = "1", features = ["full"] }

Current versions on crates.io as of June 2026:

Two notes on the feature flags. The json feature on reqwest pulls in serde_json so you can call .json() on a response, which you want the moment you hit an API instead of an HTML page. The full feature on tokio is the simplest choice for a scraper: it turns on the multi-threaded runtime and the #[tokio::main] macro. You can trim it to specific features later to cut compile time.

How do you scrape a website with reqwest and scraper?

The loop is three steps: fetch the page text with reqwest, parse it into a document with Html::parse_document, then iterate matching elements with a Selector. reqwest does the network half and scraper does the parsing half, so a complete static scrape is one short main.

This program reads the first catalogue page and prints the title, price, and rating for every book on it:

use scraper::{Html, Selector};

#[derive(Debug)]
struct Book {
    title: String,
    price: String,
    rating: String,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let url = "https://books.toscrape.com/catalogue/page-1.html";

    // 1. Fetch the page and read the body as a String.
    let html = reqwest::get(url).await?.text().await?;

    // 2. Parse the HTML once into a queryable document.
    let document = Html::parse_document(&html);

    // 3. Build selectors once, outside the loop. parse() returns a
    //    Result, so a typo in the selector fails loudly here.
    let card_sel = Selector::parse("article.product_pod").unwrap();
    let title_sel = Selector::parse("h3 a").unwrap();
    let price_sel = Selector::parse(".price_color").unwrap();
    let rating_sel = Selector::parse(".star-rating").unwrap();

    let mut books = Vec::new();

    for card in document.select(&card_sel) {
        // The full title lives in the anchor's "title" attribute,
        // not the truncated link text.
        let title = card
            .select(&title_sel)
            .next()
            .and_then(|a| a.value().attr("title"))
            .unwrap_or_default()
            .to_string();

        let price = card
            .select(&price_sel)
            .next()
            .map(|p| p.text().collect::<String>())
            .unwrap_or_default();

        // The rating is encoded in a class like "star-rating Three".
        let rating = card
            .select(&rating_sel)
            .next()
            .and_then(|r| r.value().attr("class"))
            .map(|c| c.replace("star-rating", "").trim().to_string())
            .unwrap_or_default();

        books.push(Book { title, price, rating });
    }

    for book in &books {
        println!("{:?}", book);
    }
    println!("{} books scraped", books.len());

    Ok(())
}

Each part of that maps to one job:

• reqwest::get(url).await? makes the HTTP request; the first .await? resolves the response and the second .text().await? reads the body into a String.
• Html::parse_document(&html) parses the full document once. Parsing is the expensive step, so you do it a single time and reuse the result.
• Selector::parse("...") compiles a CSS selector. It returns a Result, which is why a bad selector fails at the .unwrap() instead of silently matching nothing. Build selectors once, before the loop, not on every iteration.
• document.select(&sel) returns an iterator over matching elements. Calling .select() again on a single card scopes the search to that element’s subtree, which keeps each book’s fields together.
• .value().attr("title") reads an attribute, and .text().collect::<String>() concatenates the descendant text nodes. The Option returned by .next() is why the unwrap_or_default() is there: a missing field becomes an empty string instead of a panic.

This is the approach I would ship for a static page. The selectors are the only part that changes per site, so the pattern transfers to any target whose data is in the initial HTML. To find selectors, open the page in your browser, right-click an element, and choose Inspect.

I have not compiled this program here, because the Rust toolchain is not installed in my writing environment. The code uses the documented reqwest and scraper APIs at their June 2026 versions. Run it against the books sandbox first with cargo run, where there is no risk of a block, before you point it at a live target.

How do you parse HTML with CSS selectors in Rust?

The scraper crate gives you CSS selectors through three methods you will use constantly: select to iterate matches, text to read content, and value().attr to read attributes. CSS is the only selector language scraper supports natively, so unlike some libraries there is no XPath fallback to learn.

A few patterns I use on almost every job:

use scraper::{Html, Selector};

let document = Html::parse_document(&html);

// First match only: use .next() on the iterator.
let h1_sel = Selector::parse("h1").unwrap();
let title = document
    .select(&h1_sel)
    .next()
    .map(|el| el.text().collect::<String>())
    .unwrap_or_default();

// An attribute, for example a link target or an image source.
let link_sel = Selector::parse("article.product_pod h3 a").unwrap();
let href = document
    .select(&link_sel)
    .next()
    .and_then(|el| el.value().attr("href"));

// All matches: collect the iterator into a Vec.
let price_sel = Selector::parse(".price_color").unwrap();
let prices: Vec<String> = document
    .select(&price_sel)
    .map(|el| el.text().collect::<String>())
    .collect();

Two things worth knowing. First, select() returns a lazy iterator, so nothing is allocated until you call .next(), .collect(), or loop over it. That keeps memory flat even on large pages. Second, value().attr() returns an Option<&str> rather than panicking on a missing attribute, so the compiler forces you to decide what an absent href should do. That is the safety the type system buys you. For a deeper reference on building robust selectors, including the XPath equivalents you would use in other languages, see my XPath and CSS selectors guide.

How do you crawl multiple pages concurrently in Rust?

Reuse one Client, then fetch pages in parallel with tokio rather than one after another. Creating a fresh client per request throws away connection pooling, so you build a reqwest::Client once and clone it (clones share the same pool). To fetch concurrently, spawn each request as a tokio task and await them together. This is where Rust earns its place: the async model turns a serial crawl into a parallel one with a few lines.

use scraper::{Html, Selector};
use std::time::Duration;

async fn scrape_page(
    client: reqwest::Client,
    url: String,
) -> Result<Vec<String>, reqwest::Error> {
    let html = client.get(&url).send().await?.text().await?;
    let document = Html::parse_document(&html);
    let sel = Selector::parse("article.product_pod h3 a").unwrap();

    let titles = document
        .select(&sel)
        .filter_map(|el| el.value().attr("title").map(String::from))
        .collect();
    Ok(titles)
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Build the client once. Set a realistic User-Agent here.
    let client = reqwest::Client::builder()
        .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36")
        .timeout(Duration::from_secs(15))
        .build()?;

    let base = "https://books.toscrape.com/catalogue/page-";
    let mut handles = Vec::new();

    // Spawn pages 1..=10 as concurrent tasks.
    for page in 1..=10 {
        let client = client.clone();
        let url = format!("{base}{page}.html");
        handles.push(tokio::spawn(scrape_page(client, url)));
    }

    let mut total = 0;
    for handle in handles {
        let titles = handle.await??; // join the task, then unwrap its Result
        total += titles.len();
    }

    println!("scraped {total} titles across 10 pages", );
    Ok(())
}

Two details make this safe and polite on a real target:

• client.clone() is cheap and shares the underlying connection pool, so all ten tasks reuse keep-alive connections instead of opening fresh TCP sockets. Building a new Client inside the loop would defeat that.
• tokio::spawn hands each fetch to the runtime to run concurrently, and handle.await?? joins them back. The double ? is deliberate: the first unwraps the join (did the task run), the second unwraps the request Result (did the fetch succeed).

This fires all ten requests at once, which is fine on the sandbox but aggressive on a live site. On a production crawler, cap concurrency with a tokio::sync::Semaphore and add a delay between waves, the same way you would respect a rate limit in any language. The “spawn tasks, join them” shape is the core of every Rust crawler I have built. For the broader crawler design across languages, my Scrapy guide covers the scheduler-and-pipeline model that Rust leaves you to assemble yourself.

How do you scrape JavaScript-heavy sites in Rust?

Drive a headless browser with fantoccini, because the scraper crate cannot run JavaScript. When a page loads its data with JavaScript after the initial response, that data is absent from the HTML reqwest downloads, so your selectors find nothing. The Rust answer is fantoccini, which controls a real browser through the WebDriver protocol. You navigate to the page, let the scripts run, pull the rendered HTML out of the browser, then parse that HTML with the same scraper code you already wrote.

fantoccini talks to a WebDriver server, so you start one first. With Chrome that is chromedriver:

chromedriver --port=4444

Then connect, navigate, and read the rendered source:

use fantoccini::ClientBuilder;
use scraper::{Html, Selector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Connect to the running WebDriver server.
    let client = ClientBuilder::native()
        .connect("http://localhost:4444")
        .await?;

    // Load the page in a real browser and let its JavaScript run.
    client.goto("https://books.toscrape.com/").await?;

    // Pull the fully rendered HTML out of the live DOM.
    let html = client.source().await?;

    // Parse with the identical scraper API you use on static pages.
    let document = Html::parse_document(&html);
    let sel = Selector::parse("article.product_pod h3 a").unwrap();
    let titles: Vec<String> = document
        .select(&sel)
        .filter_map(|el| el.value().attr("title").map(String::from))
        .collect();

    println!("{} titles", titles.len());

    client.close().await?; // shut the browser session down
    Ok(())
}

What matters here:

• ClientBuilder::native().connect(...) opens a session against the WebDriver server you started. fantoccini does not launch the browser itself; chromedriver or geckodriver does, and fantoccini drives it.
• client.goto(url) loads the page in the real browser, which runs the scripts the static fetch would have skipped.
• client.source() returns the rendered HTML as a String, which you hand straight to Html::parse_document. The parsing layer is identical to the static example.
• client.close() ends the session so you do not leak browser processes across a long run.

When the data loads asynchronously after the first paint, wait for a selector before reading the source with client.wait().for_element(...), fantoccini’s explicit-wait builder. And when you need to click, fill forms, or step through a multi-page flow, client.find(Locator::Css("...")) returns an element you can .click() or .send_keys() on, the same model as Selenium. The quick test for which path you need: fetch a page with reqwest and search the body for a value you can see in the browser. When it is missing, the page is JavaScript-rendered and you move to fantoccini. My Selenium guide covers the browser-automation model in more depth if you want the cross-language picture.

How do you scrape in Rust without getting blocked?

Look like a normal browser, slow down, and spread requests across IP addresses. Parsing is the easy part. Staying unblocked across thousands of requests is a separate engineering problem. A site flags scrapers through three signals: requests that arrive too fast, headers that look automated, and too much traffic from one IP. Each has a countermeasure that works whether you use reqwest or fantoccini.

In reqwest you set headers on the client builder, so every request carries them. A default_headers map plus a user_agent covers the basics:

use reqwest::header::{HeaderMap, HeaderValue, ACCEPT, ACCEPT_LANGUAGE};

let mut headers = HeaderMap::new();
headers.insert(ACCEPT, HeaderValue::from_static("text/html,application/xhtml+xml"));
headers.insert(ACCEPT_LANGUAGE, HeaderValue::from_static("en-US,en;q=0.9"));

let client = reqwest::Client::builder()
    .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36")
    .default_headers(headers)
    .build()?;

// To route through a proxy, add:
// .proxy(reqwest::Proxy::all("http://user:pass@proxy-host:port")?)

These measures carry a small project a long way. The trouble starts at scale: rotating a large IP pool, solving CAPTCHAs, and handling per-site quirks becomes its own engineering job, and in Rust you are assembling that plumbing by hand because the ecosystem has no Scrapy-style framework that bundles it. That is where a scraper API like ChocoData fits the category, with a universal endpoint plus 453 dedicated endpoints that return the page so your Rust code keeps working unchanged while the blocking is handled upstream. You point reqwest at the API instead of the target, parse the response with scraper the same way, and skip the proxy and CAPTCHA work:

use scraper::{Html, Selector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let api_key = std::env::var("CHOCODATA_KEY")?;
    let client = reqwest::Client::new();

    // Pass the target URL and key as query params; render runs JS their side.
    let html = client
        .get("https://api.chocodata.com/v1")
        .query(&[
            ("api_key", api_key.as_str()),
            ("url", "https://books.toscrape.com/"),
            ("render", "true"),
        ])
        .send()
        .await?
        .text()
        .await?;

    // Identical scraper parsing: only the source of the HTML changed.
    let document = Html::parse_document(&html);
    let sel = Selector::parse("article.product_pod h3 a").unwrap();
    let titles: Vec<String> = document
        .select(&sel)
        .filter_map(|el| el.value().attr("title").map(String::from))
        .collect();

    println!("{} titles via the API", titles.len());
    Ok(())
}

The parsing layer never changes. You swap where the HTML comes from. Check ChocoData’s own docs for the exact parameter names and current pricing before you wire it in. I cover the full build-versus-buy tradeoff, and the legal side, in the web scraping guide and in how to scrape without getting blocked.

Where should you go next?

Start on the books sandbox, get the reqwest-plus-scraper example printing clean structs, then point it at your real target and watch for blocks. When they appear, set a User-Agent and cap your concurrency first, and reach for fantoccini or a scraper API only when you actually need them.

For the broader picture beyond Rust, my web scraping pillar guide covers the concepts that apply in every language: selectors, anti-bot defenses, the legal lines, and when to build versus buy. If you also work in Python, the BeautifulSoup and Scrapy guides cover the closest equivalents to the Rust tools here.