R Web Scraping: The Complete Guide

I have written scrapers in several languages, and R is the one I reach for when the scrape is the first step of an analysis I am already doing in R: pulling a price table to chart, harvesting study data, building a dataset for a model. This guide is the map for that path. Which package does what, a full rvest walkthrough you can run, how to parse with CSS selectors and XPath, how to handle JavaScript pages with chromote, and how to keep a scraper alive once a site pushes back. The code uses the documented API for each package at its current 2026 version. I point the examples at books.toscrape.com, a sandbox built for scraping practice.

Is R good for web scraping?

Yes, R is a strong choice for web scraping when the data feeds work you are already doing in R. The package stack is mature: rvest for fetching and parsing, xml2 and httr underneath it, chromote for JavaScript pages. A working scraper is about 10 lines, and rvest returns data that pipes straight into a tibble, so the jump from scrape to dplyr or ggplot2 has no export step in between.

Three things make R pull its weight here:

• It keeps the whole job in one environment. Scrape, clean, model, and plot all happen in the same session. No writing a CSV in Python and reading it back into R.
• rvest reads like the tidyverse. It uses the pipe and returns tibbles, so it fits the dplyr and tidyr code you already write.
• The selectors are the ones you know. rvest takes the same CSS selectors you use in the browser, and full XPath when you need it.

Python has more general scraping tutorials and a deeper bench of standalone crawlers, which I cover in the Python guide. R wins when the scraper is the front end of an analysis pipeline rather than a standalone service.

What are the main R web scraping packages?

Five packages cover the whole field, and the right one depends on whether the page needs JavaScript and how much control you want over the request. Static HTML needs only rvest. JavaScript-rendered pages need a browser engine, which means chromote.

A few notes from using all five:

• rvest is the one you reach for first, and the one most of this guide uses. It both fetches and parses, so for static HTML it is the only package you need.
• xml2 is the engine under rvest. rvest’s read_html() returns an xml2 document, and you can drop down to xml2 functions like xml_find_all() whenever you want raw node access.
• httr2 is the successor to httr. When the data lives behind a JSON API, skip HTML parsing and call the endpoint with httr2’s request pipeline.
• chromote runs a real Chrome headlessly through the Chrome DevTools Protocol. rvest’s own read_html_live() uses it under the hood, so for many JavaScript pages you never call chromote directly.
• polite layers bow() and scrape() on top of rvest to check robots.txt, set a descriptive User-Agent, and rate-limit automatically. It is the courteous default for crawling a site you do not own.

How do you install rvest in R?

Install rvest from CRAN with one line, and the dependencies come with it. rvest pulls in xml2 and httr automatically, so the static-scraping stack is a single install.packages() call.

install.packages("rvest")

For the rest of this guide, add the packages used in the JavaScript and crawling sections:

install.packages(c("rvest", "httr2", "chromote", "polite", "dplyr"))

Current versions on CRAN as of June 2026:

chromote is the one with an external requirement: it needs Chrome or Chromium on the machine, since it drives a real browser. Everything else is pure R plus system libraries that CRAN handles. Load rvest with library(rvest) at the top of your script.

How do you scrape a website with rvest step by step?

The loop is three steps: read the page into a document with read_html(), select the nodes you want with html_elements(), then pull text or attributes out of them. rvest does the fetch and the parse, so a complete static scrape is one short script.

This script reads the first catalogue page and builds a tibble of the title, price, and rating for every book on it:

library(rvest)
library(dplyr)

url <- "https://books.toscrape.com/catalogue/page-1.html"

# 1. Fetch and parse the page in one call.
page <- read_html(url)

# 2. Each book is an <article class="product_pod">.
cards <- page |> html_elements("article.product_pod")

# 3. Pull one field per book. html_element() (singular) returns the
#    first match inside each card, keeping the vectors aligned.
books <- tibble(
  title  = cards |> html_element("h3 a") |> html_attr("title"),
  price  = cards |> html_element(".price_color") |> html_text2(),
  # The rating lives in a class like "star-rating Three".
  rating = cards |> html_element(".star-rating") |> html_attr("class")
)

print(books)
cat(nrow(books), "books scraped\n")

Each part of that maps to one job:

• read_html(url) makes the HTTP request and parses the response into an xml2 document in one call.
• html_elements("article.product_pod") takes a CSS selector and returns every match as a node set. The plural form returns all matches; the singular html_element() returns the first.
• Calling html_element() (singular) on the cards node set returns exactly one node per card, so the three columns stay the same length and line up row by row in the tibble.
• html_text2() reads the visible text with whitespace cleaned up, which is what you want for display. html_text() returns the raw text including stray spacing.
• html_attr("title") reads an attribute, which is where this page stores the full book title rather than the truncated link text.

This is the approach I would ship for a static page. The selectors are the only part that changes per site, so the pattern transfers to any target whose data is in the initial HTML. To find selectors, open the page in your browser, right-click an element, and choose Inspect.

I have not run this script here because the R runtime is not installed in my writing environment. The code uses the documented rvest and dplyr APIs at their June 2026 versions. Run it against the books sandbox first, where there is no risk of a block, before you point it at a live target.

How do you parse HTML with CSS selectors and XPath in R?

rvest gives you both selector languages, and you pass XPath through the same html_elements() function. CSS is shorter for everyday selecting; XPath wins when you match on text or walk up the tree. The functions you will use constantly:

A few patterns I use on almost every job:

# CSS: the everyday case.
title <- page |> html_element("h1") |> html_text2()

# An attribute, for example a link target or image source.
href <- page |> html_element("article.product_pod h3 a") |> html_attr("href")
img  <- page |> html_element("article.product_pod img") |> html_attr("src")

# XPath: select by an exact attribute value, or by text content.
price <- page |>
  html_elements(xpath = '//p[@class="price_color"]') |>
  html_text2()

# Pull a whole HTML table straight into a tibble.
tables <- page |> html_table()
first_table <- tables[[1]]

Two things worth knowing. First, html_element() (singular) returns NA when a node is missing instead of dropping the row, so columns built from it stay aligned. The plural html_elements() silently returns a shorter vector, which knocks your columns out of sync, so use the singular form inside a per-card extract. Second, html_table() is the fastest path for tabular pages: point it at a document and it returns a list of tibbles, one per <table>. For a deeper reference on building robust selectors across both languages, see my XPath and CSS selectors guide.

How do you follow links and handle pagination in R?

Follow the “next” link until it disappears. Most paginated lists put a next-page anchor at the bottom of each page, so you scrape a page, look for that link, resolve it to an absolute URL, and repeat until there is no next link. rvest’s url_absolute() turns the relative href into a full URL using the page you are on as the base.

library(rvest)

base    <- "https://books.toscrape.com/catalogue/"
url     <- paste0(base, "page-1.html")
titles  <- character()
pages   <- 0

repeat {
  page <- read_html(url)

  titles <- c(
    titles,
    page |> html_elements("article.product_pod h3 a") |> html_attr("title")
  )
  pages <- pages + 1

  # Find the next-page link, if any.
  next_href <- page |> html_element("li.next a") |> html_attr("href")
  if (is.na(next_href)) break          # no next link means last page

  url <- url_absolute(next_href, url)  # resolve relative to current page
  Sys.sleep(1)                          # be polite: one request per second
}

cat("pages crawled:", pages, "\n")
cat("total titles: ", length(titles), "\n")

Two details make this robust on a real target:

• url_absolute(next_href, url) resolves the relative path against the current page, so you never hand-build URLs. Reading the bare href gives you something like page-2.html with no directory, which fails when the listing lives in a subfolder.
• Sys.sleep(1) paces the crawl to roughly one request per second. On a production job, wrap read_html() in tryCatch() so one failed page does not kill the run, and read the site’s robots.txt and terms first.

Against the full books.toscrape.com catalogue this pattern walks all 50 listing pages and the 1,000 books across them. The same “find the next anchor, resolve it, repeat” loop handles any next-button pagination. When you want robots.txt checks and throttling handled for you, swap read_html() for the polite package’s bow() and scrape(), which enforce both automatically.

How do you scrape JavaScript-heavy sites in R?

Use a headless browser, because rvest cannot run JavaScript. When a page loads its data with JavaScript after the initial response, that data is absent from the HTML read_html() downloads, so your selectors find nothing. The R answer is chromote, which drives a real headless Chrome through the Chrome DevTools Protocol. rvest wraps it in read_html_live(), so for most pages you stay in rvest and never touch chromote directly.

Option A: rvest’s read_html_live (the easy path)

read_html_live() loads the URL in a headless Chrome, runs the scripts, and returns a live object you query with the same html_element() functions as a static page. It needs chromote and a local Chrome install.

library(rvest)

# Loads the page in headless Chrome and runs its JavaScript.
page <- read_html_live("https://example.com")

# Optional: wait for a selector to appear before reading,
# for content that loads after the initial render.
page$wait_for("article.product_pod")

# Parse with the identical rvest API you already use.
titles <- page |>
  html_elements("article.product_pod h3 a") |>
  html_attr("title")

print(titles)
page$session$close()   # close the browser to free the process

What matters here:

• read_html_live() returns a live DOM that reflects whatever the page’s JavaScript has already rendered.
• page$wait_for("selector") blocks until the selector appears, which handles content that loads asynchronously after the first paint.
• The parsing layer is unchanged: html_elements(), html_text2(), and html_attr() work exactly as on a static page.
• page$session$close() shuts the browser down so you do not leak Chrome processes across a long run.

Option B: chromote directly (full control)

When you need to click, fill forms, or run arbitrary JavaScript, drive chromote yourself. You navigate, let the page settle, pull the rendered HTML out of Chrome, then hand that HTML to rvest’s read_html() so your parsing code stays identical.

library(chromote)
library(rvest)

b <- ChromoteSession$new()

b$Page$navigate("https://example.com")
b$Page$loadEventFired()          # wait for the load event

# Read the fully rendered HTML out of the live DOM.
html <- b$Runtime$evaluate(
  "document.documentElement.outerHTML"
)$result$value

doc <- read_html(html)           # parse with the same rvest code
# doc |> html_elements(".product") ...

b$close()

The pattern mirrors Option A: render in Chrome, grab the HTML, parse with rvest. Here are the two side by side:

The quick test for which path you need: scrape a page with read_html() and search the result for a value you can see in the browser. When it is missing, the page is JavaScript-rendered and you move to read_html_live() first, dropping to raw chromote only if you need to interact with the page. My Selenium guide covers the browser-automation model in more depth if you want the cross-language picture.

How do you scrape in R without getting blocked?

Look like a normal browser, slow down, and spread requests across IP addresses. Parsing is the easy 10 lines. Staying unblocked across thousands of requests is a separate engineering problem. A site flags scrapers through three signals: requests that arrive too fast, headers that look automated, and too much traffic from one IP. Each has a countermeasure that works whether you use rvest, chromote, or httr2.

In R you set headers cleanly with httr2, then hand the response body to rvest. httr2’s request pipeline also gives you req_retry() for backoff and req_throttle() for rate limiting in one place:

library(httr2)
library(rvest)

resp <- request("https://books.toscrape.com/") |>
  req_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36") |>
  req_retry(max_tries = 3) |>       # back off on transient failures
  req_throttle(rate = 1) |>          # at most one request per second
  req_perform()

page <- resp |> resp_body_html()     # parse with rvest as usual
titles <- page |> html_elements("article.product_pod h3 a") |> html_attr("title")

These measures carry a small project a long way. The trouble starts at scale: rotating a large IP pool, solving CAPTCHAs, and handling per-site quirks becomes its own engineering job, and it pulls focus from the analysis you actually want to run. That is where a scraper API like ChocoData fits the category, with a universal endpoint plus 453 dedicated endpoints that return the page so your R code keeps working unchanged while the blocking is handled upstream. You point httr2 at the API instead of the target, parse the response with rvest the same way, and skip the proxy and CAPTCHA plumbing:

library(httr2)
library(rvest)

resp <- request("https://api.chocodata.com/v1") |>
  req_url_query(
    api_key = Sys.getenv("CHOCODATA_KEY"),
    url     = "https://books.toscrape.com/",
    render  = "true"                 # run JavaScript on their side
  ) |>
  req_perform()

page   <- resp |> resp_body_html()   # identical rvest parsing
titles <- page |> html_elements("article.product_pod h3 a") |> html_attr("title")

The parsing layer never changes. You swap where the HTML comes from. Check ChocoData’s own docs for the exact parameter names and current pricing before you wire it in. I cover the full build-versus-buy tradeoff, and the legal side, in the web scraping guide and in how to scrape without getting blocked.

Where should you go next?

Start on the books sandbox, get the rvest example returning a clean tibble, then point it at your real target and watch for blocks. When they appear, add a User-Agent and delays with httr2 first, and reach for read_html_live() or a scraper API only when you actually need them.

For the broader picture beyond R, my web scraping pillar guide covers the concepts that apply in every language: selectors, anti-bot defenses, the legal lines, and when to build versus buy. If you also work in Python, the BeautifulSoup and Scrapy guides cover the closest equivalents to the R tools here.