Ruby Web Scraping: The Complete Guide

I have written scrapers in several languages, and Ruby is the one I reach for when the work has to live inside an app that already speaks Ruby: a Rails backend, a Sidekiq job pulling prices overnight, a small Sinatra service. This guide is the map for that path. Which gem does what, a full Nokogiri walkthrough you can run, how to parse with CSS selectors and XPath, how to handle JavaScript-heavy pages with Ferrum and Selenium, and how to keep a scraper alive once a site pushes back. The code uses the documented API for each gem at its current 2026 version. I point the examples at books.toscrape.com, a sandbox built for scraping practice.

Is Ruby good for web scraping?

Yes, Ruby is a strong choice for web scraping, particularly when your code already runs on Rails or another Ruby stack. The gem set is mature: Nokogiri for parsing, a choice of HTTP clients for fetching, Mechanize for stateful navigation, and Ferrum or Selenium for JavaScript. A working scraper is about 10 lines, and Nokogiri uses the same CSS selectors you already know from the browser.

Three things make Ruby pull its weight here:

• It fits an existing app. If your product is a Rails or Sinatra app, a scraper in the same language shares your Bundler setup, your logging, and your deploy pipeline instead of bolting on a second runtime.
• Nokogiri is genuinely pleasant. It parses messy real-world HTML without complaint and queries it with CSS or XPath, so the extraction code reads cleanly.
• The ecosystem is deep. Sidekiq and Active Job schedule recurring scrapes, and the same models that store your app data store the scraped rows with no export step.

Python has more scraping tutorials and a deeper bench of one-off scripts, which I cover in the Python guide. Ruby wins when the scraper is a feature of a larger Ruby system rather than a standalone service.

What are the main Ruby web scraping gems?

A handful of gems cover the whole field, and the right one depends on whether the page needs JavaScript and how much the scrape has to fetch versus parse. Static HTML needs a fetcher plus Nokogiri. JavaScript-rendered pages need a browser engine, which means Ferrum or Selenium.

A few notes from using them:

• Nokogiri is the constant. Whatever fetches the HTML, Nokogiri is what turns it into a queryable document, so it appears in almost every example below.
• Net::HTTP ships with Ruby, so you can scrape with no gem but Nokogiri. Its API is verbose, which is why most people add a friendlier client.
• HTTParty and http.rb are the two popular friendly clients. HTTParty is the shortest for simple GETs; http.rb reads well through method chaining and handles streaming cleanly.
• Faraday wraps an adapter in a middleware stack, so retries, logging, and response parsing become reusable layers. Reach for it when a scrape grows into a service.
• Mechanize is built on Nokogiri and adds a stateful browser-like agent: cookies, redirects, history, and form helpers. It is the tool for anything behind a login.
• Ferrum and Selenium run a real browser for JavaScript pages. Ferrum talks to Chrome directly over the Chrome DevTools Protocol with no driver binary; Selenium uses WebDriver and supports the most browsers.

How do you install Nokogiri and an HTTP client in Ruby?

Add the gems to a Gemfile and run bundle install, or install them directly with gem install. Nokogiri ships precompiled native gems for common platforms, so on a supported Ruby it installs in seconds with no system libraries to compile. It requires Ruby 3.2 or newer as of version 1.19.

For a single script, install directly:

gem install nokogiri httparty

For a project, a Gemfile pins versions and lets Bundler resolve everything at once:

# Gemfile
source "https://rubygems.org"

gem "nokogiri", "~> 1.19"     # HTML/XML parsing
gem "httparty", "~> 0.24"     # friendly HTTP client
gem "ferrum",   "~> 0.17"     # headless Chrome, for the JavaScript section

Then run bundle install. Current versions on RubyGems as of June 2026:

Ferrum and Selenium are the gems with an external requirement: they drive a real browser, so the machine needs Chrome or Chromium installed. Everything else is pure Ruby. Require what you use at the top of the script, for example require "nokogiri" and require "httparty".

How do you scrape a website with Nokogiri step by step?

The loop is three steps: fetch the page with an HTTP client, parse the body into a Nokogiri document, then pull the fields you want with CSS selectors. Nokogiri does the parsing; the client does the fetch.

This script reads the first catalogue page and prints the title, price, and rating for every book on it:

require "nokogiri"
require "httparty"

url = "https://books.toscrape.com/catalogue/page-1.html"

# 1. Fetch the page.
response = HTTParty.get(url, headers: {
  "User-Agent" => "Mozilla/5.0 (compatible; BookScraper/1.0)"
})

# 2. Parse the HTML body into a queryable document.
doc = Nokogiri::HTML(response.body)

# 3. Each book is an <article class="product_pod">.
cards = doc.css("article.product_pod")
puts "#{cards.size} books on this page"

cards.each do |card|
  title  = card.at_css("h3 a")["title"]
  price  = card.at_css(".price_color").text
  # The rating lives in a class like "star-rating Three".
  rating = card.at_css(".star-rating")["class"].split.last
  puts "#{price.ljust(8)} #{rating.ljust(6)} #{title}"
end

Each part of that maps to one job:

• HTTParty.get(url, headers: {...}) makes the HTTP request and returns a response whose .body is the raw HTML. Setting a browser-like User-Agent matters: a default or missing one is the fastest way to get blocked.
• Nokogiri::HTML(response.body) parses that HTML into a document you can query. It is forgiving of broken markup, so real-world pages parse without special handling.
• doc.css("article.product_pod") takes a CSS selector and returns every match as a node set. at_css(...) returns the single first match.
• card.at_css(".price_color").text reads the visible text of a node. card.at_css("h3 a")["title"] reads an attribute, which is where this page stores the full book title rather than the truncated link text.

This is the approach I would ship for a static page. The selectors are the only part that changes per site, so the pattern transfers to any target whose data is in the initial HTML. To find selectors, open the page in your browser, right-click an element, and choose Inspect.

I have not run this script here because the Ruby runtime is not installed in my writing environment. The code uses the documented Nokogiri and HTTParty APIs at their June 2026 versions. Run it against the books sandbox first, where there is no risk of a block, before you point it at a live target.

Which Ruby HTTP client should you use for scraping?

Pick one client and stay with it; all four fetch HTML fine, and the difference is ergonomics. Net::HTTP needs no gem but is verbose. HTTParty is the shortest for a simple GET. http.rb reads well and streams cleanly. Faraday adds a middleware stack for when the scrape becomes a service. Here is the same GET in each, so you can compare the shape:

# Net::HTTP - standard library, no gem to install.
require "net/http"
body = Net::HTTP.get(URI("https://books.toscrape.com/"))

# HTTParty - shortest for a plain GET.
require "httparty"
body = HTTParty.get("https://books.toscrape.com/").body

# http.rb - method chaining, good for headers and streaming.
require "http"
body = HTTP.headers("User-Agent" => "BookScraper/1.0")
           .get("https://books.toscrape.com/")
           .to_s

# Faraday - middleware stack: retries, logging, adapters.
require "faraday"
conn = Faraday.new do |f|
  f.request :retry, max: 3      # back off on transient failures
  f.adapter Faraday.default_adapter
end
body = conn.get("https://books.toscrape.com/").body

How I choose between them:

In every case the body is a string of HTML, which goes straight into Nokogiri::HTML(body). The parsing layer below does not care which client fetched the page, so you can swap clients later without touching your selectors.

How do you parse HTML with CSS selectors and XPath in Nokogiri?

Nokogiri gives you both selector languages, and the method names tell you which you are using. CSS is shorter for everyday selecting; XPath wins when you match on text or walk up the tree. The methods you will use constantly:

A few patterns I use on almost every job:

# CSS: the everyday case.
title = doc.at_css("h1").text.strip

# An attribute, for example a link target or image source.
href = doc.at_css("article.product_pod h3 a")["href"]
img  = doc.at_css("article.product_pod img")["src"]

# XPath: select by an exact attribute value, or by text content.
prices = doc.xpath('//p[@class="price_color"]').map(&:text)

# Iterate a node set and build an array of hashes.
books = doc.css("article.product_pod").map do |card|
  {
    title: card.at_css("h3 a")["title"],
    price: card.at_css(".price_color").text
  }
end

Two things worth knowing. First, at_css returns nil when nothing matches, so reading .text on a missing node raises a NoMethodError. Guard optional fields with the safe-navigation operator, for example card.at_css(".stock")&.text, so one absent element does not kill the run. Second, Nokogiri parses the document once and you can query it as many times as you like, so build the document with Nokogiri::HTML and reuse it across every field rather than re-parsing. For a deeper reference on building robust selectors across both languages, see my XPath and CSS selectors guide.

How do you follow links and handle pagination in Ruby?

Follow the “next” link until it disappears. Most paginated lists put a next-page anchor at the bottom of each page, so you scrape a page, look for that link, resolve it to an absolute URL, and repeat until there is no next link. Ruby’s standard URI.join turns the relative href into a full URL using the page you are on as the base.

require "nokogiri"
require "httparty"
require "uri"

base   = "https://books.toscrape.com/catalogue/page-1.html"
url    = base
titles = []
pages  = 0

while url
  response = HTTParty.get(url, headers: {
    "User-Agent" => "Mozilla/5.0 (compatible; CatalogueCrawler/1.0)"
  })
  doc = Nokogiri::HTML(response.body)

  titles += doc.css("article.product_pod h3 a").map { |a| a["title"] }
  pages  += 1

  # Find the next-page link, if any.
  next_link = doc.at_css("li.next a")
  url = next_link ? URI.join(url, next_link["href"]).to_s : nil

  sleep 1   # be polite: one request per second
end

puts "pages crawled: #{pages}"
puts "total titles:  #{titles.size}"

Two details make this robust on a real target:

• URI.join(url, next_link["href"]).to_s resolves the relative path against the current page, so you never hand-build URLs. Reading the bare href gives you something like page-2.html with no directory, which breaks when the listing lives in a subfolder.
• sleep 1 paces the crawl to roughly one request per second. On a production job, wrap the fetch in a begin/rescue block so one failed page does not kill the run, and read the site’s robots.txt and terms first.

Against the full books.toscrape.com catalogue this pattern walks all 50 listing pages and the 1,000 books across them. The same “find the next anchor, resolve it, repeat” loop handles any next-button pagination.

When should you use Mechanize instead?

Use Mechanize when the scrape needs state: a login, a session cookie, or a form you submit before the data appears. Mechanize is built on Nokogiri and wraps it in a stateful agent that keeps cookies and history across requests, follows redirects, and fills forms, so you skip the manual cookie and CSRF plumbing. For a plain GET it is overkill; for anything behind a session it saves real work.

require "mechanize"

agent = Mechanize.new
agent.user_agent_alias = "Mac Safari"   # set a realistic UA

# Log in by finding the form, filling it, and submitting.
login_page = agent.get("https://example.com/login")
form = login_page.form_with(action: /login/)
form.field_with(name: "username").value = ENV["SITE_USER"]
form.field_with(name: "password").value = ENV["SITE_PASS"]
dashboard = agent.submit(form)

# The session cookie now rides on every later request automatically.
data_page = agent.get("https://example.com/orders")

# agent.get returns a Mechanize::Page whose parser is a Nokogiri document.
orders = data_page.search("tr.order").map { |row| row.at("td.total").text }

What matters here:

• Mechanize.new creates the agent. It holds the cookie jar and history, so once you log in, every later agent.get is authenticated with no extra code.
• form_with, field_with, and submit locate a form, set its fields, and post it. Mechanize sends back any hidden fields, including CSRF tokens, exactly as the browser would.
• data_page.search(...) is Mechanize’s pass-through to Nokogiri. A Mechanize::Page exposes .parser, a normal Nokogiri document, so all the css, at_css, and xpath methods from the section above work unchanged.

Mechanize does not run JavaScript, since it parses HTML with Nokogiri. For a site that logs in through a JavaScript form or loads data after render, move to a headless browser instead.

How do you scrape JavaScript-heavy sites in Ruby?

Use a headless browser, because Nokogiri cannot run JavaScript. When a page loads its data with JavaScript after the initial response, that data is absent from the HTML your client downloads, so your selectors find nothing. Ruby gives you two options: Ferrum, which drives headless Chrome directly over the Chrome DevTools Protocol in pure Ruby, or Selenium, which drives a real browser through WebDriver.

Option A: Ferrum (pure Ruby, no driver binary)

Ferrum talks to Chrome over the CDP with no WebDriver layer, so there is no driver binary to manage. You navigate, let the page settle, pull the rendered HTML out of Chrome, then hand that HTML to Nokogiri so your parsing code stays identical.

require "ferrum"
require "nokogiri"

browser = Ferrum::Browser.new(headless: true)
browser.goto("https://example.com")

# Wait for a selector to appear before reading, for content
# that loads after the initial render.
browser.network.wait_for_idle
html = browser.body          # the fully rendered HTML

doc = Nokogiri::HTML(html)   # parse with the same Nokogiri code
# doc.css(".product") ...

browser.quit                 # close Chrome to free the process

What matters here:

• Ferrum::Browser.new(headless: true) launches a headless Chrome and connects over the CDP. It finds a system Chrome or Chromium automatically.
• browser.network.wait_for_idle blocks until network activity stops, which covers data that loads asynchronously after the first paint. For a specific element, browser.at_css("selector") polls the live DOM.
• browser.body returns the rendered HTML after scripts have run, which Nokogiri::HTML turns into the same document you already know how to query.
• browser.quit shuts Chrome down so you do not leak browser processes across a long run.

Option B: Selenium (the widest browser support)

Selenium drives an actual Chrome or Firefox through WebDriver, so it renders anything a user sees and supports the most browsers. Recent Selenium manages the driver binary for you through Selenium Manager, so there is no separate driver download.

require "selenium-webdriver"
require "nokogiri"

options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--headless=new")
driver = Selenium::WebDriver.for(:chrome, options: options)

begin
  driver.get("https://example.com")
  html = driver.page_source        # the rendered HTML

  doc = Nokogiri::HTML(html)        # same Nokogiri parsing
  # doc.css(".product") ...
ensure
  driver.quit                       # always close the browser
end

The pattern mirrors Ferrum: render in a browser, take page_source, parse with Nokogiri. Always call driver.quit in an ensure block so the browser process does not leak. Here are the two side by side:

The quick test for which path you need: fetch a page with an HTTP client and search the result for a value you can see in the browser. When it is missing, the page is JavaScript-rendered and you move to Ferrum first, reaching for Selenium when you need a browser Ferrum does not cover. My Selenium guide covers the browser-automation model in more depth if you want the cross-language picture.

How do you scrape in Ruby without getting blocked?

Look like a normal browser, slow down, and spread requests across IP addresses. Parsing is the easy 10 lines. Staying unblocked across thousands of requests is a separate engineering problem. A site flags scrapers through three signals: requests that arrive too fast, headers that look automated, and too much traffic from one IP. Each has a countermeasure that works whether you use Nokogiri with a client, Mechanize, or a headless browser.

In Ruby you set headers cleanly on any client, and Faraday’s middleware gives you retry and backoff in one place so you do not hand-roll it:

require "faraday"
require "faraday/retry"
require "nokogiri"

conn = Faraday.new(
  headers: {
    "User-Agent"      => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " \
                         "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "Accept-Language" => "en-US,en;q=0.9"
  }
) do |f|
  f.request :retry, max: 3, interval: 1, backoff_factor: 2  # back off on failures
  f.adapter Faraday.default_adapter
end

response = conn.get("https://books.toscrape.com/")
doc = Nokogiri::HTML(response.body)   # parse with Nokogiri as usual
titles = doc.css("article.product_pod h3 a").map { |a| a["title"] }

These measures carry a small project a long way. The trouble starts at scale: rotating a large IP pool, solving CAPTCHAs, and handling per-site quirks becomes its own engineering job, and it pulls focus from the data you actually want. That is where a scraper API like ChocoData fits the category, with a universal endpoint plus 453 dedicated endpoints that return the page so your Ruby code keeps working unchanged while the blocking is handled upstream. You point your client at the API instead of the target, parse the response with Nokogiri the same way, and skip the proxy and CAPTCHA plumbing:

require "httparty"
require "nokogiri"

response = HTTParty.get("https://api.chocodata.com/v1", query: {
  api_key: ENV["CHOCODATA_KEY"],
  url:     "https://books.toscrape.com/",
  render:  "true"            # run JavaScript on their side
})

doc = Nokogiri::HTML(response.body)   # identical Nokogiri parsing
titles = doc.css("article.product_pod h3 a").map { |a| a["title"] }

The parsing layer never changes. You swap where the HTML comes from. Check ChocoData’s own docs for the exact parameter names and current pricing before you wire it in. I cover the full build-versus-buy tradeoff, and the legal side, in the web scraping guide and in how to scrape without getting blocked.

Where should you go next?

Start on the books sandbox, get the Nokogiri example printing clean rows, then point it at your real target and watch for blocks. When they appear, add a User-Agent and delays with Faraday first, and reach for Ferrum or a scraper API only when you actually need them.

For the broader picture beyond Ruby, my web scraping pillar guide covers the concepts that apply in every language: selectors, anti-bot defenses, the legal lines, and when to build versus buy. If you also work in Python, the BeautifulSoup and Scrapy guides cover the closest equivalents to the Ruby tools here.