Java Web Scraping: The Complete Guide

I have written scrapers in several languages, and Java is the one I reach for when the work has to live inside an existing JVM service: a Spring backend, a Kafka pipeline, an Android data job. This guide is the map for that path. Which library does what, a full jsoup walkthrough you can compile, how to handle JavaScript-heavy pages with HtmlUnit and Selenium, and how to keep a scraper alive once a site pushes back. The code uses the documented API for each library at its current 2026 version. I point the examples at books.toscrape.com, a sandbox built for practice.

Is Java good for web scraping?

Yes, Java is a strong choice for web scraping, particularly when your code already runs on the JVM. The library stack is mature: jsoup for parsing, HtmlUnit and Selenium for JavaScript, and the standard java.net.http.HttpClient for raw requests. Java’s static types catch selector and field-mapping bugs at compile time, and its threading model handles large concurrent crawls without extra machinery.

Three things make Java pull its weight here:

• It fits an existing service. If your product is a Spring or Quarkus app, a scraper in the same language shares your build, logging, and deployment instead of bolting on a second runtime.
• jsoup is genuinely pleasant. A working scraper is about 10 lines, and the CSS-selector API reads like the one you already know from the browser.
• Concurrency is built in. Thread pools, CompletableFuture, and virtual threads (Java 21+) let one process crawl many pages without a framework.

Python has more scraping tutorials and a deeper bench of one-off scripts, which I cover in the Python guide. Java wins when the scraper is part of a larger typed system rather than a standalone script.

What are the main Java web scraping libraries?

Three tools cover the whole field, and the right one depends on whether the page needs JavaScript. Static HTML needs only jsoup. JavaScript-rendered pages need a browser engine, and there you choose between HtmlUnit (pure Java, fast) and Selenium (a real browser, maximum fidelity).

A few notes from using all four:

• jsoup is the one you reach for first, and the one most of this guide uses. It both fetches and parses, so for static HTML it is the only dependency you need.
• HtmlUnit runs JavaScript in pure Java with no browser binary to install, which makes it light and CI-friendly. It can lag real browsers on the heaviest modern sites.
• Selenium drives an actual browser, so it renders anything a user sees. The cost is speed, memory, and a driver to manage.
• HttpClient ships with the JDK since Java 11. When the data lives behind a JSON API, skip HTML parsing entirely and call the endpoint directly.

How do you set up jsoup with Maven or Gradle?

Add a single dependency and let the build tool fetch it. jsoup is published to Maven Central under the org.jsoup groupId, and the current version in June 2026 is 1.22.2.

For Maven, add this to the <dependencies> block in pom.xml:

<dependency>
  <groupId>org.jsoup</groupId>
  <artifactId>jsoup</artifactId>
  <version>1.22.2</version>
</dependency>

For Gradle, add one line to build.gradle:

implementation 'org.jsoup:jsoup:1.22.2'

That is the whole install. jsoup has no required transitive dependencies, so nothing else is pulled in. If you prefer no build tool at all, download the single jsoup-1.22.2.jar from the jsoup site and add it to your classpath, but Maven or Gradle is the path I would take on any real project.

How do you scrape a website with Java step by step?

The loop is two steps: fetch the page into a Document with Jsoup.connect(), then pull the fields you want with CSS selectors. jsoup does both, so a complete static scrape is one file.

This program fetches the first catalogue page and prints the title and price for every book on it:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;

public class BookScraper {
    public static void main(String[] args) throws IOException {
        String url = "https://books.toscrape.com/catalogue/page-1.html";

        Document doc = Jsoup.connect(url)
                .userAgent("Mozilla/5.0 (compatible; BookScraper/1.0)")
                .timeout(10_000)
                .get();

        Elements cards = doc.select("article.product_pod");
        System.out.println(cards.size() + " books on this page");

        for (Element card : cards) {
            String title = card.selectFirst("h3 a").attr("title");
            String price = card.selectFirst(".price_color").text();
            System.out.printf("%-8s %s%n", price, title);
        }
    }
}

Each part of that maps to one job:

• Jsoup.connect(url)...get() makes the HTTP request and parses the response into a Document in one call. get() issues a GET; post() exists for forms.
• .userAgent(...) sets a browser-like User-Agent. By default jsoup sends its own agent string, which many sites block, so set this on every real scrape.
• .timeout(10_000) caps the request at 10 seconds (the value is milliseconds) so a slow server does not hang your run.
• doc.select("article.product_pod") takes a CSS selector and returns every match as an Elements list. selectFirst(...) returns the single first match.
• .text() reads the visible text of an element; .attr("title") reads an attribute, which is where this page stores the full book title.

Compile and run it with a build tool, or directly with a recent JDK:

java -cp jsoup-1.22.2.jar BookScraper.java

This is the approach I would ship for a static page. The selectors are the only part that changes per site, so the pattern transfers directly to any target whose data is in the initial HTML.

How do you follow links and handle pagination in Java?

Follow the “next” link until it disappears. Most paginated lists put a next-page anchor at the bottom of each page, so you scrape a page, look for that link, resolve it to an absolute URL, and repeat until there is no next link.

jsoup resolves relative links for you when you read the abs: attribute prefix, which saves manual URL joining:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import java.io.IOException;

public class CatalogueCrawler {
    public static void main(String[] args) throws IOException, InterruptedException {
        String url = "https://books.toscrape.com/catalogue/page-1.html";
        int pages = 0;
        int total = 0;

        while (url != null) {
            Document doc = Jsoup.connect(url)
                    .userAgent("Mozilla/5.0 (compatible; CatalogueCrawler/1.0)")
                    .timeout(10_000)
                    .get();

            total += doc.select("article.product_pod").size();
            pages++;

            Element next = doc.selectFirst("li.next a");
            url = (next != null) ? next.attr("abs:href") : null;

            Thread.sleep(1000); // be polite: one request per second
        }

        System.out.println("pages crawled: " + pages);
        System.out.println("total books:   " + total);
    }
}

Two details make this robust on a real target:

• next.attr("abs:href") returns the fully resolved URL. jsoup sets the base URI for you when you fetch with Jsoup.connect(), so the abs: prefix works without any extra setup. Reading plain href would give you a relative path you would have to join yourself.
• Thread.sleep(1000) paces the crawl to roughly one request per second. On a production job, also wrap the fetch in a try/catch so one failed page does not kill the whole run, and read the site’s robots.txt and terms first.

Against the full books.toscrape.com catalogue this pattern walks all 50 listing pages and the 1,000 books across them. The same “find the next anchor, resolve it, repeat” loop handles any next-button pagination.

How do you scrape JavaScript-heavy sites in Java?

Use a browser engine, because jsoup cannot run JavaScript. When a page loads its data with JavaScript after the initial response, that data is absent from the HTML jsoup downloads, so your selectors find nothing. You have two Java options: HtmlUnit, a headless browser written in pure Java, or Selenium, which drives a real Chrome or Firefox.

Option A: HtmlUnit (pure Java, no browser binary)

HtmlUnit runs JavaScript itself, so there is no separate browser to install. Note the groupId: the project moved to org.htmlunit, and the current version is 5.1.0. The old net.sourceforge.htmlunit coordinates are the legacy line, so use the new one.

<dependency>
  <groupId>org.htmlunit</groupId>
  <artifactId>htmlunit</artifactId>
  <version>5.1.0</version>
</dependency>

You let the page run its scripts, grab the rendered HTML, then hand that HTML to jsoup so your parsing code stays identical:

import org.htmlunit.WebClient;
import org.htmlunit.html.HtmlPage;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class JsScraper {
    public static void main(String[] args) throws Exception {
        try (WebClient client = new WebClient()) {
            client.getOptions().setJavaScriptEnabled(true);
            client.getOptions().setCssEnabled(false);

            HtmlPage page = client.getPage("https://example.com");
            client.waitForBackgroundJavaScript(5000);
            String html = page.asXml();

            Document doc = Jsoup.parse(html);
            // parse with the same selectors as before:
            // doc.select(".product") ...
        }
    }
}

What matters here:

• try (WebClient client = new WebClient()) uses try-with-resources so the client closes automatically.
• setJavaScriptEnabled(true) is on by default but stated for clarity; setCssEnabled(false) skips CSS work you do not need for scraping.
• waitForBackgroundJavaScript(5000) gives async scripts up to 5 seconds to finish before you read the page. Call it after loading the page, so the timer covers the scripts that page kicked off.
• page.asXml() returns the rendered HTML, which Jsoup.parse(...) turns into the same Document you already know how to query.

Option B: Selenium (a real browser)

Selenium drives an actual Chrome or Firefox through WebDriver, so it renders anything a user sees, including the heaviest modern sites. The current version is 4.44.0, and recent Selenium manages the driver binary for you (Selenium Manager, built in since 4.6), so there is no separate driver download.

<dependency>
  <groupId>org.seleniumhq.selenium</groupId>
  <artifactId>selenium-java</artifactId>
  <version>4.44.0</version>
</dependency>

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class SeleniumScraper {
    public static void main(String[] args) {
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless=new");
        WebDriver driver = new ChromeDriver(options);

        try {
            driver.get("https://example.com");
            String html = driver.getPageSource();

            Document doc = Jsoup.parse(html);
            // doc.select(".product") ...
        } finally {
            driver.quit();
        }
    }
}

The pattern is the same as HtmlUnit: render in a browser, take getPageSource(), parse with jsoup. Always call driver.quit() in a finally block so the browser process does not leak.

Here is how the two compare:

The quick test for which path you need: fetch a page with jsoup and search the result for a value you can see in the browser. When it is missing, the page is JavaScript-rendered and you move to HtmlUnit or Selenium.

How do you avoid getting blocked when scraping in Java?

Look like a normal browser, slow down, and spread requests across IP addresses. A site flags scrapers by three signals: requests that arrive too fast, headers that look automated, and too much traffic from one IP. Each has a countermeasure that applies whether you are using jsoup, HtmlUnit, or Selenium.

These measures carry a small project a long way. The trouble starts at scale: rotating a large IP pool, solving CAPTCHAs, and handling per-site quirks becomes its own engineering job, and it pulls focus from the data you actually want. That is where a scraper API like ChocoData fits the category, with a universal endpoint plus 453 dedicated endpoints that return the page so your Java code keeps working unchanged while the blocking is handled upstream. You point Jsoup.connect() at the API instead of the target, parse the response the same way, and skip the proxy and CAPTCHA plumbing. I cover that full tradeoff, and the legal side, in the web scraping guide and in how to scrape without getting blocked.

For the scraping itself, the stack in this guide will carry you a long way. Take the pagination crawler, point it at a site you care about, adjust the selectors until the fields come out clean, and add the anti-block measures as the target pushes back.