HtmlUnit Web Scraping in Java: Setup, Forms, and Limits

HtmlUnit is a browser written entirely in Java — no Chrome binary, no WebDriver process, just a Maven dependency that fetches pages, runs their JavaScript, keeps cookies, and submits forms. It is still actively developed: version 5.3.0 shipped on 15 July 2026, and 5.x requires JDK 17. For Java teams scraping server-rendered and lightly dynamic sites, it sits between a bare HTTP client and a fleet of Chrome instances, and it is far cheaper than either misconception suggests. This guide covers current setup, the API you will actually use, and an honest account of where it stops working.

Key Takeaways

• Use org.htmlunit:htmlunit — the old net.sourceforge.htmlunit coordinates have been dead since 2023. Current release: 5.3.0 (July 2026)
• HtmlUnit 5.x requires JDK 17 or higher. Stay on the 4.x line if you're stuck on Java 8 or 11
• JavaScript runs on htmlunit-core-js, HtmlUnit's own Rhino fork — good for classic dynamic pages, unreliable for React/Vue/Angular apps
• A WebClient costs single-digit megabytes; a Chrome instance costs hundreds. Hundreds of concurrent HtmlUnit sessions in one JVM is realistic
• setThrowExceptionOnScriptError(false) and setCssEnabled(false) are the two settings every scraping setup needs — without them, real-world pages throw
• Anti-bot systems identify HtmlUnit quickly: its TLS fingerprint and DOM quirks match no shipping browser. It is not a Cloudflare workaround

Is HtmlUnit still maintained in 2026?

Yes, and more actively than its reputation suggests. The project moved from SourceForge to the HtmlUnit GitHub organisation, and the first seven months of 2026 alone produced a major release and two minor ones:

The 4.x line still exists for JDK 8 compatibility but is maintained on a sponsorship basis — treat it as a migration runway, not a destination. HtmlUnit also remains a supported Selenium backend through org.seleniumhq.selenium:htmlunit3-driver (4.46.0, July 2026) if you want the WebDriver API without a browser binary.

How do you set up HtmlUnit for web scraping?

One dependency:

<dependency>
    <groupId>org.htmlunit</groupId>
    <artifactId>htmlunit</artifactId>
    <version>5.3.0</version>
</dependency>

Gradle: implementation 'org.htmlunit:htmlunit:5.3.0'.

Two things trip people up on first run. If you find tutorials importing com.gargoylesoftware.htmlunit.*, they predate the 2023 rename — every package is now org.htmlunit.*. And if the build fails with a class-file version error, you are on a JDK below 17; either upgrade or pin 4.21.0.

A working scrape, with the configuration a scraper actually wants:

import org.htmlunit.WebClient;
import org.htmlunit.BrowserVersion;
import org.htmlunit.html.HtmlPage;
import org.htmlunit.html.HtmlElement;

try (WebClient client = new WebClient(BrowserVersion.CHROME)) {
    client.getOptions().setJavaScriptEnabled(true);
    client.getOptions().setCssEnabled(false);              // no layout needed for scraping
    client.getOptions().setThrowExceptionOnScriptError(false);
    client.getOptions().setThrowExceptionOnFailingStatusCode(false);
    client.getOptions().setTimeout(20_000);

    HtmlPage page = client.getPage("https://quotes.toscrape.com/");

    for (HtmlElement quote : page.<HtmlElement>getByXPath("//div[@class='quote']")) {
        HtmlElement text = quote.getFirstByXPath("./span[@class='text']");
        HtmlElement author = quote.getFirstByXPath("./span/small[@class='author']");
        if (text != null && author != null) {
            System.out.println(author.asNormalizedText() + ": " + text.asNormalizedText());
        }
    }
}

The four options are not optional in practice. Most real pages contain at least one script that throws, at least one 404 sub-resource, and CSS you have no use for; leaving the defaults on means your scraper dies on pages a browser renders fine. setTimeout matters too — HtmlUnit will otherwise wait indefinitely on a hung server.

BrowserVersion.CHROME (also FIREFOX, EDGE) changes the emulated headers and JavaScript quirks. It changes what the page sees; it does not change HtmlUnit's TLS fingerprint, which is the part that gets detected.

CSS selectors instead of XPath

getByXPath is the classic HtmlUnit idiom, but querySelectorAll exists and is usually more readable:

import org.htmlunit.html.DomNode;

for (DomNode node : page.querySelectorAll("div.quote span.text")) {
    System.out.println(node.asNormalizedText());
}

asNormalizedText() collapses whitespace the way a browser's rendered text does — that is almost always what you want over asXml() or raw getTextContent(). If you're translating expressions between the two styles, our XPath cheat sheet has the equivalents side by side.

How do you wait for JavaScript to finish?

The mistake that produces "HtmlUnit returns empty results" bug reports: HtmlUnit's getPage() returns as soon as the document loads, but scripts that fire on a timer or an XHR have not run yet. HtmlUnit's job queue is explicit, so you drive it directly:

HtmlPage page = client.getPage("https://example.com/dynamic");
client.waitForBackgroundJavaScript(10_000);          // ms; returns remaining job count
// or, more precisely:
client.waitForBackgroundJavaScriptStartingBefore(2_000);

This is genuinely nicer than the sleep-and-hope pattern Selenium users fall into, because HtmlUnit knows exactly how many jobs are pending and returns that count. Poll it if you need certainty:

int retries = 20;
while (client.waitForBackgroundJavaScript(500) > 0 && retries-- > 0) { /* keep waiting */ }

Forms, sessions, and logins

This is HtmlUnit's strongest use case. The WebClient is a cookie jar, so a login persists across every subsequent request on the same client:

HtmlPage loginPage = client.getPage("https://example.com/login");
HtmlForm form = loginPage.getForms().get(0);

form.getInputByName("username").type("user@example.com");
form.getInputByName("password").type("secret");
HtmlPage dashboard = form.getButtonByName("submit").click();

// Same client, still authenticated:
HtmlPage orders = client.getPage("https://example.com/orders");

.type() fires the key events a real browser would, which matters on forms with JavaScript validation; .setValueAttribute() sets the value silently and can leave such forms in an invalid state. Note the 3.x rename: getValueAttribute()/setValueAttribute() became getValue()/setValue() on most input types.

Because a WebClient is a few megabytes rather than a few hundred, running many sessions in parallel is ordinary Java concurrency — one client per thread, since WebClient is not thread-safe:

ExecutorService pool = Executors.newFixedThreadPool(16);
for (String url : urls) {
    pool.submit(() -> {
        try (WebClient client = newConfiguredClient()) {
            HtmlPage page = client.getPage(url);
            // extract…
        } catch (Exception e) { /* log and continue */ }
    });
}

Sixteen concurrent Chrome instances would need roughly 5–8 GB of RAM. Sixteen WebClients fit comfortably in a default JVM heap. That gap is the entire argument for HtmlUnit.

Where does HtmlUnit stop working?

Three walls, in the order you'll hit them:

Modern SPA frameworks. htmlunit-core-js is a maintained Rhino fork with a hand-written DOM behind it, not V8 plus Blink. It handles jQuery-era dynamic pages well and improves steadily, but React, Vue, and Angular bundles routinely throw script errors or render nothing. There is no configuration flag that fixes this — if the page is a modern SPA, HtmlUnit is the wrong tool.

Bot detection. HtmlUnit's TLS ClientHello, header ordering, and DOM implementation don't match any shipping browser, so Cloudflare, DataDome, and PerimeterX classify it quickly regardless of BrowserVersion. Changing the emulated browser changes what the page's JavaScript sees, not what the network layer reveals.

Rendering-dependent extraction. No layout engine means no element coordinates, no computed geometry, no screenshots, and no canvas-based rendering. If your extraction depends on where something appears rather than what the markup says, HtmlUnit cannot help.

HtmlUnit vs. the Java alternatives

A useful decision rule: if the page works with JavaScript disabled in your browser, jsoup is enough. If it works in a browser but not with JS off, try HtmlUnit. If HtmlUnit throws script errors or returns an empty container, go straight to a real browser or a rendering API — there is rarely a middle ground worth debugging.

For the real-browser option, Playwright has a first-party Java binding and better waiting semantics than Selenium; both cost the same memory. The pooling, memory, and detection trade-offs are the same regardless of language, and our headless browser guide covers them in one place.

Keeping Java, delegating the fetch

The fourth row of that table is worth spelling out, because it preserves the thing HtmlUnit users actually value — staying in the JVM — while removing the two things it can't do. Fetch through an API that runs real Chrome behind rotating proxies, then parse the returned HTML with jsoup:

import java.net.http.*;
import java.net.URI;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

HttpClient http = HttpClient.newHttpClient();
String target = URLEncoder.encode("https://example.com/spa", StandardCharsets.UTF_8);
String url = "https://api.webscraping.ai/html"
    + "?api_key=YOUR_API_KEY"
    + "&url=" + target
    + "&js=true&proxy=residential&wait_for=" + URLEncoder.encode(".product-grid", StandardCharsets.UTF_8);

HttpResponse<String> resp = http.send(
    HttpRequest.newBuilder(URI.create(url)).build(),
    HttpResponse.BodyHandlers.ofString());

Document doc = Jsoup.parse(resp.body());   // rendered by real Chrome upstream
String price = doc.selectFirst(".price").text();

wait_for takes a CSS selector and holds the response until that element exists — the same guarantee waitForBackgroundJavaScript gives you locally, but on a real browser. When you only need a few fields, /ai/fields skips parsing entirely and returns JSON keyed by the field names you describe in plain English, which survives markup changes that would break a selector.

WebScraping.AI charges 1 credit for a plain fetch, 5 with JavaScript rendering, and 10/25 for residential; failed requests are free, so pages that block you don't appear on the bill. There's a first-party Java SDK, and the free tier is 2,000 credits a month without a card. Where this pays off is the long-running Java pipeline — a price monitoring or job listing aggregation job that has to keep working when a target adds bot protection, without you rewriting the scraper.

Whichever route you take, be clear about what you're collecting first: is web scraping legal covers the terms-of-service and personal-data questions that decide whether the project is worth building.

Frequently asked questions

Is HtmlUnit still maintained? Yes. Version 5.3.0 was released on 15 July 2026, and the GitHub repository was still receiving commits at the end of that month. The one caveat is the JDK 17 baseline introduced in 5.0.0 — projects on Java 8 or 11 must stay on the 4.x line.

What is the correct Maven dependency for HtmlUnit? org.htmlunit:htmlunit, currently 5.3.0. The old net.sourceforge.htmlunit:htmlunit coordinates were abandoned in 2023 and every package moved from com.gargoylesoftware.htmlunit to org.htmlunit.

Can HtmlUnit run React or Angular apps? Usually not. Its JavaScript engine is htmlunit-core-js, a Rhino fork with HtmlUnit's own DOM implementation, which handles classic dynamic pages well but frequently throws on modern framework bundles. Test your specific target rather than assuming either way — and if it fails, no configuration flag will fix it.

Why does HtmlUnit return an empty page or missing elements? Almost always because background JavaScript hasn't run. Call client.waitForBackgroundJavaScript(10_000) after getPage() and check the returned pending-job count. If the content still isn't there, the page's scripts likely failed — enable script-error logging temporarily to see whether HtmlUnit's engine choked on them.

Is HtmlUnit faster than Selenium? Substantially, for the pages it can handle: no browser process to start, single-digit megabytes per session instead of hundreds, and no WebDriver round-trips. The trade is JavaScript fidelity — Selenium runs a real browser engine and HtmlUnit approximates one.

Can HtmlUnit bypass Cloudflare? No. Its TLS fingerprint and DOM quirks don't match any real browser, so it is identified quickly regardless of which BrowserVersion you emulate. Getting past serious bot protection needs a real, patched browser on a residential IP, or a scraping API that maintains that stack for you.