HtmlUnit: Web Scraping with Java's Headless Browser

HtmlUnit is a "GUI-less browser for Java programs" — a headless browser implemented entirely in Java, with no Chrome or Firefox binary behind it. It models pages, executes a good share of real-world JavaScript through the Rhino engine, handles cookies and forms, and runs happily inside a JVM process or CI pipeline. For Java teams scraping moderately dynamic sites, it occupies a sweet spot between raw HTTP clients and full Selenium infrastructure. This guide covers what it does well, working code, and its limits.

Key Takeaways

• HtmlUnit is a pure-Java headless browser — no browser binary, no WebDriver, just a Maven dependency
• It executes JavaScript via Rhino, which covers classic dynamic sites but chokes on heavy modern SPA frameworks
• Cookies, redirects, forms, and authentication are handled with a browser-like API
• It is dramatically lighter than Selenium — hundreds of concurrent "browsers" per JVM are feasible
• For React/Vue apps or bot-protected sites, use real-browser automation or a rendering API instead

Getting Started

Add HtmlUnit to your pom.xml:

<dependency>
    <groupId>org.htmlunit</groupId>
    <artifactId>htmlunit</artifactId>
    <version>4.4.0</version>
</dependency>

Note the org.htmlunit group — the project moved there from net.sourceforge.htmlunit in 2023, and the old coordinates no longer receive updates.

Fetching a page and extracting data with XPath:

import org.htmlunit.WebClient;
import org.htmlunit.html.HtmlPage;
import org.htmlunit.html.HtmlElement;

try (WebClient client = new WebClient()) {
    client.getOptions().setJavaScriptEnabled(true);
    client.getOptions().setCssEnabled(false);
    client.getOptions().setThrowExceptionOnScriptError(false);

    HtmlPage page = client.getPage("https://quotes.toscrape.com/");

    for (HtmlElement quote : page.getByXPath("//span[@class='text']")) {
        System.out.println(quote.asNormalizedText());
    }
}

Disabling CSS and script-error exceptions (most real pages have some) is the standard configuration for scraping.

Forms and Sessions

HtmlUnit's browser model makes login flows straightforward, and the WebClient keeps cookies across requests automatically:

HtmlPage loginPage = client.getPage("https://example.com/login");
HtmlForm form = loginPage.getForms().get(0);

form.getInputByName("username").type("user@example.com");
form.getInputByName("password").type("secret");
HtmlPage dashboard = form.getButtonByName("submit").click();
// client now carries the session for subsequent getPage() calls

You can also switch browser emulation (new WebClient(BrowserVersion.FIREFOX)) to change the headers and JavaScript quirks HtmlUnit presents.

Where HtmlUnit Struggles

HtmlUnit's JavaScript support comes from Rhino plus its own DOM implementation — not a real browser engine. In practice:

• Modern SPAs (React, Vue, Angular) frequently fail to render or throw script errors; HtmlUnit targets the long tail of classic dynamic sites, not framework-heavy apps
• Anti-bot systems fingerprint it easily — its TLS stack and DOM quirks don't match any real browser, so Cloudflare-class protection blocks it quickly
• Rendering fidelity — there's no layout engine, so anything depending on real rendering (canvas, element coordinates) is out

When you hit these limits from Java, the escape hatches are Selenium WebDriver with real Chrome (heavier, but a genuine browser — see our Selenium WebDriver FAQ), or delegating the fetch to a rendering API and parsing the returned HTML with jsoup:

import java.net.http.*;
import java.net.URI;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

HttpClient http = HttpClient.newHttpClient();
String url = "https://api.webscraping.ai/html"
    + "?api_key=YOUR_API_KEY"
    + "&url=" + java.net.URLEncoder.encode("https://example.com/spa", "UTF-8")
    + "&js=true&proxy=residential";

HttpResponse<String> resp = http.send(
    HttpRequest.newBuilder(URI.create(url)).build(),
    HttpResponse.BodyHandlers.ofString());

Document doc = Jsoup.parse(resp.body()); // rendered by real Chrome upstream

This keeps your pipeline in Java while real browsers, proxy rotation, and block handling happen on the API side.

HtmlUnit vs. the Java Alternatives

Conclusion

HtmlUnit remains the pragmatic middle option for Java scraping: far more capable than a bare HTTP client, far cheaper than a fleet of Chrome instances. Use it for server-rendered and lightly dynamic sites, especially where forms and sessions are involved. When a target needs a real rendering engine or hides behind serious bot protection, pair jsoup with WebScraping.AI — you get Chrome-rendered HTML through rotating residential proxies with one HTTP request, and your Java parsing code stays exactly as it is.