jsoup Java Web Scraping: HTML Parsing and CSS Selectors

jsoup is the standard Java HTML parser: it turns real-world, messy HTML into a clean DOM you query with CSS selectors, and doubles as a lightweight HTTP client and an HTML sanitizer. This guide covers the full workflow — Maven/Gradle setup and Java compatibility, fetching and parsing, selector patterns, tables and attributes, character encoding, caching, proxies and country-specific pages, login sessions, cleaning untrusted HTML, Android usage, and where jsoup stops (JavaScript) and what to do about it.

Key Takeaways

• Add org.jsoup:jsoup from Maven Central; current releases need Java 8+ and run on Android API 21+
• Jsoup.connect(url).get() fetches and parses in one call — but set userAgent() and timeout() on every connection; the defaults get you blocked and stalled
• Selection is CSS selectors (select("div.product > h2 a[href]")), not XPath — first() returns null, select() returns an empty list
• jsoup silently truncates bodies at 1 MB by default (maxBodySize(0) to disable) — the classic "why is my page cut off" bug
• Use absUrl("href") instead of attr("href") to resolve relative links against the page URL
• jsoup never executes JavaScript — for rendered pages, pair it with a headless browser or a rendering API and keep jsoup for the parsing

Adding jsoup to your project (Maven, Gradle, Android)

Maven:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.21.2</version>
</dependency>

Gradle (Kotlin DSL works the same with parentheses):

implementation 'org.jsoup:jsoup:1.21.2'

Check Maven Central for the latest version — jsoup releases steadily and updates regularly matter because it parses untrusted input. For a quick script without a build tool, download the jar and add it to the classpath: java -cp jsoup-1.21.2.jar:. Scrape.

Which Java versions are compatible?

Modern jsoup (1.16 and later) requires Java 8 or higher and is tested through the current LTS releases (11, 17, 21, 25) — it's a zero-dependency library, so there's no transitive matrix to fight. The jar is modular (module org.jsoup) for JPMS projects, and works on Android API 21+ with no extra configuration. If you're trapped on Java 7 or ancient Android, you'd have to pin a years-old 1.13.x release — better to treat that as a platform-upgrade signal, since old parsers accumulate security fixes they'll never receive.

Fetching and parsing a page

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

Document doc = Jsoup.connect("https://example.com/products")
        .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/141.0.0.0 Safari/537.36")
        .referrer("https://www.google.com/")
        .timeout(10_000)                       // milliseconds; default is 30s
        .get();

System.out.println(doc.title());

Always set a real userAgent — jsoup's default identifies itself as Java, which many sites block outright. get() throws HttpStatusException on non-2xx responses (call .ignoreHttpErrors(true) to inspect error pages instead) and SocketTimeoutException on timeout.

Parsing HTML you already have — from a string or file — skips the network entirely:

Document doc = Jsoup.parse(html);                                  // String
Document fileDoc = Jsoup.parse(new File("page.html"), "UTF-8");
Document fragment = Jsoup.parseBodyFragment("<p>Just a snippet</p>");

Selecting elements with CSS selectors

jsoup's query language is CSS selectors — the same syntax you'd use in a stylesheet or querySelectorAll (our CSS selectors cheat sheet is a handy reference):

import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

Elements products = doc.select("div.product-card");               // all matches (never null)
Element first = doc.selectFirst("h1");                            // first match or NULL

for (Element card : products) {
    String name  = card.selectFirst(".name").text();
    String price = card.selectFirst(".price").text();
    String link  = card.selectFirst("a").absUrl("href");          // absolute URL — see below
}

The selector patterns that cover most scraping work:

select() returns an empty Elements collection when nothing matches, but selectFirst() (and first()) return null — guard them or use Optional.ofNullable. There's no XPath support; if a snippet you're porting uses XPath, translate it to a CSS selector (most translate directly) or see our XPath cheat sheet for the mapping patterns.

Extracting attributes, links, and absolute URLs

attr("href") returns the attribute as written — usually a relative path. absUrl() resolves it against the document's base URI:

Element link = doc.selectFirst("a.next-page");
link.attr("href");        // "/products?page=2"
link.absUrl("href");      // "https://example.com/products?page=2"

This works automatically when the document came from Jsoup.connect(). When parsing a string, pass the base URI yourself: Jsoup.parse(html, "https://example.com/") — otherwise absUrl returns an empty string.

Scraping tables

Element table = doc.selectFirst("table#prices");
List<String> headers = table.select("th").eachText();

List<Map<String, String>> rows = new ArrayList<>();
for (Element tr : table.select("tbody tr")) {
    Elements cells = tr.select("td");
    if (cells.size() < headers.size()) continue;      // skip spacer/summary rows

    Map<String, String> row = new LinkedHashMap<>();
    for (int i = 0; i < headers.size(); i++) {
        row.put(headers.get(i), cells.get(i).text());
    }
    rows.add(row);
}

eachText(), eachAttr("href"), and text() on an Elements collection are the compact extractors — text() on multiple elements joins their text with spaces.

Malformed HTML and XML parsing

jsoup parses like a browser: unclosed tags get closed, misnested elements get reparented, and you always receive a complete html > head + body tree — that's why it succeeds on tag soup that strict XML parsers reject. If you need to see the input's structure as written rather than normalized (or you're parsing actual XML, where <item> shouldn't be second-guessed), switch parsers:

import org.jsoup.parser.Parser;

Document xml = Jsoup.parse(xmlString, "", Parser.xmlParser());   // no HTML normalization
Elements items = xml.select("channel > item > title");           // selectors work the same

The XML parser doesn't validate, resolve entities against a DTD, or support namespaces beyond treating ns|tag selectors as literal names — for schema validation or XPath over XML, use a dedicated XML stack; jsoup's XML mode is for scraping feeds and sitemaps.

Character encoding

From Jsoup.connect(), encoding is handled for you (HTTP headers, then <meta charset>). Encoding bugs almost always come from file and stream parsing:

// Let jsoup detect from BOM / meta tags — pass null charset
Document doc = Jsoup.parse(new FileInputStream("page.html"), null, "https://example.com/");

// Or force one when you know the source lies
Document win = Jsoup.parse(new FileInputStream("legacy.html"), "windows-1251", baseUri);

If you see mojibake (привет, ’), the bytes were decoded with the wrong charset before jsoup — e.g., you read the file into a String with the platform default charset and then called Jsoup.parse(string). Parse from the stream and let jsoup detect, and set output explicitly if you re-serialize: doc.outputSettings().charset("UTF-8").

Timeouts, retries, and the 1 MB trap

Three connection settings prevent most production surprises:

Document doc = Jsoup.connect(url)
        .timeout(15_000)           // connect + read timeout, ms (0 = infinite — never do that)
        .maxBodySize(0)            // DEFAULT IS 1MB (older versions 2MB) — 0 removes the cap
        .ignoreContentType(true)   // when an endpoint returns JSON/text you still want to read
        .get();

maxBodySize deserves emphasis: jsoup silently truncates responses beyond the limit, which surfaces as selectors that mysteriously match nothing in the bottom half of large pages. jsoup has no built-in retry; wrap the fetch:

Document fetchWithRetry(String url, int attempts) throws IOException, InterruptedException {
    IOException last = null;
    for (int i = 0; i < attempts; i++) {
        try {
            return Jsoup.connect(url).userAgent(UA).timeout(15_000).get();
        } catch (SocketTimeoutException | HttpStatusException e) {
            last = e instanceof IOException ? (IOException) e : last;
            Thread.sleep((long) Math.pow(2, i) * 1000);       // 1s, 2s, 4s...
        }
    }
    throw last;
}

Retry 5xx and timeouts; don't blindly retry 4xx — a 403 on retry is the same 403, and a tight retry loop on a 429 makes the rate limiting worse.

Caching responses between runs

Java hands you nothing here by default. Jsoup.connect() rides on HttpURLConnection, whose ResponseCache hook has no implementation shipped in the JDK, and java.net.http.HttpClient (JDK 11+) has no HTTP cache at all. Every run re-downloads every page unless you cache it yourself — which during development means hammering a site you're still writing selectors against.

Inside one process, a TTL map covers it:

record Cached(String html, Instant fetchedAt) {}

private final Map<String, Cached> cache = new ConcurrentHashMap<>();
private static final Duration TTL = Duration.ofMinutes(30);

String fetch(String url) throws IOException {
    Cached hit = cache.get(url);
    if (hit != null && Duration.between(hit.fetchedAt(), Instant.now()).compareTo(TTL) < 0) {
        return hit.html();
    }
    String html = Jsoup.connect(url).userAgent(UA).timeout(15_000).execute().body();
    cache.put(url, new Cached(html, Instant.now()));
    return html;
}

A plain ConcurrentHashMap grows until the heap does, so swap it for Caffeine as soon as the URL set is unbounded — Caffeine.newBuilder().maximumSize(10_000).expireAfterWrite(Duration.ofMinutes(30)).build() gives you eviction and size limits for one line. To survive a restart, write the HTML to disk or Redis keyed by URL instead.

Conditional requests are the cheaper half of caching: keep the ETag or Last-Modified from each response and let the server tell you nothing changed.

Connection.Response res = Jsoup.connect(url)
        .header("If-None-Match", storedEtag)     // or "If-Modified-Since", storedDate
        .ignoreHttpErrors(true)                  // else jsoup throws on the 304
        .execute();

if (res.statusCode() == 304) return cachedHtml;  // unchanged — nothing downloaded

Document doc = res.parse();
String etag = res.header("ETag");

The ignoreHttpErrors(true) is required: jsoup treats any non-2xx, non-redirect status as an error, so a 304 raises HttpStatusException instead of returning a response. A 304 costs a round trip and a few hundred bytes rather than a full page — the politest thing a recurring scraper can do to a server it revisits.

Sessions, cookies, and logging in

Connection.Response exposes cookies; newSession() (jsoup 1.14+) carries them across requests automatically. A typical form login with a CSRF token:

Connection session = Jsoup.newSession()
        .userAgent(UA)
        .timeout(15_000);

// 1. Load the login page and grab the CSRF token
Document loginPage = session.newRequest("https://example.com/login").get();
String csrf = loginPage.selectFirst("input[name=_token]").attr("value");

// 2. Submit credentials — cookies from step 1 are sent automatically
session.newRequest("https://example.com/login")
        .data("email", email, "password", password, "_token", csrf)
        .post();

// 3. Authenticated pages now work within the session
Document dashboard = session.newRequest("https://example.com/account").get();

This handles cookie-based auth. It won't get past JavaScript-driven logins (SPA auth flows, OAuth popups, CAPTCHA gates) — those need a real browser in the loop.

Using jsoup with a proxy

Route requests through a proxy per connection, or via a Proxy object you reuse:

// Simplest: host + port on the connection
Document doc = Jsoup.connect("https://example.com/")
        .proxy("proxy.example.com", 8080)
        .userAgent(UA)
        .get();

// Or a java.net.Proxy instance (also how you do SOCKS)
Proxy socks = new Proxy(Proxy.Type.SOCKS, new InetSocketAddress("127.0.0.1", 9050));
Document viaSocks = Jsoup.connect("https://example.com/").proxy(socks).get();

Authenticated proxies are the sharp edge: jsoup rides on HttpURLConnection, so credentials go through the JVM-wide Authenticator, and modern JDKs disable Basic auth for HTTPS tunnels by default — clear jdk.http.auth.tunneling.disabledSchemes or requests fail with 407 no matter what:

System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");   // allow Basic for CONNECT

Authenticator.setDefault(new Authenticator() {
    @Override protected PasswordAuthentication getPasswordAuthentication() {
        if (getRequestorType() == RequestorType.PROXY) {
            return new PasswordAuthentication("user", "pass".toCharArray());
        }
        return null;
    }
});

Alternatively, set JVM-wide proxy system properties (-Dhttps.proxyHost=... -Dhttps.proxyPort=...) — coarser, but no code changes. For scraping at volume you'll want rotating proxies rather than one static exit; our proxy provider comparison covers the options (or let a scraping API manage the pool entirely — see below).

Country-specific and geo-restricted pages

Many sites serve different prices, catalogues, or article text depending on where the request comes from, and some return a block page outside their market. The exit IP is what decides that, so the answer is the proxy mechanism above with an exit in the country you actually want, plus an Accept-Language header that matches it:

Document doc = Jsoup.connect("https://example.de/produkte")
        .proxy("de-proxy.example.com", 8080)          // German exit IP
        .header("Accept-Language", "de-DE,de;q=0.9")
        .userAgent(UA)
        .get();

With a scraping API the country is a request parameter rather than a pool you assemble yourself — &country=de on our endpoints, chosen from us, gb, de, fr, it, es, ca, jp, kr, in, hk, tr, ru. Either way, this only changes where the request appears to originate. It won't get you past a login or a paywall, and content that's geo-restricted for licensing reasons is usually restricted for reasons worth respecting — is web scraping legal covers where the lines sit.

Cleaning and sanitizing untrusted HTML

jsoup's third role: a whitelist-based sanitizer for user-supplied HTML. This is the correct tool when you accept rich text and must prevent XSS:

import org.jsoup.safety.Safelist;

String safe = Jsoup.clean(userHtml, Safelist.basic());
// basic(): formatting tags + links (rel=nofollow enforced); no images, no divs

String richer = Jsoup.clean(userHtml, Safelist.relaxed()
        .addAttributes("a", "target")
        .preserveRelativeLinks(false));

Progressively stricter presets: relaxed() → basic() → simpleText() → none() (text only). The cleaner parses the input and rebuilds output from the whitelist, so obfuscated payloads (<script> in attributes, javascript: URLs, mangled tags) don't survive. Always sanitize at render/storage boundaries rather than trusting upstream filtering.

jsoup on Android

jsoup works on Android (API 21+) as a plain dependency, with one hard rule: never fetch on the main thread — NetworkOnMainThreadException is immediate. Use coroutines (Kotlin) or an executor (Java):

// Kotlin coroutine
lifecycleScope.launch {
    val doc = withContext(Dispatchers.IO) {
        Jsoup.connect("https://example.com/").userAgent(UA).get()
    }
    titleView.text = doc.title()
}

Add the INTERNET permission in the manifest. On constrained devices, keep maxBodySize bounded and parse only what you need — a full DOM of a heavy page is real memory on a phone.

JavaScript-rendered pages: jsoup's hard limit

jsoup parses the HTML the server sends — it has no JavaScript engine, so React/Vue/Angular apps and lazy-loaded content give it an empty shell. Your options, in order of increasing weight:

1. Find the underlying API: DevTools → Network → XHR; scraping the JSON endpoint beats HTML parsing every time it's available
2. Java-native rendering: HtmlUnit executes much real-world JavaScript in-process; Selenium/Playwright drive a real Chrome for full fidelity
3. A rendering API: fetch through a service that runs a real browser and returns final HTML — then parse with jsoup exactly as before

Selenium for the fetch, jsoup for the parse

When you do reach for Selenium, keep the extraction in jsoup — driver.getPageSource() hands you the rendered DOM as HTML and every selector you've already written keeps working:

WebDriver driver = new ChromeDriver(new ChromeOptions().addArguments("--headless=new"));
try {
    driver.get("https://example.com/spa-products");

    new WebDriverWait(driver, Duration.ofSeconds(15))
            .until(ExpectedConditions.presenceOfElementLocated(By.cssSelector(".product-card")));

    Document doc = Jsoup.parse(driver.getPageSource(), driver.getCurrentUrl());
    // tables, absUrl(), everything above — unchanged
} finally {
    driver.quit();
}

Two things carry most of the reliability: wait on the element you need with WebDriverWait and ExpectedConditions rather than sleeping a fixed number of seconds, and let Selenium Manager (built in since Selenium 4.6) fetch the driver binary — the WebDriverManager dependency and manual chromedriver paths that older Java tutorials open with are no longer needed. Passing driver.getCurrentUrl() as the base URI is what keeps absUrl() working after redirects.

The wider Java scraping ecosystem

jsoup is the parsing layer, and a complete scraper usually pairs it with one of these:

• HTTP clients — jsoup's own fetcher is fine for straightforward jobs. When you need connection pooling, HTTP/2, or async, fetch with java.net.http.HttpClient (JDK 11+), OkHttp, or Apache HttpClient, then pass the body to Jsoup.parse().
• HtmlUnit — a browser implemented in pure Java, which runs a lot of real-world JavaScript without a Chrome process.
• Selenium and Playwright for Java — real browsers, full fidelity, heaviest footprint. The pattern above applies to both.
• WebMagic — the closest Java has to Scrapy: a crawler framework covering scheduling, a page-processor pipeline, and persistence, with jsoup-style selectors for extraction. Check its activity against your timeline before adopting, though — 1.0.3 (February 2025) is still the newest release and commits have been sparse since. crawler4j, the other name in this slot, last released in 2018 and has had no commits since 2021.

Frameworks earn their keep on crawls with real queueing and politeness requirements. For a fixed list of URLs, jsoup plus an ExecutorService is less machinery to maintain.

jsoup for web scraping at scale

The productive pattern is splitting fetch from parse: something browser-grade gets the HTML, jsoup does what it's best at. WebScraping.AI renders pages in a real browser with rotating proxies behind a single GET — from Java, that's the standard HttpClient plus your existing jsoup code:

HttpClient http = HttpClient.newHttpClient();
String api = "https://api.webscraping.ai/html?api_key=" + apiKey
        + "&url=" + URLEncoder.encode("https://example.com/spa-products", StandardCharsets.UTF_8)
        + "&js=true";                                   // real browser rendering, proxies included

String html = http.send(HttpRequest.newBuilder(URI.create(api)).build(),
        HttpResponse.BodyHandlers.ofString()).body();

Document doc = Jsoup.parse(html, "https://example.com/");   // selectors work unchanged

No proxy pools, no browser processes, no Authenticator gymnastics — and the /ai/fields endpoint can skip selectors altogether, returning structured JSON from plain-English field descriptions. That's the difference between a scraper you babysit and one that survives a redesign, which matters most for continuous jobs like job listing aggregation. The API reference documents the rest of the parameters (proxy, country, wait_for, device).

Frequently asked questions

Is jsoup free for commercial use? Yes — MIT license. Use it in commercial and closed-source products freely; just keep the license notice.

Does jsoup support XPath? No — selection is CSS selectors only (plus the Elements traversal API). Nearly every scraping XPath has a direct CSS equivalent (//div[@class='x'] → div.x); for the rare axis CSS can't express (like selecting a parent), traverse with .parent() / .closest() after selecting.

Which Java version does jsoup require? Java 8 or newer for all current releases, tested through the latest LTS versions, plus Android API 21+. It has zero runtime dependencies, so it won't drag anything else into your build.

Can jsoup handle JavaScript-rendered websites? No — it never executes JavaScript. Check whether the data hides in a JSON <script> block or an XHR endpoint first; otherwise render with HtmlUnit/Selenium or a rendering API and hand the resulting HTML to jsoup.

Why does jsoup only return part of my page? Almost always maxBodySize: jsoup caps downloads at 1 MB by default and truncates silently. Set .maxBodySize(0) on the connection. The other culprit is content loaded by JavaScript after the initial HTML — which was never in the response to begin with.

jsoup vs Selenium — which one for scraping? Different layers: jsoup is a parser with a simple fetcher; Selenium is browser automation. Static pages → jsoup alone (orders of magnitude faster, no browser). JavaScript apps and login flows → Selenium (or a rendering API) to obtain the HTML, then jsoup to parse it. Combining both is the normal architecture, not a workaround.