jsoup Java Guide: HTML Parsing, CSS Selectors, and Scraping

jsoup is the standard Java HTML parser: it turns real-world, messy HTML into a clean DOM you query with CSS selectors, and doubles as a lightweight HTTP client and an HTML sanitizer. This guide covers the full workflow — Maven/Gradle setup and Java compatibility, fetching and parsing, selector patterns, tables and attributes, character encoding, proxies, login sessions, cleaning untrusted HTML, Android usage, and where jsoup stops (JavaScript) and what to do about it.

Key Takeaways

• Add org.jsoup:jsoup from Maven Central; current releases need Java 8+ and run on Android API 21+
• Jsoup.connect(url).get() fetches and parses in one call — but set userAgent() and timeout() on every connection; the defaults get you blocked and stalled
• Selection is CSS selectors (select("div.product > h2 a[href]")), not XPath — first() returns null, select() returns an empty list
• jsoup silently truncates bodies at 1 MB by default (maxBodySize(0) to disable) — the classic "why is my page cut off" bug
• Use absUrl("href") instead of attr("href") to resolve relative links against the page URL
• jsoup never executes JavaScript — for rendered pages, pair it with a headless browser or a rendering API and keep jsoup for the parsing

Adding jsoup to your project (Maven, Gradle, Android)

Maven:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.21.2</version>
</dependency>

Gradle (Kotlin DSL works the same with parentheses):

implementation 'org.jsoup:jsoup:1.21.2'

Check Maven Central for the latest version — jsoup releases steadily and updates regularly matter because it parses untrusted input. For a quick script without a build tool, download the jar and add it to the classpath: java -cp jsoup-1.21.2.jar:. Scrape.

Which Java versions are compatible?

Modern jsoup (1.16 and later) requires Java 8 or higher and is tested through the current LTS releases (11, 17, 21, 25) — it's a zero-dependency library, so there's no transitive matrix to fight. The jar is modular (module org.jsoup) for JPMS projects, and works on Android API 21+ with no extra configuration. If you're trapped on Java 7 or ancient Android, you'd have to pin a years-old 1.13.x release — better to treat that as a platform-upgrade signal, since old parsers accumulate security fixes they'll never receive.

Fetching and parsing a page

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

Document doc = Jsoup.connect("https://example.com/products")
        .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/126.0 Safari/537.36")
        .referrer("https://www.google.com/")
        .timeout(10_000)                       // milliseconds; default is 30s
        .get();

System.out.println(doc.title());

Always set a real userAgent — jsoup's default identifies itself as Java, which many sites block outright. get() throws HttpStatusException on non-2xx responses (call .ignoreHttpErrors(true) to inspect error pages instead) and SocketTimeoutException on timeout.

Parsing HTML you already have — from a string or file — skips the network entirely:

Document doc = Jsoup.parse(html);                                  // String
Document fileDoc = Jsoup.parse(new File("page.html"), "UTF-8");
Document fragment = Jsoup.parseBodyFragment("<p>Just a snippet</p>");

Selecting elements with CSS selectors

jsoup's query language is CSS selectors — the same syntax you'd use in a stylesheet or querySelectorAll (our CSS selectors FAQ is a handy reference):

import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

Elements products = doc.select("div.product-card");               // all matches (never null)
Element first = doc.selectFirst("h1");                            // first match or NULL

for (Element card : products) {
    String name  = card.selectFirst(".name").text();
    String price = card.selectFirst(".price").text();
    String link  = card.selectFirst("a").absUrl("href");          // absolute URL — see below
}

The selector patterns that cover most scraping work:

select() returns an empty Elements collection when nothing matches, but selectFirst() (and first()) return null — guard them or use Optional.ofNullable. There's no XPath support; if a snippet you're porting uses XPath, translate it to a CSS selector (most translate directly) or see our XPath cheat sheet for the mapping patterns.

Extracting attributes, links, and absolute URLs

attr("href") returns the attribute as written — usually a relative path. absUrl() resolves it against the document's base URI:

Element link = doc.selectFirst("a.next-page");
link.attr("href");        // "/products?page=2"
link.absUrl("href");      // "https://example.com/products?page=2"

This works automatically when the document came from Jsoup.connect(). When parsing a string, pass the base URI yourself: Jsoup.parse(html, "https://example.com/") — otherwise absUrl returns an empty string.

Scraping tables

Element table = doc.selectFirst("table#prices");
List<String> headers = table.select("th").eachText();

List<Map<String, String>> rows = new ArrayList<>();
for (Element tr : table.select("tbody tr")) {
    Elements cells = tr.select("td");
    if (cells.size() < headers.size()) continue;      // skip spacer/summary rows

    Map<String, String> row = new LinkedHashMap<>();
    for (int i = 0; i < headers.size(); i++) {
        row.put(headers.get(i), cells.get(i).text());
    }
    rows.add(row);
}

eachText(), eachAttr("href"), and text() on an Elements collection are the compact extractors — text() on multiple elements joins their text with spaces.

Malformed HTML and XML parsing

jsoup parses like a browser: unclosed tags get closed, misnested elements get reparented, and you always receive a complete html > head + body tree — that's why it succeeds on tag soup that strict XML parsers reject. If you need to see the input's structure as written rather than normalized (or you're parsing actual XML, where <item> shouldn't be second-guessed), switch parsers:

import org.jsoup.parser.Parser;

Document xml = Jsoup.parse(xmlString, "", Parser.xmlParser());   // no HTML normalization
Elements items = xml.select("channel > item > title");           // selectors work the same

The XML parser doesn't validate, resolve entities against a DTD, or support namespaces beyond treating ns|tag selectors as literal names — for schema validation or XPath over XML, use a dedicated XML stack; jsoup's XML mode is for scraping feeds and sitemaps.

Character encoding

From Jsoup.connect(), encoding is handled for you (HTTP headers, then <meta charset>). Encoding bugs almost always come from file and stream parsing:

// Let jsoup detect from BOM / meta tags — pass null charset
Document doc = Jsoup.parse(new FileInputStream("page.html"), null, "https://example.com/");

// Or force one when you know the source lies
Document win = Jsoup.parse(new FileInputStream("legacy.html"), "windows-1251", baseUri);

If you see mojibake (привет, ’), the bytes were decoded with the wrong charset before jsoup — e.g., you read the file into a String with the platform default charset and then called Jsoup.parse(string). Parse from the stream and let jsoup detect, and set output explicitly if you re-serialize: doc.outputSettings().charset("UTF-8").

Timeouts, retries, and the 1 MB trap

Three connection settings prevent most production surprises:

Document doc = Jsoup.connect(url)
        .timeout(15_000)           // connect + read timeout, ms (0 = infinite — never do that)
        .maxBodySize(0)            // DEFAULT IS 1MB (older versions 2MB) — 0 removes the cap
        .ignoreContentType(true)   // when an endpoint returns JSON/text you still want to read
        .get();

maxBodySize deserves emphasis: jsoup silently truncates responses beyond the limit, which surfaces as selectors that mysteriously match nothing in the bottom half of large pages. jsoup has no built-in retry; wrap the fetch:

Document fetchWithRetry(String url, int attempts) throws IOException, InterruptedException {
    IOException last = null;
    for (int i = 0; i < attempts; i++) {
        try {
            return Jsoup.connect(url).userAgent(UA).timeout(15_000).get();
        } catch (SocketTimeoutException | HttpStatusException e) {
            last = e instanceof IOException ? (IOException) e : last;
            Thread.sleep((long) Math.pow(2, i) * 1000);       // 1s, 2s, 4s...
        }
    }
    throw last;
}

Retry 5xx and timeouts; don't blindly retry 4xx — a 403 on retry is the same 403, and a tight retry loop on a 429 makes the rate limiting worse.

Sessions, cookies, and logging in

Connection.Response exposes cookies; newSession() (jsoup 1.14+) carries them across requests automatically. A typical form login with a CSRF token:

Connection session = Jsoup.newSession()
        .userAgent(UA)
        .timeout(15_000);

// 1. Load the login page and grab the CSRF token
Document loginPage = session.newRequest("https://example.com/login").get();
String csrf = loginPage.selectFirst("input[name=_token]").attr("value");

// 2. Submit credentials — cookies from step 1 are sent automatically
session.newRequest("https://example.com/login")
        .data("email", email, "password", password, "_token", csrf)
        .post();

// 3. Authenticated pages now work within the session
Document dashboard = session.newRequest("https://example.com/account").get();

This handles cookie-based auth. It won't get past JavaScript-driven logins (SPA auth flows, OAuth popups, CAPTCHA gates) — those need a real browser in the loop.

Using jsoup with a proxy

Route requests through a proxy per connection, or via a Proxy object you reuse:

// Simplest: host + port on the connection
Document doc = Jsoup.connect("https://example.com/")
        .proxy("proxy.example.com", 8080)
        .userAgent(UA)
        .get();

// Or a java.net.Proxy instance (also how you do SOCKS)
Proxy socks = new Proxy(Proxy.Type.SOCKS, new InetSocketAddress("127.0.0.1", 9050));
Document viaSocks = Jsoup.connect("https://example.com/").proxy(socks).get();

Authenticated proxies are the sharp edge: jsoup rides on HttpURLConnection, so credentials go through the JVM-wide Authenticator, and modern JDKs disable Basic auth for HTTPS tunnels by default — clear jdk.http.auth.tunneling.disabledSchemes or requests fail with 407 no matter what:

System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");   // allow Basic for CONNECT

Authenticator.setDefault(new Authenticator() {
    @Override protected PasswordAuthentication getPasswordAuthentication() {
        if (getRequestorType() == RequestorType.PROXY) {
            return new PasswordAuthentication("user", "pass".toCharArray());
        }
        return null;
    }
});

Alternatively, set JVM-wide proxy system properties (-Dhttps.proxyHost=... -Dhttps.proxyPort=...) — coarser, but no code changes. For scraping at volume you'll want rotating proxies rather than one static exit; our proxy provider comparison covers the options (or let a scraping API manage the pool entirely — see below).

Cleaning and sanitizing untrusted HTML

jsoup's third role: a whitelist-based sanitizer for user-supplied HTML. This is the correct tool when you accept rich text and must prevent XSS:

import org.jsoup.safety.Safelist;

String safe = Jsoup.clean(userHtml, Safelist.basic());
// basic(): formatting tags + links (rel=nofollow enforced); no images, no divs

String richer = Jsoup.clean(userHtml, Safelist.relaxed()
        .addAttributes("a", "target")
        .preserveRelativeLinks(false));

Progressively stricter presets: relaxed() → basic() → simpleText() → none() (text only). The cleaner parses the input and rebuilds output from the whitelist, so obfuscated payloads (<script> in attributes, javascript: URLs, mangled tags) don't survive. Always sanitize at render/storage boundaries rather than trusting upstream filtering.

jsoup on Android

jsoup works on Android (API 21+) as a plain dependency, with one hard rule: never fetch on the main thread — NetworkOnMainThreadException is immediate. Use coroutines (Kotlin) or an executor (Java):

// Kotlin coroutine
lifecycleScope.launch {
    val doc = withContext(Dispatchers.IO) {
        Jsoup.connect("https://example.com/").userAgent(UA).get()
    }
    titleView.text = doc.title()
}

Add the INTERNET permission in the manifest. On constrained devices, keep maxBodySize bounded and parse only what you need — a full DOM of a heavy page is real memory on a phone.

JavaScript-rendered pages: jsoup's hard limit

jsoup parses the HTML the server sends — it has no JavaScript engine, so React/Vue/Angular apps and lazy-loaded content give it an empty shell. Your options, in order of increasing weight:

1. Find the underlying API: DevTools → Network → XHR; scraping the JSON endpoint beats HTML parsing every time it's available
2. Java-native rendering: HtmlUnit executes much real-world JavaScript in-process; Selenium/Playwright drive a real Chrome for full fidelity
3. A rendering API: fetch through a service that runs a real browser and returns final HTML — then parse with jsoup exactly as before

jsoup for web scraping at scale

The productive pattern is splitting fetch from parse: something browser-grade gets the HTML, jsoup does what it's best at. WebScraping.AI renders pages in a real browser with rotating proxies behind a single GET — from Java, that's the standard HttpClient plus your existing jsoup code:

HttpClient http = HttpClient.newHttpClient();
String api = "https://api.webscraping.ai/html?api_key=" + apiKey
        + "&url=" + URLEncoder.encode("https://example.com/spa-products", StandardCharsets.UTF_8)
        + "&js=true";                                   // real browser rendering, proxies included

String html = http.send(HttpRequest.newBuilder(URI.create(api)).build(),
        HttpResponse.BodyHandlers.ofString()).body();

Document doc = Jsoup.parse(html, "https://example.com/");   // selectors work unchanged

No proxy pools, no browser processes, no Authenticator gymnastics — and the /ai/fields endpoint can skip selectors altogether, returning structured JSON from plain-English field descriptions.

Frequently asked questions

Is jsoup free for commercial use? Yes — MIT license. Use it in commercial and closed-source products freely; just keep the license notice.

Does jsoup support XPath? No — selection is CSS selectors only (plus the Elements traversal API). Nearly every scraping XPath has a direct CSS equivalent (//div[@class='x'] → div.x); for the rare axis CSS can't express (like selecting a parent), traverse with .parent() / .closest() after selecting.

Which Java version does jsoup require? Java 8 or newer for all current releases, tested through the latest LTS versions, plus Android API 21+. It has zero runtime dependencies, so it won't drag anything else into your build.

Can jsoup handle JavaScript-rendered websites? No — it never executes JavaScript. Check whether the data hides in a JSON <script> block or an XHR endpoint first; otherwise render with HtmlUnit/Selenium or a rendering API and hand the resulting HTML to jsoup.

Why does jsoup only return part of my page? Almost always maxBodySize: jsoup caps downloads at 1 MB by default and truncates silently. Set .maxBodySize(0) on the connection. The other culprit is content loaded by JavaScript after the initial HTML — which was never in the response to begin with.

jsoup vs Selenium — which one for scraping? Different layers: jsoup is a parser with a simple fetcher; Selenium is browser automation. Static pages → jsoup alone (orders of magnitude faster, no browser). JavaScript apps and login flows → Selenium (or a rendering API) to obtain the HTML, then jsoup to parse it. Combining both is the normal architecture, not a workaround.