---
title: "jsoup Java Web Scraping: HTML Parsing and CSS Selectors"
description: "Java web scraping with jsoup: Maven setup, CSS selectors, tables, encoding, proxies and country targeting, caching, login sessions, and JavaScript pages."
url: https://webscraping.ai/blog/jsoup-java-guide
markdown_index: https://webscraping.ai/llms.txt
---
# jsoup Java Web Scraping: HTML Parsing and CSS Selectors

[jsoup](https://jsoup.org/) is the standard Java HTML parser: it turns real-world, messy HTML into a clean DOM you query with CSS selectors, and doubles as a lightweight HTTP client and an HTML sanitizer. This guide covers the full workflow — Maven/Gradle setup and Java compatibility, fetching and parsing, selector patterns, tables and attributes, character encoding, caching, proxies and country-specific pages, login sessions, cleaning untrusted HTML, Android usage, and where jsoup stops (JavaScript) and what to do about it.

## Key Takeaways

- Add `org.jsoup:jsoup` from Maven Central; current releases need **Java 8+** and run on Android API 21+
- `Jsoup.connect(url).get()` fetches and parses in one call — but set `userAgent()` and `timeout()` on every connection; the defaults get you blocked and stalled
- Selection is CSS selectors (`select("div.product > h2 a[href]")`), not XPath — `first()` returns null, `select()` returns an empty list
- jsoup silently truncates bodies at **1 MB by default** (`maxBodySize(0)` to disable) — the classic "why is my page cut off" bug
- Use `absUrl("href")` instead of `attr("href")` to resolve relative links against the page URL
- jsoup never executes JavaScript — for rendered pages, pair it with a headless browser or a rendering API and keep jsoup for the parsing

## Adding jsoup to your project (Maven, Gradle, Android)

Maven:

```xml
<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.21.2</version>
</dependency>
```

Gradle (Kotlin DSL works the same with parentheses):

```groovy
implementation 'org.jsoup:jsoup:1.21.2'
```

Check [Maven Central](https://central.sonatype.com/artifact/org.jsoup/jsoup) for the latest version — jsoup releases steadily and updates regularly matter because it parses untrusted input. For a quick script without a build tool, download the jar and add it to the classpath: `java -cp jsoup-1.21.2.jar:. Scrape`.

### Which Java versions are compatible?

Modern jsoup (1.16 and later) requires **Java 8 or higher** and is tested through the current LTS releases (11, 17, 21, 25) — it's a zero-dependency library, so there's no transitive matrix to fight. The jar is modular (`module org.jsoup`) for JPMS projects, and works on **Android API 21+** with no extra configuration. If you're trapped on Java 7 or ancient Android, you'd have to pin a years-old 1.13.x release — better to treat that as a platform-upgrade signal, since old parsers accumulate security fixes they'll never receive.

## Fetching and parsing a page

```java
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

Document doc = Jsoup.connect("https://example.com/products")
        .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/141.0.0.0 Safari/537.36")
        .referrer("https://www.google.com/")
        .timeout(10_000) // milliseconds; default is 30s
        .get();

System.out.println(doc.title());
```

Always set a real `userAgent` — jsoup's default identifies itself as Java, which many sites block outright. `get()` throws `HttpStatusException` on non-2xx responses (call `.ignoreHttpErrors(true)` to inspect error pages instead) and `SocketTimeoutException` on timeout.

Parsing HTML you already have — from a string or file — skips the network entirely:

```java
Document doc = Jsoup.parse(html); // String
Document fileDoc = Jsoup.parse(new File("page.html"), "UTF-8");
Document fragment = Jsoup.parseBodyFragment("<p>Just a snippet</p>");
```

## Selecting elements with CSS selectors

jsoup's query language is CSS selectors — the same syntax you'd use in a stylesheet or `querySelectorAll` (our [CSS selectors cheat sheet](https://webscraping.ai/blog/css-selectors-cheat-sheet) is a handy reference):

```java
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

Elements products = doc.select("div.product-card"); // all matches (never null)
Element first = doc.selectFirst("h1"); // first match or NULL

for (Element card : products) {
    String name = card.selectFirst(".name").text();
    String price = card.selectFirst(".price").text();
    String link = card.selectFirst("a").absUrl("href"); // absolute URL — see below
}
```

The selector patterns that cover most scraping work:

| Goal | Selector |
| --- | --- |
| By id / class | `#main`, `.price` |
| Attribute presence / value | `img[data-src]`, `a[rel=nofollow]` |
| Attribute prefix | `a[href^=https://]` |
| Direct child vs descendant | `ul.menu > li` vs `ul.menu li` |
| Element containing text | `h2:contains(Reviews)` |
| Structural | `tr:nth-child(2n)`, `li:first-child` |
| Element that has a descendant | `div:has(> img.hero)` |

`select()` returns an empty `Elements` collection when nothing matches, but `selectFirst()` (and `first()`) return **null** — guard them or use `Optional.ofNullable`. There's no XPath support; if a snippet you're porting uses XPath, translate it to a CSS selector (most translate directly) or see our [XPath cheat sheet](https://webscraping.ai/blog/xpath-cheat-sheet) for the mapping patterns.

## Extracting attributes, links, and absolute URLs

`attr("href")` returns the attribute _as written_ — usually a relative path. `absUrl()` resolves it against the document's base URI:

```java
Element link = doc.selectFirst("a.next-page");
link.attr("href"); // "/products?page=2"
link.absUrl("href"); // "https://example.com/products?page=2"
```

This works automatically when the document came from `Jsoup.connect()`. When parsing a string, pass the base URI yourself: `Jsoup.parse(html, "https://example.com/")` — otherwise `absUrl` returns an empty string.

## Scraping tables

```java
Element table = doc.selectFirst("table#prices");
List<String> headers = table.select("th").eachText();

List<Map<String, String>> rows = new ArrayList<>();
for (Element tr : table.select("tbody tr")) {
    Elements cells = tr.select("td");
    if (cells.size() < headers.size()) continue; // skip spacer/summary rows

    Map<String, String> row = new LinkedHashMap<>();
    for (int i = 0; i < headers.size(); i++) {
        row.put(headers.get(i), cells.get(i).text());
    }
    rows.add(row);
}
```

`eachText()`, `eachAttr("href")`, and `text()` on an `Elements` collection are the compact extractors — `text()` on multiple elements joins their text with spaces.

## Malformed HTML and XML parsing

jsoup parses like a browser: unclosed tags get closed, misnested elements get reparented, and you always receive a complete `html > head + body` tree — that's why it succeeds on tag soup that strict XML parsers reject. If you need to see the input's structure _as written_ rather than normalized (or you're parsing actual XML, where `<item>` shouldn't be second-guessed), switch parsers:

```java
import org.jsoup.parser.Parser;

Document xml = Jsoup.parse(xmlString, "", Parser.xmlParser()); // no HTML normalization
Elements items = xml.select("channel > item > title"); // selectors work the same
```

The XML parser doesn't validate, resolve entities against a DTD, or support namespaces beyond treating `ns|tag` selectors as literal names — for schema validation or XPath over XML, use a dedicated XML stack; jsoup's XML mode is for scraping feeds and sitemaps.

## Character encoding

From `Jsoup.connect()`, encoding is handled for you (HTTP headers, then `<meta charset>`). Encoding bugs almost always come from file and stream parsing:

```java
// Let jsoup detect from BOM / meta tags — pass null charset
Document doc = Jsoup.parse(new FileInputStream("page.html"), null, "https://example.com/");

// Or force one when you know the source lies
Document win = Jsoup.parse(new FileInputStream("legacy.html"), "windows-1251", baseUri);
```

If you see mojibake (`привет`, `’`), the bytes were decoded with the wrong charset **before** jsoup — e.g., you read the file into a `String` with the platform default charset and then called `Jsoup.parse(string)`. Parse from the stream and let jsoup detect, and set output explicitly if you re-serialize: `doc.outputSettings().charset("UTF-8")`.

## Timeouts, retries, and the 1 MB trap

Three connection settings prevent most production surprises:

```java
Document doc = Jsoup.connect(url)
        .timeout(15_000) // connect + read timeout, ms (0 = infinite — never do that)
        .maxBodySize(0) // DEFAULT IS 1MB (older versions 2MB) — 0 removes the cap
        .ignoreContentType(true) // when an endpoint returns JSON/text you still want to read
        .get();
```

`maxBodySize` deserves emphasis: jsoup **silently truncates** responses beyond the limit, which surfaces as selectors that mysteriously match nothing in the bottom half of large pages. jsoup has no built-in retry; wrap the fetch:

```java
Document fetchWithRetry(String url, int attempts) throws IOException, InterruptedException {
    IOException last = null;
    for (int i = 0; i < attempts; i++) {
        try {
            return Jsoup.connect(url).userAgent(UA).timeout(15_000).get();
        } catch (SocketTimeoutException | HttpStatusException e) {
            last = e instanceof IOException ? (IOException) e : last;
            Thread.sleep((long) Math.pow(2, i) * 1000); // 1s, 2s, 4s...
        }
    }
    throw last;
}
```

Retry 5xx and timeouts; don't blindly retry 4xx — a 403 on retry is the same 403, and a tight retry loop on a 429 makes the rate limiting worse.

## Caching responses between runs

Java hands you nothing here by default. `Jsoup.connect()` rides on `HttpURLConnection`, whose `ResponseCache` hook has no implementation shipped in the JDK, and `java.net.http.HttpClient` (JDK 11+) has no HTTP cache at all. Every run re-downloads every page unless you cache it yourself — which during development means hammering a site you're still writing selectors against.

Inside one process, a TTL map covers it:

```java
record Cached(String html, Instant fetchedAt) {}

private final Map<String, Cached> cache = new ConcurrentHashMap<>();
private static final Duration TTL = Duration.ofMinutes(30);

String fetch(String url) throws IOException {
    Cached hit = cache.get(url);
    if (hit != null && Duration.between(hit.fetchedAt(), Instant.now()).compareTo(TTL) < 0) {
        return hit.html();
    }
    String html = Jsoup.connect(url).userAgent(UA).timeout(15_000).execute().body();
    cache.put(url, new Cached(html, Instant.now()));
    return html;
}
```

A plain `ConcurrentHashMap` grows until the heap does, so swap it for [Caffeine](https://github.com/ben-manes/caffeine) as soon as the URL set is unbounded — `Caffeine.newBuilder().maximumSize(10_000).expireAfterWrite(Duration.ofMinutes(30)).build()` gives you eviction and size limits for one line. To survive a restart, write the HTML to disk or Redis keyed by URL instead.

Conditional requests are the cheaper half of caching: keep the `ETag` or `Last-Modified` from each response and let the server tell you nothing changed.

```java
Connection.Response res = Jsoup.connect(url)
        .header("If-None-Match", storedEtag) // or "If-Modified-Since", storedDate
        .ignoreHttpErrors(true) // else jsoup throws on the 304
        .execute();

if (res.statusCode() == 304) return cachedHtml; // unchanged — nothing downloaded

Document doc = res.parse();
String etag = res.header("ETag");
```

The `ignoreHttpErrors(true)` is required: jsoup treats any non-2xx, non-redirect status as an error, so a 304 raises `HttpStatusException` instead of returning a response. A 304 costs a round trip and a few hundred bytes rather than a full page — the politest thing a recurring scraper can do to a server it revisits.

## Sessions, cookies, and logging in

`Connection.Response` exposes cookies; `newSession()` (jsoup 1.14+) carries them across requests automatically. A typical form login with a CSRF token:

```java
Connection session = Jsoup.newSession()
        .userAgent(UA)
        .timeout(15_000);

// 1. Load the login page and grab the CSRF token
Document loginPage = session.newRequest("https://example.com/login").get();
String csrf = loginPage.selectFirst("input[name=_token]").attr("value");

// 2. Submit credentials — cookies from step 1 are sent automatically
session.newRequest("https://example.com/login")
        .data("email", email, "password", password, "_token", csrf)
        .post();

// 3. Authenticated pages now work within the session
Document dashboard = session.newRequest("https://example.com/account").get();
```

This handles cookie-based auth. It won't get past JavaScript-driven logins (SPA auth flows, OAuth popups, CAPTCHA gates) — those need a real browser in the loop.

## Using jsoup with a proxy

Route requests through a proxy per connection, or via a `Proxy` object you reuse:

```java
// Simplest: host + port on the connection
Document doc = Jsoup.connect("https://example.com/")
        .proxy("proxy.example.com", 8080)
        .userAgent(UA)
        .get();

// Or a java.net.Proxy instance (also how you do SOCKS)
Proxy socks = new Proxy(Proxy.Type.SOCKS, new InetSocketAddress("127.0.0.1", 9050));
Document viaSocks = Jsoup.connect("https://example.com/").proxy(socks).get();
```

Authenticated proxies are the sharp edge: jsoup rides on `HttpURLConnection`, so credentials go through the JVM-wide `Authenticator`, and modern JDKs **disable Basic auth for HTTPS tunnels by default** — clear `jdk.http.auth.tunneling.disabledSchemes` or requests fail with 407 no matter what:

```java
System.setProperty("jdk.http.auth.tunneling.disabledSchemes", ""); // allow Basic for CONNECT

Authenticator.setDefault(new Authenticator() {
    @Override protected PasswordAuthentication getPasswordAuthentication() {
        if (getRequestorType() == RequestorType.PROXY) {
            return new PasswordAuthentication("user", "pass".toCharArray());
        }
        return null;
    }
});
```

Alternatively, set JVM-wide proxy system properties (`-Dhttps.proxyHost=... -Dhttps.proxyPort=...`) — coarser, but no code changes. For scraping at volume you'll want rotating proxies rather than one static exit; our [proxy provider comparison](https://webscraping.ai/blog/best-proxy-providers-for-web-scraping) covers the options (or let a scraping API manage the pool entirely — see below).

### Country-specific and geo-restricted pages

Many sites serve different prices, catalogues, or article text depending on where the request comes from, and some return a block page outside their market. The exit IP is what decides that, so the answer is the proxy mechanism above with an exit in the country you actually want, plus an `Accept-Language` header that matches it:

```java
Document doc = Jsoup.connect("https://example.de/produkte")
        .proxy("de-proxy.example.com", 8080) // German exit IP
        .header("Accept-Language", "de-DE,de;q=0.9")
        .userAgent(UA)
        .get();
```

With a scraping API the country is a request parameter rather than a pool you assemble yourself — `&country=de` on our endpoints, chosen from `us`, `gb`, `de`, `fr`, `it`, `es`, `ca`, `jp`, `kr`, `in`, `hk`, `tr`, `ru`. Either way, this only changes where the request appears to originate. It won't get you past a login or a paywall, and content that's geo-restricted for licensing reasons is usually restricted for reasons worth respecting — [is web scraping legal](https://webscraping.ai/blog/is-web-scraping-legal) covers where the lines sit.

## Cleaning and sanitizing untrusted HTML

jsoup's third role: a whitelist-based sanitizer for user-supplied HTML. This is the correct tool when you accept rich text and must prevent XSS:

```java
import org.jsoup.safety.Safelist;

String safe = Jsoup.clean(userHtml, Safelist.basic());
// basic(): formatting tags + links (rel=nofollow enforced); no images, no divs

String richer = Jsoup.clean(userHtml, Safelist.relaxed()
        .addAttributes("a", "target")
        .preserveRelativeLinks(false));
```

Progressively stricter presets: `relaxed()` → `basic()` → `simpleText()` → `none()` (text only). The cleaner parses the input and rebuilds output from the whitelist, so obfuscated payloads (`<script>` in attributes, `javascript:` URLs, mangled tags) don't survive. Always sanitize at render/storage boundaries rather than trusting upstream filtering.

## jsoup on Android

jsoup works on Android (API 21+) as a plain dependency, with one hard rule: **never fetch on the main thread** — `NetworkOnMainThreadException` is immediate. Use coroutines (Kotlin) or an executor (Java):

```kotlin
// Kotlin coroutine
lifecycleScope.launch {
    val doc = withContext(Dispatchers.IO) {
        Jsoup.connect("https://example.com/").userAgent(UA).get()
    }
    titleView.text = doc.title()
}
```

Add the `INTERNET` permission in the manifest. On constrained devices, keep `maxBodySize` bounded and parse only what you need — a full DOM of a heavy page is real memory on a phone.

## JavaScript-rendered pages: jsoup's hard limit

jsoup parses the HTML the server sends — it has no JavaScript engine, so React/Vue/Angular apps and lazy-loaded content give it an empty shell. Your options, in order of increasing weight:

1. **Find the underlying API** : DevTools → Network → XHR; scraping the JSON endpoint beats HTML parsing every time it's available
2. **Java-native rendering** : [HtmlUnit](https://webscraping.ai/blog/htmlunit-web-scraping-java) executes much real-world JavaScript in-process; Selenium/Playwright drive a real Chrome for full fidelity
3. **A rendering API** : fetch through a service that runs a real browser and returns final HTML — then parse with jsoup exactly as before

| | jsoup | HtmlUnit | Selenium + Chrome |
| --- | --- | --- | --- |
| JavaScript | None | Good (Rhino-based) | Full (real browser) |
| Speed / footprint | Milliseconds, tiny | Moderate | Heavy (browser process) |
| Selector API | Excellent (CSS) | DOM + XPath | DOM, CSS, XPath |
| Best for | Static HTML, sanitizing, parsing fetched markup | Mid-complexity JS sites in pure Java | SPAs, logins, anti-bot flows |

### Selenium for the fetch, jsoup for the parse

When you do reach for Selenium, keep the extraction in jsoup — `driver.getPageSource()` hands you the rendered DOM as HTML and every selector you've already written keeps working:

```java
WebDriver driver = new ChromeDriver(new ChromeOptions().addArguments("--headless=new"));
try {
    driver.get("https://example.com/spa-products");

    new WebDriverWait(driver, Duration.ofSeconds(15))
            .until(ExpectedConditions.presenceOfElementLocated(By.cssSelector(".product-card")));

    Document doc = Jsoup.parse(driver.getPageSource(), driver.getCurrentUrl());
    // tables, absUrl(), everything above — unchanged
} finally {
    driver.quit();
}
```

Two things carry most of the reliability: wait on the element you need with `WebDriverWait` and `ExpectedConditions` rather than sleeping a fixed number of seconds, and let Selenium Manager (built in since Selenium 4.6) fetch the driver binary — the `WebDriverManager` dependency and manual `chromedriver` paths that older Java tutorials open with are no longer needed. Passing `driver.getCurrentUrl()` as the base URI is what keeps `absUrl()` working after redirects.

## The wider Java scraping ecosystem

jsoup is the parsing layer, and a complete scraper usually pairs it with one of these:

- **HTTP clients** — jsoup's own fetcher is fine for straightforward jobs. When you need connection pooling, HTTP/2, or async, fetch with `java.net.http.HttpClient` (JDK 11+), OkHttp, or Apache HttpClient, then pass the body to `Jsoup.parse()`.
- **[HtmlUnit](https://webscraping.ai/blog/htmlunit-web-scraping-java)** — a browser implemented in pure Java, which runs a lot of real-world JavaScript without a Chrome process.
- **Selenium and Playwright for Java** — real browsers, full fidelity, heaviest footprint. The pattern above applies to both.
- **[WebMagic](https://github.com/code4craft/webmagic)** — the closest Java has to Scrapy: a crawler framework covering scheduling, a page-processor pipeline, and persistence, with jsoup-style selectors for extraction. Check its activity against your timeline before adopting, though — 1.0.3 (February 2025) is still the newest release and commits have been sparse since. [crawler4j](https://github.com/yasserg/crawler4j), the other name in this slot, last released in 2018 and has had no commits since 2021.

Frameworks earn their keep on crawls with real queueing and politeness requirements. For a fixed list of URLs, jsoup plus an `ExecutorService` is less machinery to maintain.

## jsoup for web scraping at scale

The productive pattern is splitting fetch from parse: something browser-grade gets the HTML, jsoup does what it's best at. [WebScraping.AI](https://webscraping.ai/) renders pages in a real browser with rotating proxies behind a single GET — from Java, that's the standard `HttpClient` plus your existing jsoup code:

```java
HttpClient http = HttpClient.newHttpClient();
String api = "https://api.webscraping.ai/html?api_key=" + apiKey
        + "&url=" + URLEncoder.encode("https://example.com/spa-products", StandardCharsets.UTF_8)
        + "&js=true"; // real browser rendering, proxies included

String html = http.send(HttpRequest.newBuilder(URI.create(api)).build(),
        HttpResponse.BodyHandlers.ofString()).body();

Document doc = Jsoup.parse(html, "https://example.com/"); // selectors work unchanged
```

No proxy pools, no browser processes, no `Authenticator` gymnastics — and the [/ai/fields endpoint](https://webscraping.ai/ai-web-scraping) can skip selectors altogether, returning structured JSON from plain-English field descriptions. That's the difference between a scraper you babysit and one that survives a redesign, which matters most for continuous jobs like [job listing aggregation](https://webscraping.ai/use-cases/job-listing-aggregation). The [API reference](https://webscraping.ai/docs) documents the rest of the parameters (`proxy`, `country`, `wait_for`, `device`).

## Frequently asked questions

**Is jsoup free for commercial use?** Yes — MIT license. Use it in commercial and closed-source products freely; just keep the license notice.

**Does jsoup support XPath?** No — selection is CSS selectors only (plus the `Elements` traversal API). Nearly every scraping XPath has a direct CSS equivalent (`//div[@class='x']` → `div.x`); for the rare axis CSS can't express (like selecting a parent), traverse with `.parent()` / `.closest()` after selecting.

**Which Java version does jsoup require?** Java 8 or newer for all current releases, tested through the latest LTS versions, plus Android API 21+. It has zero runtime dependencies, so it won't drag anything else into your build.

**Can jsoup handle JavaScript-rendered websites?** No — it never executes JavaScript. Check whether the data hides in a JSON `<script>` block or an XHR endpoint first; otherwise render with HtmlUnit/Selenium or a rendering API and hand the resulting HTML to jsoup.

**Why does jsoup only return part of my page?** Almost always `maxBodySize`: jsoup caps downloads at 1 MB by default and truncates silently. Set `.maxBodySize(0)` on the connection. The other culprit is content loaded by JavaScript after the initial HTML — which was never in the response to begin with.

**jsoup vs Selenium — which one for scraping?** Different layers: jsoup is a parser with a simple fetcher; Selenium is browser automation. Static pages → jsoup alone (orders of magnitude faster, no browser). JavaScript apps and login flows → Selenium (or a rendering API) to obtain the HTML, then jsoup to parse it. Combining both is the normal architecture, not a workaround.