Rust Web Scraping: the scraper Crate, Crawlers, and Headless Chrome

The default Rust web scraping stack in 2026 is reqwest for fetching and scraper for parsing, with a headless browser bolted on only when a page needs JavaScript. This guide covers the parsing layer, the concurrency patterns that make Rust worth the extra compile time, the three headless-browser crates and how to choose between them, and the two scraper gotchas that cost every newcomer an afternoon.

Key Takeaways

• reqwest + scraper is the stack: scraper 0.27 wraps Servo's html5ever and selectors, so CSS selectors behave exactly as they do in Firefox
• scraper::Html is not Send — holding a parsed document across an .await will not compile. Extract owned Strings inside a scope, then await
• Compile Selector::parse once and reuse it. Parsing a selector inside a per-element loop is the most common Rust scraping performance bug
• chromiumoxide gets roughly 1.4M downloads per 90 days vs headless_chrome's 0.9M — it is the async/tokio-native choice; headless_chrome is the simpler blocking one
• Rust has no production-grade XPath-over-HTML crate. Convert your XPath expressions to CSS selectors before you start
• None of this defeats Cloudflare. Rust makes your pipeline fast and cheap; it does nothing about fingerprinting

What is the standard Rust web scraping stack?

Every layer has one obvious default and one or two credible alternates. Versions and 90-day download counts below are from crates.io as of July 2026 — download volume is the most honest signal of what the ecosystem actually uses:

Start with cargo add reqwest --features gzip and cargo add scraper tokio --features tokio/full. Everything below assumes that.

How do you parse HTML with the scraper crate?

scraper gives you Html::parse_document and Selector::parse, and that is essentially the whole API:

use scraper::{Html, Selector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::builder()
        .user_agent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36")
        .build()?;

    let body = client
        .get("https://quotes.toscrape.com/")
        .send()
        .await?
        .error_for_status()?
        .text()
        .await?;

    let doc = Html::parse_document(&body);
    let quote_sel = Selector::parse("div.quote").unwrap();
    let text_sel = Selector::parse("span.text").unwrap();
    let author_sel = Selector::parse("small.author").unwrap();

    for quote in doc.select(&quote_sel) {
        let text = quote.select(&text_sel).next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();
        let author = quote.select(&author_sel).next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();
        println!("{author}: {text}");
    }
    Ok(())
}

Three things worth internalising:

• select is scoped. Calling .select() on an ElementRef searches within that element, which is how you keep record boundaries straight instead of zipping three flat lists together and praying they line up.
• .text() yields descendant text nodes, not a string — .collect::<String>() joins them. Nested markup like <span>Price: <b>$9</b></span> needs the collect; .next() alone gives you "Price: ".
• Attributes come from .value().attr(...), which returns Option<&str>:

let link_sel = Selector::parse("a.product-link").unwrap();
let urls: Vec<String> = doc
    .select(&link_sel)
    .filter_map(|a| a.value().attr("href"))
    .map(|href| href.to_string())          // own it before doc is dropped
    .collect();

The two gotchas

Selector::parse returns a borrowed error. SelectorErrorKind<'_> borrows the selector string, so ? into Box<dyn Error> fails to compile with a lifetime error. Use .unwrap() for selectors you wrote as literals (a hardcoded selector failing to parse is a bug, not a runtime condition), or .map_err(|e| e.to_string())? when the selector comes from configuration.

Html is !Send. The parse tree uses reference-counted nodes internally, so a Selector or Html held across an .await makes the whole future non-Send and tokio::spawn rejects it — the notorious "future cannot be sent between threads safely" error. The fix is to finish parsing before you await anything:

async fn fetch_titles(client: &reqwest::Client, url: &str) -> Result<Vec<String>, reqwest::Error> {
    let body = client.get(url).send().await?.text().await?;

    // Scoped: doc and selector are dropped before this function's future yields again.
    let titles = {
        let doc = Html::parse_document(&body);
        let sel = Selector::parse("h2.title").unwrap();
        doc.select(&sel).map(|e| e.text().collect::<String>()).collect()
    };

    Ok(titles)   // owned Strings cross the await boundary fine
}

Also hoist selector compilation out of loops. Selector::parse runs a real CSS parser; calling it once per row on a 5,000-row page is measurable. For selectors used across functions, a static SEL: LazyLock<Selector> = LazyLock::new(|| Selector::parse("td.price").unwrap()); compiles it once for the process.

What about XPath?

Rust does not have a good answer here. skyscraper is the only XPath-for-HTML crate under active development and it sees under a thousand downloads a quarter; the libxml bindings work but drag in a C library. If you are porting a scraper from Python or Java, budget time to rewrite the expressions — most translate mechanically, and our XPath cheat sheet lists the CSS equivalent for the common patterns.

How do you crawl many pages concurrently in Rust?

This is the part that justifies choosing Rust. Bounded concurrency with buffer_unordered keeps thousands of in-flight requests in kilobytes of memory each:

use futures::stream::{self, StreamExt};

let client = reqwest::Client::builder()
    .timeout(std::time::Duration::from_secs(20))
    .build()?;

let results: Vec<Vec<String>> = stream::iter(urls)
    .map(|url| {
        let client = client.clone();          // clones share the connection pool
        async move { fetch_titles(&client, &url).await.unwrap_or_default() }
    })
    .buffer_unordered(16)                     // never more than 16 requests in flight
    .collect()
    .await;

Two rules that matter more than the crate choice:

1. Bound it. join_all over 50,000 URLs exhausts file descriptors and gets you blocked, in that order. buffer_unordered(n) where n is 8–32 per host is the polite range.
2. Always set a timeout. reqwest has no default request timeout — one hung server pins a task forever. Twenty seconds is a reasonable scraping default.

If you want retries, connection tuning, and error classification, those live in the HTTP layer rather than here; the reqwest guide covers reqwest-retry, pool sizing, and reading is_timeout() / is_connect().

For a crawler with frontier management, robots.txt handling, and depth limits already built, spider is the framework option. It is a much larger dependency and a much larger API surface — worth it for a real crawler, overkill for scraping a known list of URLs.

Which Rust headless browser crate should you use?

When the HTML you get back is an empty <div id="root">, you need a real browser. Rust has four viable options and they split cleanly by protocol and by async model:

The blocking headless_chrome API is genuinely the easiest to read:

use headless_chrome::{Browser, LaunchOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let browser = Browser::new(
        LaunchOptions::default_builder().headless(true).build()?
    )?;

    let tab = browser.new_tab()?;
    tab.navigate_to("https://example.com/spa")?;
    tab.wait_for_element(".loaded-content")?;   // waits, with a default timeout

    let html = tab.get_content()?;
    // hand off to scraper as usual
    Ok(())
}

chromiumoxide is the one to reach for in a tokio pipeline, because it doesn't force you to shove blocking calls into spawn_blocking. Its shape is a Browser plus a handler future you spawn once, then page.goto(...).await, page.find_element(...).await, page.content().await.

The caveats apply to all four equally, and they are the reason this section is short: each Chrome instance costs a few hundred megabytes and a second or two of startup, so a browser-per-URL crawler is roughly 100× more expensive than the reqwest path. Run browsers only for the pages that need them. The economics and the pooling patterns are the same in every language — our headless browser guide covers them, and Playwright remains the better-supported option if you're free to put the rendering step in Node or Python.

What Rust does not fix

A default headless Chrome — driven from Rust or anywhere else — announces itself through TLS fingerprint, navigator.webdriver, missing codecs, and a dozen other signals. Rust's memory safety and throughput are irrelevant to a Cloudflare challenge that returns in 50ms. Neither is reqwest immune: its TLS ClientHello is a JA3/JA4 fingerprint that does not match any shipped browser, so a "perfect" User-Agent header on a reqwest request is often more suspicious than an honest one.

The practical answers, in order of cost:

1. Rotate real proxies and slow down. Fixes rate limiting and IP-reputation blocks, which are most blocks.
2. Run a real, patched browser with a residential exit. Fixes fingerprinting, at hundreds of megabytes per worker.
3. Delegate the fetch to a service that already maintains both, and keep Rust for the parts Rust is good at.

Before you invest in any of them, be clear about what you're allowed to collect — is web scraping legal walks through the terms-of-service and personal-data questions that decide whether a project is worth building at all.

Delegating the fetch from Rust

Option 3 keeps your Rust pipeline exactly as it is — reqwest in, scraper out — and moves rendering, proxy rotation, and block handling behind an HTTP call:

let html = client
    .get("https://api.webscraping.ai/html")
    .query(&[
        ("api_key", api_key.as_str()),
        ("url", "https://example.com/protected-page"),
        ("js", "true"),
        ("proxy", "residential"),
        ("wait_for", ".product-grid"),   // wait for the selector that matters
    ])
    .send()
    .await?
    .error_for_status()?
    .text()
    .await?;

let doc = Html::parse_document(&html);   // parse with scraper exactly as before

You can skip the parsing step entirely when the target is a handful of fields. /ai/fields returns JSON that deserialises straight into a struct, which pairs unusually well with serde:

#[derive(serde::Deserialize, Debug)]
struct Product { name: String, price: String, in_stock: String }

let product: Product = client
    .get("https://api.webscraping.ai/ai/fields")
    .query(&[
        ("api_key", api_key.as_str()),
        ("url", "https://example.com/product/42"),
        ("fields[name]", "Product name"),
        ("fields[price]", "Price with currency symbol"),
        ("fields[in_stock]", "Whether the item is in stock, yes or no"),
    ])
    .send().await?
    .error_for_status()?
    .json().await?;

That is the whole integration — no browser to manage, no selectors to repair when the markup shifts. WebScraping.AI charges 1 credit for a plain datacenter fetch and 5 with JavaScript rendering, residential is 10/25, and failed requests cost nothing, so a crawler that hits a wall on a third of its URLs isn't paying for the misses. Every parameter is listed in the docs, AI field extraction covers the /ai/fields endpoint in more depth, and our Rust FAQ answers the narrower questions this guide skipped.

Rust's real strengths show up on the other side of that call: deserialising, deduplicating, and writing a few million records per hour without a GC pause, which is exactly the shape of a price monitoring or product data aggregation pipeline.

Frequently asked questions

Is Rust a good language for web scraping? For high-volume crawling, yes — bounded-concurrency fan-out with reqwest costs kilobytes per in-flight request, so a single process handles concurrency levels that need a worker fleet elsewhere. For a one-off script, Python is still faster to write and has a far larger scraping ecosystem. The honest split: Rust when throughput or long-running memory behaviour is the constraint, Python when developer time is.

Does the scraper crate support XPath? No. scraper is CSS-selectors only, via Servo's selectors crate. The skyscraper crate implements XPath for HTML but sees very little use; libxml bindings work at the cost of a C dependency. Rewriting expressions as CSS selectors is the path of least resistance.

Why does my scraper code fail to compile inside tokio::spawn? scraper::Html and Selector are !Send, so any future holding one across an .await cannot be spawned onto a multi-threaded runtime. Scope the parsing so the document is dropped before the next await point and return owned Strings or your own structs.

chromiumoxide or headless_chrome? chromiumoxide if your pipeline is async — it is tokio-native and currently the more-downloaded of the two. headless_chrome if you want a blocking API and the smallest possible amount of code. Both speak the DevTools Protocol to the same Chrome binary; the difference is ergonomics, not capability.

Can Rust scrape JavaScript-rendered pages without a browser? Not in general. There is no Rust equivalent of a JavaScript engine wired to a DOM that survives real-world SPA code. Two things often work instead: find the JSON API the page's own JavaScript calls and hit it directly with reqwest, or look for __NEXT_DATA__ / other embedded state in the HTML. Both are faster than any browser. When neither exists, you need a real browser or a rendering API.

How do I handle cookies and logins in Rust? reqwest::ClientBuilder::cookie_store(true) (with the cookies feature) persists Set-Cookie across requests on that client, which covers session-based logins. Post the form with .form(&[("username", u), ("password", p)]) and every later request on the same client carries the session.