Rust Web Scraping: the scraper Crate, TLS, PDFs, XML, and Headless Chrome

The default Rust web scraping stack is reqwest for fetching and scraper for parsing, with a headless browser bolted on only when a page genuinely needs JavaScript. This guide covers that core path, then the parts that trip people up once a scraper leaves the tutorial stage: TLS certificate configuration, XML and PDF sources, the headless-browser crates, bounded concurrency with tokio, and error handling that survives a long crawl.

Key Takeaways

• reqwest + scraper is the stack. scraper 0.27 wraps Servo's html5ever and selectors, so CSS selectors behave exactly as they do in Firefox
• scraper::Html is not Send — holding a parsed document across an .await will not compile. Extract owned Strings inside a scope, then await
• Compile Selector::parse once and reuse it. Parsing a selector inside a per-element loop is the most common Rust scraping performance bug
• With the rustls-tls feature, reqwest trusts a bundled Mozilla root store and ignores your system store — the usual cause of "works with curl, fails in Rust" on corporate networks
• Rust's PDF ecosystem handles text extraction (pdf-extract, lopdf) but has no equivalent of pdfplumber or PyMuPDF for tables, layout, or OCR
• Rust has no production-grade XPath-over-HTML crate. Convert your XPath expressions to CSS selectors before you start
• None of this defeats Cloudflare. Rust makes your pipeline fast and cheap; it does nothing about fingerprinting

Is Rust worth it for scraping?

Be honest about the tradeoff before committing to it. Rust's advantages in a scraper are real but narrow: bounded-concurrency fan-out costs kilobytes per in-flight request rather than a thread or a coroutine with a large frame, so one process comfortably holds concurrency levels that need a worker fleet elsewhere; and there is no GC pause when you are parsing and writing millions of records an hour. That matters for a long-running crawler and for the post-fetch data pipeline.

The costs are equally real. The scraping ecosystem is a fraction of Python's — no Scrapy equivalent with middleware and pipelines, no XPath worth using, no mature anti-bot toolkit, and a much thinner set of Stack Overflow answers when something breaks. Compile times slow the edit-run loop that scraper development is mostly made of, and the borrow checker charges you for the exploratory phase where you don't yet know the shape of the data.

The split that holds up in practice: Rust when throughput, memory behaviour, or a single deployable binary is the constraint; Python when developer time is. A scraper you'll run once against a hundred pages should be Python and Beautiful Soup. A crawler that runs continuously against a million URLs is a good Rust project. A common and underrated middle path is to write the fetch-and-parse layer in Rust and keep the exploratory analysis in Python.

The standard stack

Every layer has one obvious default and one or two credible alternates. Versions and 90-day download counts below are from crates.io as of August 2026 — download volume is the most honest signal of what the ecosystem actually uses:

Start with cargo add reqwest --features gzip and cargo add scraper tokio --features tokio/full. Everything below assumes that.

Fetching pages with reqwest

The fetch layer is deliberately boring, and that's the point — build one Client, reuse it everywhere, and set a timeout:

let client = reqwest::Client::builder()
    .user_agent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36")
    .timeout(std::time::Duration::from_secs(20))
    .build()?;

let body = client
    .get("https://quotes.toscrape.com/")
    .send()
    .await?
    .error_for_status()?     // 4xx/5xx become Err; without this they're Ok
    .text()
    .await?;

Two things bite newcomers here. reqwest has no default request timeout, so one hung server pins a task forever; and a 404 is a successful HTTP exchange as far as send() is concerned, which is why .error_for_status() exists. Cloning a Client is cheap and shares the underlying connection pool, so pass clones into tasks rather than building new clients.

Retries, proxy configuration, cookie stores, streaming downloads, and error classification all live in the HTTP layer — the reqwest guide covers them in depth. The rest of this guide assumes you have bytes in hand.

Parsing HTML with the scraper crate

scraper gives you Html::parse_document and Selector::parse, and that is essentially the whole API:

use scraper::{Html, Selector};

let doc = Html::parse_document(&body);
let quote_sel = Selector::parse("div.quote").unwrap();
let text_sel = Selector::parse("span.text").unwrap();
let author_sel = Selector::parse("small.author").unwrap();

for quote in doc.select(&quote_sel) {
    let text = quote.select(&text_sel).next()
        .map(|e| e.text().collect::<String>())
        .unwrap_or_default();
    let author = quote.select(&author_sel).next()
        .map(|e| e.text().collect::<String>())
        .unwrap_or_default();
    println!("{author}: {text}");
}

Three things worth internalising:

• select is scoped. Calling .select() on an ElementRef searches within that element, which is how you keep record boundaries straight instead of zipping three flat lists together and praying they line up.
• .text() yields descendant text nodes, not a string — .collect::<String>() joins them. Nested markup like <span>Price: <b>$9</b></span> needs the collect; .next() alone gives you "Price: ".
• Attributes come from .value().attr(...), which returns Option<&str>:

let link_sel = Selector::parse("a.product-link").unwrap();
let urls: Vec<String> = doc
    .select(&link_sel)
    .filter_map(|a| a.value().attr("href"))
    .map(|href| href.to_string())          // own it before doc is dropped
    .collect();

Html::parse_document runs a full HTML5 parse — it will insert <html>/<body> and fix up malformed markup the way a browser does. When you're parsing an AJAX response that is a bare fragment, Html::parse_fragment skips that and keeps the node tree closer to what you passed in.

The two gotchas

Selector::parse returns a borrowed error. SelectorErrorKind<'_> borrows the selector string, so ? into Box<dyn Error> fails to compile with a lifetime error. Use .unwrap() for selectors you wrote as literals (a hardcoded selector failing to parse is a bug, not a runtime condition), or .map_err(|e| e.to_string())? when the selector comes from configuration.

Html is !Send. The parse tree uses reference-counted nodes internally, so a Selector or Html held across an .await makes the whole future non-Send and tokio::spawn rejects it — the notorious "future cannot be sent between threads safely" error. The fix is to finish parsing before you await anything:

async fn fetch_titles(client: &reqwest::Client, url: &str) -> Result<Vec<String>, reqwest::Error> {
    let body = client.get(url).send().await?.text().await?;

    // Scoped: doc and selector are dropped before this function's future yields again.
    let titles = {
        let doc = Html::parse_document(&body);
        let sel = Selector::parse("h2.title").unwrap();
        doc.select(&sel).map(|e| e.text().collect::<String>()).collect()
    };

    Ok(titles)   // owned Strings cross the await boundary fine
}

Also hoist selector compilation out of loops. Selector::parse runs a real CSS parser; calling it once per row on a 5,000-row page is measurable. For selectors used across functions, a static SEL: LazyLock<Selector> = LazyLock::new(|| Selector::parse("td.price").unwrap()); compiles it once for the process.

What about XPath?

Rust does not have a good answer here. skyscraper is the only XPath-for-HTML crate under active development and it sees under a thousand downloads a quarter; the libxml bindings work but drag in a C library. If you are porting a scraper from Python or Java, budget time to rewrite the expressions — most translate mechanically, and our XPath cheat sheet lists the CSS equivalent for the common patterns.

SSL/TLS certificates

Certificate handling is where Rust scrapers fail in ways that look nothing like a certificate problem. The first thing to know is that reqwest has two TLS backends and they trust different things.

default-tls uses the platform's native TLS library (SChannel on Windows, Security.framework on macOS, OpenSSL on Linux) and therefore your system trust store. rustls-tls is a pure-Rust stack that removes the OpenSSL build dependency — the standard fix for cross-compilation and slim Docker images — but it ships with a bundled Mozilla root store and does not read your system store at all.

That's the trap. On a corporate network with a TLS-inspecting proxy, or against an internal service signed by a private CA, curl works and your Rust binary returns invalid peer certificate: UnknownIssuer, because the CA you installed into the OS is invisible to rustls. Two fixes:

# Option 1: make rustls read the system store
reqwest = { version = "0.13", default-features = false, features = ["rustls-tls-native-roots"] }

// Option 2: add the CA explicitly (works with either backend)
let pem = std::fs::read("corporate-ca.pem")?;
let cert = reqwest::Certificate::from_pem(&pem)?;

let client = reqwest::Client::builder()
    .add_root_certificate(cert)
    .build()?;

If your CA file is a bundle — several BEGIN CERTIFICATE blocks concatenated, which is what most corporate exports look like — use Certificate::from_pem_bundle(&pem)?, which returns a Vec<Certificate>. from_pem reads only the first certificate in the file and silently ignores the rest, which produces the same UnknownIssuer error you were trying to fix.

Client certificates, for APIs that require mutual TLS:

let identity = reqwest::Identity::from_pem(
    &[std::fs::read("client-cert.pem")?, std::fs::read("client-key.pem")?].concat()
)?;

let client = reqwest::Client::builder().identity(identity).build()?;

Should you use danger_accept_invalid_certs?

The honest answer is: rarely, and never as a blanket default. .danger_accept_invalid_certs(true) disables the entire chain of trust — expiry, issuer, and hostname — which means anything on the path can transparently intercept and modify your traffic. On a scraper that runs from a datacenter through third-party proxies, that is not a theoretical concern: the proxy is exactly the position an attacker would want.

It's defensible in three situations: local development against a self-signed service you control, a one-off fetch from a site whose certificate has genuinely expired and whose content you don't trust anyway, and reproducing a TLS failure to confirm that's what it is. In each case scope it to that one client, not the one you use for everything.

Two narrower options usually solve the actual problem with less blast radius. .danger_accept_invalid_hostnames(true) keeps chain validation while allowing a hostname mismatch, which is the common case when you're fetching by IP. And a site with an incomplete chain — a valid certificate whose intermediate the server forgot to send — is fixed properly by adding the intermediate with add_root_certificate. Some genuinely old targets fail because they only speak TLS 1.0/1.1; .min_tls_version(reqwest::tls::Version::TLS_1_0) re-enables that, and it's a far smaller concession than turning verification off.

One thing certificate configuration does not do: make you look like a browser. Your TLS ClientHello is itself a fingerprint (JA3/JA4), and reqwest's doesn't match any shipped browser regardless of how you configure the trust store. See what Rust doesn't fix below.

Crawling concurrently with tokio

This is the part that justifies choosing Rust. Bounded concurrency with buffer_unordered keeps thousands of in-flight requests in kilobytes of memory each:

use futures::stream::{self, StreamExt};

let results: Vec<Vec<String>> = stream::iter(urls)
    .map(|url| {
        let client = client.clone();          // clones share the connection pool
        async move { fetch_titles(&client, &url).await.unwrap_or_default() }
    })
    .buffer_unordered(16)                     // never more than 16 requests in flight
    .collect()
    .await;

Two rules that matter more than the crate choice:

1. Bound it. join_all over 50,000 URLs exhausts file descriptors and gets you blocked, in that order. buffer_unordered(n) where n is 8–32 per host is the polite range.
2. Keep CPU work off the async runtime. Parsing a large document, extracting PDF text, or decompressing a big response are blocking operations. Inside a tokio task they stall every other task on that worker thread. Anything that runs for more than a millisecond or so belongs in tokio::task::spawn_blocking — which has the pleasant side effect of solving the !Send problem for scraper, since the closure owns its document start to finish.

For per-host politeness rather than global concurrency, a Semaphore keyed by host is the usual shape: acquire a permit before the request, hold it across the fetch, and let the crawl fan out freely across different domains while staying gentle on each one.

For a crawler with frontier management, robots.txt handling, and depth limits already built, spider is the framework option. It is a much larger dependency and a much larger API surface — worth it for a real crawler, overkill for scraping a known list of URLs.

Error handling that survives a long crawl

The default in a scraper is to keep going. A crawl of 100,000 URLs will hit dead hosts, malformed HTML, and missing fields, and none of those should stop the run — but you need to know which happened, and how often.

The convention that has settled in the Rust ecosystem: thiserror for libraries, anyhow for applications. A library defines a typed error enum so callers can match on the variant; an application wants a single error type that carries context and prints nicely.

use thiserror::Error;

#[derive(Error, Debug)]
pub enum ScrapeError {
    #[error("HTTP request failed: {0}")]
    Http(#[from] reqwest::Error),

    #[error("selector matched nothing: {selector} on {url}")]
    Missing { selector: String, url: String },

    #[error("blocked by anti-bot (status {status}) on {url}")]
    Blocked { status: u16, url: String },
}

The Missing and Blocked variants are the ones that earn their keep. Bundling every failure into "something went wrong" means you can't tell a site that changed its markup from a site that started blocking you, and those need opposite responses — the first is a code fix, the second is a proxy or rate change.

At the top level, anyhow plus .context() turns an unhelpful error into something you can act on from a log line:

use anyhow::{Context, Result};

async fn scrape_product(client: &reqwest::Client, url: &str) -> Result<Product> {
    let body = client.get(url).send().await
        .with_context(|| format!("fetching {url}"))?
        .error_for_status()
        .with_context(|| format!("bad status from {url}"))?
        .text().await?;

    parse_product(&body).with_context(|| format!("parsing {url}"))
}

Then decide per-error whether to retry. reqwest::Error exposes is_timeout(), is_connect(), and is_body(), and the useful split is transient versus permanent: timeouts and connection failures are worth a backoff and another attempt; a 404 is not, and a 403 means retrying the same way will fail the same way. Collect failures instead of propagating them — partition the results at the end of a batch, log the failure counts by variant, and let the successful records through.

Scraping XML sources

Sitemaps, RSS feeds, and product catalogues arrive as XML often enough that it's worth knowing the two shapes. Note that XML parsers are strict where HTML parsers are forgiving — feed HTML to quick-xml and it will reject unclosed <br> tags. Use scraper for HTML, these for XML.

quick-xml for streaming. The right choice for large documents, because it never builds a tree:

use quick_xml::events::Event;
use quick_xml::Reader;

let mut reader = Reader::from_str(&xml);
reader.config_mut().trim_text(true);

let mut urls = Vec::new();
let mut in_loc = false;

loop {
    match reader.read_event()? {
        Event::Start(e) if e.name().as_ref() == b"loc" => in_loc = true,
        Event::End(e) if e.name().as_ref() == b"loc" => in_loc = false,
        Event::Text(e) if in_loc => urls.push(e.unescape()?.into_owned()),
        Event::Eof => break,
        _ => {}
    }
}

Two API details that older tutorials get wrong: configuration moved onto config_mut() (the standalone reader.trim_text(true) method is gone), and Reader::from_str uses read_event() with no buffer — read_event_into(&mut buf) is for readers built over BufRead, which is what you want when streaming a file or response body rather than a String you already hold.

quick-xml with serde for structured documents. Enable the serialize feature and the whole loop above collapses into a struct definition. @ prefixes an attribute, $text is the element's text content:

#[derive(serde::Deserialize)]
struct Urlset {
    url: Vec<UrlEntry>,
}

#[derive(serde::Deserialize)]
struct UrlEntry {
    loc: String,
    lastmod: Option<String>,
}

let sitemap: Urlset = quick_xml::de::from_str(&xml)?;

roxmltree is the third option: a read-only DOM with an unusually clean API (doc.descendants().filter(|n| n.has_tag_name("item"))), good when you need to navigate around a small document rather than stream it. serde-xml-rs also still exists for serde deserialization, but quick-xml's serde support is better maintained and you're likely to have the crate already.

Namespaces are the recurring headache in real feeds. Both quick-xml and roxmltree expose namespace-aware lookups; matching on the local tag name alone works until a document uses two namespaces with the same local name, and then it silently mixes them.

Extracting data from PDFs

Plenty of the data worth scraping — filings, price lists, government reports — only exists as a PDF. Rust can handle the common case, and you should know where the ceiling is before you build on it.

pdf-extract for text. One function, plain text out:

// From a path
let text = pdf_extract::extract_text("report.pdf")?;

// From bytes you just downloaded
let bytes = client.get(url).send().await?.bytes().await?;
let text = pdf_extract::extract_text_from_mem(&bytes)?;

Note the two entry points: extract_text takes a path, extract_text_from_mem takes &[u8]. Passing downloaded bytes to extract_text is the most common compile error here. And run it under spawn_blocking in an async scraper — extraction on a large document is seconds of CPU, not microseconds.

From there it's ordinary text processing. Regex against the extracted text is how most structured extraction actually gets done:

let invoice = Regex::new(r"Invoice\s*#?:?\s*([A-Z0-9-]+)")?
    .captures(&text)
    .and_then(|c| c.get(1))
    .map(|m| m.as_str().to_string());

lopdf for structure. When you need page counts, metadata, embedded attachments, or to split and merge documents, lopdf exposes the PDF object model directly — Document::load(path), then doc.get_pages(), doc.trailer, and the object graph underneath. It also offers page-scoped text extraction, which is useful when you only want page 3 of a 400-page filing.

pdfium-render binds Google's PDFium, the engine in Chrome. It gives the highest-fidelity text extraction and can render pages to images, at the cost of shipping the PDFium shared library alongside your binary — a real deployment consideration for a container image, and the reason it isn't the default recommendation.

Where Rust's PDF ecosystem stops

Be realistic about this before you plan around it:

• No table extraction. There is no Rust equivalent of pdfplumber or Camelot. You get a flat text stream and reconstruct columns yourself from whitespace and ordering, which is fragile and specific to each document layout.
• Reading order is not guaranteed. PDF stores positioned glyphs, not paragraphs. Multi-column layouts frequently extract interleaved, and no pure-Rust crate reflows them reliably. pdfium-render is noticeably better here.
• Scanned PDFs need OCR, which means shelling out to Tesseract or calling a service — there is no in-ecosystem answer.
• Encrypted PDFs are hit-and-miss. Standard password-protected files often work; unusual encryption handlers often don't.

If your pipeline is mostly PDF and mostly tables, this is a legitimate reason to put that one stage in Python and keep Rust for the rest.

Headless browsers in Rust

When the HTML you get back is an empty <div id="root">, you need a real browser. Rust has four viable options and they split cleanly by protocol and by async model:

The blocking headless_chrome API is genuinely the easiest to read:

use headless_chrome::{Browser, LaunchOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let browser = Browser::new(
        LaunchOptions::default_builder().headless(true).build()?
    )?;

    let tab = browser.new_tab()?;
    tab.navigate_to("https://example.com/spa")?;
    tab.wait_for_element(".loaded-content")?;   // waits, with a default timeout

    let html = tab.get_content()?;
    // hand off to scraper as usual
    Ok(())
}

chromiumoxide is the one to reach for in a tokio pipeline, because it doesn't force you to shove blocking calls into spawn_blocking. Its shape is a Browser plus a handler future you spawn once, then page.goto(...).await, page.find_element(...).await, page.content().await. The handler future is the part newcomers miss — forget to spawn it and every subsequent call hangs, with no error to explain why.

Now the maturity framing, because it's the thing to weigh before building on any of them. All four are community-maintained crates with a fraction of the contributor base behind Playwright or Puppeteer, and it shows in the same places each time: auto-waiting is thinner, so you write more explicit waits; the error messages when a CDP call fails are protocol-level rather than helpful; and features that arrive in Playwright within weeks of a Chrome release can take a year or never land at all. headless_chrome in particular is stable but slow-moving. None of this makes them unusable — they drive Chrome through the same protocol as everything else — but it does mean you'll spend more time on the browser layer than you would in Node or Python.

The economics apply to all four equally: each Chrome instance costs a few hundred megabytes and a second or two of startup, so a browser-per-URL crawler is roughly 100× more expensive than the reqwest path. Run browsers only for the pages that need them. Our headless browser guide covers pooling and the reuse patterns, and Playwright remains the better-supported option if you're free to put the rendering step in Node or Python.

What Rust does not fix

A default headless Chrome — driven from Rust or anywhere else — announces itself through TLS fingerprint, navigator.webdriver, missing codecs, and a dozen other signals. Rust's memory safety and throughput are irrelevant to a Cloudflare challenge that returns in 50ms. Neither is reqwest immune: its TLS ClientHello is a JA3/JA4 fingerprint that does not match any shipped browser, so a "perfect" User-Agent header on a reqwest request is often more suspicious than an honest one.

The practical answers, in order of cost:

1. Rotate real proxies and slow down. Fixes rate limiting and IP-reputation blocks, which are most blocks.
2. Run a real, patched browser with a residential exit. Fixes fingerprinting, at hundreds of megabytes per worker.
3. Delegate the fetch to a service that already maintains both, and keep Rust for the parts Rust is good at.

Before you invest in any of them, be clear about what you're allowed to collect — is web scraping legal walks through the terms-of-service and personal-data questions that decide whether a project is worth building at all.

Delegating the fetch from Rust

Option 3 keeps your Rust pipeline exactly as it is — reqwest in, scraper out — and moves rendering, proxy rotation, and block handling behind an HTTP call:

let html = client
    .get("https://api.webscraping.ai/html")
    .query(&[
        ("api_key", api_key.as_str()),
        ("url", "https://example.com/protected-page"),
        ("js", "true"),
        ("proxy", "residential"),
        ("wait_for", ".product-grid"),   // wait for the selector that matters
    ])
    .send()
    .await?
    .error_for_status()?
    .text()
    .await?;

let doc = Html::parse_document(&html);   // parse with scraper exactly as before

You can skip the parsing step entirely when the target is a handful of fields. /ai/fields returns JSON that deserialises straight into a struct, which pairs unusually well with serde:

#[derive(serde::Deserialize, Debug)]
struct Product { name: String, price: String, in_stock: String }

let product: Product = client
    .get("https://api.webscraping.ai/ai/fields")
    .query(&[
        ("api_key", api_key.as_str()),
        ("url", "https://example.com/product/42"),
        ("fields[name]", "Product name"),
        ("fields[price]", "Price with currency symbol"),
        ("fields[in_stock]", "Whether the item is in stock, yes or no"),
    ])
    .send().await?
    .error_for_status()?
    .json().await?;

That is the whole integration — no browser to manage, no selectors to repair when the markup shifts. WebScraping.AI charges 1 credit for a plain datacenter fetch and 5 with JavaScript rendering, residential is 10/25, and failed requests cost nothing, so a crawler that hits a wall on a third of its URLs isn't paying for the misses. Every parameter is listed in the docs, and AI field extraction covers the /ai/fields endpoint in more depth.

Rust's real strengths show up on the other side of that call: deserialising, deduplicating, and writing a few million records per hour without a GC pause, which is exactly the shape of a price monitoring or product data aggregation pipeline.

Rust vs Python vs Go for scraping

Go is Rust's closest competitor for this workload and usually the more pragmatic choice: goroutines give you the same cheap concurrency and the same single-binary deploy, with much faster iteration and no borrow checker in your way during the exploratory phase. Rust wins where the post-fetch work is heavy — millions of records an hour, tight memory budgets, no GC pauses — or where the scraper is one component of a larger Rust system.

Python wins on everything except runtime cost, and runtime cost is usually not the binding constraint. If you're unsure which you're in, start in Python, measure, and port the hot stage if you actually find one.

Frequently asked questions

Is Rust a good language for web scraping? For high-volume crawling, yes — bounded-concurrency fan-out with reqwest costs kilobytes per in-flight request, so a single process handles concurrency levels that need a worker fleet elsewhere. For a one-off script, Python is still faster to write and has a far larger scraping ecosystem. The honest split: Rust when throughput or long-running memory behaviour is the constraint, Python when developer time is.

Does the scraper crate support XPath? No. scraper is CSS-selectors only, via Servo's selectors crate. The skyscraper crate implements XPath for HTML but sees very little use; libxml bindings work at the cost of a C dependency. Rewriting expressions as CSS selectors is the path of least resistance.

Why does my scraper code fail to compile inside tokio::spawn? scraper::Html and Selector are !Send, so any future holding one across an .await cannot be spawned onto a multi-threaded runtime. Scope the parsing so the document is dropped before the next await point and return owned Strings or your own structs, or move the whole parse into spawn_blocking.

Why do I get UnknownIssuer errors in Rust when curl works? Almost always the rustls-tls feature: it trusts a bundled Mozilla root store, not your operating system's, so a corporate CA you installed system-wide is invisible to it. Switch to rustls-tls-native-roots, or add the CA explicitly with add_root_certificate — using Certificate::from_pem_bundle if the file contains more than one certificate.

Is danger_accept_invalid_certs safe to use in a scraper? Not as a default. It disables expiry, issuer, and hostname checks together, which lets anything on the network path intercept and modify your traffic — a real risk when your requests already route through third-party proxies. Prefer danger_accept_invalid_hostnames for a hostname mismatch, or add the missing CA. If you do need it, scope it to one client rather than the one you use everywhere.

Can Rust extract tables from PDFs? Not well. pdf-extract and lopdf give you a text stream, and there is no Rust equivalent of pdfplumber or Camelot for reconstructing table structure. You rebuild columns from whitespace and ordering yourself, per layout. If tables are the point of the project, run that stage in Python.

chromiumoxide or headless_chrome? chromiumoxide if your pipeline is async — it is tokio-native and currently the more-downloaded of the two. headless_chrome if you want a blocking API and the smallest possible amount of code. Both speak the DevTools Protocol to the same Chrome binary; the difference is ergonomics, not capability.

Can Rust scrape JavaScript-rendered pages without a browser? Not in general. There is no Rust equivalent of a JavaScript engine wired to a DOM that survives real-world SPA code. Two things often work instead: find the JSON API the page's own JavaScript calls and hit it directly with reqwest, or look for __NEXT_DATA__ / other embedded state in the HTML. Both are faster than any browser. When neither exists, you need a real browser or a rendering API.

How do I handle cookies and logins in Rust? reqwest::ClientBuilder::cookie_store(true) (with the cookies feature) persists Set-Cookie across requests on that client, which covers session-based logins. Post the form with .form(&[("username", u), ("password", p)]) and every later request on the same client carries the session.