--- title: "Rust Web Scraping: the scraper Crate, TLS, PDFs, XML, and Headless Chrome" description: "Rust web scraping with current crates: the scraper crate, TLS certificates, XML and PDF parsing, headless Chrome, tokio concurrency, and error handling." url: https://webscraping.ai/blog/web-scraping-with-rust markdown_index: https://webscraping.ai/llms.txt --- # Rust Web Scraping: the scraper Crate, TLS, PDFs, XML, and Headless Chrome The default Rust web scraping stack is [reqwest](https://webscraping.ai/blog/rust-reqwest-guide) for fetching and [`scraper`](https://crates.io/crates/scraper) for parsing, with a headless browser bolted on only when a page genuinely needs JavaScript. This guide covers that core path, then the parts that trip people up once a scraper leaves the tutorial stage: TLS certificate configuration, XML and PDF sources, the headless-browser crates, bounded concurrency with tokio, and error handling that survives a long crawl. ## Key Takeaways - `reqwest` + `scraper` is the stack. `scraper` 0.27 wraps Servo's `html5ever` and `selectors`, so CSS selectors behave exactly as they do in Firefox - `scraper::Html` is **not `Send`** — holding a parsed document across an `.await` will not compile. Extract owned `String`s inside a scope, then await - Compile `Selector::parse` once and reuse it. Parsing a selector inside a per-element loop is the most common Rust scraping performance bug - With the `rustls-tls` feature, reqwest trusts a **bundled** Mozilla root store and ignores your system store — the usual cause of "works with curl, fails in Rust" on corporate networks - Rust's PDF ecosystem handles text extraction (`pdf-extract`, `lopdf`) but has no equivalent of pdfplumber or PyMuPDF for tables, layout, or OCR - Rust has **no** production-grade XPath-over-HTML crate. Convert your XPath expressions to CSS selectors before you start - None of this defeats Cloudflare. Rust makes your pipeline fast and cheap; it does nothing about fingerprinting ## Is Rust worth it for scraping? Be honest about the tradeoff before committing to it. Rust's advantages in a scraper are real but narrow: bounded-concurrency fan-out costs kilobytes per in-flight request rather than a thread or a coroutine with a large frame, so one process comfortably holds concurrency levels that need a worker fleet elsewhere; and there is no GC pause when you are parsing and writing millions of records an hour. That matters for a long-running crawler and for the post-fetch data pipeline. The costs are equally real. The scraping ecosystem is a fraction of Python's — no Scrapy equivalent with middleware and pipelines, no XPath worth using, no mature anti-bot toolkit, and a much thinner set of Stack Overflow answers when something breaks. Compile times slow the edit-run loop that scraper development is mostly made of, and the borrow checker charges you for the exploratory phase where you don't yet know the shape of the data. The split that holds up in practice: **Rust when throughput, memory behaviour, or a single deployable binary is the constraint; Python when developer time is.** A scraper you'll run once against a hundred pages should be [Python and Beautiful Soup](https://webscraping.ai/blog/beautiful-soup-python). A crawler that runs continuously against a million URLs is a good Rust project. A common and underrated middle path is to write the fetch-and-parse layer in Rust and keep the exploratory analysis in Python. ### The standard stack Every layer has one obvious default and one or two credible alternates. Versions and 90-day download counts below are from crates.io as of August 2026 — download volume is the most honest signal of what the ecosystem actually uses: | Layer | Crate | Version | 90-day downloads | Notes | | --- | --- | --- | --- | --- | | HTTP | `reqwest` | 0.13.4 | 155M | Async-first, blocking feature available | | HTML parsing | `scraper` | 0.27.0 | 7.0M | CSS selectors on Servo's parser | | XML parsing | `quick-xml` | 0.41.0 | 88.8M | Streaming, plus serde deserialization | | XML (DOM) | `roxmltree` | 0.21.1 | 17.3M | Read-only tree, simplest API | | PDF text | `pdf-extract` | 0.12.0 | 2.1M | One function, plain text out | | PDF structure | `lopdf` | 0.44.0 | 7.0M | Low-level document model | | Browser (CDP, async) | `chromiumoxide` | 0.9.1 | 1.4M | tokio-native, DevTools Protocol | | Browser (CDP, blocking) | `headless_chrome` | 1.0.22 | 0.9M | Simplest API, synchronous | | Browser (WebDriver) | `fantoccini` | 0.22.1 | 0.5M | Async, works with Firefox too | | Full crawler framework | `spider` | 2.53.4 | 66k | Batteries-included crawling | Start with `cargo add reqwest --features gzip` and `cargo add scraper tokio --features tokio/full`. Everything below assumes that. ## Fetching pages with reqwest The fetch layer is deliberately boring, and that's the point — build one `Client`, reuse it everywhere, and set a timeout: ```rust let client = reqwest::Client::builder() .user_agent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36") .timeout(std::time::Duration::from_secs(20)) .build()?; let body = client .get("https://quotes.toscrape.com/") .send() .await? .error_for_status()? // 4xx/5xx become Err; without this they're Ok .text() .await?; ``` Two things bite newcomers here. `reqwest` has **no default request timeout** , so one hung server pins a task forever; and a 404 is a successful HTTP exchange as far as `send()` is concerned, which is why `.error_for_status()` exists. Cloning a `Client` is cheap and shares the underlying connection pool, so pass clones into tasks rather than building new clients. Retries, proxy configuration, cookie stores, streaming downloads, and error classification all live in the HTTP layer — the [reqwest guide](https://webscraping.ai/blog/rust-reqwest-guide) covers them in depth. The rest of this guide assumes you have bytes in hand. ## Parsing HTML with the scraper crate `scraper` gives you `Html::parse_document` and `Selector::parse`, and that is essentially the whole API: ```rust use scraper::{Html, Selector}; let doc = Html::parse_document(&body); let quote_sel = Selector::parse("div.quote").unwrap(); let text_sel = Selector::parse("span.text").unwrap(); let author_sel = Selector::parse("small.author").unwrap(); for quote in doc.select("e_sel) { let text = quote.select(&text_sel).next() .map(|e| e.text().collect::()) .unwrap_or_default(); let author = quote.select(&author_sel).next() .map(|e| e.text().collect::()) .unwrap_or_default(); println!("{author}: {text}"); } ``` Three things worth internalising: - **`select` is scoped.** Calling `.select()` on an `ElementRef` searches within that element, which is how you keep record boundaries straight instead of zipping three flat lists together and praying they line up. - **`.text()` yields descendant text nodes**, not a string — `.collect::()` joins them. Nested markup like `Price: $9` needs the collect; `.next()` alone gives you `"Price: "`. - **Attributes come from `.value().attr(...)`**, which returns `Option<&str>`: ```rust let link_sel = Selector::parse("a.product-link").unwrap(); let urls: Vec = doc .select(&link_sel) .filter_map(|a| a.value().attr("href")) .map(|href| href.to_string()) // own it before doc is dropped .collect(); ``` `Html::parse_document` runs a full HTML5 parse — it will insert ``/`` and fix up malformed markup the way a browser does. When you're parsing an AJAX response that is a bare fragment, `Html::parse_fragment` skips that and keeps the node tree closer to what you passed in. ### The two gotchas **`Selector::parse` returns a borrowed error.** `SelectorErrorKind<'_>` borrows the selector string, so `?` into `Box` fails to compile with a lifetime error. Use `.unwrap()` for selectors you wrote as literals (a hardcoded selector failing to parse is a bug, not a runtime condition), or `.map_err(|e| e.to_string())?` when the selector comes from configuration. **`Html` is `!Send`.** The parse tree uses reference-counted nodes internally, so a `Selector` or `Html` held across an `.await` makes the whole future non-`Send` and `tokio::spawn` rejects it — the notorious "future cannot be sent between threads safely" error. The fix is to finish parsing before you await anything: ```rust async fn fetch_titles(client: &reqwest::Client, url: &str) -> Result, reqwest::Error> { let body = client.get(url).send().await?.text().await?; // Scoped: doc and selector are dropped before this function's future yields again. let titles = { let doc = Html::parse_document(&body); let sel = Selector::parse("h2.title").unwrap(); doc.select(&sel).map(|e| e.text().collect::()).collect() }; Ok(titles) // owned Strings cross the await boundary fine } ``` Also hoist selector compilation out of loops. `Selector::parse` runs a real CSS parser; calling it once per row on a 5,000-row page is measurable. For selectors used across functions, a `static SEL: LazyLock = LazyLock::new(|| Selector::parse("td.price").unwrap());` compiles it once for the process. ### What about XPath? Rust does not have a good answer here. `skyscraper` is the only XPath-for-HTML crate under active development and it sees under a thousand downloads a quarter; the `libxml` bindings work but drag in a C library. If you are porting a scraper from Python or Java, budget time to rewrite the expressions — most translate mechanically, and our [XPath cheat sheet](https://webscraping.ai/blog/xpath-cheat-sheet) lists the CSS equivalent for the common patterns. ## SSL/TLS certificates Certificate handling is where Rust scrapers fail in ways that look nothing like a certificate problem. The first thing to know is that reqwest has **two** TLS backends and they trust different things. `default-tls` uses the platform's native TLS library (SChannel on Windows, Security.framework on macOS, OpenSSL on Linux) and therefore your system trust store. `rustls-tls` is a pure-Rust stack that removes the OpenSSL build dependency — the standard fix for cross-compilation and slim Docker images — but it ships with a **bundled Mozilla root store** and does not read your system store at all. That's the trap. On a corporate network with a TLS-inspecting proxy, or against an internal service signed by a private CA, `curl` works and your Rust binary returns `invalid peer certificate: UnknownIssuer`, because the CA you installed into the OS is invisible to rustls. Two fixes: ```toml # Option 1: make rustls read the system store reqwest = { version = "0.13", default-features = false, features = ["rustls-tls-native-roots"] } ``` ```rust // Option 2: add the CA explicitly (works with either backend) let pem = std::fs::read("corporate-ca.pem")?; let cert = reqwest::Certificate::from_pem(&pem)?; let client = reqwest::Client::builder() .add_root_certificate(cert) .build()?; ``` If your CA file is a **bundle** — several `BEGIN CERTIFICATE` blocks concatenated, which is what most corporate exports look like — use `Certificate::from_pem_bundle(&pem)?`, which returns a `Vec`. `from_pem` reads only the first certificate in the file and silently ignores the rest, which produces the same `UnknownIssuer` error you were trying to fix. Client certificates, for APIs that require mutual TLS: ```rust let identity = reqwest::Identity::from_pem( &[std::fs::read("client-cert.pem")?, std::fs::read("client-key.pem")?].concat() )?; let client = reqwest::Client::builder().identity(identity).build()?; ``` ### Should you use danger\_accept\_invalid\_certs? The honest answer is: rarely, and never as a blanket default. `.danger_accept_invalid_certs(true)` disables the entire chain of trust — expiry, issuer, and hostname — which means anything on the path can transparently intercept and modify your traffic. On a scraper that runs from a datacenter through third-party proxies, that is not a theoretical concern: the proxy is exactly the position an attacker would want. It's defensible in three situations: local development against a self-signed service you control, a one-off fetch from a site whose certificate has genuinely expired and whose content you don't trust anyway, and reproducing a TLS failure to confirm that's what it is. In each case scope it to that one client, not the one you use for everything. Two narrower options usually solve the actual problem with less blast radius. `.danger_accept_invalid_hostnames(true)` keeps chain validation while allowing a hostname mismatch, which is the common case when you're fetching by IP. And a site with an incomplete chain — a valid certificate whose intermediate the server forgot to send — is fixed properly by adding the intermediate with `add_root_certificate`. Some genuinely old targets fail because they only speak TLS 1.0/1.1; `.min_tls_version(reqwest::tls::Version::TLS_1_0)` re-enables that, and it's a far smaller concession than turning verification off. One thing certificate configuration does _not_ do: make you look like a browser. Your TLS ClientHello is itself a fingerprint (JA3/JA4), and reqwest's doesn't match any shipped browser regardless of how you configure the trust store. See [what Rust doesn't fix](https://webscraping.ai/blog/web-scraping-with-rust#what-rust-does-not-fix) below. ## Crawling concurrently with tokio This is the part that justifies choosing Rust. Bounded concurrency with `buffer_unordered` keeps thousands of in-flight requests in kilobytes of memory each: ```rust use futures::stream::{self, StreamExt}; let results: Vec> = stream::iter(urls) .map(|url| { let client = client.clone(); // clones share the connection pool async move { fetch_titles(&client, &url).await.unwrap_or_default() } }) .buffer_unordered(16) // never more than 16 requests in flight .collect() .await; ``` Two rules that matter more than the crate choice: 1. **Bound it.** `join_all` over 50,000 URLs exhausts file descriptors and gets you blocked, in that order. `buffer_unordered(n)` where `n` is 8–32 per host is the polite range. 2. **Keep CPU work off the async runtime.** Parsing a large document, extracting PDF text, or decompressing a big response are blocking operations. Inside a tokio task they stall every other task on that worker thread. Anything that runs for more than a millisecond or so belongs in `tokio::task::spawn_blocking` — which has the pleasant side effect of solving the `!Send` problem for `scraper`, since the closure owns its document start to finish. For per-host politeness rather than global concurrency, a `Semaphore` keyed by host is the usual shape: acquire a permit before the request, hold it across the fetch, and let the crawl fan out freely across different domains while staying gentle on each one. For a crawler with frontier management, robots.txt handling, and depth limits already built, `spider` is the framework option. It is a much larger dependency and a much larger API surface — worth it for a real crawler, overkill for scraping a known list of URLs. ## Error handling that survives a long crawl The default in a scraper is to keep going. A crawl of 100,000 URLs will hit dead hosts, malformed HTML, and missing fields, and none of those should stop the run — but you need to know which happened, and how often. The convention that has settled in the Rust ecosystem: **`thiserror` for libraries, `anyhow` for applications.** A library defines a typed error enum so callers can match on the variant; an application wants a single error type that carries context and prints nicely. ```rust use thiserror::Error; #[derive(Error, Debug)] pub enum ScrapeError { #[error("HTTP request failed: {0}")] Http(#[from] reqwest::Error), #[error("selector matched nothing: {selector} on {url}")] Missing { selector: String, url: String }, #[error("blocked by anti-bot (status {status}) on {url}")] Blocked { status: u16, url: String }, } ``` The `Missing` and `Blocked` variants are the ones that earn their keep. Bundling every failure into "something went wrong" means you can't tell a site that changed its markup from a site that started blocking you, and those need opposite responses — the first is a code fix, the second is a proxy or rate change. At the top level, `anyhow` plus `.context()` turns an unhelpful error into something you can act on from a log line: ```rust use anyhow::{Context, Result}; async fn scrape_product(client: &reqwest::Client, url: &str) -> Result { let body = client.get(url).send().await .with_context(|| format!("fetching {url}"))? .error_for_status() .with_context(|| format!("bad status from {url}"))? .text().await?; parse_product(&body).with_context(|| format!("parsing {url}")) } ``` Then decide per-error whether to retry. `reqwest::Error` exposes `is_timeout()`, `is_connect()`, and `is_body()`, and the useful split is transient versus permanent: timeouts and connection failures are worth a backoff and another attempt; a 404 is not, and a 403 means retrying the same way will fail the same way. Collect failures instead of propagating them — `partition` the results at the end of a batch, log the failure counts by variant, and let the successful records through. ## Scraping XML sources Sitemaps, RSS feeds, and product catalogues arrive as XML often enough that it's worth knowing the two shapes. Note that XML parsers are strict where HTML parsers are forgiving — feed HTML to `quick-xml` and it will reject unclosed `
` tags. Use `scraper` for HTML, these for XML. **`quick-xml` for streaming.** The right choice for large documents, because it never builds a tree: ```rust use quick_xml::events::Event; use quick_xml::Reader; let mut reader = Reader::from_str(&xml); reader.config_mut().trim_text(true); let mut urls = Vec::new(); let mut in_loc = false; loop { match reader.read_event()? { Event::Start(e) if e.name().as_ref() == b"loc" => in_loc = true, Event::End(e) if e.name().as_ref() == b"loc" => in_loc = false, Event::Text(e) if in_loc => urls.push(e.unescape()?.into_owned()), Event::Eof => break, _ => {} } } ``` Two API details that older tutorials get wrong: configuration moved onto `config_mut()` (the standalone `reader.trim_text(true)` method is gone), and `Reader::from_str` uses `read_event()` with no buffer — `read_event_into(&mut buf)` is for readers built over `BufRead`, which is what you want when streaming a file or response body rather than a `String` you already hold. **`quick-xml` with serde for structured documents.** Enable the `serialize` feature and the whole loop above collapses into a struct definition. `@` prefixes an attribute, `$text` is the element's text content: ```rust #[derive(serde::Deserialize)] struct Urlset { url: Vec, } #[derive(serde::Deserialize)] struct UrlEntry { loc: String, lastmod: Option, } let sitemap: Urlset = quick_xml::de::from_str(&xml)?; ``` `roxmltree` is the third option: a read-only DOM with an unusually clean API (`doc.descendants().filter(|n| n.has_tag_name("item"))`), good when you need to navigate around a small document rather than stream it. `serde-xml-rs` also still exists for serde deserialization, but `quick-xml`'s serde support is better maintained and you're likely to have the crate already. Namespaces are the recurring headache in real feeds. Both `quick-xml` and `roxmltree` expose namespace-aware lookups; matching on the local tag name alone works until a document uses two namespaces with the same local name, and then it silently mixes them. ## Extracting data from PDFs Plenty of the data worth scraping — filings, price lists, government reports — only exists as a PDF. Rust can handle the common case, and you should know where the ceiling is before you build on it. **`pdf-extract` for text.** One function, plain text out: ```rust // From a path let text = pdf_extract::extract_text("report.pdf")?; // From bytes you just downloaded let bytes = client.get(url).send().await?.bytes().await?; let text = pdf_extract::extract_text_from_mem(&bytes)?; ``` Note the two entry points: `extract_text` takes a path, `extract_text_from_mem` takes `&[u8]`. Passing downloaded bytes to `extract_text` is the most common compile error here. And run it under `spawn_blocking` in an async scraper — extraction on a large document is seconds of CPU, not microseconds. From there it's ordinary text processing. Regex against the extracted text is how most structured extraction actually gets done: ```rust let invoice = Regex::new(r"Invoice\s*#?:?\s*([A-Z0-9-]+)")? .captures(&text) .and_then(|c| c.get(1)) .map(|m| m.as_str().to_string()); ``` **`lopdf` for structure.** When you need page counts, metadata, embedded attachments, or to split and merge documents, `lopdf` exposes the PDF object model directly — `Document::load(path)`, then `doc.get_pages()`, `doc.trailer`, and the object graph underneath. It also offers page-scoped text extraction, which is useful when you only want page 3 of a 400-page filing. **`pdfium-render`** binds Google's PDFium, the engine in Chrome. It gives the highest-fidelity text extraction and can render pages to images, at the cost of shipping the PDFium shared library alongside your binary — a real deployment consideration for a container image, and the reason it isn't the default recommendation. ### Where Rust's PDF ecosystem stops Be realistic about this before you plan around it: - **No table extraction.** There is no Rust equivalent of `pdfplumber` or Camelot. You get a flat text stream and reconstruct columns yourself from whitespace and ordering, which is fragile and specific to each document layout. - **Reading order is not guaranteed.** PDF stores positioned glyphs, not paragraphs. Multi-column layouts frequently extract interleaved, and no pure-Rust crate reflows them reliably. `pdfium-render` is noticeably better here. - **Scanned PDFs need OCR** , which means shelling out to Tesseract or calling a service — there is no in-ecosystem answer. - **Encrypted PDFs** are hit-and-miss. Standard password-protected files often work; unusual encryption handlers often don't. If your pipeline is mostly PDF and mostly tables, this is a legitimate reason to put that one stage in Python and keep Rust for the rest. ## Headless browsers in Rust When the HTML you get back is an empty `
`, you need a real browser. Rust has four viable options and they split cleanly by protocol and by async model: | Crate | Protocol | Async | Pick it when | | --- | --- | --- | --- | | `chromiumoxide` | DevTools (CDP) | Yes (tokio) | You are already async and want the fastest, most direct Chrome control | | `headless_chrome` | DevTools (CDP) | No (blocking) | You want the shortest path to working code and don't need async | | `fantoccini` | WebDriver | Yes | You need Firefox, or a Selenium Grid / existing WebDriver infrastructure | | `thirtyfour` | WebDriver | Yes | You're porting Selenium test code and want familiar `By::Css` ergonomics | The blocking `headless_chrome` API is genuinely the easiest to read: ```rust use headless_chrome::{Browser, LaunchOptions}; fn main() -> Result<(), Box> { let browser = Browser::new( LaunchOptions::default_builder().headless(true).build()? )?; let tab = browser.new_tab()?; tab.navigate_to("https://example.com/spa")?; tab.wait_for_element(".loaded-content")?; // waits, with a default timeout let html = tab.get_content()?; // hand off to scraper as usual Ok(()) } ``` `chromiumoxide` is the one to reach for in a tokio pipeline, because it doesn't force you to shove blocking calls into `spawn_blocking`. Its shape is a `Browser` plus a handler future you spawn once, then `page.goto(...).await`, `page.find_element(...).await`, `page.content().await`. The handler future is the part newcomers miss — forget to spawn it and every subsequent call hangs, with no error to explain why. Now the maturity framing, because it's the thing to weigh before building on any of them. All four are community-maintained crates with a fraction of the contributor base behind Playwright or Puppeteer, and it shows in the same places each time: auto-waiting is thinner, so you write more explicit waits; the error messages when a CDP call fails are protocol-level rather than helpful; and features that arrive in Playwright within weeks of a Chrome release can take a year or never land at all. `headless_chrome` in particular is stable but slow-moving. None of this makes them unusable — they drive Chrome through the same protocol as everything else — but it does mean you'll spend more time on the browser layer than you would in Node or Python. The economics apply to all four equally: each Chrome instance costs a few hundred megabytes and a second or two of startup, so a browser-per-URL crawler is roughly 100× more expensive than the `reqwest` path. Run browsers only for the pages that need them. Our [headless browser guide](https://webscraping.ai/blog/headless-browser-guide) covers pooling and the reuse patterns, and [Playwright](https://webscraping.ai/blog/playwright-web-scraping) remains the better-supported option if you're free to put the rendering step in Node or Python. ## What Rust does not fix A default headless Chrome — driven from Rust or anywhere else — announces itself through TLS fingerprint, `navigator.webdriver`, missing codecs, and a dozen other signals. Rust's memory safety and throughput are irrelevant to a Cloudflare challenge that returns in 50ms. Neither is `reqwest` immune: its TLS ClientHello is a JA3/JA4 fingerprint that does not match any shipped browser, so a "perfect" `User-Agent` header on a `reqwest` request is often _more_ suspicious than an honest one. The practical answers, in order of cost: 1. **Rotate real proxies** and slow down. Fixes rate limiting and IP-reputation blocks, which are most blocks. 2. **Run a real, patched browser** with a residential exit. Fixes fingerprinting, at hundreds of megabytes per worker. 3. **Delegate the fetch** to a service that already maintains both, and keep Rust for the parts Rust is good at. Before you invest in any of them, be clear about what you're allowed to collect — [is web scraping legal](https://webscraping.ai/blog/is-web-scraping-legal) walks through the terms-of-service and personal-data questions that decide whether a project is worth building at all. ## Delegating the fetch from Rust Option 3 keeps your Rust pipeline exactly as it is — `reqwest` in, `scraper` out — and moves rendering, proxy rotation, and block handling behind an HTTP call: ```rust let html = client .get("https://api.webscraping.ai/html") .query(&[ ("api_key", api_key.as_str()), ("url", "https://example.com/protected-page"), ("js", "true"), ("proxy", "residential"), ("wait_for", ".product-grid"), // wait for the selector that matters ]) .send() .await? .error_for_status()? .text() .await?; let doc = Html::parse_document(&html); // parse with scraper exactly as before ``` You can skip the parsing step entirely when the target is a handful of fields. `/ai/fields` returns JSON that deserialises straight into a struct, which pairs unusually well with serde: ```rust #[derive(serde::Deserialize, Debug)] struct Product { name: String, price: String, in_stock: String } let product: Product = client .get("https://api.webscraping.ai/ai/fields") .query(&[ ("api_key", api_key.as_str()), ("url", "https://example.com/product/42"), ("fields[name]", "Product name"), ("fields[price]", "Price with currency symbol"), ("fields[in_stock]", "Whether the item is in stock, yes or no"), ]) .send().await? .error_for_status()? .json().await?; ``` That is the whole integration — no browser to manage, no selectors to repair when the markup shifts. [WebScraping.AI](https://webscraping.ai/) charges 1 credit for a plain datacenter fetch and 5 with JavaScript rendering, residential is 10/25, and failed requests cost nothing, so a crawler that hits a wall on a third of its URLs isn't paying for the misses. Every parameter is listed in the [docs](https://webscraping.ai/docs), and [AI field extraction](https://webscraping.ai/ai-web-scraping) covers the `/ai/fields` endpoint in more depth. Rust's real strengths show up on the other side of that call: deserialising, deduplicating, and writing a few million records per hour without a GC pause, which is exactly the shape of a [price monitoring](https://webscraping.ai/use-cases/price-monitoring) or [product data aggregation](https://webscraping.ai/use-cases/product-data-aggregation) pipeline. ## Rust vs Python vs Go for scraping | | Rust | Python | Go | | --- | --- | --- | --- | | Ecosystem | Thin — `scraper`, no Scrapy equivalent | By far the largest | Moderate — Colly is solid | | Concurrency cost | Kilobytes per in-flight request | Heavy threads, or asyncio | ~KB per goroutine | | HTML parsing | `scraper` (CSS only) | lxml, BeautifulSoup, full XPath | goquery (CSS only) | | Browser automation | Community CDP crates | Playwright, Selenium, first-class | chromedp, decent | | Iteration speed | Slow — compile on every change | Fastest | Fast | | Deployment | Single static binary | Interpreter + dependencies | Single static binary | | Anti-bot tooling | Essentially none | Largest selection | Limited | Go is Rust's closest competitor for this workload and usually the more pragmatic choice: goroutines give you the same cheap concurrency and the same single-binary deploy, with much faster iteration and no borrow checker in your way during the exploratory phase. Rust wins where the post-fetch work is heavy — millions of records an hour, tight memory budgets, no GC pauses — or where the scraper is one component of a larger Rust system. Python wins on everything except runtime cost, and runtime cost is usually not the binding constraint. If you're unsure which you're in, start in [Python](https://webscraping.ai/blog/web-scraping-with-python), measure, and port the hot stage if you actually find one. ## Frequently asked questions **Is Rust a good language for web scraping?** For high-volume crawling, yes — bounded-concurrency fan-out with `reqwest` costs kilobytes per in-flight request, so a single process handles concurrency levels that need a worker fleet elsewhere. For a one-off script, Python is still faster to write and has a far larger scraping ecosystem. The honest split: Rust when throughput or long-running memory behaviour is the constraint, Python when developer time is. **Does the scraper crate support XPath?** No. `scraper` is CSS-selectors only, via Servo's `selectors` crate. The `skyscraper` crate implements XPath for HTML but sees very little use; `libxml` bindings work at the cost of a C dependency. Rewriting expressions as CSS selectors is the path of least resistance. **Why does my scraper code fail to compile inside tokio::spawn?** `scraper::Html` and `Selector` are `!Send`, so any future holding one across an `.await` cannot be spawned onto a multi-threaded runtime. Scope the parsing so the document is dropped before the next await point and return owned `String`s or your own structs, or move the whole parse into `spawn_blocking`. **Why do I get UnknownIssuer errors in Rust when curl works?** Almost always the `rustls-tls` feature: it trusts a bundled Mozilla root store, not your operating system's, so a corporate CA you installed system-wide is invisible to it. Switch to `rustls-tls-native-roots`, or add the CA explicitly with `add_root_certificate` — using `Certificate::from_pem_bundle` if the file contains more than one certificate. **Is danger\_accept\_invalid\_certs safe to use in a scraper?** Not as a default. It disables expiry, issuer, and hostname checks together, which lets anything on the network path intercept and modify your traffic — a real risk when your requests already route through third-party proxies. Prefer `danger_accept_invalid_hostnames` for a hostname mismatch, or add the missing CA. If you do need it, scope it to one client rather than the one you use everywhere. **Can Rust extract tables from PDFs?** Not well. `pdf-extract` and `lopdf` give you a text stream, and there is no Rust equivalent of pdfplumber or Camelot for reconstructing table structure. You rebuild columns from whitespace and ordering yourself, per layout. If tables are the point of the project, run that stage in Python. **chromiumoxide or headless\_chrome?** `chromiumoxide` if your pipeline is async — it is tokio-native and currently the more-downloaded of the two. `headless_chrome` if you want a blocking API and the smallest possible amount of code. Both speak the DevTools Protocol to the same Chrome binary; the difference is ergonomics, not capability. **Can Rust scrape JavaScript-rendered pages without a browser?** Not in general. There is no Rust equivalent of a JavaScript engine wired to a DOM that survives real-world SPA code. Two things often work instead: find the JSON API the page's own JavaScript calls and hit it directly with `reqwest`, or look for ` __NEXT_DATA__ ` / other embedded state in the HTML. Both are faster than any browser. When neither exists, you need a real browser or a rendering API. **How do I handle cookies and logins in Rust?** `reqwest::ClientBuilder::cookie_store(true)` (with the `cookies` feature) persists `Set-Cookie` across requests on that client, which covers session-based logins. Post the form with `.form(&[("username", u), ("password", p)])` and every later request on the same client carries the session.