Web Scraping with Rust: reqwest, scraper, and headless_chrome

Rust is an increasingly popular choice for scraping infrastructure: no garbage-collector pauses, tiny memory footprints per task, and fearless concurrency for crawling thousands of pages in parallel. The ecosystem has matured into a standard stack — reqwest for HTTP, scraper for HTML parsing, and headless_chrome or fantoccini when you need a real browser. This guide walks through all three layers with working code.

Key Takeaways

• reqwest + scraper covers server-rendered sites with excellent performance
• The scraper crate uses Servo's real CSS engine, so selectors behave exactly like a browser's
• headless_chrome drives a real Chrome instance via DevTools Protocol for JavaScript-heavy pages
• tokio makes highly concurrent crawlers straightforward and memory-cheap
• Rust doesn't exempt you from anti-bot systems — protected sites still need rotating proxies or a scraping API

The HTTP Layer: reqwest

Add the essentials to Cargo.toml:

[dependencies]
reqwest = { version = "0.12", features = ["gzip"] }
scraper = "0.20"
tokio = { version = "1", features = ["full"] }

Fetching and parsing a page:

use scraper::{Html, Selector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::builder()
        .user_agent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36")
        .build()?;

    let html = client
        .get("https://quotes.toscrape.com/")
        .send()
        .await?
        .text()
        .await?;

    let doc = Html::parse_document(&html);
    let quote_sel = Selector::parse(".quote .text").unwrap();

    for quote in doc.select(&quote_sel) {
        println!("{}", quote.text().collect::<String>());
    }
    Ok(())
}

Because scraper is built on Servo's html5ever and selectors crates, parsing is spec-correct — malformed HTML is handled the same way Firefox handles it.

Concurrent Crawling with tokio

Rust's async model makes a polite-but-fast crawler compact:

use futures::stream::{self, StreamExt};

let urls: Vec<String> = /* your URL list */;
let results: Vec<_> = stream::iter(urls)
    .map(|url| {
        let client = client.clone();
        async move { client.get(&url).send().await?.text().await }
    })
    .buffer_unordered(10) // 10 concurrent requests
    .collect()
    .await;

Each in-flight request costs kilobytes, not megabytes — this is where Rust pulls decisively ahead of Python for large crawls.

JavaScript Pages: the headless_chrome Crate

For client-side-rendered sites, headless_chrome controls a real Chrome/Chromium instance over the DevTools Protocol — think Puppeteer, but Rust:

use headless_chrome::{Browser, LaunchOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let browser = Browser::new(LaunchOptions::default_builder()
        .headless(true)
        .build()?)?;

    let tab = browser.new_tab()?;
    tab.navigate_to("https://example.com/spa")?;
    tab.wait_for_element(".loaded-content")?;

    let html = tab.get_content()?;
    println!("{}", &html[..500.min(html.len())]);
    Ok(())
}

It supports navigation, waiting for elements, clicking, typing, screenshots, and request interception. The main alternative is fantoccini, which speaks WebDriver instead of DevTools and works with Firefox too. headless_chrome is the more direct Puppeteer equivalent; fantoccini is async-first and fits tokio pipelines more naturally.

The usual headless-browser caveats apply: each Chrome instance eats hundreds of megabytes, and a default headless Chrome is trivially fingerprintable by serious anti-bot systems.

Dealing With Blocks

None of Rust's performance helps when Cloudflare returns a challenge page in 50ms. For protected targets, the pragmatic setup is Rust for orchestration and parsing, with fetching delegated to a scraping API that handles proxies, fingerprints, and JavaScript rendering:

let html = client
    .get("https://api.webscraping.ai/html")
    .query(&[
        ("api_key", "YOUR_API_KEY"),
        ("url", "https://example.com/protected-page"),
        ("js", "true"),
        ("proxy", "residential"),
    ])
    .send()
    .await?
    .text()
    .await?;
// parse with scraper as usual

You keep Rust's concurrency and type safety for the pipeline; the API absorbs the anti-bot arms race.

Conclusion

Rust's scraping stack is production-ready: reqwest and scraper for the fast path, headless_chrome or fantoccini when JavaScript is unavoidable, and tokio to run it all at high concurrency for very little memory. Combine it with a web scraping API for protected sites and you get the best of both worlds — Rust-speed processing with none of the block-evasion maintenance. Our Rust FAQ section and reqwest FAQ cover the narrower questions this guide skipped.