---
title: "Rust Reqwest Guide: HTTP Requests, JSON, Timeouts, and Retries"
description: "Practical reqwest tutorial for Rust: async and blocking clients, JSON with serde, timeouts, error handling, retries, streaming, mocking, and debugging."
url: https://webscraping.ai/blog/rust-reqwest-guide
markdown_index: https://webscraping.ai/llms.txt
---
# Rust Reqwest Guide: HTTP Requests, JSON, Timeouts, and Retries

[reqwest](https://docs.rs/reqwest) is Rust's batteries-included HTTP client: built on [hyper](https://hyper.rs/), async-first with an optional blocking API, and handling TLS, JSON, cookies, redirects, and connection pooling out of the box. It's the crate almost every Rust project doing HTTP reaches for first. This guide goes from `cargo add reqwest` to the production topics — timeouts, typed error handling, retries with middleware, streaming, mocking in tests, and how to see what your client is actually sending.

## Key Takeaways

- Use one `Client` for your whole program — it holds the connection pool; creating a client per request throws pooling away
- reqwest is async by default (bring tokio); the `blocking` feature gives a synchronous API for CLIs and scripts
- JSON in and out is one `.json()` call each way with the `json` feature and serde derive
- reqwest doesn't fail on HTTP 404/500 — call `.error_for_status()` if you want statuses as errors
- Retries aren't built in: the `reqwest-middleware` + `reqwest-retry` crates add exponential backoff cleanly
- `RUST_LOG=reqwest=debug` plus `.connection_verbose(true)` answers most "what is it actually sending?" questions

## Installing reqwest

```bash
cargo add reqwest --features json
cargo add tokio --features full # async runtime
cargo add serde --features derive # for typed JSON
```

Features you'll actually toggle:

| Feature | What it enables |
| --- | --- |
| `json` | `.json()` request/response helpers via serde |
| `blocking` | Synchronous `reqwest::blocking` API |
| `cookies` | Automatic cookie store per client |
| `stream` | Response/request bodies as async streams |
| `multipart` | File-upload form bodies |
| `gzip`, `brotli`, `deflate`, `zstd` | Transparent decompression |
| `rustls-tls` | Pure-Rust TLS instead of the platform's native TLS |
| `socks` | SOCKS4/5 proxy support |

`rustls-tls` deserves a note: it removes the OpenSSL system dependency, which is the standard fix for cross-compilation and slim Docker builds (`default-features = false, features = ["rustls-tls"]`).

## GET requests and query parameters

```rust
use std::collections::HashMap;

#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
    let client = reqwest::Client::new();

    let resp = client
        .get("https://httpbin.org/get")
        .query(&[("q", "web scraping"), ("page", "2")]) // ?q=web+scraping&page=2
        .header("Accept", "application/json")
        .send()
        .await?;

    println!("status: {}", resp.status());
    let body = resp.text().await?;
    println!("{body}");
    Ok(())
}
```

`.query()` takes anything serializable as key/value pairs — slices of tuples, `HashMap`s, or a serde struct — and percent-encodes for you. Values that need encoding (`spaces`, `&`, unicode) are handled; never format query strings by hand.

The one-liner `reqwest::get(url)` exists for quick experiments, but it builds a throwaway client each call — in real code, **create one `Client` and reuse it everywhere** (it's cheaply clonable; clones share the same pool).

## Blocking client (no async required)

For CLIs, build scripts, and simple tools, skip tokio entirely with the `blocking` feature:

```rust
fn main() -> Result<(), reqwest::Error> {
    let client = reqwest::blocking::Client::new();
    let body = client.get("https://httpbin.org/get").send()?.text()?;
    println!("{body}");
    Ok(())
}
```

Same builder API, no `.await`. Two rules: don't call blocking reqwest inside an async runtime (it panics with "Cannot drop a runtime in a context where blocking is not allowed" — it spins up its own tokio internally), and prefer the async client anywhere you're already async.

## JSON with serde

```rust
use serde::{Deserialize, Serialize};

#[derive(Serialize)]
struct NewPost { title: String, body: String }

#[derive(Deserialize, Debug)]
struct Post { id: u64, title: String }

let created: Post = client
    .post("https://api.example.com/posts")
    .json(&NewPost { title: "hi".into(), body: "…".into() }) // serializes + sets Content-Type
    .send()
    .await?
    .json() // deserializes the response
    .await?;
```

Deserialization failures surface as `reqwest::Error` with `is_decode() == true` — which is also what you get when a server returns an HTML error page where JSON was expected. When debugging "expected value at line 1", read the body as text first and look at what actually came back. For exploratory JSON without defining structs, deserialize into `serde_json::Value`.

Form posts are symmetric: `.form(&[("user", "vlad")])` sends `application/x-www-form-urlencoded`. Multipart uploads (with the `multipart` feature):

```rust
let form = reqwest::multipart::Form::new()
    .text("description", "quarterly report")
    .part("file", reqwest::multipart::Part::bytes(std::fs::read("report.pdf")?)
        .file_name("report.pdf")
        .mime_str("application/pdf")?);
let resp = client.post(url).multipart(form).send().await?;
```

## Configuring the client: timeouts, redirects, cookies, proxies

All the durable settings live on `ClientBuilder`:

```rust
use std::time::Duration;

let client = reqwest::Client::builder()
    .timeout(Duration::from_secs(30)) // total per-request deadline
    .connect_timeout(Duration::from_secs(5)) // TCP/TLS handshake budget
    .redirect(reqwest::redirect::Policy::limited(5))
    .cookie_store(true) // needs the `cookies` feature
    .user_agent("my-scraper/1.0")
    .proxy(reqwest::Proxy::all("http://user:pass@proxy.example.com:3128")?)
    .build()?;
```

- **Timeouts** : there is no default request timeout — a hung server hangs your task. Always set one; override per request with `.timeout(...)` on the request builder. Timeouts surface as errors where `.is_timeout()` is true.
- **Redirects** : followed by default up to 10 hops. `Policy::none()` lets you inspect 3xx responses yourself (`resp.status().is_redirection()` + the `Location` header) — the usual choice when scraping login flows, where a 302's `Set-Cookie` matters. Note: reqwest strips `Authorization` headers on cross-host redirects — that's by design, not a bug.
- **Cookies** : with `cookie_store(true)`, `Set-Cookie` responses persist and replay automatically across requests to the same client, which is all that session-based logins need. For pre-seeding cookies, use `reqwest::cookie::Jar` and `ClientBuilder::cookie_provider`.
- **Proxies** : `Proxy::all()` routes everything; `Proxy::http()`/`Proxy::https()` split by scheme; `socks5h://` URLs work with the `socks` feature (the `h` resolves DNS through the proxy). Without explicit configuration, reqwest also honors `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` environment variables. One `Proxy` is one exit IP — rotation is yours to build on top; our [proxy provider comparison](https://webscraping.ai/blog/best-proxy-providers-for-web-scraping) covers where to get pools worth rotating.

Connection pooling is on by default and per-client; tune it only when you have evidence:

```rust
let client = reqwest::Client::builder()
    .pool_max_idle_per_host(20) // default is unlimited idle
    .pool_idle_timeout(Duration::from_secs(90))
    .tcp_keepalive(Duration::from_secs(60))
    .build()?;
```

For high-concurrency scraping against one host, `pool_max_idle_per_host` roughly matching your concurrency keeps handshakes rare; servers that drop idle connections mid-reuse (sporadic "connection closed before message completed") are fixed by lowering `pool_idle_timeout`.

## Error handling done properly

`reqwest::Error` is one type with classification methods, and — important — **HTTP error statuses are not Rust errors** :

```rust
match client.get(url).send().await {
    Ok(resp) if resp.status().is_success() => {
        let body = resp.text().await?;
        // …
    }
    Ok(resp) => eprintln!("HTTP {}: server answered, unhappily", resp.status()),
    Err(e) if e.is_timeout() => eprintln!("timed out: {e}"),
    Err(e) if e.is_connect() => eprintln!("connection failed (DNS/TCP/TLS): {e}"),
    Err(e) => eprintln!("other transport error: {e}"),
}
```

When "non-2xx should just be an error" is the right policy, `resp.error_for_status()?` converts 4xx/5xx into a `reqwest::Error` (with `.status()` populated) and passes success responses through. In application code, wrap transport errors into your own error enum with `thiserror`, or bubble everything as `anyhow::Error` in binaries — both compose fine with `?` since `reqwest::Error` implements `std::error::Error`.

## Retries with exponential backoff

reqwest has no built-in retry; the ecosystem answer is `reqwest-middleware` + `reqwest-retry`:

```rust
use reqwest_middleware::ClientBuilder;
use reqwest_retry::{RetryTransientMiddleware, policies::ExponentialBackoff};

let retry_policy = ExponentialBackoff::builder().build_with_max_retries(3);
let client = ClientBuilder::new(reqwest::Client::new())
    .with(RetryTransientMiddleware::new_with_policy(retry_policy))
    .build();

let resp = client.get(url).send().await?; // now retries 5xx/connect errors with backoff
```

It retries transient failures (connection errors, 5xx, 429) and leaves deterministic ones alone. For custom rules — say, retrying only idempotent methods, or honoring `Retry-After` yourself — a hand-rolled loop is a dozen lines with `tokio::time::sleep`; keep jitter in the delay so a fleet of scrapers doesn't synchronize its hammering.

## Streaming large downloads

Buffering a 2 GB file through `.bytes()` holds 2 GB in RAM. Stream instead (feature `stream`, plus the `futures` crate for `StreamExt`):

```rust
use futures_util::StreamExt;
use tokio::io::AsyncWriteExt;

let resp = client.get(file_url).send().await?.error_for_status()?;
let mut out = tokio::fs::File::create("dump.bin").await?;
let mut stream = resp.bytes_stream();
while let Some(chunk) = stream.next().await {
    out.write_all(&chunk?).await?;
}
```

The same idea powers progress bars (`content_length()` for the total), resumable downloads (`Range` headers), and bounded-memory scraping of huge exports. Uploads stream too: `Body::wrap_stream()` accepts any `Stream` of bytes.

## Concurrency: fetching many URLs

The idiomatic bounded-concurrency fan-out:

```rust
use futures_util::{stream, StreamExt};

let client = reqwest::Client::new();
let results: Vec<_> = stream::iter(urls)
    .map(|url| {
        let client = client.clone(); // clones share the pool
        async move {
            let resp = client.get(&url).send().await?;
            Ok::<_, reqwest::Error>((url, resp.status()))
        }
    })
    .buffer_unordered(20) // at most 20 in flight
    .collect()
    .await;
```

`buffer_unordered` is the knob that keeps you fast _and_ polite — unbounded `join_all` over ten thousand URLs is how you get banned and how you exhaust file descriptors, in that order.

## Logging and debugging

reqwest logs through `tracing`/`log`. Wire up any subscriber and raise the filter:

```bash
RUST_LOG=reqwest=debug,hyper=debug cargo run
```

```rust
tracing_subscriber::fmt::init(); // or env_logger::init();
```

That shows request lifecycles, redirects, and pool activity. Two more tools when that's not enough:

- `.connection_verbose(true)` on `ClientBuilder` — logs read/write events on connections (`trace` level), close to a wire dump
- Point the client at a local debugging proxy (mitmproxy: `Proxy::all("http://127.0.0.1:8080")` + accepting its CA, or `danger_accept_invalid_certs(true)` strictly in a debug build) to inspect full requests and responses in a UI

Printing the almost-final request without sending it also works: `client.get(url).build()?` returns the `Request`, whose method/URL/headers you can log.

## Testing: mock the server, not the client

Because reqwest talks real HTTP, the clean testing pattern is a local mock server — no traits to abstract, just a URL you control. With `wiremock`:

```rust
use wiremock::{MockServer, Mock, ResponseTemplate};
use wiremock::matchers::{method, path};

#[tokio::test]
async fn fetches_user() {
    let server = MockServer::start().await;
    Mock::given(method("GET")).and(path("/users/1"))
        .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
            "id": 1, "name": "Ada"
        })))
        .mount(&server)
        .await;

    let user: User = reqwest::get(format!("{}/users/1", server.uri()))
        .await.unwrap().json().await.unwrap();
    assert_eq!(user.name, "Ada");
}
```

`mockito` and `httpmock` fill the same role with different ergonomics. The structural requirement for all of them: your code must take the base URL as configuration rather than hardcoding it — which is a refactor worth doing anyway.

## reqwest vs the other Rust HTTP clients

| Crate | Model | When to pick it |
| --- | --- | --- |
| **reqwest** | Async (+ blocking), high-level, on hyper | Default choice; richest feature set and ecosystem |
| **hyper** | Async, low-level | Building infrastructure (proxies, servers, custom clients); you assemble TLS/pooling/redirects yourself |
| **ureq** | Sync only, minimal deps | CLIs and tools where you want tiny builds and no async runtime |
| **curl** (bindings) | Sync, libcurl | Need a libcurl behavior exactly; FFI to the most battle-tested client alive |

If you're deciding for a scraper: reqwest, plus the [scraper](https://crates.io/crates/scraper) crate for CSS-selector HTML parsing — that pairing is covered end-to-end in our [Rust web scraping guide](https://webscraping.ai/blog/web-scraping-with-rust).

## Scraping with reqwest, and when it's not enough

reqwest happily pulls static HTML at very high throughput. What it can't do is execute JavaScript or make a datacenter IP look residential — the two walls every serious scraper hits. When the response is an empty `<div id="root">` or a bot-detection page, the fix isn't more headers; it's a rendering pipeline with rotating proxies. [WebScraping.AI](https://webscraping.ai/) provides exactly that behind an HTTP API, so the Rust side stays plain reqwest:

```rust
#[derive(serde::Deserialize)]
struct Product { name: String, price: String }

let product: Product = client
    .get("https://api.webscraping.ai/ai/fields")
    .query(&[
        ("api_key", api_key.as_str()),
        ("url", "https://example.com/product/42"),
        ("fields[name]", "Product name"),
        ("fields[price]", "Price with currency"),
    ])
    .send().await?
    .error_for_status()?
    .json().await?;
```

The [/html endpoint](https://webscraping.ai/docs) returns rendered pages for your own parsing; [/ai/fields](https://webscraping.ai/ai-web-scraping) skips parsing entirely and returns the structured data — a good division of labor with Rust doing what it's best at: fast, safe processing of the results. It's the shape most continuous pipelines end up with, whether that's [price monitoring](https://webscraping.ai/use-cases/price-monitoring) or feeding a [RAG knowledge base](https://webscraping.ai/use-cases/rag-knowledge-base): a `buffer_unordered` fan-out over the API, results deserialized straight into your structs.

## Frequently asked questions

**Is reqwest async or blocking?** Async-first: the primary API returns futures and expects a tokio runtime. The `blocking` feature adds `reqwest::blocking::Client` with the same builder API for synchronous code. Don't mix them in one call stack — blocking calls inside an async context panic; use the async client there instead.

**Why does reqwest return Ok for a 404 response?** Because HTTP worked: request out, response in. Transport failures (DNS, TCP, TLS, timeout) are `Err`; status codes are data on the `Ok` response. Call `.error_for_status()` on the response when you want 4xx/5xx converted into errors — it's a deliberate one-liner, not a missing feature.

**How do I set headers on every request?** `ClientBuilder::default_headers` with a `HeaderMap` for arbitrary headers, plus dedicated builders for common ones (`user_agent(...)`). Per-request `.header(...)` overrides defaults on collision. Headers with sensitive values can be marked `set_sensitive(true)` so logging middleware redacts them.

**Does reqwest support HTTP/2?** Yes — negotiated automatically via TLS ALPN when the server supports it, with no configuration. You can inspect the negotiated version with `resp.version()`. HTTP/3 exists behind an unstable feature flag; don't build on it yet.

**How do I add a retry to reqwest?** Use `reqwest-middleware` with `reqwest-retry` for policy-based exponential backoff on transient failures, or write a small loop with `tokio::time::sleep` for custom logic. Retry only idempotent requests blindly; for POSTs, make sure the server tolerates replays (idempotency keys) first.

**Why is my reqwest client slow compared to curl?** Almost always one of: creating a new `Client` per request (no connection reuse — hoist it), missing `--release` (debug builds are dramatically slower at TLS), or DNS — try `hickory-dns` via the corresponding feature. With a reused client and a release build, reqwest benchmarks at the same order as curl.