How do I customize the user-agent string in Reqwest?

In Rust, when you're using the Reqwest library to make HTTP requests, you may want to customize the User-Agent string to identify your HTTP client. The User-Agent string is a header that is sent with each request, which allows the server to identify the type of client making the request.

To customize the User-Agent string in Reqwest, you can use the header method of the RequestBuilder to set the User-Agent header to your desired value. Here's an example:

use reqwest::header::{HeaderMap, USER_AGENT};

#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
    let client = reqwest::Client::new();

    // Create a custom User-Agent string
    let custom_user_agent = "MyCustomUserAgent/1.0";

    // Add it to a HeaderMap
    let mut headers = HeaderMap::new();
    headers.insert(USER_AGENT, custom_user_agent.parse().unwrap());

    // Use the HeaderMap in the request
    let res = client.get("https://httpbin.org/get")
        .headers(headers)
        .send()
        .await?;

    // Print the response
    println!("Status: {}", res.status());
    println!("Headers:\n{:#?}", res.headers());
    println!("Body:\n{}", res.text().await?);

    Ok(())
}

In this example, we are using the header::HeaderMap to create a set of headers, then inserting our custom User-Agent string into that set. The headers method of the RequestBuilder takes this map and applies it to the request.

Remember that servers might use the User-Agent string to apply different logic, for instance, to tailor responses for different browsers or to block requests from bots. Always respect the terms of service of the websites you are scraping and avoid misrepresenting your User-Agent string in a misleading or malicious manner.

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon