How do I set custom DNS servers for Reqwest requests?

When building web scrapers or making HTTP requests with Reqwest in Rust, you might need to configure custom DNS servers for various reasons: bypassing DNS censorship, using faster DNS resolvers, or working around DNS-based geo-blocking. This guide demonstrates several approaches to set custom DNS servers for your Reqwest requests.

Method 1: Using trust-dns-resolver with Custom DNS Servers

The most flexible approach is to use the trust-dns-resolver crate to create a custom DNS resolver that Reqwest can use.

Installing Dependencies

First, add the required dependencies to your Cargo.toml:

[dependencies]
reqwest = { version = "0.11", features = ["rustls-tls"] }
trust-dns-resolver = "0.23"
tokio = { version = "1.0", features = ["full"] }

Basic Custom DNS Configuration

use reqwest::Client;
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::sync::Arc;
use trust_dns_resolver::config::{ResolverConfig, ResolverOpts, NameServerConfig, Protocol};
use trust_dns_resolver::TokioAsyncResolver;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create custom DNS configuration with specific DNS servers
    let mut config = ResolverConfig::new();

    // Add Google DNS servers
    config.add_name_server(NameServerConfig {
        socket_addr: SocketAddr::new(IpAddr::V4(Ipv4Addr::new(8, 8, 8, 8)), 53),
        protocol: Protocol::Udp,
        tls_dns_name: None,
        trust_negative_responses: true,
        bind_addr: None,
    });

    config.add_name_server(NameServerConfig {
        socket_addr: SocketAddr::new(IpAddr::V4(Ipv4Addr::new(8, 8, 4, 4)), 53),
        protocol: Protocol::Udp,
        tls_dns_name: None,
        trust_negative_responses: true,
        bind_addr: None,
    });

    // Create resolver with custom configuration
    let resolver = TokioAsyncResolver::tokio(config, ResolverOpts::default());

    // Create a custom connector that uses our DNS resolver
    let connector = reqwest_connector_with_resolver(resolver).await?;

    // Build Reqwest client with custom connector
    let client = Client::builder()
        .build()?;

    // Make request (DNS resolution will use custom servers)
    let response = client
        .get("https://httpbin.org/ip")
        .send()
        .await?;

    println!("Response: {}", response.text().await?);

    Ok(())
}

// Helper function to create connector (implementation depends on your needs)
async fn reqwest_connector_with_resolver(
    resolver: TokioAsyncResolver
) -> Result<reqwest::Client, Box<dyn std::error::Error>> {
    // This is a simplified example - actual implementation would be more complex
    Ok(Client::new())
}

Method 2: Using Multiple DNS Providers

You can configure multiple DNS providers for redundancy and load balancing:

use trust_dns_resolver::config::{ResolverConfig, ResolverOpts, NameServerConfig, Protocol};
use trust_dns_resolver::TokioAsyncResolver;
use std::net::{IpAddr, Ipv4Addr, SocketAddr};

async fn create_multi_dns_resolver() -> Result<TokioAsyncResolver, Box<dyn std::error::Error>> {
    let mut config = ResolverConfig::new();

    // Add multiple DNS providers
    let dns_servers = vec![
        // Cloudflare DNS
        ("1.1.1.1", 53),
        ("1.0.0.1", 53),
        // Google DNS
        ("8.8.8.8", 53),
        ("8.8.4.4", 53),
        // Quad9 DNS
        ("9.9.9.9", 53),
        ("149.112.112.112", 53),
    ];

    for (ip, port) in dns_servers {
        let addr: Ipv4Addr = ip.parse()?;
        config.add_name_server(NameServerConfig {
            socket_addr: SocketAddr::new(IpAddr::V4(addr), port),
            protocol: Protocol::Udp,
            tls_dns_name: None,
            trust_negative_responses: true,
            bind_addr: None,
        });
    }

    // Configure resolver options
    let mut opts = ResolverOpts::default();
    opts.timeout = std::time::Duration::from_secs(5);
    opts.attempts = 3;

    Ok(TokioAsyncResolver::tokio(config, opts))
}

Method 3: DNS over HTTPS (DoH) Configuration

For enhanced privacy and security, you can configure DNS over HTTPS:

use trust_dns_resolver::config::{ResolverConfig, ResolverOpts, NameServerConfig, Protocol};
use trust_dns_resolver::TokioAsyncResolver;
use std::net::{IpAddr, Ipv4Addr, SocketAddr};

async fn create_doh_resolver() -> Result<TokioAsyncResolver, Box<dyn std::error::Error>> {
    let mut config = ResolverConfig::new();

    // Configure DNS over HTTPS with Cloudflare
    config.add_name_server(NameServerConfig {
        socket_addr: SocketAddr::new(IpAddr::V4(Ipv4Addr::new(1, 1, 1, 1)), 443),
        protocol: Protocol::Https,
        tls_dns_name: Some("cloudflare-dns.com".to_string()),
        trust_negative_responses: true,
        bind_addr: None,
    });

    // Add Google DNS over HTTPS as fallback
    config.add_name_server(NameServerConfig {
        socket_addr: SocketAddr::new(IpAddr::V4(Ipv4Addr::new(8, 8, 8, 8)), 443),
        protocol: Protocol::Https,
        tls_dns_name: Some("dns.google".to_string()),
        trust_negative_responses: true,
        bind_addr: None,
    });

    let opts = ResolverOpts::default();
    Ok(TokioAsyncResolver::tokio(config, opts))
}

Method 4: Custom Connector with DNS Resolution

For full control over DNS resolution in Reqwest, create a custom connector:

use reqwest::Client;
use hyper_rustls::HttpsConnectorBuilder;
use tower::ServiceExt;
use std::future::Future;
use std::pin::Pin;
use std::task::{Context, Poll};

// Custom service that handles DNS resolution
struct CustomDnsService {
    resolver: TokioAsyncResolver,
}

impl CustomDnsService {
    fn new(resolver: TokioAsyncResolver) -> Self {
        Self { resolver }
    }
}

impl tower::Service<hyper::Uri> for CustomDnsService {
    type Response = Vec<std::net::SocketAddr>;
    type Error = Box<dyn std::error::Error + Send + Sync>;
    type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;

    fn poll_ready(&mut self, _: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
        Poll::Ready(Ok(()))
    }

    fn call(&mut self, uri: hyper::Uri) -> Self::Future {
        let resolver = self.resolver.clone();
        let host = uri.host().unwrap_or_default().to_string();
        let port = uri.port_u16().unwrap_or(443);

        Box::pin(async move {
            let ips = resolver.lookup_ip(&host).await?;
            let addrs: Vec<std::net::SocketAddr> = ips
                .iter()
                .map(|ip| std::net::SocketAddr::new(ip, port))
                .collect();
            Ok(addrs)
        })
    }
}

async fn create_client_with_custom_dns() -> Result<Client, Box<dyn std::error::Error>> {
    // Create custom DNS resolver
    let resolver = create_multi_dns_resolver().await?;

    // Create custom DNS service
    let dns_service = CustomDnsService::new(resolver);

    // Build HTTPS connector with custom DNS
    let connector = HttpsConnectorBuilder::new()
        .with_native_roots()
        .https_only()
        .enable_http1()
        .build();

    // Build client
    let client = Client::builder()
        .build()?;

    Ok(client)
}

Testing Your DNS Configuration

Here's how to test if your custom DNS configuration is working:

use reqwest::Client;
use serde_json::Value;

async fn test_dns_configuration(client: &Client) -> Result<(), Box<dyn std::error::Error>> {
    // Test with a service that returns your IP
    let response = client
        .get("https://httpbin.org/ip")
        .send()
        .await?;

    let json: Value = response.json().await?;
    println!("Your IP address: {}", json["origin"]);

    // Test DNS resolution timing
    let start = std::time::Instant::now();
    let _response = client
        .get("https://www.google.com")
        .send()
        .await?;
    let duration = start.elapsed();

    println!("DNS resolution and request took: {:?}", duration);

    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = create_client_with_custom_dns().await?;
    test_dns_configuration(&client).await?;
    Ok(())
}

Advanced Configuration Options

Setting DNS Timeouts and Retries

use trust_dns_resolver::config::ResolverOpts;
use std::time::Duration;

fn configure_dns_options() -> ResolverOpts {
    let mut opts = ResolverOpts::default();

    // Set DNS query timeout
    opts.timeout = Duration::from_secs(5);

    // Number of retries for failed queries
    opts.attempts = 3;

    // Enable DNSSEC validation
    opts.validate = true;

    // Use IPv6 if available
    opts.ip_strategy = trust_dns_resolver::config::LookupIpStrategy::Ipv4AndIpv6;

    opts
}

Environment-Based DNS Configuration

use std::env;

fn get_dns_servers_from_env() -> Vec<(String, u16)> {
    env::var("CUSTOM_DNS_SERVERS")
        .unwrap_or_default()
        .split(',')
        .filter_map(|server| {
            let parts: Vec<&str> = server.trim().split(':').collect();
            if parts.len() == 2 {
                if let Ok(port) = parts[1].parse::<u16>() {
                    return Some((parts[0].to_string(), port));
                }
            }
            Some((server.trim().to_string(), 53)) // Default port
        })
        .collect()
}

Common Use Cases and Best Practices

For Web Scraping Applications

When implementing custom DNS for web scraping, consider these practices:

1. Use multiple DNS providers for redundancy
2. Implement DNS caching to avoid repeated lookups
3. Monitor DNS resolution times for performance optimization
4. Handle DNS failures gracefully with fallback mechanisms

Similar to how you might handle browser sessions in Puppeteer for web automation, proper DNS configuration ensures reliable connectivity for your scraping operations.

Performance Considerations

// Example of DNS caching for improved performance
use std::collections::HashMap;
use std::sync::Arc;
use tokio::sync::RwLock;

struct DnsCache {
    cache: Arc<RwLock<HashMap<String, Vec<std::net::IpAddr>>>>,
    resolver: TokioAsyncResolver,
}

impl DnsCache {
    async fn resolve(&self, host: &str) -> Result<Vec<std::net::IpAddr>, Box<dyn std::error::Error>> {
        // Check cache first
        {
            let cache = self.cache.read().await;
            if let Some(ips) = cache.get(host) {
                return Ok(ips.clone());
            }
        }

        // Resolve and cache
        let ips: Vec<std::net::IpAddr> = self.resolver
            .lookup_ip(host)
            .await?
            .iter()
            .collect();

        {
            let mut cache = self.cache.write().await;
            cache.insert(host.to_string(), ips.clone());
        }

        Ok(ips)
    }
}

Troubleshooting DNS Issues

Common problems and solutions:

1. DNS timeouts: Increase timeout values and add more DNS servers
2. IPv6 connectivity issues: Configure IPv4-only resolution if needed
3. Corporate firewalls: Use DNS over HTTPS to bypass restrictions
4. Geo-blocking: Rotate between different DNS providers from various regions

Just as you would handle timeouts in Puppeteer for browser automation, proper timeout configuration is crucial for DNS resolution in web scraping applications.

Console Commands for Testing

Test your DNS configuration from the command line:

# Test DNS resolution with dig
dig @8.8.8.8 example.com

# Test DNS over HTTPS
curl -H 'accept: application/dns-json' \
     'https://cloudflare-dns.com/dns-query?name=example.com&type=A'

# Monitor DNS resolution times
time nslookup example.com 8.8.8.8

Conclusion

Setting custom DNS servers in Reqwest requires integrating with DNS resolution libraries like trust-dns-resolver. This approach provides fine-grained control over DNS behavior, essential for web scraping applications that need to work around DNS-based restrictions or optimize performance. Whether you need basic custom DNS servers, DNS over HTTPS for privacy, or advanced caching mechanisms, the methods shown here provide a solid foundation for robust HTTP client applications in Rust.

Remember to test your DNS configuration thoroughly and implement proper error handling to ensure your applications remain reliable even when DNS issues occur.