How do I implement request caching in Rust web scraping?

Request caching is a crucial optimization technique for web scraping applications that helps reduce server load, improve response times, and avoid unnecessary network requests. In Rust, you can implement various caching strategies using built-in data structures, file systems, or external cache stores like Redis.

Why Use Request Caching?

Request caching provides several benefits for web scraping applications:

• Performance improvement: Cached responses are served instantly without network delays
• Reduced server load: Fewer requests to target websites prevent rate limiting
• Cost optimization: Lower bandwidth usage and API call costs
• Offline capabilities: Access to previously fetched data when network is unavailable
• Resilience: Fallback data when websites are temporarily unavailable

Basic In-Memory Caching

The simplest caching approach uses Rust's HashMap to store responses in memory:

use reqwest;
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio;

#[derive(Clone)]
pub struct CacheEntry {
    data: String,
    timestamp: Instant,
    ttl: Duration,
}

impl CacheEntry {
    pub fn new(data: String, ttl: Duration) -> Self {
        Self {
            data,
            timestamp: Instant::now(),
            ttl,
        }
    }

    pub fn is_expired(&self) -> bool {
        self.timestamp.elapsed() > self.ttl
    }
}

#[derive(Clone)]
pub struct MemoryCache {
    store: Arc<Mutex<HashMap<String, CacheEntry>>>,
    default_ttl: Duration,
}

impl MemoryCache {
    pub fn new(default_ttl: Duration) -> Self {
        Self {
            store: Arc::new(Mutex::new(HashMap::new())),
            default_ttl,
        }
    }

    pub fn get(&self, key: &str) -> Option<String> {
        let mut store = self.store.lock().unwrap();

        if let Some(entry) = store.get(key) {
            if !entry.is_expired() {
                return Some(entry.data.clone());
            } else {
                store.remove(key);
            }
        }
        None
    }

    pub fn set(&self, key: String, data: String) {
        let mut store = self.store.lock().unwrap();
        let entry = CacheEntry::new(data, self.default_ttl);
        store.insert(key, entry);
    }

    pub fn clear_expired(&self) {
        let mut store = self.store.lock().unwrap();
        store.retain(|_, entry| !entry.is_expired());
    }
}

// Usage example
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cache = MemoryCache::new(Duration::from_secs(300)); // 5 minutes TTL
    let client = reqwest::Client::new();

    let url = "https://httpbin.org/json";

    // Check cache first
    if let Some(cached_data) = cache.get(url) {
        println!("Cache hit: {}", cached_data);
    } else {
        // Fetch from network
        let response = client.get(url).send().await?;
        let data = response.text().await?;

        // Store in cache
        cache.set(url.to_string(), data.clone());
        println!("Cache miss, fetched: {}", data);
    }

    Ok(())
}

Advanced Caching with Reqwest Middleware

For more sophisticated caching, you can create a middleware layer that automatically handles caching for all HTTP requests:

use reqwest::{Client, Request, Response};
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use reqwest::header::{HeaderMap, HeaderValue};

pub struct CachedClient {
    client: Client,
    cache: Arc<Mutex<HashMap<String, CachedResponse>>>,
    default_ttl: Duration,
}

#[derive(Clone)]
struct CachedResponse {
    status: u16,
    headers: HeaderMap,
    body: Vec<u8>,
    timestamp: Instant,
    ttl: Duration,
}

impl CachedResponse {
    fn is_expired(&self) -> bool {
        self.timestamp.elapsed() > self.ttl
    }
}

impl CachedClient {
    pub fn new(default_ttl: Duration) -> Self {
        Self {
            client: Client::new(),
            cache: Arc::new(Mutex::new(HashMap::new())),
            default_ttl,
        }
    }

    pub async fn get(&self, url: &str) -> Result<Response, reqwest::Error> {
        let cache_key = self.generate_cache_key("GET", url, &HeaderMap::new());

        // Check cache first
        if let Some(cached) = self.get_from_cache(&cache_key) {
            return Ok(self.response_from_cache(cached));
        }

        // Fetch from network
        let response = self.client.get(url).send().await?;

        // Store in cache
        self.store_response(&cache_key, &response).await;

        Ok(response)
    }

    fn generate_cache_key(&self, method: &str, url: &str, headers: &HeaderMap) -> String {
        use std::collections::hash_map::DefaultHasher;
        use std::hash::{Hash, Hasher};

        let mut hasher = DefaultHasher::new();
        method.hash(&mut hasher);
        url.hash(&mut hasher);

        // Include relevant headers in cache key
        for (name, value) in headers {
            name.as_str().hash(&mut hasher);
            value.as_bytes().hash(&mut hasher);
        }

        format!("{:x}", hasher.finish())
    }

    fn get_from_cache(&self, key: &str) -> Option<CachedResponse> {
        let mut cache = self.cache.lock().unwrap();

        if let Some(cached) = cache.get(key) {
            if !cached.is_expired() {
                return Some(cached.clone());
            } else {
                cache.remove(key);
            }
        }
        None
    }

    async fn store_response(&self, key: &str, response: &Response) {
        if let Ok(body) = response.bytes().await {
            let cached = CachedResponse {
                status: response.status().as_u16(),
                headers: response.headers().clone(),
                body: body.to_vec(),
                timestamp: Instant::now(),
                ttl: self.default_ttl,
            };

            let mut cache = self.cache.lock().unwrap();
            cache.insert(key.to_string(), cached);
        }
    }

    fn response_from_cache(&self, cached: CachedResponse) -> Response {
        // Note: This is a simplified example
        // In practice, you'd need to properly reconstruct the Response
        unimplemented!("Response reconstruction from cache")
    }
}

File-Based Caching

For persistent caching across application restarts, implement file-based storage:

use serde::{Deserialize, Serialize};
use std::fs;
use std::path::Path;
use std::time::{Duration, SystemTime, UNIX_EPOCH};

#[derive(Serialize, Deserialize)]
struct FileCache {
    data: String,
    timestamp: u64,
    ttl_seconds: u64,
}

impl FileCache {
    fn new(data: String, ttl: Duration) -> Self {
        Self {
            data,
            timestamp: SystemTime::now()
                .duration_since(UNIX_EPOCH)
                .unwrap()
                .as_secs(),
            ttl_seconds: ttl.as_secs(),
        }
    }

    fn is_expired(&self) -> bool {
        let now = SystemTime::now()
            .duration_since(UNIX_EPOCH)
            .unwrap()
            .as_secs();
        now > (self.timestamp + self.ttl_seconds)
    }
}

pub struct FileCacheManager {
    cache_dir: String,
    default_ttl: Duration,
}

impl FileCacheManager {
    pub fn new(cache_dir: &str, default_ttl: Duration) -> Self {
        // Create cache directory if it doesn't exist
        if !Path::new(cache_dir).exists() {
            fs::create_dir_all(cache_dir).unwrap();
        }

        Self {
            cache_dir: cache_dir.to_string(),
            default_ttl,
        }
    }

    pub fn get(&self, key: &str) -> Option<String> {
        let file_path = self.get_file_path(key);

        if let Ok(content) = fs::read_to_string(&file_path) {
            if let Ok(cache_entry) = serde_json::from_str::<FileCache>(&content) {
                if !cache_entry.is_expired() {
                    return Some(cache_entry.data);
                } else {
                    // Remove expired cache file
                    let _ = fs::remove_file(&file_path);
                }
            }
        }
        None
    }

    pub fn set(&self, key: &str, data: String) -> Result<(), Box<dyn std::error::Error>> {
        let file_path = self.get_file_path(key);
        let cache_entry = FileCache::new(data, self.default_ttl);
        let json_content = serde_json::to_string(&cache_entry)?;
        fs::write(file_path, json_content)?;
        Ok(())
    }

    fn get_file_path(&self, key: &str) -> String {
        use std::collections::hash_map::DefaultHasher;
        use std::hash::{Hash, Hasher};

        let mut hasher = DefaultHasher::new();
        key.hash(&mut hasher);
        let hash = format!("{:x}", hasher.finish());

        format!("{}/{}.cache", self.cache_dir, hash)
    }

    pub fn clear_expired(&self) -> Result<(), Box<dyn std::error::Error>> {
        for entry in fs::read_dir(&self.cache_dir)? {
            let entry = entry?;
            if let Ok(content) = fs::read_to_string(entry.path()) {
                if let Ok(cache_entry) = serde_json::from_str::<FileCache>(&content) {
                    if cache_entry.is_expired() {
                        fs::remove_file(entry.path())?;
                    }
                }
            }
        }
        Ok(())
    }
}

// Usage example
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cache = FileCacheManager::new("./cache", Duration::from_secs(3600));
    let client = reqwest::Client::new();

    let url = "https://httpbin.org/json";

    if let Some(cached_data) = cache.get(url) {
        println!("File cache hit: {}", cached_data);
    } else {
        let response = client.get(url).send().await?;
        let data = response.text().await?;
        cache.set(url, data.clone())?;
        println!("Cached to file: {}", data);
    }

    Ok(())
}

Redis-Based Caching

For distributed applications, use Redis for shared caching:

# Cargo.toml
[dependencies]
redis = "0.23"
tokio = { version = "1.0", features = ["full"] }
reqwest = "0.11"
serde_json = "1.0"

use redis::AsyncCommands;
use std::time::Duration;

pub struct RedisCache {
    client: redis::Client,
    default_ttl: Duration,
}

impl RedisCache {
    pub fn new(redis_url: &str, default_ttl: Duration) -> Result<Self, redis::RedisError> {
        let client = redis::Client::open(redis_url)?;
        Ok(Self {
            client,
            default_ttl,
        })
    }

    pub async fn get(&self, key: &str) -> Result<Option<String>, redis::RedisError> {
        let mut conn = self.client.get_async_connection().await?;
        conn.get(key).await
    }

    pub async fn set(&self, key: &str, value: &str) -> Result<(), redis::RedisError> {
        let mut conn = self.client.get_async_connection().await?;
        conn.set_ex(key, value, self.default_ttl.as_secs() as usize).await
    }

    pub async fn exists(&self, key: &str) -> Result<bool, redis::RedisError> {
        let mut conn = self.client.get_async_connection().await?;
        conn.exists(key).await
    }

    pub async fn delete(&self, key: &str) -> Result<(), redis::RedisError> {
        let mut conn = self.client.get_async_connection().await?;
        conn.del(key).await
    }
}

// Web scraper with Redis caching
pub struct CachedScraper {
    client: reqwest::Client,
    cache: RedisCache,
}

impl CachedScraper {
    pub fn new(redis_url: &str) -> Result<Self, redis::RedisError> {
        Ok(Self {
            client: reqwest::Client::new(),
            cache: RedisCache::new(redis_url, Duration::from_secs(3600))?,
        })
    }

    pub async fn fetch_url(&self, url: &str) -> Result<String, Box<dyn std::error::Error>> {
        // Check cache first
        if let Some(cached_content) = self.cache.get(url).await? {
            println!("Redis cache hit for: {}", url);
            return Ok(cached_content);
        }

        // Fetch from network
        println!("Fetching from network: {}", url);
        let response = self.client.get(url).send().await?;
        let content = response.text().await?;

        // Store in cache
        self.cache.set(url, &content).await?;

        Ok(content)
    }
}

Smart Caching Strategies

Implement intelligent caching based on HTTP headers and response characteristics:

use reqwest::header::{CACHE_CONTROL, ETAG, LAST_MODIFIED};
use std::time::Duration;

pub struct SmartCache {
    storage: MemoryCache,
}

impl SmartCache {
    pub fn new() -> Self {
        Self {
            storage: MemoryCache::new(Duration::from_secs(3600)),
        }
    }

    pub fn determine_ttl(&self, response: &reqwest::Response) -> Duration {
        // Check Cache-Control header
        if let Some(cache_control) = response.headers().get(CACHE_CONTROL) {
            if let Ok(cache_control_str) = cache_control.to_str() {
                // Parse max-age directive
                for directive in cache_control_str.split(',') {
                    let directive = directive.trim();
                    if directive.starts_with("max-age=") {
                        if let Ok(seconds) = directive[8..].parse::<u64>() {
                            return Duration::from_secs(seconds);
                        }
                    }
                    // Handle no-cache directive
                    if directive == "no-cache" || directive == "no-store" {
                        return Duration::from_secs(0);
                    }
                }
            }
        }

        // Default TTL based on content type
        if let Some(content_type) = response.headers().get("content-type") {
            if let Ok(content_type_str) = content_type.to_str() {
                return match content_type_str {
                    ct if ct.contains("text/html") => Duration::from_secs(300),      // 5 minutes
                    ct if ct.contains("application/json") => Duration::from_secs(60), // 1 minute
                    ct if ct.contains("image/") => Duration::from_secs(3600),        // 1 hour
                    _ => Duration::from_secs(1800), // 30 minutes default
                };
            }
        }

        Duration::from_secs(1800) // Default 30 minutes
    }

    pub fn generate_etag_key(&self, url: &str, etag: Option<&str>) -> String {
        match etag {
            Some(etag_value) => format!("{}:{}", url, etag_value),
            None => url.to_string(),
        }
    }
}

Cache Management and Cleanup

Implement proper cache management with size limits and cleanup:

use std::collections::HashMap;
use std::sync::{Arc, Mutex};

pub struct ManagedCache {
    storage: Arc<Mutex<HashMap<String, CacheEntry>>>,
    max_size: usize,
    max_memory: usize, // in bytes
}

impl ManagedCache {
    pub fn new(max_size: usize, max_memory: usize) -> Self {
        Self {
            storage: Arc::new(Mutex::new(HashMap::new())),
            max_size,
            max_memory,
        }
    }

    pub fn set(&self, key: String, data: String) {
        let mut storage = self.storage.lock().unwrap();

        // Check if we need to evict entries
        self.evict_if_needed(&mut storage, data.len());

        let entry = CacheEntry::new(data, Duration::from_secs(3600));
        storage.insert(key, entry);
    }

    fn evict_if_needed(&self, storage: &mut HashMap<String, CacheEntry>, new_entry_size: usize) {
        // Remove expired entries first
        storage.retain(|_, entry| !entry.is_expired());

        // Check size limit
        while storage.len() >= self.max_size {
            if let Some(oldest_key) = self.find_oldest_entry(storage) {
                storage.remove(&oldest_key);
            } else {
                break;
            }
        }

        // Check memory limit
        let current_memory = self.calculate_memory_usage(storage);
        if current_memory + new_entry_size > self.max_memory {
            // Implement LRU eviction
            self.evict_lru_entries(storage, new_entry_size);
        }
    }

    fn find_oldest_entry(&self, storage: &HashMap<String, CacheEntry>) -> Option<String> {
        storage.iter()
            .min_by_key(|(_, entry)| entry.timestamp)
            .map(|(key, _)| key.clone())
    }

    fn calculate_memory_usage(&self, storage: &HashMap<String, CacheEntry>) -> usize {
        storage.iter()
            .map(|(key, entry)| key.len() + entry.data.len())
            .sum()
    }

    fn evict_lru_entries(&self, storage: &mut HashMap<String, CacheEntry>, needed_space: usize) {
        let mut freed_space = 0;
        let mut entries_to_remove = Vec::new();

        // Sort by timestamp (oldest first)
        let mut sorted_entries: Vec<_> = storage.iter().collect();
        sorted_entries.sort_by_key(|(_, entry)| entry.timestamp);

        for (key, entry) in sorted_entries {
            entries_to_remove.push(key.clone());
            freed_space += key.len() + entry.data.len();

            if freed_space >= needed_space {
                break;
            }
        }

        for key in entries_to_remove {
            storage.remove(&key);
        }
    }
}

Testing Your Cache Implementation

Create comprehensive tests for your caching system:

#[cfg(test)]
mod tests {
    use super::*;
    use std::time::Duration;

    #[test]
    fn test_memory_cache_basic_operations() {
        let cache = MemoryCache::new(Duration::from_secs(60));

        // Test cache miss
        assert_eq!(cache.get("test_key"), None);

        // Test cache set and hit
        cache.set("test_key".to_string(), "test_value".to_string());
        assert_eq!(cache.get("test_key"), Some("test_value".to_string()));
    }

    #[test]
    fn test_cache_expiration() {
        let cache = MemoryCache::new(Duration::from_millis(100));

        cache.set("test_key".to_string(), "test_value".to_string());
        assert_eq!(cache.get("test_key"), Some("test_value".to_string()));

        // Wait for expiration
        std::thread::sleep(Duration::from_millis(150));
        assert_eq!(cache.get("test_key"), None);
    }

    #[tokio::test]
    async fn test_file_cache_persistence() {
        let temp_dir = "./test_cache";
        let cache = FileCacheManager::new(temp_dir, Duration::from_secs(60));

        cache.set("test_key", "test_value".to_string()).unwrap();
        assert_eq!(cache.get("test_key"), Some("test_value".to_string()));

        // Clean up
        std::fs::remove_dir_all(temp_dir).ok();
    }
}

Best Practices and Considerations

When implementing request caching in Rust web scraping:

1. Choose appropriate TTL values: Balance freshness with performance based on content type and update frequency
2. Implement cache invalidation: Provide mechanisms to clear stale or invalid cache entries
3. Handle cache misses gracefully: Always have fallback logic when cache lookups fail
4. Monitor cache performance: Track hit rates, memory usage, and response times
5. Consider distributed caching: Use Redis or similar for multi-instance applications
6. Respect robots.txt and rate limits: Caching should complement, not replace, ethical scraping practices
7. Implement proper error handling: Cache failures shouldn't break your scraping flow
8. Use compression: Store compressed data to reduce memory usage
9. Implement cache warming: Pre-populate frequently accessed data
10. Monitor cache size: Implement eviction policies to prevent memory leaks

Performance Optimization Tips

• Use efficient serialization: Choose fast serialization formats like bincode for file-based caching
• Implement connection pooling: When using Redis, maintain connection pools for better performance
• Use async operations: Leverage Rust's async/await for non-blocking cache operations
• Consider cache partitioning: Distribute cache across multiple stores for better scalability
• Implement cache compression: Use compression algorithms to reduce storage requirements

Request caching significantly improves the performance and reliability of Rust web scraping applications. Whether you choose in-memory, file-based, or distributed caching depends on your specific requirements for persistence, scalability, and performance. Similar optimization techniques are often used when handling browser sessions in Puppeteer or managing network requests in Puppeteer for JavaScript-based scraping solutions.