Table of contents

How do I handle different content types in responses with Reqwest?

When using the reqwest library in Rust, handling different content types in responses requires inspecting the Content-Type header and parsing the response body accordingly. Reqwest provides convenient built-in methods for common content types like JSON and text, while other formats require manual parsing.

Basic Approach

The general workflow for handling different content types:

  1. Send Request: Make an HTTP request using reqwest
  2. Check Content-Type: Inspect the Content-Type header in the response
  3. Parse Response: Use the appropriate parsing method based on the content type
  4. Handle Errors: Implement proper error handling for each content type

Setup

Add reqwest with required features to your Cargo.toml:

[dependencies]
reqwest = { version = "0.11", features = ["json", "blocking"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
tokio = { version = "1.0", features = ["full"] }

Comprehensive Content Type Handling

Blocking Example

use reqwest::blocking::Client;
use reqwest::header::{HeaderValue, CONTENT_TYPE};
use reqwest::Error;
use serde_json::Value;

fn handle_response_by_content_type(url: &str) -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new();
    let response = client.get(url).send()?.error_for_status()?;

    // Get Content-Type header
    let content_type = response
        .headers()
        .get(CONTENT_TYPE)
        .and_then(|ct| ct.to_str().ok())
        .unwrap_or("");

    match content_type {
        // JSON responses
        ct if ct.contains("application/json") => {
            let json: Value = response.json()?;
            println!("JSON data: {:#}", json);
        }

        // XML responses  
        ct if ct.contains("application/xml") || ct.contains("text/xml") => {
            let xml_text = response.text()?;
            println!("XML content: {}", xml_text);
            // Use xml-rs or quick-xml crate for parsing
        }

        // HTML responses
        ct if ct.contains("text/html") => {
            let html = response.text()?;
            println!("HTML content length: {}", html.len());
            // Use scraper or select crate for HTML parsing
        }

        // Plain text responses
        ct if ct.contains("text/plain") => {
            let text = response.text()?;
            println!("Text content: {}", text);
        }

        // Form data responses
        ct if ct.contains("application/x-www-form-urlencoded") => {
            let form_text = response.text()?;
            println!("Form data: {}", form_text);
            // Parse key=value&key2=value2 format manually or use url crate
        }

        // Binary content (images, files, etc.)
        ct if ct.contains("image/") || ct.contains("application/octet-stream") => {
            let bytes = response.bytes()?;
            println!("Binary content size: {} bytes", bytes.len());
            // Save to file or process binary data
        }

        // Default case for unknown content types
        _ => {
            let bytes = response.bytes()?;
            println!("Unknown content type '{}', got {} bytes", content_type, bytes.len());
        }
    }

    Ok(())
}

Async Example

use reqwest::Client;
use reqwest::header::CONTENT_TYPE;
use serde_json::Value;

async fn handle_response_async(url: &str) -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new();
    let response = client.get(url).send().await?.error_for_status()?;

    let content_type = response
        .headers()
        .get(CONTENT_TYPE)
        .and_then(|ct| ct.to_str().ok())
        .unwrap_or("");

    match content_type {
        ct if ct.contains("application/json") => {
            let json: Value = response.json().await?;
            println!("Async JSON: {:#}", json);
        }

        ct if ct.contains("text/") => {
            let text = response.text().await?;
            println!("Async text length: {}", text.len());
        }

        _ => {
            let bytes = response.bytes().await?;
            println!("Async binary size: {} bytes", bytes.len());
        }
    }

    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    handle_response_async("https://api.example.com/data").await?;
    Ok(())
}

Advanced Content Type Handling

Structured JSON with Serde

use serde::{Deserialize, Serialize};

#[derive(Deserialize, Debug)]
struct ApiResponse {
    id: u32,
    name: String,
    data: Vec<String>,
}

async fn parse_structured_json(url: &str) -> Result<ApiResponse, reqwest::Error> {
    let client = Client::new();
    let response = client.get(url).send().await?;

    // Directly deserialize to struct if content type is JSON
    let api_data: ApiResponse = response.json().await?;
    Ok(api_data)
}

Content Type with Charset Handling

fn parse_content_type_header(ct_header: &HeaderValue) -> (String, Option<String>) {
    let ct_str = ct_header.to_str().unwrap_or("");
    let parts: Vec<&str> = ct_str.split(';').collect();

    let mime_type = parts[0].trim().to_lowercase();
    let charset = parts.iter()
        .find(|part| part.trim().starts_with("charset="))
        .map(|part| part.split('=').nth(1).unwrap_or("").trim().to_string());

    (mime_type, charset)
}

fn handle_with_charset(response: reqwest::blocking::Response) -> Result<(), Box<dyn std::error::Error>> {
    if let Some(ct_header) = response.headers().get(CONTENT_TYPE) {
        let (mime_type, charset) = parse_content_type_header(ct_header);

        match mime_type.as_str() {
            "text/html" | "text/plain" => {
                let text = response.text()?;
                println!("Text with charset {:?}: {}", charset, text);
            }
            _ => {
                let bytes = response.bytes()?;
                println!("Binary content: {} bytes", bytes.len());
            }
        }
    }

    Ok(())
}

Error Handling Best Practices

use reqwest::Error as ReqwestError;
use serde_json::Error as JsonError;

#[derive(Debug)]
enum ContentError {
    Network(ReqwestError),
    Json(JsonError),
    UnsupportedContentType(String),
}

impl From<ReqwestError> for ContentError {
    fn from(err: ReqwestError) -> Self {
        ContentError::Network(err)
    }
}

impl From<JsonError> for ContentError {
    fn from(err: JsonError) -> Self {
        ContentError::Json(err)
    }
}

async fn robust_content_handler(url: &str) -> Result<(), ContentError> {
    let client = Client::new();
    let response = client.get(url).send().await?.error_for_status()?;

    let content_type = response
        .headers()
        .get(CONTENT_TYPE)
        .and_then(|ct| ct.to_str().ok())
        .unwrap_or("");

    match content_type {
        ct if ct.contains("application/json") => {
            let json: Value = response.json().await?;
            println!("Parsed JSON successfully");
            Ok(())
        }
        ct if ct.contains("text/") => {
            let _text = response.text().await?;
            println!("Parsed text successfully");
            Ok(())
        }
        ct => Err(ContentError::UnsupportedContentType(ct.to_string()))
    }
}

Practical Usage Examples

File Download Based on Content Type

use std::fs::File;
use std::io::Write;

async fn download_by_content_type(url: &str, filename: &str) -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new();
    let response = client.get(url).send().await?;

    let content_type = response
        .headers()
        .get(CONTENT_TYPE)
        .and_then(|ct| ct.to_str().ok())
        .unwrap_or("");

    let bytes = response.bytes().await?;

    let extension = match content_type {
        ct if ct.contains("application/json") => "json",
        ct if ct.contains("text/html") => "html",
        ct if ct.contains("text/plain") => "txt",
        ct if ct.contains("image/png") => "png",
        ct if ct.contains("image/jpeg") => "jpg",
        _ => "bin",
    };

    let full_filename = format!("{}.{}", filename, extension);
    let mut file = File::create(&full_filename)?;
    file.write_all(&bytes)?;

    println!("Downloaded {} as {}", content_type, full_filename);
    Ok(())
}

Key Points

  • Always check status: Use error_for_status() before processing content
  • Handle charset: Content-Type headers may include charset information
  • Use appropriate methods: json() for JSON, text() for text, bytes() for binary
  • Error handling: Implement robust error handling for network and parsing errors
  • Async support: Prefer async methods for better performance in concurrent applications
  • Memory efficiency: Use bytes() for large files to avoid string conversion overhead

This comprehensive approach ensures your Rust applications can handle any content type gracefully while maintaining good performance and error handling.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

What is the best way to debug a Reqwest client?
Is there a way to customize the connection pool settings in Reqwest?
How can I mock Reqwest clients for testing purposes?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon