How do I Serialize and Deserialize JSON in C# When Scraping APIs?

JSON (JavaScript Object Notation) is the most common data format for web APIs. When scraping APIs in C#, you'll constantly need to deserialize JSON responses into C# objects and sometimes serialize C# objects into JSON for POST requests. C# offers two primary libraries for JSON handling: the modern System.Text.Json (built-in since .NET Core 3.0) and the popular third-party library Newtonsoft.Json (Json.NET).

Using System.Text.Json (Modern Approach)

System.Text.Json is the recommended approach for new .NET projects as it's built-in, high-performance, and doesn't require external dependencies.

Basic Deserialization

To deserialize JSON into C# objects, first define a class that matches the JSON structure:

using System.Text.Json;
using System.Net.Http;

// Define a class matching the API response structure
public class Product
{
    public int Id { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
    public string Category { get; set; }
}

// Deserialize JSON from an API response
public async Task<Product> ScrapeProductAsync(string url)
{
    using var client = new HttpClient();
    var jsonResponse = await client.GetStringAsync(url);

    // Deserialize JSON string to Product object
    var product = JsonSerializer.Deserialize<Product>(jsonResponse);
    return product;
}

Handling Complex Nested JSON

When dealing with nested JSON structures from APIs, create corresponding nested classes:

public class ApiResponse
{
    public bool Success { get; set; }
    public DataContainer Data { get; set; }
    public MetaInfo Meta { get; set; }
}

public class DataContainer
{
    public List<Product> Products { get; set; }
    public int TotalCount { get; set; }
}

public class MetaInfo
{
    public int Page { get; set; }
    public int PerPage { get; set; }
}

// Deserialize complex nested JSON
public async Task<ApiResponse> ScrapeApiAsync(string url)
{
    using var client = new HttpClient();
    var jsonResponse = await client.GetStringAsync(url);

    var response = JsonSerializer.Deserialize<ApiResponse>(jsonResponse);
    Console.WriteLine($"Found {response.Data.Products.Count} products");
    return response;
}

Custom Property Naming with JsonPropertyName

APIs often use different naming conventions (like snake_case or camelCase) than C# conventions (PascalCase). Use the [JsonPropertyName] attribute:

using System.Text.Json.Serialization;

public class Product
{
    [JsonPropertyName("product_id")]
    public int Id { get; set; }

    [JsonPropertyName("product_name")]
    public string Name { get; set; }

    [JsonPropertyName("unit_price")]
    public decimal Price { get; set; }

    [JsonPropertyName("is_available")]
    public bool IsAvailable { get; set; }
}

Serialization for POST Requests

When scraping APIs, you often need to send JSON data in POST requests:

public async Task<string> PostSearchQueryAsync(string apiUrl)
{
    var searchRequest = new
    {
        query = "laptops",
        filters = new
        {
            minPrice = 500,
            maxPrice = 2000,
            category = "electronics"
        },
        page = 1,
        limit = 50
    };

    using var client = new HttpClient();

    // Serialize object to JSON string
    var jsonContent = JsonSerializer.Serialize(searchRequest);
    var content = new StringContent(jsonContent, Encoding.UTF8, "application/json");

    var response = await client.PostAsync(apiUrl, content);
    return await response.Content.ReadAsStringAsync();
}

Configuring JsonSerializerOptions

For more control over serialization/deserialization behavior:

public async Task<Product> ScrapeWithOptionsAsync(string url)
{
    var options = new JsonSerializerOptions
    {
        PropertyNameCaseInsensitive = true,  // Ignore case differences
        WriteIndented = true,                 // Pretty-print JSON
        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
        NumberHandling = JsonNumberHandling.AllowReadingFromString
    };

    using var client = new HttpClient();
    var jsonResponse = await client.GetStringAsync(url);

    var product = JsonSerializer.Deserialize<Product>(jsonResponse, options);
    return product;
}

Using Newtonsoft.Json (Json.NET)

Newtonsoft.Json is a mature, feature-rich library that's still widely used, especially in legacy projects.

Installation

dotnet add package Newtonsoft.Json

Basic Deserialization with Newtonsoft.Json

using Newtonsoft.Json;

public class Product
{
    [JsonProperty("product_id")]
    public int Id { get; set; }

    [JsonProperty("product_name")]
    public string Name { get; set; }
}

public async Task<Product> ScrapeProductAsync(string url)
{
    using var client = new HttpClient();
    var jsonResponse = await client.GetStringAsync(url);

    // Deserialize using Newtonsoft.Json
    var product = JsonConvert.DeserializeObject<Product>(jsonResponse);
    return product;
}

Handling Dynamic JSON with JObject

When the JSON structure is unpredictable or varies between requests, use JObject:

using Newtonsoft.Json.Linq;

public async Task<Dictionary<string, object>> ScrapeDynamicJsonAsync(string url)
{
    using var client = new HttpClient();
    var jsonResponse = await client.GetStringAsync(url);

    // Parse JSON without a predefined class
    var jsonObject = JObject.Parse(jsonResponse);

    var results = new Dictionary<string, object>();

    // Access properties dynamically
    results["title"] = jsonObject["title"]?.ToString();
    results["price"] = jsonObject["pricing"]?["current"]?.Value<decimal>();

    // Check if property exists before accessing
    if (jsonObject["availability"] != null)
    {
        results["inStock"] = jsonObject["availability"]["inStock"]?.Value<bool>();
    }

    return results;
}

Advanced Serialization Settings

public async Task<string> SerializeWithSettingsAsync(object data)
{
    var settings = new JsonSerializerSettings
    {
        NullValueHandling = NullValueHandling.Ignore,
        Formatting = Formatting.Indented,
        DateFormatString = "yyyy-MM-dd",
        ContractResolver = new CamelCasePropertyNamesContractResolver()
    };

    var json = JsonConvert.SerializeObject(data, settings);
    return json;
}

Practical Web Scraping Example

Here's a complete example demonstrating JSON handling while scraping an e-commerce API:

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using System.Threading.Tasks;

public class ProductScraper
{
    private readonly HttpClient _httpClient;
    private readonly JsonSerializerOptions _jsonOptions;

    public ProductScraper()
    {
        _httpClient = new HttpClient();
        _httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0");

        _jsonOptions = new JsonSerializerOptions
        {
            PropertyNameCaseInsensitive = true,
            PropertyNamingPolicy = JsonNamingPolicy.CamelCase
        };
    }

    public async Task<List<Product>> ScrapeProductsAsync(string apiUrl, string category)
    {
        try
        {
            // Create request payload
            var requestData = new { category, page = 1, limit = 100 };
            var jsonRequest = JsonSerializer.Serialize(requestData, _jsonOptions);
            var content = new StringContent(jsonRequest, Encoding.UTF8, "application/json");

            // Send POST request
            var response = await _httpClient.PostAsync(apiUrl, content);
            response.EnsureSuccessStatusCode();

            // Read and deserialize response
            var jsonResponse = await response.Content.ReadAsStringAsync();
            var apiResponse = JsonSerializer.Deserialize<ApiResponse>(jsonResponse, _jsonOptions);

            return apiResponse?.Data?.Products ?? new List<Product>();
        }
        catch (HttpRequestException ex)
        {
            Console.WriteLine($"HTTP Error: {ex.Message}");
            return new List<Product>();
        }
        catch (JsonException ex)
        {
            Console.WriteLine($"JSON Parsing Error: {ex.Message}");
            return new List<Product>();
        }
    }
}

// Usage
var scraper = new ProductScraper();
var products = await scraper.ScrapeProductsAsync("https://api.example.com/products", "electronics");
foreach (var product in products)
{
    Console.WriteLine($"{product.Name}: ${product.Price}");
}

Error Handling and Edge Cases

Always implement robust error handling when working with JSON from external APIs:

public async Task<Product> SafeDeserializeAsync(string url)
{
    try
    {
        using var client = new HttpClient();
        client.Timeout = TimeSpan.FromSeconds(30);

        var jsonResponse = await client.GetStringAsync(url);

        // Validate JSON before deserializing
        if (string.IsNullOrWhiteSpace(jsonResponse))
        {
            throw new Exception("Empty response received");
        }

        var product = JsonSerializer.Deserialize<Product>(jsonResponse);

        // Validate deserialized object
        if (product == null)
        {
            throw new JsonException("Deserialization returned null");
        }

        return product;
    }
    catch (JsonException ex)
    {
        Console.WriteLine($"Invalid JSON format: {ex.Message}");
        throw;
    }
    catch (HttpRequestException ex)
    {
        Console.WriteLine($"Network error: {ex.Message}");
        throw;
    }
    catch (TaskCanceledException)
    {
        Console.WriteLine("Request timed out");
        throw;
    }
}

Performance Optimization

For high-volume web scraping, consider these optimizations:

using System.Text.Json;

// Reuse JsonSerializerOptions instance
private static readonly JsonSerializerOptions SharedOptions = new()
{
    PropertyNameCaseInsensitive = true
};

// Use async streams for large datasets
public async IAsyncEnumerable<Product> StreamProductsAsync(string url)
{
    using var client = new HttpClient();
    using var stream = await client.GetStreamAsync(url);

    await foreach (var product in JsonSerializer.DeserializeAsyncEnumerable<Product>(stream, SharedOptions))
    {
        if (product != null)
        {
            yield return product;
        }
    }
}

Choosing Between System.Text.Json and Newtonsoft.Json

Use System.Text.Json when: - Building new .NET Core/.NET 5+ applications - Performance is critical (it's faster and uses less memory) - You want minimal dependencies

Use Newtonsoft.Json when: - Working with legacy .NET Framework projects - You need advanced features like LINQ to JSON (JObject, JArray) - Your project already uses it extensively - You need more flexible type handling

Conclusion

JSON serialization and deserialization are fundamental skills for API web scraping in C#. System.Text.Json provides excellent performance and is recommended for modern applications, while Newtonsoft.Json offers more features and flexibility. Both libraries handle the heavy lifting of converting between JSON strings and C# objects, allowing you to focus on extracting and processing data. When working with complex APIs, remember to handle errors gracefully, validate data, and optimize for performance when dealing with large datasets.

For more advanced scenarios, you might also want to explore how to handle AJAX requests using Puppeteer for JavaScript-heavy sites, or learn about handling authentication in Puppeteer when dealing with protected APIs.