Table of contents

How do I enable logging for HttpClient (C#) requests and responses?

Logging HttpClient requests and responses in C# is essential for debugging API calls, monitoring performance, and troubleshooting network issues. This guide covers multiple approaches from basic built-in logging to advanced custom implementations.

Method 1: Built-in HttpClient Logging

Configuration via appsettings.json

The simplest approach is to enable built-in logging through configuration:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "System.Net.Http.HttpClient": "Debug",
      "System.Net.Http.HttpClient.Default.LogicalHandler": "Debug",
      "System.Net.Http.HttpClient.Default.ClientHandler": "Debug"
    }
  }
}

.NET 6+ Program.cs Setup

var builder = WebApplication.CreateBuilder(args);

// Add HttpClient with logging
builder.Services.AddHttpClient();

// Optional: Configure specific log levels
builder.Logging.AddFilter("System.Net.Http.HttpClient", LogLevel.Debug);

var app = builder.Build();

Service Implementation

public class ApiService
{
    private readonly HttpClient _httpClient;
    private readonly ILogger<ApiService> _logger;

    public ApiService(IHttpClientFactory httpClientFactory, ILogger<ApiService> logger)
    {
        _httpClient = httpClientFactory.CreateClient();
        _logger = logger;
    }

    public async Task<string> GetDataAsync(string url)
    {
        _logger.LogInformation("Making request to {Url}", url);

        var response = await _httpClient.GetAsync(url);
        response.EnsureSuccessStatusCode();

        var content = await response.Content.ReadAsStringAsync();
        _logger.LogInformation("Received {ContentLength} characters", content.Length);

        return content;
    }
}

Method 2: Custom Logging Handler

For more control over what gets logged, create a custom DelegatingHandler:

public class LoggingHandler : DelegatingHandler
{
    private readonly ILogger<LoggingHandler> _logger;

    public LoggingHandler(ILogger<LoggingHandler> logger)
    {
        _logger = logger;
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        // Log request
        _logger.LogInformation("HTTP {Method} {Uri}", request.Method, request.RequestUri);

        if (request.Content != null)
        {
            var requestBody = await request.Content.ReadAsStringAsync();
            _logger.LogDebug("Request Body: {RequestBody}", requestBody);
        }

        var stopwatch = Stopwatch.StartNew();

        // Send request
        var response = await base.SendAsync(request, cancellationToken);

        stopwatch.Stop();

        // Log response
        _logger.LogInformation("HTTP {Method} {Uri} responded {StatusCode} in {ElapsedMilliseconds}ms",
            request.Method, request.RequestUri, (int)response.StatusCode, stopwatch.ElapsedMilliseconds);

        if (response.Content != null)
        {
            var responseBody = await response.Content.ReadAsStringAsync();
            _logger.LogDebug("Response Body: {ResponseBody}", responseBody);
        }

        return response;
    }
}

Register the Custom Handler

// In Program.cs or Startup.cs
builder.Services.AddTransient<LoggingHandler>();

builder.Services.AddHttpClient<ApiService>(client =>
{
    client.BaseAddress = new Uri("https://api.example.com/");
})
.AddHttpMessageHandler<LoggingHandler>();

Method 3: Named HttpClient with Logging

For different logging configurations per API:

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        // Named client with specific logging
        services.AddHttpClient("ApiClient", client =>
        {
            client.BaseAddress = new Uri("https://api.example.com/");
            client.DefaultRequestHeaders.Add("User-Agent", "MyApp/1.0");
        })
        .AddHttpMessageHandler<LoggingHandler>();

        // Another client with different configuration
        services.AddHttpClient("InternalApi", client =>
        {
            client.BaseAddress = new Uri("https://internal.api.com/");
        })
        .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler()
        {
            UseProxy = false // Example: disable proxy for internal calls
        });
    }
}

Using Named Clients

public class MultiApiService
{
    private readonly HttpClient _externalClient;
    private readonly HttpClient _internalClient;

    public MultiApiService(IHttpClientFactory httpClientFactory)
    {
        _externalClient = httpClientFactory.CreateClient("ApiClient");
        _internalClient = httpClientFactory.CreateClient("InternalApi");
    }

    public async Task<string> GetExternalDataAsync()
    {
        var response = await _externalClient.GetAsync("data");
        return await response.Content.ReadAsStringAsync();
    }
}

Method 4: Logging with Serilog

For structured logging with Serilog:

public class SerilogLoggingHandler : DelegatingHandler
{
    private readonly ILogger _logger = Log.ForContext<SerilogLoggingHandler>();

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var requestId = Guid.NewGuid().ToString("N")[..8];

        _logger.Information("HTTP Request {RequestId}: {Method} {Uri}",
            requestId, request.Method, request.RequestUri);

        var response = await base.SendAsync(request, cancellationToken);

        _logger.Information("HTTP Response {RequestId}: {StatusCode} {ReasonPhrase}",
            requestId, (int)response.StatusCode, response.ReasonPhrase);

        return response;
    }
}

What Gets Logged

With Debug level logging enabled, you'll see:

info: System.Net.Http.HttpClient.Default.LogicalHandler[100]
      Start processing HTTP request GET https://api.example.com/data

info: System.Net.Http.HttpClient.Default.ClientHandler[100]
      Sending HTTP request GET https://api.example.com/data

info: System.Net.Http.HttpClient.Default.ClientHandler[101]
      Received HTTP response after 156ms - OK

info: System.Net.Http.HttpClient.Default.LogicalHandler[101]
      End processing HTTP request after 159ms - OK

Security Considerations

⚠️ Important Security Notes:

  • Never log sensitive data like API keys, passwords, or personal information
  • Use LogLevel.Debug only in development environments
  • Consider implementing log filtering for production:
public class SafeLoggingHandler : DelegatingHandler
{
    private readonly ILogger<SafeLoggingHandler> _logger;
    private readonly IWebHostEnvironment _env;

    public SafeLoggingHandler(ILogger<SafeLoggingHandler> logger, IWebHostEnvironment env)
    {
        _logger = logger;
        _env = env;
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        // Only log request/response bodies in development
        var shouldLogBodies = _env.IsDevelopment();

        if (shouldLogBodies && request.Content != null)
        {
            var body = await request.Content.ReadAsStringAsync();
            // Filter sensitive data
            var safeBody = FilterSensitiveData(body);
            _logger.LogDebug("Request: {Body}", safeBody);
        }

        var response = await base.SendAsync(request, cancellationToken);

        return response;
    }

    private string FilterSensitiveData(string content)
    {
        // Implement your filtering logic here
        return content.Replace("\"password\":\"[^\"]*\"", "\"password\":\"***\"");
    }
}

Performance Considerations

  • Built-in logging has minimal performance impact
  • Custom handlers add slight overhead for each request
  • Avoid logging large response bodies in production
  • Use asynchronous logging when possible
  • Consider using IMemoryCache for frequently accessed endpoints

This comprehensive approach gives you full visibility into your HttpClient operations while maintaining security and performance best practices.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon