How can I set request headers for an HttpClient (C#) request?

There are several ways to set request headers when using HttpClient in C#. The method you choose depends on whether you need headers applied to all requests or just specific ones.

Method 1: Using DefaultRequestHeaders (Global Headers)

Use DefaultRequestHeaders when you want to apply the same headers to all requests made by an HttpClient instance. This is ideal for headers like User-Agent, Authorization tokens, or API keys.

using System;
using System.Net.Http;
using System.Threading.Tasks;

class Program
{
    static async Task Main(string[] args)
    {
        using var httpClient = new HttpClient();

        // Set headers that apply to all requests
        httpClient.DefaultRequestHeaders.Add("User-Agent", "MyApp/1.0");
        httpClient.DefaultRequestHeaders.Add("Accept", "application/json");
        httpClient.DefaultRequestHeaders.Add("X-API-Key", "your-api-key-here");

        // All requests will include these headers
        var response = await httpClient.GetAsync("https://api.example.com/data");
        var content = await response.Content.ReadAsStringAsync();
        Console.WriteLine(content);
    }
}

Method 2: Using HttpRequestMessage (Per-Request Headers)

Use HttpRequestMessage when you need different headers for individual requests or want more control over specific requests.

using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;

class Program
{
    static async Task Main(string[] args)
    {
        using var httpClient = new HttpClient();
        using var request = new HttpRequestMessage(HttpMethod.Get, "https://api.example.com/data");

        // Set headers for this specific request
        request.Headers.Add("User-Agent", "MyApp/1.0");
        request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
        request.Headers.Add("X-Request-ID", Guid.NewGuid().ToString());

        var response = await httpClient.SendAsync(request);
        var content = await response.Content.ReadAsStringAsync();
        Console.WriteLine(content);
    }
}

Common Header Examples

Authorization Headers

// Bearer token
httpClient.DefaultRequestHeaders.Authorization = 
    new AuthenticationHeaderValue("Bearer", "your-jwt-token");

// Basic authentication
var credentials = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes("username:password"));
httpClient.DefaultRequestHeaders.Authorization = 
    new AuthenticationHeaderValue("Basic", credentials);

// API key in header
httpClient.DefaultRequestHeaders.Add("X-API-Key", "your-api-key");

Custom Headers for Web Scraping

// Common headers for web scraping
httpClient.DefaultRequestHeaders.Add("User-Agent", 
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36");
httpClient.DefaultRequestHeaders.Add("Accept", 
    "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
httpClient.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.5");
httpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate");
httpClient.DefaultRequestHeaders.Add("Referer", "https://www.google.com/");

Conditional Headers

using var request = new HttpRequestMessage(HttpMethod.Get, url);

// Add header only if condition is met
if (!string.IsNullOrEmpty(apiKey))
{
    request.Headers.Add("X-API-Key", apiKey);
}

// Multiple values for same header
request.Headers.Add("Accept", "application/json");
request.Headers.Add("Accept", "application/xml");

Content Headers vs Request Headers

When working with POST/PUT requests, distinguish between request headers and content headers:

using var request = new HttpRequestMessage(HttpMethod.Post, "https://api.example.com/data");

// Request headers
request.Headers.Add("Authorization", "Bearer token");
request.Headers.Add("User-Agent", "MyApp/1.0");

// Content headers (when sending data)
var jsonContent = new StringContent("{\"name\":\"value\"}", System.Text.Encoding.UTF8, "application/json");
request.Content = jsonContent;

// Content headers are set automatically, but you can modify them
request.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");

Using HttpClientFactory (Recommended Approach)

For production applications, use HttpClientFactory with configured headers:

// In Startup.cs or Program.cs
services.AddHttpClient("ApiClient", client =>
{
    client.BaseAddress = new Uri("https://api.example.com/");
    client.DefaultRequestHeaders.Add("User-Agent", "MyApp/1.0");
    client.DefaultRequestHeaders.Add("Accept", "application/json");
});

// In your service class
public class ApiService
{
    private readonly HttpClient _httpClient;

    public ApiService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient("ApiClient");
    }

    public async Task<string> GetDataAsync()
    {
        var response = await _httpClient.GetAsync("data");
        return await response.Content.ReadAsStringAsync();
    }
}

Error Handling and Best Practices

try
{
    using var httpClient = new HttpClient();
    httpClient.DefaultRequestHeaders.Add("User-Agent", "MyApp/1.0");

    var response = await httpClient.GetAsync("https://api.example.com/data");
    response.EnsureSuccessStatusCode(); // Throws if not 2xx status

    var content = await response.Content.ReadAsStringAsync();
    Console.WriteLine(content);
}
catch (HttpRequestException ex)
{
    Console.WriteLine($"Request failed: {ex.Message}");
}
catch (TaskCanceledException ex)
{
    Console.WriteLine($"Request timed out: {ex.Message}");
}

Key Points to Remember

• Use DefaultRequestHeaders for headers that apply to all requests
• Use HttpRequestMessage.Headers for request-specific headers
  
• Use HttpClientFactory in production applications to avoid socket exhaustion
• Always handle exceptions appropriately
• Be mindful of rate limiting when setting custom headers for web scraping
• Some headers (like Content-Length) are set automatically and cannot be manually overridden