How can I set custom headers for requests in Puppeteer-Sharp?

Puppeteer-Sharp provides two main approaches for setting custom HTTP headers: global page-level headers using SetExtraHttpHeadersAsync() and per-request headers using request interception. Here's a comprehensive guide to both methods.

Method 1: Global Headers with SetExtraHttpHeadersAsync

The SetExtraHttpHeadersAsync() method sets headers for all subsequent requests made by a page. This is the simplest approach when you need the same headers for all requests.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using PuppeteerSharp;

class Program
{
    public static async Task Main(string[] args)
    {
        // Download Chromium if needed
        await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);

        // Launch browser
        var browser = await Puppeteer.LaunchAsync(new LaunchOptions
        {
            Headless = true
        });

        var page = await browser.NewPageAsync();

        // Set global headers for all requests
        var headers = new Dictionary<string, string>
        {
            ["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
            ["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
            ["Accept-Language"] = "en-US,en;q=0.5",
            ["Authorization"] = "Bearer your-api-token",
            ["X-Custom-Header"] = "custom-value"
        };

        await page.SetExtraHttpHeadersAsync(headers);

        // All requests will now include the custom headers
        await page.GoToAsync("https://httpbin.org/headers");

        await browser.CloseAsync();
    }
}

Method 2: Per-Request Headers with Request Interception

For more granular control, use request interception to modify headers on a per-request basis:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using PuppeteerSharp;

class Program
{
    public static async Task Main(string[] args)
    {
        await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);

        var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = true });
        var page = await browser.NewPageAsync();

        // Enable request interception
        await page.SetRequestInterceptionAsync(true);

        // Handle each request individually
        page.Request += async (sender, e) =>
        {
            var request = e.Request;
            var headers = new Dictionary<string, string>(request.Headers);

            // Add different headers based on request type or URL
            if (request.Url.Contains("api"))
            {
                headers["Authorization"] = "Bearer api-token";
                headers["Content-Type"] = "application/json";
            }
            else if (request.ResourceType == ResourceType.Image)
            {
                headers["Accept"] = "image/webp,image/apng,image/*,*/*;q=0.8";
            }
            else
            {
                headers["User-Agent"] = "Custom Bot 1.0";
            }

            // Continue request with modified headers
            await request.ContinueAsync(new Payload
            {
                Headers = headers
            });
        };

        await page.GoToAsync("https://example.com");
        await browser.CloseAsync();
    }
}

Common Use Cases and Examples

Setting Authentication Headers

var authHeaders = new Dictionary<string, string>
{
    ["Authorization"] = "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
    ["X-API-Key"] = "your-api-key"
};

await page.SetExtraHttpHeadersAsync(authHeaders);

Custom User Agent and Browser Headers

var browserHeaders = new Dictionary<string, string>
{
    ["User-Agent"] = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
    ["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    ["Accept-Language"] = "en-US,en;q=0.9",
    ["Accept-Encoding"] = "gzip, deflate, br",
    ["DNT"] = "1",
    ["Connection"] = "keep-alive",
    ["Upgrade-Insecure-Requests"] = "1"
};

await page.SetExtraHttpHeadersAsync(browserHeaders);

Conditional Headers with Request Interception

await page.SetRequestInterceptionAsync(true);

page.Request += async (sender, e) =>
{
    var request = e.Request;
    var headers = new Dictionary<string, string>(request.Headers);

    // Only add auth headers to API requests
    if (request.Url.Contains("/api/"))
    {
        headers["Authorization"] = "Bearer " + GetAccessToken();
    }

    // Add referer for external requests
    if (!request.Url.StartsWith("https://your-domain.com"))
    {
        headers["Referer"] = "https://your-domain.com";
    }

    await request.ContinueAsync(new Payload { Headers = headers });
};

Important Notes

• Global vs Per-Request: Use SetExtraHttpHeadersAsync() for consistent headers across all requests, and request interception for dynamic or conditional headers.
• Header Overrides: Headers set via SetExtraHttpHeadersAsync() can be overridden by request interception.
• Performance: Request interception adds overhead since every request is processed by your event handler.
• Case Sensitivity: HTTP header names are case-insensitive, but some servers may be particular about casing.
• Reserved Headers: Some headers like Host, Content-Length, and Connection are controlled by the browser and cannot be overridden.

Error Handling

Always wrap header operations in try-catch blocks:

try
{
    await page.SetExtraHttpHeadersAsync(customHeaders);
    await page.GoToAsync(url);
}
catch (PuppeteerException ex)
{
    Console.WriteLine($"Error setting headers: {ex.Message}");
}

This approach gives you complete control over HTTP headers in Puppeteer-Sharp, whether you need simple global headers or complex per-request customization.