How can I set custom headers for requests in Puppeteer-Sharp?

In Puppeteer-Sharp, a .NET port of the Puppeteer headless Chrome Node.js API, you can set custom headers for requests by using the SetExtraHttpHeadersAsync method of the Page class. This method accepts a dictionary where the keys are the header names and the values are the corresponding header values.

Here's an example of how to set custom headers for all requests on a page in Puppeteer-Sharp:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using PuppeteerSharp;

class Program
{
    public static async Task Main(string[] args)
    {
        // Download the Chromium revision if it does not already exist
        await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);

        // Launch the browser
        var browser = await Puppeteer.LaunchAsync(new LaunchOptions
        {
            Headless = true // Set to false if you want to see the browser
        });

        // Create a new page
        var page = await browser.NewPageAsync();

        // Define custom headers
        var customHeaders = new Dictionary<string, string>
        {
            ["Custom-Header-1"] = "Value1",
            ["Custom-Header-2"] = "Value2"
        };

        // Set the custom headers for the page
        await page.SetExtraHttpHeadersAsync(customHeaders);

        // Navigate to a URL
        await page.GoToAsync("http://example.com");

        // Do something with the page, like taking a screenshot or extracting content
        // ...

        // Close browser
        await browser.CloseAsync();
    }
}

In the example above, we launch a headless browser, create a new page, and then set custom HTTP headers using SetExtraHttpHeadersAsync. These headers will be sent with every request made by the page.

It's important to note that SetExtraHttpHeadersAsync sets headers for all subsequent requests made by the page. If you want to modify headers for individual requests or responses, you will need to use request interception with Request.ContinueAsync and modify the request object on the fly.

Here's an example of setting headers for an individual request using request interception:

// Enable request interception
await page.SetRequestInterceptionAsync(true);

// Add event listener for the request event
page.Request += async (sender, e) =>
{
    // Modify headers for a specific request
    var headers = new Dictionary<string, string>(e.Request.Headers)
    {
        ["Custom-Header"] = "Value"
    };

    // Continue the request with the new headers
    await e.Request.ContinueAsync(new Payload
    {
        Headers = headers
    });
};

// Navigate to a URL
await page.GoToAsync("http://example.com");

In this second example, we first enable request interception, then add an event listener for the Request event, where we modify the headers of the request before continuing it. This gives you fine-grained control over the headers of individual requests.

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon