PuppeteerSharp Web Scraping: Headless Chrome in C#

PuppeteerSharp brings headless Chrome automation to .NET: it's a C# port of Google's Puppeteer that drives a real browser over the DevTools Protocol, so your scraper sees pages the way Chrome renders them — JavaScript executed, XHR data applied, DOM complete. This guide covers the whole scraping workflow in C#: installation, selectors, waiting and timeouts, forms and logins, file downloads, screenshots and PDFs, running on Linux and Docker, and an honest comparison with Playwright for .NET and Selenium.

Key Takeaways

• dotnet add package PuppeteerSharp installs the library from NuGet; new BrowserFetcher().DownloadAsync() fetches a matching Chrome build on first run
• The API mirrors Node.js Puppeteer with .NET conventions: page.goto() becomes page.GoToAsync(), everything is async/await over Tasks
• Prefer WaitForSelectorAsync/WaitForNetworkIdleAsync over Task.Delay — fixed sleeps are the number-one cause of flaky scrapers
• Set page.DefaultTimeout and page.DefaultNavigationTimeout once instead of passing timeouts to every call
• Reuse one browser across many pages and wrap everything in await using — leaked Chrome processes are the classic PuppeteerSharp memory bug
• A real browser doesn't beat anti-bot walls: Cloudflare and DataDome fingerprint headless Chrome regardless of the language driving it — plan for proxies or a scraping API on protected targets

What is PuppeteerSharp?

PuppeteerSharp (published on NuGet as PuppeteerSharp) is an open-source .NET port of Puppeteer, created and maintained by Darío Kondratiuk. Like the original, it controls Chrome or Chromium over the DevTools Protocol — and recent versions can also drive Firefox via WebDriver BiDi. The API tracks upstream Puppeteer closely, which means two useful things: almost any Puppeteer recipe you find translates to C# nearly line-for-line, and the Puppeteer documentation doubles as a reference for the .NET API.

The differences are exactly what you'd expect from idiomatic .NET:

Use PuppeteerSharp when your stack is .NET and the target site renders content with JavaScript. For static HTML, a plain HttpClient with Html Agility Pack is 10–50× cheaper per page — don't launch Chrome for markup that's already in the HTTP response. Our headless browser guide covers when a real browser is and isn't worth the overhead.

Installation and first scrape

Install the NuGet package into any modern .NET project:

dotnet add package PuppeteerSharp

Unlike the Node version, PuppeteerSharp doesn't bundle a browser with the package — you download one explicitly (once) with BrowserFetcher. The download lands in the user profile by default and is skipped if already present:

using PuppeteerSharp;

// Download a compatible Chrome build on first run (no-op afterwards)
await new BrowserFetcher().DownloadAsync();

await using var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = true
});
await using var page = await browser.NewPageAsync();

await page.GoToAsync("https://example.com");
Console.WriteLine(await page.GetTitleAsync());

var html = await page.GetContentAsync(); // rendered HTML, after JavaScript

Two setup tips that save debugging time later:

• Watch it work while developing. Headless = false with SlowMo = 50 in LaunchOptions shows the browser executing your steps — the fastest way to see why a selector misses.
• Use an existing Chrome instead of downloading. Set ExecutablePath in LaunchOptions to a system Chrome/Chromium — the standard move in locked-down CI images (more in the Docker section).

Selectors and extracting data

PuppeteerSharp queries the live DOM with CSS selectors (the syntax itself is covered in our CSS selectors FAQ):

// One element / all elements
var heading = await page.QuerySelectorAsync("h1");
var rows = await page.QuerySelectorAllAsync("table tr");

// Evaluate JavaScript in the page and get plain data back — the workhorse
var title = await page.EvaluateFunctionAsync<string>(
    "() => document.querySelector('h1').innerText.trim()");

var products = await page.EvaluateFunctionAsync<Product[]>(@"
    () => [...document.querySelectorAll('.product')].map(el => ({
        name: el.querySelector('.name')?.innerText.trim(),
        price: el.querySelector('.price')?.innerText.trim(),
        url: el.querySelector('a')?.href
    }))");

public record Product(string Name, string Price, string Url);

EvaluateFunctionAsync<T> runs your JavaScript inside the browser and deserializes the JSON result straight into a C# type — for list pages it's usually one call instead of a loop of element handles. On individual handles, element.EvaluateFunctionAsync<string>("el => el.innerText") extracts values the same way.

XPath still works through the xpath/ selector prefix (page.QuerySelectorAllAsync("xpath///div[@class='price']")), but CSS covers nearly every scraping case — see the XPath cheat sheet if you're migrating old XPath-based scrapers.

Waiting, timeouts, and delays

Most "PuppeteerSharp is flaky" reports are waiting bugs: the code queried the DOM before the site's JavaScript finished building it. The fix is never Task.Delay — it's waiting for an observable condition:

// Wait until the element exists (default timeout: 30s)
await page.WaitForSelectorAsync(".results .product");

// Wait for an arbitrary page condition
await page.WaitForFunctionAsync(
    "() => document.querySelectorAll('.product').length >= 20");

// Wait for network traffic to settle (good for AJAX-heavy pages)
await page.WaitForNetworkIdleAsync();

// Click that triggers navigation: start waiting BEFORE clicking
var nav = page.WaitForNavigationAsync();
await page.ClickAsync("a.next-page");
await nav;

Navigation itself takes a WaitUntil option: Load (default), DOMContentLoaded (fastest), or Networkidle0/Networkidle2 for pages that keep loading data after the load event:

await page.GoToAsync("https://example.com/spa", new NavigationOptions
{
    WaitUntil = new[] { WaitUntilNavigation.Networkidle2 },
    Timeout = 60_000
});

Rather than passing timeouts everywhere, set the defaults once per page — SetDefaultNavigationTimeoutAsync covers GoToAsync/WaitForNavigationAsync, and DefaultTimeout covers everything else:

page.DefaultTimeout = 30_000;                      // waits, selectors
page.DefaultNavigationTimeout = 60_000;            // page loads

Every timeout throws a WaitTaskTimeoutException/NavigationException subclass of PuppeteerException — catch it, log the URL, and retry with backoff rather than letting one slow page kill a crawl.

Clicking, typing, and multiple pages

Interaction methods dispatch real input events, so client-side frameworks react exactly as they do for a human:

await page.ClickAsync("button.load-more");
await page.TypeAsync("input[name='q']", "wireless headphones", new TypeOptions { Delay = 50 });
await page.Keyboard.PressAsync("Enter");
await page.SelectAsync("select#sort", "price_asc");

One browser can run many pages — that's the unit of concurrency. Throttle with a SemaphoreSlim so you don't open fifty tabs at once:

var throttle = new SemaphoreSlim(4); // 4 concurrent tabs
var tasks = urls.Select(async url =>
{
    await throttle.WaitAsync();
    try
    {
        await using var p = await browser.NewPageAsync();
        await p.GoToAsync(url);
        return await p.EvaluateFunctionAsync<string>("() => document.title");
    }
    finally { throttle.Release(); }
});
var titles = await Task.WhenAll(tasks);

Around 100–400 MB of RAM per concurrent page is a realistic budget; on long crawls, restart the browser every few hundred pages to keep Chrome's memory growth in check.

Viewport, mobile emulation, and user agents

Sites serve different markup to different screens, so pin the environment your scraper sees. SetViewportAsync controls size and pixel density, SetUserAgentAsync the UA string, and SetExtraHttpHeadersAsync any additional headers:

// Desktop at standard density
await page.SetViewportAsync(new ViewPortOptions { Width = 1920, Height = 1080 });

// Mobile emulation: small viewport, touch, 3x scale factor, mobile UA
await page.SetViewportAsync(new ViewPortOptions
{
    Width = 390, Height = 844,
    DeviceScaleFactor = 3,
    IsMobile = true,
    HasTouch = true
});
await page.SetUserAgentAsync(
    "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 " +
    "(KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1");

await page.SetExtraHttpHeadersAsync(new Dictionary<string, string>
{
    ["Accept-Language"] = "en-US,en;q=0.9"
});

DeviceScaleFactor matters for screenshots too: 2 or 3 produces retina-quality captures without changing layout. Keep the user agent, viewport, and Accept-Language consistent with each other — a desktop UA on a 390px viewport is exactly the kind of mismatch anti-bot heuristics flag.

Authentication, cookies, and sessions

For form-based logins, drive the form like a user, then persist the session cookies so subsequent runs skip the login page:

// Log in once
await page.GoToAsync("https://example.com/login");
await page.TypeAsync("#email", Environment.GetEnvironmentVariable("SCRAPE_USER"));
await page.TypeAsync("#password", Environment.GetEnvironmentVariable("SCRAPE_PASS"));
var loggedIn = page.WaitForNavigationAsync();
await page.ClickAsync("button[type='submit']");
await loggedIn;

// Save the session
var cookies = await page.GetCookiesAsync();
await File.WriteAllTextAsync("cookies.json", JsonSerializer.Serialize(cookies));

// Later run: restore instead of logging in again
var saved = JsonSerializer.Deserialize<CookieParam[]>(
    await File.ReadAllTextAsync("cookies.json"));
await page.SetCookieAsync(saved);
await page.GoToAsync("https://example.com/account");

HTTP Basic auth and authenticated proxies both go through page.AuthenticateAsync(new Credentials { Username = ..., Password = ... }). Two cautions: store credentials in configuration, not code, and only automate logins on accounts you own or have permission to use — scraping behind a login is also where terms-of-service and legal exposure concentrate (see our legality guide).

File downloads

Chrome won't download files in headless mode until you tell it where to put them. PuppeteerSharp exposes the DevTools Protocol directly for this:

var downloadPath = Path.Combine(Environment.CurrentDirectory, "downloads");
Directory.CreateDirectory(downloadPath);

await page.Client.SendAsync("Page.setDownloadBehavior", new
{
    behavior = "allow",
    downloadPath
});

await page.ClickAsync("a.download-csv");

There's no completion event, so poll the directory until the file stops growing and Chrome's .crdownload temp file disappears. For plain file URLs, skip the browser entirely — once you have the session cookies, passing them to HttpClient downloads the file with a fraction of the overhead.

Screenshots and PDFs

// Full-page screenshot
await page.ScreenshotAsync("page.png", new ScreenshotOptions { FullPage = true });

// Single element
var chart = await page.QuerySelectorAsync(".chart");
await chart.ScreenshotAsync("chart.png");

// PDF (headless mode only)
await page.PdfAsync("report.pdf", new PdfOptions { PrintBackground = true });

PdfAsync renders the print stylesheet, so pages can look different than on screen — PrintBackground = true and an explicit Format fix the most common surprises. This trio (rendered HTML, screenshots, PDFs) is why PuppeteerSharp shows up in .NET reporting stacks, not just scrapers.

Running on Linux, macOS, and Docker

PuppeteerSharp is cross-platform — the usual friction is Chrome's system dependencies on slim Linux images. In containers, the standard launch flags are:

var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = true,
    ExecutablePath = "/usr/bin/chromium",   // system package instead of BrowserFetcher
    Args = new[] { "--no-sandbox", "--disable-dev-shm-usage" }
});

A minimal Dockerfile installs Chromium from the distro so all shared libraries come with it:

FROM mcr.microsoft.com/dotnet/runtime:8.0
RUN apt-get update && apt-get install -y chromium fonts-liberation \
    && rm -rf /var/lib/apt/lists/*
COPY bin/Release/net8.0/publish/ /app/
WORKDIR /app
ENTRYPOINT ["dotnet", "Scraper.dll"]

--disable-dev-shm-usage avoids crashes from Docker's default 64 MB /dev/shm; --no-sandbox is required when the container runs as root (prefer a non-root user where you can). On developer macOS/Windows machines, the plain BrowserFetcher download works with no extra flags.

PuppeteerSharp vs Playwright vs Selenium in .NET

All three drive real browsers from C#; the differences are ergonomics and ecosystem:

Honest guidance: for a brand-new .NET scraping or testing project, Playwright is the safer default — Microsoft backing, auto-waiting, and WebKit coverage. PuppeteerSharp wins when you're translating existing Puppeteer code or recipes, want the DevTools Protocol exposed directly, or prefer its smaller API surface. Selenium remains the choice for cross-browser test grids. None of the three is better at avoiding bot detection — that fight is decided by IP reputation and fingerprinting, not the automation library.

When PuppeteerSharp isn't enough

PuppeteerSharp solves JavaScript rendering. It doesn't solve blocked IPs, Cloudflare challenges, CAPTCHAs, or the ops cost of running a Chrome fleet from your .NET services. When those become the problem, point your code at a rendering API instead of the target site — WebScraping.AI runs the page in real Chrome with rotating datacenter or residential proxies and returns the result over plain HTTP:

using var client = new HttpClient();

// Fully rendered HTML through a managed browser + residential proxies
var html = await client.GetStringAsync(
    "https://api.webscraping.ai/html" +
    $"?api_key={apiKey}&url={Uri.EscapeDataString("https://example.com/product/42")}" +
    "&js=true&proxy=residential");

// Or skip parsing entirely — structured fields from any page
var fields = await client.GetFromJsonAsync<Dictionary<string, string>>(
    "https://api.webscraping.ai/ai/fields" +
    $"?api_key={apiKey}&url={Uri.EscapeDataString("https://example.com/product/42")}" +
    "&fields[name]=Product name&fields[price]=Price with currency");

A common hybrid: prototype locally with PuppeteerSharp, then swap GoToAsync + GetContentAsync for the /html endpoint in production and keep your parsing logic — whether that's EvaluateFunctionAsync ported to Html Agility Pack or the AI extraction endpoints doing the parsing for you.

Frequently asked questions

Is PuppeteerSharp free, and is it still maintained? Yes — MIT-licensed and actively maintained, with releases tracking upstream Puppeteer closely. It's one of the most-downloaded browser automation packages on NuGet.

How is PuppeteerSharp different from Puppeteer? Same architecture and near-identical API, different runtime: C#/.NET Tasks instead of Node.js Promises, PascalCase *Async methods instead of camelCase, NuGet instead of npm, and an explicit BrowserFetcher browser download instead of one bundled at install time. Puppeteer recipes translate almost mechanically — our Puppeteer guide is effectively a companion to this one.

Should I use PuppeteerSharp or Playwright for .NET? Starting fresh, Playwright — official Microsoft support, auto-waiting, three browser engines. Choose PuppeteerSharp when you're porting Puppeteer code, need raw DevTools Protocol access, or want a lighter dependency. Both scrape JavaScript sites equally well.

Does PuppeteerSharp run on Linux and in Docker? Yes — it's .NET-cross-platform. In Docker, install Chromium from the distro (so its dependencies come along), set ExecutablePath, and launch with --no-sandbox --disable-dev-shm-usage. See the Docker section above for a working Dockerfile.

Why does my scraper work locally but time out on the server? Usually one of three things: missing Chrome dependencies on the slim server image, a smaller /dev/shm in containers (add --disable-dev-shm-usage), or slower network making default 30-second timeouts too tight — raise page.DefaultNavigationTimeout and wait on Networkidle2 for AJAX-heavy pages.

Can PuppeteerSharp bypass Cloudflare or CAPTCHAs? Not reliably. Headless Chrome is fingerprintable regardless of whether Node.js or .NET drives it, and the C# ecosystem doesn't have a maintained equivalent of the stealth plugins. For protected targets, a scraping API that owns the challenge layer — which is what WebScraping.AI does — is usually cheaper than building evasion into your scraper.