C# Web Scraping: The Complete Guide

C# is an underrated scraping language. It has a fast, well-designed HTTP client, two mature HTML parsers, first-class bindings for every headless browser worth using, and a concurrency model — async/await over a real thread pool — that beats what most scripting languages offer for running hundreds of requests at once. What it lacks is Python's all-in-one framework: there is no Scrapy for .NET, so you assemble the stack yourself.

This guide is the map of that stack. It covers which library to pick for which job, the HTTP-layer details that decide whether a scraper survives production (timeouts above all), a real comparison of Html Agility Pack against AngleSharp, browser automation with Playwright for .NET, honest status on the older .NET scraping libraries people still search for, and the async patterns that make C# scraping fast without melting the target server.

Key Takeaways

• There is no single "C# scraping library" — you compose an HTTP client, a parser, and (only when needed) a headless browser
• HttpClient + AngleSharp is the right default for a new project; WebClient is obsolete and should not appear in new code
• Html Agility Pack is XPath-first and battle-tested; AngleSharp is spec-compliant, CSS-selector-native, and async by design
• Playwright for .NET is the strongest browser-automation option in the ecosystem: Microsoft-maintained, auto-waiting, three browser engines
• The default HttpClient timeout is 100 seconds, which is far too long for scraping — set a real one
• Bound concurrency with SemaphoreSlim or Parallel.ForEachAsync, not by firing every request at once
• Parsers and browsers don't solve blocked IPs or bot detection — that's an infrastructure problem, not a library choice

The .NET scraping stack, at a glance

Every C# scraper is some combination of three layers: fetch, parse, and — when the page needs JavaScript — render. Here is how the realistic options compare:

The decision is usually two questions deep:

1. Does the data appear in the raw HTML? Right-click, View Source, search for a value you want. If it's there, use HttpClient plus a parser and stay away from browsers — they cost roughly 100× the CPU and memory per page.
2. If not, is it in a JSON API the page calls? Open DevTools → Network → Fetch/XHR. Sites that render client-side almost always fetch from an endpoint you can call directly with HttpClient, which is faster and more stable than driving a browser.

Only when both answers are no does a headless browser earn its cost.

Examples below target .NET 8 and run unchanged on newer versions. Every one is a complete, compilable snippet — top-level statements, System.Net.Http in scope by default.

The HTTP layer: HttpClient

HttpClient is the fetch layer for every non-browser approach, and getting it wrong is the most common source of scrapers that work on ten pages and fall over on ten thousand.

Reuse one client

HttpClient is designed to be created once and reused. A new HttpClient() per request leaks sockets into TIME_WAIT and eventually throws SocketException under load — the single most reported .NET HTTP bug:

// One instance for the app's lifetime
private static readonly HttpClient Http = new(new SocketsHttpHandler
{
    AutomaticDecompression = DecompressionMethods.All,
    PooledConnectionLifetime = TimeSpan.FromMinutes(2),
    ConnectTimeout = TimeSpan.FromSeconds(10),
})
{
    Timeout = TimeSpan.FromSeconds(30),
};

In anything with dependency injection — ASP.NET Core, a worker service, a generic host — use IHttpClientFactory instead of the static field. It pools handlers, rotates them so DNS changes are picked up, and lets you attach retry and circuit-breaker policies in one line:

builder.Services.AddHttpClient("scraper", c =>
{
    c.Timeout = TimeSpan.FromSeconds(30);
    c.DefaultRequestHeaders.UserAgent.ParseAdd(
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36");
})
.AddStandardResilienceHandler();   // Microsoft.Extensions.Http.Resilience

Two settings above matter specifically for scraping. AutomaticDecompression sends Accept-Encoding and transparently unzips responses — browsers always do this, and servers notice clients that don't. A realistic User-Agent is the cheapest anti-blocking measure there is; the .NET default advertises itself as a .NET runtime.

Our C# HttpClient guide covers the full surface — POST bodies, authentication, cookies, multipart uploads, streaming downloads, and HttpClientHandler configuration.

Timeouts: the setting that decides whether your scraper finishes

HttpClient's default timeout is 100 seconds. On a crawl of ten thousand URLs, a handful of dead hosts at 100 seconds each is the difference between a job that finishes in minutes and one that appears to hang. Set a client-wide timeout, and use a CancellationToken when a single request needs a tighter one:

// Per-request deadline, independent of the client's Timeout
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(8));

try
{
    var response = await Http.GetAsync(url, cts.Token);
    response.EnsureSuccessStatusCode();
    var html = await response.Content.ReadAsStringAsync(cts.Token);
}
catch (TaskCanceledException ex) when (ex.InnerException is TimeoutException)
{
    Console.WriteLine($"Timed out: {url}");   // our deadline fired
}
catch (HttpRequestException ex)
{
    Console.WriteLine($"Request failed: {url} — {ex.StatusCode}");
}

Three things about HttpClient timeouts that surprise people:

• A timeout arrives as TaskCanceledException, not TimeoutException. Since .NET 5 the inner exception is a TimeoutException when the deadline fired, which is how you tell a timeout apart from a caller cancelling. Catch both cases explicitly — a bare catch (TaskCanceledException) will swallow your own shutdown signal.
• Timeout covers the whole operation, including reading the response body — not just the connection. A slow trickle of bytes counts against it, which is what you want for scraping. For a separate connection deadline, use SocketsHttpHandler.ConnectTimeout as in the snippet above.
• Timeout is immutable after the first request. Changing it later throws InvalidOperationException. Per-request control has to come from a CancellationToken.

A practical starting point for scraping: 10 seconds to connect, 30 seconds total, with retries on timeout rather than a longer deadline. Slow-but-alive hosts are usually better retried later than waited on.

HttpClient vs WebClient (and HttpWebRequest)

If you find WebClient in a tutorial or in your own codebase, the guidance is unambiguous: WebClient is obsolete. It's marked [Obsolete] in .NET 6 and later (diagnostic SYSLIB0014) and produces a build warning; Microsoft's own documentation tells you to use HttpClient instead. HttpWebRequest is likewise legacy — it still exists, but on modern .NET it's a compatibility shim implemented over the same sockets stack.

The one thing WebClient genuinely had going for it was terse one-liners — new WebClient().DownloadString(url). Modern HttpClient matches that: await Http.GetStringAsync(url). There is no remaining reason to prefer it, and porting is mostly mechanical:

// Old
var html = new WebClient().DownloadString(url);
new WebClient().DownloadFile(url, "page.html");

// Modern equivalent
var html = await Http.GetStringAsync(url);
await using var file = File.Create("page.html");
await (await Http.GetStreamAsync(url)).CopyToAsync(file);

Parsing HTML: Html Agility Pack vs AngleSharp

Neither of .NET's two main parsers runs JavaScript — they turn a string of HTML into a queryable tree. The choice between them comes down to query language and how faithfully you want the tree to match what a browser would build.

Html Agility Pack (HAP) is the older and more downloaded of the two. It's tolerant, fast, and XPath-first:

using HtmlAgilityPack;

var html = await Http.GetStringAsync("https://example.com/products");

var doc = new HtmlDocument();
doc.LoadHtml(html);

// SelectNodes returns null — not an empty collection — when nothing matches
var nodes = doc.DocumentNode.SelectNodes("//div[@class='product']");
foreach (var node in nodes ?? Enumerable.Empty<HtmlNode>())
{
    var name = node.SelectSingleNode(".//h2")?.InnerText.Trim();
    var price = node.SelectSingleNode(".//span[@class='price']")?.InnerText.Trim();
    Console.WriteLine($"{name}: {price}");
}

AngleSharp implements the WHATWG HTML5 parsing specification, so its DOM matches what Chrome builds from the same bytes. Its API is the W3C DOM you already know from JavaScript, with CSS selectors built in:

using AngleSharp;

var config = Configuration.Default.WithDefaultLoader();
var context = BrowsingContext.New(config);
var document = await context.OpenAsync("https://example.com/products");

foreach (var el in document.QuerySelectorAll("div.product"))
{
    var name = el.QuerySelector("h2")?.TextContent.Trim();
    var price = el.QuerySelector("span.price")?.TextContent.Trim();
    Console.WriteLine($"{name}: {price}");
}

Note that AngleSharp can fetch the page itself via WithDefaultLoader(), while HAP's equivalent (HtmlWeb.Load) is synchronous. In practice most scrapers fetch with a configured HttpClient — for the headers, proxy, and timeout control — and hand the string to the parser:

var html = await Http.GetStringAsync(url);
var document = await BrowsingContext.New().OpenAsync(req => req.Content(html));

Head to head:

Honest guidance: use AngleSharp for new projects. Spec-compliant parsing means your selectors match what you saw in DevTools, CSS selectors are what you copy out of the browser anyway, and the async API fits the rest of a .NET scraper. Html Agility Pack stays the right answer when XPath is your query language (it's genuinely more expressive for structural queries like "the table cell after the one containing 'Price'"), when you're maintaining existing HAP code, or when you need the lightest possible memory footprint across millions of documents. Both are MIT-licensed, actively maintained, and neither will ever execute JavaScript.

The null from SelectNodes deserves a specific warning — it's the most common NullReferenceException in .NET scraping code, and it fires exactly when a site changes its markup. Our Html Agility Pack guide covers that and the other HAP-specific traps (InnerText entity handling, the historical <form> parsing quirk, table extraction) in depth.

Playwright for .NET

When the data genuinely isn't in the HTML, Playwright is the default browser-automation choice for C#. Microsoft maintains the .NET bindings alongside the JavaScript ones, so features land at the same time rather than months later, and it drives Chromium, Firefox, and WebKit through one API.

Setup

dotnet new console -n Scraper && cd Scraper
dotnet add package Microsoft.Playwright
dotnet build

# Downloads the browser binaries (once per machine)
pwsh bin/Debug/net8.0/playwright.ps1 install

The install step needs PowerShell — dotnet tool install --global PowerShell if pwsh isn't on your machine. On CI, playwright.ps1 install --with-deps chromium also pulls the Linux system libraries Chromium needs, which saves a round of missing-shared-object errors.

A first scrape

using Microsoft.Playwright;

using var playwright = await Playwright.CreateAsync();
await using var browser = await playwright.Chromium.LaunchAsync(
    new BrowserTypeLaunchOptions { Headless = true });

var page = await browser.NewPageAsync();
await page.GotoAsync("https://example.com/products");

foreach (var card in await page.Locator(".product").AllAsync())
{
    var name = await card.Locator("h2").InnerTextAsync();
    var price = await card.Locator(".price").InnerTextAsync();
    Console.WriteLine($"{name}: {price}");
}

Let codegen write your selectors

Playwright's recorder opens a browser, follows your clicks, and emits C#. It is the fastest way to get working selectors for an unfamiliar site — and it prefers resilient locators (roles, labels, text) over brittle CSS paths:

pwsh bin/Debug/net8.0/playwright.ps1 codegen https://example.com

Waiting, done properly

The reason Playwright is pleasant to write against is auto-waiting: every action (ClickAsync, InnerTextAsync, FillAsync) waits for the element to exist, be visible, and be stable before acting. Most explicit waits people write are unnecessary. When you do need one, wait for a condition rather than a duration:

// Wait for content that arrives via XHR after load
await page.Locator(".results-loaded").WaitForAsync(
    new LocatorWaitForOptions { Timeout = 15_000 });

// Wait for the network call itself — more precise than a selector
var response = await page.RunAndWaitForResponseAsync(
    async () => await page.ClickAsync("#load-more"),
    r => r.Url.Contains("/api/products") && r.Status == 200);

var json = await response.JsonAsync();

// Global default for this page (milliseconds)
page.SetDefaultTimeout(15_000);

Avoid Task.Delay and WaitUntilState.NetworkIdle where you can. Fixed sleeps are simultaneously too slow on fast pages and too short on slow ones, and NetworkIdle never settles on pages with polling or analytics beacons — Playwright's own documentation discourages it.

Contexts, sessions, and parallelism

A BrowserContext is an isolated profile — its own cookies, storage, and cache — and it's much cheaper than launching another browser. That makes contexts the right unit for both session reuse and parallelism:

// Log in once, save the session, reuse it forever
var context = await browser.NewContextAsync();
var page = await context.NewPageAsync();
await page.GotoAsync("https://example.com/login");
await page.FillAsync("#username", user);
await page.FillAsync("#password", pass);
await page.ClickAsync("button[type=submit]");
await page.WaitForURLAsync("**/dashboard");
await context.StorageStateAsync(new() { Path = "state.json" });

// Later runs skip the login entirely
var restored = await browser.NewContextAsync(
    new BrowserNewContextOptions { StorageStatePath = "state.json" });

Running four to eight contexts in parallel against one browser instance is usually the throughput sweet spot; past that, memory rather than CPU becomes the limit.

PuppeteerSharp and Selenium

PuppeteerSharp is a faithful C# port of Puppeteer, driving Chrome over the DevTools Protocol. It's actively maintained and a fine choice if you're translating existing Puppeteer code or want CDP access directly — see our PuppeteerSharp guide. For a greenfield project, Playwright's auto-waiting and multi-engine support make it the safer default.

Selenium WebDriver has the largest C# install base of the three, almost entirely from test automation. As a scraping tool it's the weakest option: no auto-waiting (you write explicit WebDriverWait conditions yourself), a separate driver binary to version-match against the browser, and a slower protocol. The honest case for Selenium in C# is infrastructure you already own — an existing Selenium Grid, a test suite whose page objects you want to reuse, or a browser Playwright doesn't support. Starting fresh, pick Playwright.

None of the three is meaningfully better at avoiding bot detection. That fight is decided by IP reputation and TLS/browser fingerprinting, not by which automation library sends the commands.

IronWebScraper and ScrapySharp

Two library names come up in older "C# web scraping" articles and Stack Overflow answers. Both deserve a straight answer before you spend an afternoon on them.

ScrapySharp is a Scrapy-inspired wrapper that added CSS selectors and a fake-browser session on top of Html Agility Pack. It is abandoned: the last NuGet release is 3.0.0 from October 2018, with no updates in the years since. Its main draw — CSS selectors over HAP's tree — is now better served by AngleSharp natively, or by Fizzler if you want to stay on HAP. Don't start new work on it, and treat existing ScrapySharp code as a migration candidate.

IronWebScraper is a different situation: it's a commercial crawling framework from Iron Software, and it is actively maintained, shipping monthly releases. It offers what the rest of the .NET ecosystem doesn't — a Scrapy-style crawl loop with request scheduling, throttling, and pipelines built in, rather than parts you assemble. The tradeoffs are that it requires a paid license for production use (the free key is development-only) and that adoption is small: roughly 142,000 total NuGet downloads against tens of millions for Html Agility Pack, which means far less community troubleshooting when you hit an edge case. If you want the framework ergonomics and the license fits your budget, it's a legitimate choice. If you'd rather not pay or not depend on a single vendor, HttpClient + AngleSharp + the concurrency patterns in the next section covers the same ground in a few dozen lines.

Async and parallel scraping patterns

This is where C# earns its place. async/await over the .NET thread pool handles thousands of concurrent HTTP requests without a thread per request — but unbounded concurrency is a way to get rate-limited and IP-banned. Bound it.

Bounded concurrency with SemaphoreSlim

The classic pattern: launch everything, but let only N run at a time.

var gate = new SemaphoreSlim(8);           // 8 concurrent requests

var tasks = urls.Select(async url =>
{
    await gate.WaitAsync();
    try
    {
        return await Http.GetStringAsync(url);
    }
    catch (HttpRequestException)
    {
        return null;                        // one failure shouldn't sink the batch
    }
    finally
    {
        gate.Release();                     // always, even on exception
    }
});

var pages = (await Task.WhenAll(tasks)).Where(p => p is not null).ToList();

Two things worth internalizing. Task.WhenAll throws only the first exception even when several tasks fail, so catching inside each task — as above — is usually what you want for scraping. And gate.Release() belongs in a finally; miss it on an exception path and your scraper silently deadlocks after N failures.

Parallel.ForEachAsync

Since .NET 6 the framework has a cleaner form of the same thing, with cancellation built in:

await Parallel.ForEachAsync(
    urls,
    new ParallelOptions { MaxDegreeOfParallelism = 8 },
    async (url, ct) =>
    {
        var html = await Http.GetStringAsync(url, ct);
        var doc = await BrowsingContext.New().OpenAsync(r => r.Content(html), ct);
        var title = doc.QuerySelector("h1")?.TextContent;
        Console.WriteLine($"{url}: {title}");
    });

This is the shortest correct answer for "scrape a known list of URLs" and should be your default.

Channels for producer/consumer crawls

When the URL list isn't known up front — a crawl that discovers links as it goes — System.Threading.Channels gives you a lock-free queue that plugs straight into await foreach:

using System.Threading.Channels;

var channel = Channel.CreateUnbounded<string>();
var seen = new HashSet<string>();
await channel.Writer.WriteAsync("https://example.com/page/1");

var workers = Enumerable.Range(0, 8).Select(async _ =>
{
    await foreach (var url in channel.Reader.ReadAllAsync())
    {
        var html = await Http.GetStringAsync(url);
        var doc = await BrowsingContext.New().OpenAsync(r => r.Content(html));

        foreach (var link in doc.QuerySelectorAll("a.next[href]"))
        {
            var next = link.GetAttribute("href")!;
            lock (seen)
            {
                if (!seen.Add(next)) continue;
            }
            await channel.Writer.WriteAsync(next);
        }
    }
});

await Task.WhenAll(workers);

The subtlety in crawls like this is knowing when you're done — the channel is only empty because workers are between items, not because the crawl finished. Track in-flight work with a counter and call channel.Writer.Complete() when it hits zero, or bound the crawl by depth or page count.

Politeness

Concurrency limits are also courtesy limits. A few conventions that keep scrapers welcome: honor robots.txt, add a small delay between requests to the same host rather than only capping global concurrency, respect Retry-After on 429 responses, and back off exponentially on 5xx. Hammering a rate limiter earns bans, not data — see our notes on web scraping legality for the rules beyond etiquette.

Beyond HTML: files, XML, PDFs, and certificates

Real scraping jobs rarely stop at HTML.

Downloading files should stream rather than buffer. HttpCompletionOption.ResponseHeadersRead returns as soon as headers arrive, so a 2 GB dataset never lands in memory:

using var response = await Http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();

await using var source = await response.Content.ReadAsStreamAsync();
await using var target = File.Create("dataset.zip");
await source.CopyToAsync(target);

The suggested filename, when the server provides one, is in response.Content.Headers.ContentDisposition?.FileNameStar ?? .FileName — always sanitize it before using it as a path.

XML and RSS feeds are the easiest targets in .NET: XDocument plus LINQ to XML handles most of them in a few lines, and XmlReader streams documents too large to hold in memory. Watch for namespaces — an XNamespace prefix is required on every element name once a feed declares one, and forgetting it is why Descendants("item") silently returns nothing.

var xml = await Http.GetStringAsync("https://example.com/feed.xml");
var feed = XDocument.Parse(xml);

foreach (var item in feed.Descendants("item"))
    Console.WriteLine(item.Element("title")?.Value);

PDFs need a dedicated library — .NET has none built in. PdfPig (Apache 2.0) and iText 7 (AGPL, or commercial) are the two common picks; PdfPig is the easier starting point for text and table extraction, and its licence doesn't force your own project open. Scanned PDFs contain images rather than text and need OCR on top, which is a different and much larger problem.

SSL certificate errors show up as HttpRequestException wrapping an AuthenticationException, usually on sites with expired, self-signed, or incomplete certificate chains. The advice you'll find first — accept everything — is a real security hole, because it disables the protection TLS exists to provide and makes you interceptable on any network:

// Development and known-bad hosts only. Never a blanket production setting.
var handler = new HttpClientHandler
{
    ServerCertificateCustomValidationCallback = (request, cert, chain, errors) =>
        request.RequestUri?.Host == "known-broken.example.com" || errors == SslPolicyErrors.None,
};

Scoping the exception to a specific host, as above, keeps validation on everywhere else. If a target's certificate is broken for everyone, that's the target's bug, and a per-host exception documents it honestly.

When the page fights back

Everything above assumes the server answers. Increasingly it doesn't: Cloudflare, DataDome, PerimeterX, and their peers fingerprint the TLS handshake, header ordering, and browser environment. A .NET TLS stack does not look like Chrome no matter which headers you set, and a headless browser announces itself in dozens of subtle ways. Add rotating proxies, CAPTCHA solving, and the ops cost of running a Chrome fleet, and the scraper stops being a parsing problem.

WebScraping.AI handles that layer behind a plain HTTP endpoint — real Chromium rendering, rotating datacenter and residential proxies — so your existing C# code keeps working with a different URL:

var apiKey = Environment.GetEnvironmentVariable("WEBSCRAPING_AI_API_KEY");
var target = Uri.EscapeDataString("https://example.com/spa-products");

// Rendered HTML — hand it to AngleSharp or HAP exactly as before
var html = await Http.GetStringAsync(
    $"https://api.webscraping.ai/html?api_key={apiKey}&url={target}&js=true&proxy=residential");

// Or skip parsing entirely: describe the fields in English
var fields = await Http.GetFromJsonAsync<Dictionary<string, string>>(
    $"https://api.webscraping.ai/ai/fields?api_key={apiKey}&url={target}" +
    "&fields[name]=Product name&fields[price]=Price with currency");

There's also an official .NET SDK if you'd rather have typed requests, a typed exception hierarchy, and CancellationToken support than build query strings by hand:

// dotnet add package WebScrapingAI
using WebScrapingAI;

using var client = new WebScrapingAIClient();   // reads WEBSCRAPING_AI_API_KEY

string html = await client.HtmlAsync(new HtmlRequest { Url = url, Js = true });

FieldsResult result = await client.FieldsAsync(new FieldsRequest
{
    Url = url,
    Fields = new Dictionary<string, string>
    {
        ["price"] = "Current price with currency",
        ["stock"] = "In stock or out of stock",
    },
});
Console.WriteLine(result.Result?["price"]);

The /text and /selected endpoints return clean text and CSS-selected fragments, and /ai/question answers a question about a page in plain language. Full parameter list — proxy, country, wait_for, device, headers — is in the API documentation. Typical jobs this fits: price monitoring across retailers that render client-side, or B2B lead generation from directories behind bot protection.

Frequently asked questions

Is C# good for web scraping? Yes, particularly for scrapers that need to run continuously, concurrently, or inside an existing .NET system. async/await handles high concurrency cleanly, the parsers are mature, and static typing catches extraction bugs at compile time rather than in production. What C# lacks is Python's batteries-included framework — there's no Scrapy equivalent outside the commercial IronWebScraper — so you assemble the fetch, parse, and concurrency layers yourself. For a one-off script, Python is quicker to start; for a service you'll maintain for years, C# holds up better.

What's the best C# library for web scraping? There isn't one library — there's a stack. For static sites: HttpClient for fetching, AngleSharp for parsing. For JavaScript-rendered sites: Playwright for .NET. Html Agility Pack replaces AngleSharp if you prefer XPath. That combination covers the overwhelming majority of scraping work in .NET.

Do I need a headless browser to scrape JavaScript-rendered sites? Often not. Check the Network tab first: most client-rendered pages fetch their data from a JSON endpoint that HttpClient can call directly, which is faster, more stable, and easier to parse than the rendered DOM. Reach for a browser when the data is genuinely assembled in the page, when you need to interact with it, or when the API is signed in a way you can't reproduce.

How do I fix "The SSL connection could not be established" in C#? It usually means an expired certificate, a self-signed certificate, or an incomplete chain on the target. Diagnose it first — openssl s_client -connect host:443 shows the chain the server actually sends. Only then decide: fix the trust store if the CA is missing locally, or add a host-scoped ServerCertificateCustomValidationCallback exception. Don't disable validation globally.

How many concurrent requests should a C# scraper make? Start at 5–10 per target host and measure. The right number is bounded by the target's tolerance, not your machine's — .NET will happily saturate a link that gets you blocked in minutes. Use SemaphoreSlim or Parallel.ForEachAsync to cap it, add a delay between requests to the same host, and back off when you see 429s.

Is Html Agility Pack still maintained? Yes — it's actively developed, MIT-licensed, and the most-downloaded HTML parser on NuGet. "Use AngleSharp for new projects" is a recommendation about spec fidelity and API ergonomics, not a warning that HAP is dying.

Can I scrape sites that require a login? Yes. With HttpClient, attach a CookieContainer to the handler, POST the credentials, and the session cookie rides along automatically. With Playwright, log in once and save the session with StorageStateAsync, then restore it on later runs. Logins behind CAPTCHAs or JavaScript challenges need a browser plus proxy infrastructure, or a rendering API.

Does scraping with C# require .NET on Linux? No — .NET runs cross-platform, and scrapers deploy comfortably to Linux containers. The one thing to plan for is browser automation: Playwright and PuppeteerSharp need Chromium's shared libraries in the image, which playwright.ps1 install --with-deps handles on Debian and Ubuntu bases.