Html Agility Pack: C# HTML Parsing and Web Scraping Guide

Html Agility Pack (HAP) is the most-downloaded HTML parser on NuGet — a .NET library that reads real-world, broken HTML into a queryable DOM and lets you search it with XPath or LINQ. This guide covers the workflow end to end: installing, loading documents, selecting nodes by XPath, class, and attribute, extracting clean text and tables, modifying and sanitizing markup, the classic gotchas (null SelectNodes, <form> parsing, InnerText entities), and when AngleSharp is the better pick.

Key Takeaways

• Install with dotnet add package HtmlAgilityPack; load HTML with HtmlDocument.LoadHtml() — prefer HttpClient for fetching over the bundled HtmlWeb
• SelectNodes() returns null, not an empty list, when nothing matches — the number-one source of NullReferenceException in HAP code
• HAP speaks XPath and LINQ natively; CSS selectors need an extension package (Fizzler) or a different parser (AngleSharp)
• Match classes with contains(concat(' ', normalize-space(@class), ' '), ' name ') — a naive contains(@class, 'name') also matches name-large
• InnerText neither strips <script> contents nor decodes entities reliably — use a text-node walk plus HtmlEntity.DeEntitize() for clean text
• HAP tolerates malformed HTML by design, but <form> elements parse as empty by default (HtmlNode.ElementsFlags) — know the quirks before debugging

Installing Html Agility Pack

dotnet add package HtmlAgilityPack

Or in the .csproj directly:

<PackageReference Include="HtmlAgilityPack" Version="1.12.*" />

HAP targets .NET Standard 2.0, so it runs on .NET Framework 4.6.1+, all modern .NET (6/8/10), Xamarin, and Unity. It's MIT-licensed and free for commercial use.

Loading HTML: strings, files, and URLs

using HtmlAgilityPack;

// From a string (the common case when you fetch with HttpClient)
var doc = new HtmlDocument();
doc.LoadHtml(html);

// From a file
var fileDoc = new HtmlDocument();
fileDoc.Load("page.html");

// From a URL with the built-in loader
var web = new HtmlWeb();
var urlDoc = web.Load("https://example.com/");

For anything beyond a quick script, fetch with HttpClient and pass the string to LoadHtml — you control headers, timeouts, retries, and connection pooling, none of which HtmlWeb handles well:

using var http = new HttpClient();
http.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/126.0");

var html = await http.GetStringAsync("https://example.com/products");
var doc = new HtmlDocument();
doc.LoadHtml(html);

(Our C# HttpClient guide covers that fetching side in depth.)

Selecting nodes with XPath

XPath is HAP's native query language. The two workhorses:

// All matches — or NULL if none
HtmlNodeCollection rows = doc.DocumentNode.SelectNodes("//table[@id='results']//tr");

// First match — or null
HtmlNode title = doc.DocumentNode.SelectSingleNode("//h1");

SelectNodes returns null when nothing matches, not an empty collection. Every HAP codebase eventually grows this pattern:

var links = doc.DocumentNode.SelectNodes("//a[@href]") ?? Enumerable.Empty<HtmlNode>();
foreach (var link in links)
{
    Console.WriteLine($"{link.InnerText.Trim()} -> {link.GetAttributeValue("href", "")}");
}

Common XPath patterns you'll reuse constantly:

One subtlety when querying within a node: node.SelectNodes("//td") searches the whole document, because // is anchored at the root. Use a leading dot for a relative search: node.SelectNodes(".//td"). For a deeper XPath reference, see our XPath cheat sheet.

Selecting by class and id

// By id — direct helper
HtmlNode main = doc.GetElementbyId("main");   // note the lowercase 'b' — historical API quirk

// By class — the correct way
var cards = doc.DocumentNode.SelectNodes(
    "//div[contains(concat(' ', normalize-space(@class), ' '), ' product-card ')]");

Why the concat/normalize-space dance? Because class is a space-separated list and XPath 1.0 has no "has class" function. contains(@class, 'card') happily matches class="card-header" and class="discard"; wrapping both haystack and needle in spaces makes the match token-exact.

If you'd rather write CSS selectors, add Fizzler (doc.DocumentNode.QuerySelectorAll(".product-card > h2")) — or use AngleSharp, which supports them natively (comparison below).

Filtering by attributes — XPath or LINQ

HAP nodes are fully LINQ-queryable, and for complex conditions LINQ often reads better than a heroic XPath expression:

// XPath: attribute predicates
var inStock = doc.DocumentNode.SelectNodes("//div[@data-status='in-stock' and @data-price]");

// LINQ: same query, composable and debuggable
var cheapInStock = doc.DocumentNode
    .Descendants("div")
    .Where(d => d.GetAttributeValue("data-status", "") == "in-stock"
             && d.GetAttributeValue("data-price", 0m) < 20m)
    .ToList();

GetAttributeValue(name, defaultValue) is the safe accessor — it never throws on a missing attribute and picks its return type from the default you pass. Descendants() / Elements() / Ancestors() give you the axes; unlike SelectNodes, LINQ methods return empty sequences rather than null, which makes them pleasant to chain.

Extracting text (and the InnerText traps)

InnerText concatenates every text node underneath an element. Two things bite people:

1. Script and style contents are text nodes too — document.InnerText on a full page includes JavaScript source and CSS.
2. Entities are not decoded — you get &, –,   literally.

Clean plain-text extraction looks like this:

// Strip non-content elements first
foreach (var node in doc.DocumentNode.SelectNodes("//script|//style|//noscript|//comment()")
                     ?? Enumerable.Empty<HtmlNode>())
{
    node.Remove();
}

// Collect text nodes, decode entities, normalize whitespace
var text = string.Join(" ",
    doc.DocumentNode.DescendantsAndSelf()
        .Where(n => n.NodeType == HtmlNodeType.Text)
        .Select(n => HtmlEntity.DeEntitize(n.InnerText))
        .Select(t => t.Trim())
        .Where(t => t.Length > 0));

For a single element, HtmlEntity.DeEntitize(node.InnerText).Trim() is usually enough. There's no built-in "render like a browser" text mode (line breaks for <br> and <p>) — if you need readable formatted output, walk the tree and emit newlines on block elements, or hand the HTML to a converter library.

Scraping tables

The bread-and-butter task, with the relative-path gotcha handled:

var table = doc.DocumentNode.SelectSingleNode("//table[@id='prices']");
var headers = table.SelectNodes(".//th")!.Select(th => th.InnerText.Trim()).ToList();

foreach (var row in table.SelectNodes(".//tbody/tr") ?? Enumerable.Empty<HtmlNode>())
{
    var cells = row.SelectNodes(".//td");
    if (cells == null || cells.Count < headers.Count) continue;   // skip spacer rows

    var record = headers.Zip(cells, (h, c) => (h, v: HtmlEntity.DeEntitize(c.InnerText).Trim()))
                        .ToDictionary(x => x.h, x => x.v);
    // record["Price"], record["Product"], ...
}

Modifying documents: adding and removing nodes

HAP writes HTML as well as it reads it:

var container = doc.GetElementbyId("content");

// Create from markup and append
var banner = HtmlNode.CreateNode("<div class='banner'>Archived copy</div>");
container.PrependChild(banner);

// Create programmatically
var link = doc.CreateElement("a");
link.SetAttributeValue("href", "/source");
link.InnerHtml = "Original source";
container.AppendChild(link);

// Insert relative to siblings
container.InsertAfter(doc.CreateElement("hr"), banner);

// Remove and replace
doc.DocumentNode.SelectSingleNode("//aside")?.Remove();

doc.Save("modified.html");                 // or doc.DocumentNode.OuterHtml for the string

Cleaning HTML content

Typical cleanup — dropping unwanted elements and attributes before storing scraped content:

// Remove scripts, styles, iframes, comments
foreach (var n in doc.DocumentNode.SelectNodes("//script|//style|//iframe|//comment()")
                  ?? Enumerable.Empty<HtmlNode>())
    n.Remove();

// Strip event handlers and inline styles everywhere
foreach (var el in doc.DocumentNode.Descendants().Where(d => d.NodeType == HtmlNodeType.Element))
{
    foreach (var attr in el.Attributes.Where(a => a.Name.StartsWith("on") || a.Name == "style").ToList())
        attr.Remove();
}

One important boundary: this is cleanup, not security sanitization. If the HTML is untrusted and will be re-rendered to users, use a purpose-built sanitizer (HtmlSanitizer, which itself builds on AngleSharp) — hand-rolled tag stripping is how XSS slips through.

Handling malformed HTML and common quirks

Tolerating broken markup is HAP's founding feature — unclosed tags, misnested elements, and tag-soup all produce a usable DOM. The knobs and quirks worth knowing when output looks wrong:

var doc = new HtmlDocument
{
    OptionFixNestedTags = true,          // repair misnested tags
    OptionAutoCloseOnEnd = true,         // close dangling tags at EOF
};
doc.LoadHtml(html);

// See what the parser had to fix
foreach (var err in doc.ParseErrors)
    Console.WriteLine($"{err.Line}:{err.LinePosition} {err.Code} - {err.Reason}");

• <form> parses as an empty node by default — historical behavior because forms can legally overlap table markup. Its inputs appear as siblings, not children. Fix before loading: HtmlNode.ElementsFlags.Remove("form");
• <option> contents behave similarly via the same ElementsFlags table.
• Encoding looks garbled? Load streams rather than pre-decoded strings when possible (doc.Load(stream) auto-detects from BOM/meta), or set OptionReadEncoding = false when servers lie in their headers. Mojibake like ’ means the bytes were decoded with the wrong charset before HAP saw them.
• Whole document inside one giant node? You probably loaded a fragment with Load() (file path expected) instead of LoadHtml() (string expected).

Memory and performance

An HtmlDocument is a plain managed object graph — there's nothing to dispose; it's garbage-collected when unreferenced. Practical tips for high-volume parsing:

• Parse, extract into your own POCOs, and let the document go out of scope — don't cache HtmlNode references long-term, as each pins the entire DOM in memory
• Prefer anchored XPath (//table[@id='x']//tr) over repeated whole-document scans; run one SelectNodes and iterate rather than re-querying in a loop
• A parsed DOM costs several times the raw HTML size in memory — for a 100 MB export, process documents one at a time rather than keeping a list of parsed docs
• HAP has no streaming/SAX mode; if you genuinely need to process HTML too large for a DOM, split the input or fall back to targeted regex/Span scanning for that one field

Html Agility Pack vs AngleSharp (and friends)

Honest guidance: for a new C# scraping project, AngleSharp's spec-compliant parsing and CSS selectors are a nicer foundation, and it's what HtmlSanitizer builds on. HAP remains the pragmatic choice when XPath is your query language, when you're maintaining existing HAP code, or when you want the massive install base and answer coverage. Both parse only — neither executes JavaScript.

Html Agility Pack for web scraping

HAP parses whatever HTML you hand it — the hard part of scraping is getting HTML worth parsing. JavaScript-rendered SPAs return empty shells to HttpClient, and bot protection blocks clean requests no matter how correct your parser is. WebScraping.AI handles that fetch side — real browser rendering with rotating proxies — and returns HTML that drops straight into your existing HAP code:

using var http = new HttpClient();
var url = "https://api.webscraping.ai/html" +
          $"?api_key={apiKey}&url={Uri.EscapeDataString("https://example.com/spa-products")}&js=true";

var html = await http.GetStringAsync(url);   // rendered in a real browser, proxies included

var doc = new HtmlDocument();
doc.LoadHtml(html);                          // your selectors work unchanged
var prices = doc.DocumentNode.SelectNodes("//span[@class='price']");

If you'd rather skip selectors entirely, the /ai/fields endpoint returns structured JSON from field descriptions in plain English.

Frequently asked questions

Is Html Agility Pack free for commercial use? Yes — MIT license, free everywhere including commercial products. The project accepts sponsorships but there's no paid tier for the library itself.

Why does SelectNodes return null instead of an empty list? Long-standing API design that predates modern .NET conventions, kept for backward compatibility. Guard with ?? Enumerable.Empty<HtmlNode>() or null-conditional access (nodes?.Count ?? 0), or use the LINQ axes (Descendants()), which do return empty sequences.

Does Html Agility Pack support CSS selectors? Not natively — XPath and LINQ are built in. Add the Fizzler.Systems.HtmlAgilityPack package for QuerySelector/QuerySelectorAll over HAP documents, or choose AngleSharp if CSS selectors are your primary interface.

Can Html Agility Pack execute JavaScript? No. HAP parses static markup only; content that a browser builds with JavaScript simply isn't in the HTML it sees. Render the page first — with a headless browser like PuppeteerSharp or a rendering API (see the scraping section above) — then parse the resulting HTML with HAP.

Why is InnerText showing & and other entities? InnerText returns raw text-node content without entity decoding. Wrap it: HtmlEntity.DeEntitize(node.InnerText). Combine with stripping <script>/<style> elements first, or their source code lands in your "text" too.

Html Agility Pack or AngleSharp — which should I choose? New project with no XPath investment: AngleSharp (browser-grade parsing, CSS selectors, W3C-style DOM). Existing HAP codebase, XPath-centric team, or need for the largest body of Stack Overflow answers: HAP. For pure scraping throughput the difference is rarely decisive — pick by API preference.