C# Web Scraping: The Complete Guide

I have written scrapers in several languages, and C# is the one I reach for when the work lives inside an existing .NET service: an ASP.NET backend, a worker process, a desktop tool. This guide is the map for that path. Which library does what, a full HttpClient plus HtmlAgilityPack walkthrough you can compile, the same job in AngleSharp with CSS selectors, how to handle JavaScript-heavy pages with Playwright and Selenium, and how to keep a scraper alive once a site pushes back. The code uses the documented API for each library at its current 2026 version. I point the examples at books.toscrape.com, a sandbox built for scraping practice.

Is C# good for web scraping?

Yes, C# is a strong choice for web scraping, particularly when your code already runs on .NET. The stack is mature: HttpClient ships with the runtime, HtmlAgilityPack and AngleSharp parse HTML, and Playwright or Selenium drive a browser for JavaScript. Static types catch selector and field-mapping bugs at compile time, and async/await handles large concurrent crawls without extra machinery.

Three things make C# pull its weight here:

• It fits an existing service. If your product is an ASP.NET app or a .NET worker, a scraper in the same language shares your build, logging, configuration, and deployment instead of bolting on a second runtime.
• The HTTP client is built in. HttpClient is part of the base class library, so for static pages and JSON APIs you add one parsing dependency and nothing else.
• Async is first-class. async/await and Task let one process fetch many pages concurrently with back-pressure, no framework required.

Python has more scraping tutorials and a deeper bench of throwaway scripts, which I cover in the Python guide. C# wins when the scraper is part of a larger typed system rather than a standalone script.

What are the main C# web scraping libraries?

Four tools cover the whole field, and the right one depends on whether the page needs JavaScript and which selector style you like. Static HTML needs HttpClient plus a parser (HtmlAgilityPack or AngleSharp). JavaScript-rendered pages need a browser engine, and there you choose between Playwright and Selenium.

A few notes from using all of them:

• HtmlAgilityPack is the long-standing C# default. It is forgiving with malformed HTML and uses XPath, which is powerful for structural queries. A CSS-selector add-on exists if you prefer that syntax.
• AngleSharp builds a real W3C DOM and uses the same CSS selectors you know from the browser and from document.querySelector. It is the more modern, standards-driven option.
• Playwright for .NET is the browser tool I reach for first now. It manages its own browser binaries, has a clean async API, and auto-waits for elements.
• Selenium drives an actual browser through WebDriver and remains the most widely deployed option, with the largest body of existing examples.

HttpClient is the common front door for the two parsers: it fetches the HTML, then HtmlAgilityPack or AngleSharp turns that string into something you can query.

How do you set up a C# scraper with the .NET CLI?

Create a console project and add the packages with dotnet add package. The .NET CLI handles project creation and NuGet restore, so the whole setup is four commands. These are the current stable versions on NuGet as of June 2026.

dotnet new console -n BookScraper
cd BookScraper
dotnet add package HtmlAgilityPack --version 1.12.4
dotnet add package CsvHelper --version 33.1.0

HttpClient needs no package; it lives in System.Net.Http in the runtime. Pin versions with --version on a real project so a later release does not silently change behavior under you. Omit the flag to take the latest stable.

How do you scrape a website with C# step by step?

The loop is two steps: fetch the page into a string with HttpClient, then load that string into an HtmlDocument and pull fields with XPath. HtmlAgilityPack does the parsing; HttpClient does the fetch.

This program fetches the first catalogue page and prints the title and price for every book on it:

using System;
using System.Net.Http;
using System.Threading.Tasks;
using HtmlAgilityPack;

class BookScraper
{
    static readonly HttpClient Http = new HttpClient();

    static async Task Main()
    {
        const string url = "https://books.toscrape.com/catalogue/page-1.html";

        Http.DefaultRequestHeaders.UserAgent.ParseAdd(
            "Mozilla/5.0 (compatible; BookScraper/1.0)");

        string html = await Http.GetStringAsync(url);

        var doc = new HtmlDocument();
        doc.LoadHtml(html);

        var cards = doc.DocumentNode.SelectNodes("//article[@class='product_pod']");
        Console.WriteLine($"{cards.Count} books on this page");

        foreach (var card in cards)
        {
            string title = card.SelectSingleNode(".//h3/a")
                               .GetAttributeValue("title", "");
            string price = card.SelectSingleNode(".//p[@class='price_color']")
                               .InnerText.Trim();
            Console.WriteLine($"{price,-8} {HtmlEntity.DeEntitize(title)}");
        }
    }
}

Each part maps to one job:

• new HttpClient() as a static readonly field reuses one client for the process. Creating a fresh HttpClient per request exhausts sockets under load, so share one instance.
• DefaultRequestHeaders.UserAgent.ParseAdd(...) sets a browser-like User-Agent. The default has none, and many sites reject requests without one, so set it on every real scrape.
• await Http.GetStringAsync(url) issues the GET and returns the body as a string. For status-code control, use GetAsync and read response.Content.
• doc.LoadHtml(html) parses the string into a node tree. HtmlAgilityPack tolerates broken markup, so real-world HTML loads without throwing.
• SelectNodes("//article[@class='product_pod']") runs an XPath query over the document and returns every match. SelectSingleNode(...) returns the first match.
• GetAttributeValue("title", "") reads an attribute with a default fallback, and InnerText reads the visible text. HtmlEntity.DeEntitize(...) turns & and friends back into real characters.

Run it from the project directory:

dotnet run

This is the approach I would ship for a static page. The XPath strings are the only part that changes per site, so the pattern transfers to any target whose data is in the initial HTML.

How do you parse HTML with AngleSharp and CSS selectors?

Use AngleSharp when you would rather write CSS selectors than XPath. AngleSharp builds a real W3C DOM, so QuerySelector and QuerySelectorAll behave like they do in a browser. Add the package first:

dotnet add package AngleSharp --version 1.5.0

The same book scrape, with AngleSharp fetching and parsing in one step through its BrowsingContext:

using System;
using System.Threading.Tasks;
using AngleSharp;
using AngleSharp.Dom;

class AngleScraper
{
    static async Task Main()
    {
        var config = Configuration.Default.WithDefaultLoader();
        var context = BrowsingContext.New(config);

        const string url = "https://books.toscrape.com/catalogue/page-1.html";
        using IDocument doc = await context.OpenAsync(url);

        var cards = doc.QuerySelectorAll("article.product_pod");
        Console.WriteLine($"{cards.Length} books on this page");

        foreach (var card in cards)
        {
            string title = card.QuerySelector("h3 a")?.GetAttribute("title") ?? "";
            string price = card.QuerySelector(".price_color")?.TextContent.Trim() ?? "";
            Console.WriteLine($"{price,-8} {title}");
        }
    }
}

What is different from HtmlAgilityPack:

• Configuration.Default.WithDefaultLoader() enables AngleSharp’s built-in document loader so OpenAsync can fetch over HTTP. Without the loader, AngleSharp parses strings you supply but does not download anything.
• context.OpenAsync(url) fetches and parses in one call and returns an IDocument. To parse a string you already have, use context.OpenAsync(req => req.Content(html)).
• QuerySelectorAll("article.product_pod") takes a CSS selector and returns a live element list. QuerySelector(...) returns the first match or null, which is why the code uses the ?. null-conditional operator.
• GetAttribute("title") reads an attribute and TextContent reads the text, the DOM property names you already know from JavaScript.

Both libraries handle messy markup. Pick HtmlAgilityPack for XPath, AngleSharp for CSS selectors and a browser-accurate DOM. The selector strings are the only meaningful difference in day-to-day use, which I go deeper on in the XPath and CSS selectors guide.

How do you follow links and handle pagination in C#?

Follow the “next” link until it disappears. Most paginated lists put a next-page anchor at the bottom of each page, so you scrape a page, look for that link, resolve it to an absolute URL, and repeat until there is no next link.

This crawler walks the whole catalogue with HtmlAgilityPack, resolving each relative next link against the current page URL:

using System;
using System.Net.Http;
using System.Threading.Tasks;
using HtmlAgilityPack;

class CatalogueCrawler
{
    static readonly HttpClient Http = new HttpClient();

    static async Task Main()
    {
        var url = new Uri("https://books.toscrape.com/catalogue/page-1.html");
        int pages = 0, total = 0;

        Http.DefaultRequestHeaders.UserAgent.ParseAdd(
            "Mozilla/5.0 (compatible; CatalogueCrawler/1.0)");

        while (url != null)
        {
            string html = await Http.GetStringAsync(url);
            var doc = new HtmlDocument();
            doc.LoadHtml(html);

            total += doc.DocumentNode
                        .SelectNodes("//article[@class='product_pod']").Count;
            pages++;

            var next = doc.DocumentNode.SelectSingleNode("//li[@class='next']/a");
            url = next != null
                ? new Uri(url, next.GetAttributeValue("href", ""))
                : null;

            await Task.Delay(1000); // be polite: one request per second
        }

        Console.WriteLine($"pages crawled: {pages}");
        Console.WriteLine($"total books:   {total}");
    }
}

Two details make this robust on a real target:

• new Uri(url, next.GetAttributeValue("href", "")) resolves the relative href against the current page URL. HtmlAgilityPack does not join URLs for you the way some parsers do, so the Uri two-argument constructor does the work and handles ../ paths correctly.
• await Task.Delay(1000) paces the crawl to roughly one request per second. On a production job, wrap the fetch in a try/catch so one failed page does not kill the run, and read the site’s robots.txt and terms first.

Against the full books.toscrape.com catalogue this pattern walks all 50 listing pages and the 1,000 books across them. The same “find the next anchor, resolve it, repeat” loop handles any next-button pagination.

How do you save scraped data to CSV in C#?

Map each record to a class and let CsvHelper write the file. Define a type for one row, collect a list of them while you scrape, then hand the list to a CsvWriter. CsvHelper handles quoting, escaping, and headers.

using System.Collections.Generic;
using System.Globalization;
using System.IO;
using CsvHelper;

record Book(string Title, decimal Price);

class CsvExport
{
    static void Write(List<Book> books, string path)
    {
        using var writer = new StreamWriter(path);
        using var csv = new CsvWriter(writer, CultureInfo.InvariantCulture);
        csv.WriteRecords(books);
    }
}

What matters here:

• record Book(string Title, decimal Price) defines the row shape in one line. CsvHelper reads the property names for the header row, so the CSV columns become Title,Price.
• CultureInfo.InvariantCulture fixes the number and delimiter format so the output is identical on every machine. A culture that uses a comma decimal separator would otherwise clash with the CSV comma.
• WriteRecords(books) writes the header and every row in one call. To append instead, configure HasHeaderRecord = false and open the stream in append mode.

Parse prices to decimal while scraping with decimal.TryParse(text, NumberStyles.Currency, CultureInfo.InvariantCulture, out var value) so the CSV holds real numbers you can sum, not strings with currency symbols.

How do you scrape JavaScript-heavy sites in C#?

Use a browser engine, because neither HtmlAgilityPack nor AngleSharp runs JavaScript. When a page loads its data with scripts after the initial response, that data is absent from the HTML those parsers receive, so your selectors find nothing. You have two main C# options: Playwright for .NET or Selenium. Both render the page, then you parse the rendered HTML with the parser you already know.

Option A: Playwright for .NET

Playwright manages its own browser binaries and has a clean async API, which makes it the lighter setup. Add the package, build, then run its install script to download the browsers:

dotnet add package Microsoft.Playwright --version 1.60.0
dotnet build
pwsh bin/Debug/net8.0/playwright.ps1 install

Launch Chromium headless, navigate, and read the rendered HTML into HtmlAgilityPack so your parsing code stays identical:

using System;
using System.Threading.Tasks;
using Microsoft.Playwright;
using HtmlAgilityPack;

class PlaywrightScraper
{
    static async Task Main()
    {
        using var playwright = await Playwright.CreateAsync();
        await using var browser = await playwright.Chromium.LaunchAsync(
            new BrowserTypeLaunchOptions { Headless = true });

        var page = await browser.NewPageAsync();
        await page.GotoAsync("https://books.toscrape.com",
            new PageGotoOptions { WaitUntil = WaitUntilState.NetworkIdle });

        string html = await page.ContentAsync();

        var doc = new HtmlDocument();
        doc.LoadHtml(html);
        // parse with the same XPath as before:
        // doc.DocumentNode.SelectNodes("//article[@class='product_pod']") ...
        Console.WriteLine($"rendered {html.Length} bytes");
    }
}

What matters here:

• using var playwright and await using var browser dispose both objects automatically, so the browser process closes when the method exits.
• WaitUntil = WaitUntilState.NetworkIdle waits until network activity settles before reading, which covers data fetched by background scripts. For a specific element, prefer await page.Locator(".product").First.WaitForAsync().
• await page.ContentAsync() returns the fully rendered HTML, which LoadHtml(...) turns into the same document you already query.

Option B: Selenium WebDriver

Selenium drives an actual Chrome or Firefox and has the largest body of existing examples. Recent Selenium manages the driver binary for you (Selenium Manager, built in since 4.6), so there is no separate driver download.

dotnet add package Selenium.WebDriver

using System;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using HtmlAgilityPack;

class SeleniumScraper
{
    static void Main()
    {
        var options = new ChromeOptions();
        options.AddArgument("--headless=new");
        using var driver = new ChromeDriver(options);

        driver.Navigate().GoToUrl("https://books.toscrape.com");
        string html = driver.PageSource;

        var doc = new HtmlDocument();
        doc.LoadHtml(html);
        // doc.DocumentNode.SelectNodes("//article[@class='product_pod']") ...
        Console.WriteLine($"rendered {html.Length} bytes");
    }
}

The pattern matches Playwright: render in a browser, take PageSource, parse with HtmlAgilityPack. The using declaration on the driver calls Dispose() for you so the browser process does not leak. Here is how the two compare:

The quick test for which path you need: fetch a page with HttpClient and search the result for a value you can see in the browser. When it is missing, the page is JavaScript-rendered and you move to Playwright or Selenium. I cover the browser-automation path in more depth in the Selenium guide.

How do you avoid getting blocked when scraping in C#?

Look like a normal browser, slow down, and spread requests across IP addresses. A site flags scrapers by three signals: requests that arrive too fast, headers that look automated, and too much traffic from one IP. Each has a countermeasure that applies whether you use HttpClient, Playwright, or Selenium.

In HttpClient, route through a proxy by passing an HttpClientHandler { Proxy = new WebProxy("http://host:port") } to the constructor, and set extra headers on DefaultRequestHeaders to match a real browser.

These measures carry a small project a long way. The trouble starts at scale: rotating a large IP pool, solving CAPTCHAs, and handling per-site quirks becomes its own engineering job, and it pulls focus from the data you actually want. That is where a scraper API like ChocoData fits the category, with a universal endpoint plus 453 dedicated endpoints that return the page so your C# code keeps working unchanged while the blocking is handled upstream. You point HttpClient at the API instead of the target, parse the response with HtmlAgilityPack the same way, and skip the proxy and CAPTCHA plumbing. I cover that full tradeoff, and the legal side, in the web scraping guide and in how to scrape without getting blocked.

For the scraping itself, the stack in this guide will carry you a long way. Take the pagination crawler, point it at a site you care about, adjust the XPath until the fields come out clean, write the rows to CSV, and add the anti-block measures as the target pushes back.