← All posts

C# Web Scraping: HttpClient, AngleSharp, and the linksc SDK

GUIDES · JUL 18, 2026 · 5 MIN READ

Quick answer: For C# web scraping, use HttpClient to download pages and either AngleSharp or HtmlAgilityPack to parse the HTML. AngleSharp gives you a real DOM with CSS selectors, and HtmlAgilityPack leans on XPath. That stack is great for static, cooperative sites. When JavaScript rendering, proxies, or blocking get in the way, the linksc .NET SDK (NuGet Linksc) returns clean markdown from any URL through one async call.

.NET is a strong platform for scrapers. HttpClient is fast and built in, the async model is first-class, and you have two mature parsers to choose from. As with every language, the raw libraries hand you HTML and leave the hard parts, rendering and staying unblocked, to you. Here are both paths with runnable code.

The from-scratch stack: HttpClient + a parser

Pick a parser. AngleSharp models the page as a browser-like DOM and supports CSS selectors, which most people find natural. HtmlAgilityPack is the long-standing option and uses XPath.

dotnet add package AngleSharp
# or
dotnet add package HtmlAgilityPack

Here is a complete AngleSharp example that downloads a page and extracts article titles and links.

using System;
using System.Net.Http;
using System.Threading.Tasks;
using AngleSharp;
using AngleSharp.Dom;

class Program
{
    static async Task Main()
    {
        using var http = new HttpClient();
        http.DefaultRequestHeaders.Add("User-Agent",
            "Mozilla/5.0 (compatible; MyBot/1.0)");

        string html = await http.GetStringAsync("https://example.com/blog");

        var context = BrowsingContext.New(Configuration.Default);
        var doc = await context.OpenAsync(req => req.Content(html));

        foreach (var a in doc.QuerySelectorAll("article h2 a"))
        {
            var href = a.GetAttribute("href");
            Console.WriteLine($"{a.TextContent.Trim()} -> {href}");
        }
    }
}

QuerySelectorAll is the same API you would use in browser JavaScript, so selectors port over cleanly. If you prefer XPath, the HtmlAgilityPack version parses with HtmlDocument and queries doc.DocumentNode.SelectNodes("//article//h2/a").

One thing that trips people up: do not create a new HttpClient per request. Use a single shared instance or, in ASP.NET, IHttpClientFactory. Creating many clients exhausts sockets under load. Set a timeout while you are at it.

using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(20) };

foreach (var url in urls)
{
    string body = await http.GetStringAsync(url);
    // process body
    await Task.Delay(500); // be polite
}

Where the from-scratch approach runs out of road

HttpClient plus a parser is the right tool when the site serves complete HTML, does not fight bots, and does not need a browser to render. That describes plenty of the web, but often not the sites people actually target.

The friction is familiar:

  • JavaScript rendering. HttpClient fetches the initial HTML. If content is painted client-side, your parser sees an empty shell. You would need Playwright for .NET, which is heavier and slower.
  • Blocking. A single datacenter IP moving fast gets a 403 or 429. You then own proxy rotation, retries, and header realism.
  • Parsing drift. Selectors and XPath expressions break on every redesign.

If you are new to block responses, our HTTP status codes guide explains what each one signals, and what a web scraping API is covers the managed alternative in general terms.

The low-effort path: the linksc .NET SDK

When the goal is data rather than plumbing, the linksc .NET SDK handles rendering, proxies, and parsing server-side and returns clean markdown. It is async throughout and built on HttpClient. The client reads LINKSC_API_KEY from the environment if you do not pass a key.

dotnet add package Linksc

using System;
using System.Threading.Tasks;
using Linksc;

class Program
{
    static async Task Main()
    {
        var client = new LinkscClient("lsc_your_key_here");

        var page = await client.FetchAsync(
            "https://example.com/blog",
            new FetchOptions { Format = "markdown" });

        Console.WriteLine(page.Content);
    }
}

page.Content is already markdown, so there is no selector to maintain and nothing to re-fix after a redesign. Search uses the q field and returns a SerpData payload.

var client = new LinkscClient("lsc_your_key_here");

var results = await client.SearchAsync(
    "c# web scraping tutorial",
    new SearchOptions { Engine = "google" });

Console.WriteLine(results.SerpData);

Under the hood this is a POST to https://api.link.sc/v1/fetch or /v1/search with an x-api-key header. Here is the raw call if you want to see the contract:

curl -X POST https://api.link.sc/v1/fetch \
  -H "x-api-key: lsc_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/blog", "format": "markdown"}'

Which should you use?

Factor HttpClient + parser linksc .NET SDK
Setup HttpClient + AngleSharp/HAP, write selectors One NuGet package, no selectors
JavaScript rendering Manual (Playwright) Handled server-side
Proxies and blocking You build and maintain Handled server-side
Output Raw HTML, you parse Clean markdown or JSON
Concurrency async/await, you manage async/await, infra managed
Cost Free libraries, your infra time Credit-based, free tier available
Best when Static cooperative sites, full control You want data without infra

Be honest about which side you are on. For a handful of static pages with no external service, AngleSharp or HtmlAgilityPack are genuinely good and cost nothing. If you are feeding an LLM or building an agent and do not want to run a proxy pool, the SDK wins once you count engineering time.

Retries and error handling

The linksc SDK retries automatically on 502, 503, and transport errors. If you build your own with HttpClient, add the same guard and honor Retry-After on a 429.

static async Task<string> FetchWithRetry(HttpClient http, string url)
{
    for (int attempt = 0; attempt < 3; attempt++)
    {
        var resp = await http.GetAsync(url);
        if (resp.IsSuccessStatusCode)
            return await resp.Content.ReadAsStringAsync();

        if ((int)resp.StatusCode == 429)
        {
            var wait = TimeSpan.FromSeconds(Math.Pow(2, attempt));
            await Task.Delay(wait); // exponential backoff
        }
    }
    throw new Exception("failed after retries");
}

A note on ethics

Scrape public data, respect robots.txt, keep your request rate reasonable, and never route around a login or paywall. .NET makes concurrent requests trivial, so pace yourself and send a real User-Agent. A polite client is both the decent choice and the reliable way to stay unblocked.

.NET gives you a fast, well-supported foundation for scraping. Use HttpClient with AngleSharp or HtmlAgilityPack when you want full control over simple sites, and reach for the linksc SDK when rendering and blocking would otherwise consume your schedule. The free tier is 500 credits a month, with plans at link.sc/pricing.

Building a .NET app or agent that needs clean web data without managing proxies? Get a free link.sc key and fetch any URL as markdown.