Rust Web Scraping: reqwest, scraper, and the linksc Crate

Quick answer: For Rust web scraping, use reqwest (async HTTP over tokio) to download pages and the scraper crate to query the HTML with CSS selectors. That combo works great for static, cooperative sites. When you hit JavaScript rendering, proxies, or aggressive blocking, reach for the async linksc crate, which returns clean markdown from any URL in one call so you skip the parsing and infrastructure entirely.

Rust is an excellent language for scrapers. It is fast, the memory story is predictable under high concurrency, and the async ecosystem around tokio is mature. The catch is that the raw crates only get you clean HTML. Everything that makes real-world scraping hard, rendering, rotating IPs, and staying unblocked, is left to you. This guide covers both paths honestly.

The from-scratch stack: reqwest + scraper

Start a project and pull in the crates you need.

# Cargo.toml
[dependencies]
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["rustls-tls"] }
scraper = "0.20"
anyhow = "1"

Here is a complete async scraper that downloads a page and extracts every article title and link.

use scraper::{Html, Selector};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let body = reqwest::Client::new()
        .get("https://example.com/blog")
        .header("User-Agent", "Mozilla/5.0 (compatible; MyBot/1.0)")
        .send()
        .await?
        .text()
        .await?;

    let doc = Html::parse_document(&body);
    let item = Selector::parse("article h2 a").unwrap();

    for el in doc.select(&item) {
        let title = el.text().collect::<String>();
        let href = el.value().attr("href").unwrap_or("");
        println!("{} -> {}", title.trim(), href);
    }

    Ok(())
}

A few notes on this code. The scraper crate is built on html5ever, the same parser Servo uses, so it handles malformed HTML the way a browser does. Selector::parse returns a Result because a bad CSS selector is a runtime error, not a compile-time one. And reqwest with the rustls-tls feature avoids a system OpenSSL dependency, which keeps your builds portable.

For anything more than a single page, set a shared client and add polite delays.

use std::time::Duration;

let client = reqwest::Client::builder()
    .timeout(Duration::from_secs(20))
    .build()?;

for url in urls {
    let resp = client.get(url).send().await?;
    // process...
    tokio::time::sleep(Duration::from_millis(500)).await;
}

Reuse one reqwest::Client across all requests. It pools connections internally, so creating a fresh client per request throws away that benefit and can exhaust file descriptors under load.

Where the from-scratch approach runs out of road

The reqwest + scraper stack is the right tool when the site serves complete HTML, does not fight bots, and does not require a browser to render its content. That describes a lot of the web, but not the parts people usually want.

Here is where it gets painful:

• JavaScript rendering. reqwest fetches the initial HTML. If the content is painted by client-side JavaScript, you get an empty shell. You would need a headless browser like a chromiumoxide setup, which is heavy and slow.
• Blocking. A single datacenter IP hammering a site gets a 403 or a 429 fast. You then own proxy rotation, retry logic, and header fingerprinting.
• Parsing drift. Every site has a different DOM, so your selectors break whenever the site ships a redesign.

If you want to understand the browser-versus-plain-HTTP tradeoff in depth, we wrote about it in curl vs headless vs stealth browser. And when you start seeing block responses, our HTTP status codes guide explains what each one means for a scraper.

The low-effort path: the linksc crate

When the goal is data, not infrastructure, the linksc crate handles rendering, proxies, and parsing on the server side and hands you clean markdown. It is async, built on reqwest with rustls, and reads your key from the LINKSC_API_KEY environment variable if you do not pass one.

# Cargo.toml
[dependencies]
tokio = { version = "1", features = ["full"] }
linksc = "0.1"
anyhow = "1"

use linksc::Client;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Reads LINKSC_API_KEY from the environment if no key is passed.
    let client = Client::new("lsc_your_key_here");

    let page = client
        .fetch("https://example.com/blog", &[("format", "markdown")])
        .await?;

    println!("{}", page.content);
    Ok(())
}

That page.content is already markdown. There is no selector to write and nothing to re-fix when the site redesigns, because you are consuming rendered, cleaned content rather than raw DOM.

Search works the same way. Note the field is q, and the response carries a serp_data payload.

use linksc::Client;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = Client::new("lsc_your_key_here");

    let results = client
        .search("rust async web scraping", &[("engine", "google")])
        .await?;

    println!("{:#?}", results.serp_data);
    Ok(())
}

Under the hood this is a POST to https://api.link.sc/v1/fetch or /v1/search with an x-api-key header. If you ever want to see the wire format, here is the equivalent curl.

curl -X POST https://api.link.sc/v1/fetch \
  -H "x-api-key: lsc_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/blog", "format": "markdown"}'

Which should you use?

Factor	reqwest + scraper	linksc crate
Setup	Two crates, write selectors	One crate, no selectors
JavaScript rendering	Manual (headless browser)	Handled server-side
Proxies and blocking	You build and maintain	Handled server-side
Output	Raw HTML, you parse	Clean markdown or JSON
Cost	Free crates, your infra time	Credit-based, free tier available
Best when	Static cooperative sites, full control	You want data without infra

Be honest with yourself about which column you are in. If you are scraping a handful of static pages and you enjoy owning the details, reqwest and scraper are genuinely good and cost nothing. If you are feeding an LLM or an agent and you would rather not run a proxy pool, the crate is the cheaper choice once you count engineering hours.

Retries and error handling

Whatever path you choose, transient failures are normal. The linksc crate retries automatically on 502, 503, and transport errors. If you roll your own with reqwest, add the same guardrail yourself.

async fn fetch_with_retry(client: &reqwest::Client, url: &str) -> anyhow::Result<String> {
    for attempt in 0..3u32 {
        match client.get(url).send().await {
            Ok(r) if r.status().is_success() => return Ok(r.text().await?),
            Ok(r) if r.status().as_u16() == 429 => {
                // Respect the server. Back off before retrying.
                let wait = 2u64.pow(attempt);
                tokio::time::sleep(std::time::Duration::from_secs(wait)).await;
            }
            _ => tokio::time::sleep(std::time::Duration::from_secs(1)).await,
        }
    }
    anyhow::bail!("failed after retries")
}

A note on ethics

Scrape public data, respect robots.txt, keep your request rate reasonable, and never route around a login or paywall. Rust makes it easy to generate enormous concurrent load, so the burden is on you to be a polite client. A short delay between requests and a real User-Agent go a long way.

Rust gives you a fast, reliable foundation for scraping. Use the raw crates when you want full control over simple sites, and lean on the linksc crate when rendering and blocking would otherwise eat your week. Free tier is 500 credits a month, and you can compare plans at link.sc/pricing.

---

Building a Rust scraper or agent that needs clean web data without the proxy headache? Get a free link.sc key and fetch any URL as markdown.