Java Web Scraping: HttpClient, Jsoup, and the linksc SDK

Quick answer: For Java web scraping, use the built-in java.net.http.HttpClient to download pages and Jsoup to parse the HTML with CSS-style selectors. That stack is solid for static, cooperative sites. When you run into JavaScript rendering, proxy rotation, or blocking, the linksc Java SDK (Maven sc.link:linksc) returns clean markdown from any URL in one call, with both synchronous and async APIs.

Java has had a mature scraping story for years. HttpClient shipped with the JDK in Java 11, so you no longer need a third-party HTTP library for most work, and Jsoup remains the gold standard for parsing. The honest limitation is the same one every language hits: the raw libraries give you HTML, and everything hard about real scraping lives outside the parser. Here are both approaches, with runnable code.

The from-scratch stack: HttpClient + Jsoup

You only need Jsoup as an external dependency. HttpClient is part of the JDK.

<!-- pom.xml -->
<dependency>
  <groupId>org.jsoup</groupId>
  <artifactId>jsoup</artifactId>
  <version>1.18.1</version>
</dependency>

Here is a complete program that downloads a page and extracts article titles and links.

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

public class Scraper {
    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newHttpClient();

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://example.com/blog"))
            .header("User-Agent", "Mozilla/5.0 (compatible; MyBot/1.0)")
            .GET()
            .build();

        HttpResponse<String> response =
            client.send(request, HttpResponse.BodyHandlers.ofString());

        Document doc = Jsoup.parse(response.body());
        for (Element a : doc.select("article h2 a")) {
            System.out.println(a.text() + " -> " + a.absUrl("href"));
        }
    }
}

A couple of things worth knowing. Jsoup's select uses CSS selectors, and absUrl("href") resolves relative links against the document base for you, which saves a lot of string wrangling. HttpClient follows redirects only if you tell it to, so add .followRedirects(HttpClient.Redirect.NORMAL) on the builder if the site uses them.

Jsoup can also fetch and parse in one step with Jsoup.connect(url).get(), but pairing it with HttpClient gives you finer control over headers, timeouts, and connection reuse, which matters at scale.

For multiple pages, reuse a single client and add a delay.

HttpClient client = HttpClient.newBuilder()
    .followRedirects(HttpClient.Redirect.NORMAL)
    .build();

for (String url : urls) {
    HttpRequest req = HttpRequest.newBuilder(URI.create(url)).GET().build();
    HttpResponse<String> resp = client.send(req, HttpResponse.BodyHandlers.ofString());
    // process resp.body()
    Thread.sleep(500); // be polite
}

Where the from-scratch approach runs out of road

The HttpClient + Jsoup stack is the right tool when the site serves complete HTML, does not fight bots, and does not need a browser to render. That covers a large slice of the web, but the interesting targets often fall outside it.

The friction points are predictable:

• JavaScript rendering. HttpClient fetches the raw HTML. If the page paints its content client-side, Jsoup parses an empty shell. You would need Selenium or Playwright for Java, which is heavier and slower.
• Blocking. One datacenter IP at speed earns a 403 or 429 quickly. You then own proxy rotation, retries, and header realism.
• Parsing drift. Selectors break on every redesign, so a scraper is never truly finished.

If block responses are new to you, our HTTP status codes guide walks through what each code means for a scraper, and web scraping vs API covers when you should not be scraping at all.

The low-effort path: the linksc Java SDK

When you want data rather than infrastructure, the linksc Java SDK does rendering, proxies, and parsing server-side and returns clean markdown. It uses Jackson for JSON, and it offers both a blocking API and a CompletableFuture async API. The client reads LINKSC_API_KEY from the environment if you do not pass a key.

<!-- pom.xml -->
<dependency>
  <groupId>sc.link</groupId>
  <artifactId>linksc</artifactId>
  <version>1.0.0</version>
</dependency>

Synchronous fetch:

import sc.link.LinkscClient;
import sc.link.FetchResult;

public class Fetch {
    public static void main(String[] args) {
        LinkscClient client = new LinkscClient("lsc_your_key_here");

        FetchResult page = client.fetch(
            "https://example.com/blog",
            Map.of("format", "markdown"));

        System.out.println(page.getContent());
    }
}

page.getContent() is already markdown, so there is no selector to write and nothing to re-fix after a redesign. Search uses the q field and returns a serpData payload.

import sc.link.LinkscClient;
import sc.link.SearchResult;

LinkscClient client = new LinkscClient("lsc_your_key_here");

SearchResult results = client.search(
    "java web scraping tutorial",
    Map.of("engine", "google"));

System.out.println(results.getSerpData());

For high-throughput jobs, the async API keeps threads free while requests are in flight.

import java.util.concurrent.CompletableFuture;

CompletableFuture<FetchResult> future =
    client.fetchAsync("https://example.com/blog", Map.of("format", "markdown"));

future.thenAccept(page -> System.out.println(page.getContent()))
      .join();

Under the hood this is a POST to https://api.link.sc/v1/fetch or /v1/search with an x-api-key header. The raw call looks like this if you want to see the contract:

curl -X POST https://api.link.sc/v1/search \
  -H "x-api-key: lsc_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"q": "java web scraping tutorial", "engine": "google"}'

Which should you use?

Factor	HttpClient + Jsoup	linksc Java SDK
Setup	JDK client + Jsoup, write selectors	One dependency, no selectors
JavaScript rendering	Manual (Selenium/Playwright)	Handled server-side
Proxies and blocking	You build and maintain	Handled server-side
Output	Raw HTML, you parse	Clean markdown or JSON
Concurrency	Threads or virtual threads	Sync and CompletableFuture async
Cost	Free libraries, your infra time	Credit-based, free tier available
Best when	Static cooperative sites, full control	You want data without infra

If you are parsing a few static pages and want zero external services, HttpClient and Jsoup are excellent and cost nothing. If you are feeding an LLM pipeline or an agent and do not want to run a proxy pool, the SDK is cheaper once you price in your own time.

Retries and error handling

The linksc SDK retries automatically on 502, 503, and transport failures. If you build your own with HttpClient, add the same behavior and honor Retry-After on a 429.

int wait = 1;
for (int attempt = 0; attempt < 3; attempt++) {
    HttpResponse<String> resp =
        client.send(req, HttpResponse.BodyHandlers.ofString());
    if (resp.statusCode() == 200) break;
    if (resp.statusCode() == 429) {
        Thread.sleep(wait * 1000L);
        wait *= 2; // exponential backoff
    }
}

A note on ethics

Scrape public data, respect robots.txt, keep your request rate reasonable, and never route around a login or paywall. Java makes it easy to spin up many virtual threads, so pace yourself and send a real User-Agent. Being a polite client is both the right thing to do and the surest way to stay unblocked.

Java gives you a dependable base for scraping. Use HttpClient and Jsoup when you want full control over simple sites, and reach for the linksc SDK when rendering and blocking would otherwise dominate your time. The free tier is 500 credits a month, with plans at link.sc/pricing.

---

Building a Java service or agent that needs clean web data without managing proxies? Get a free link.sc key and fetch any URL as markdown.