Go Web Scraping: A Practical Guide (net/http, goquery, and an SDK)

Quick answer: For Go web scraping you have two solid paths. Roll your own with the standard library net/http client plus the goquery package for HTML parsing, and lean on goroutines for concurrency. Or, when the sites you hit render JavaScript or block datacenter IPs, call a fetch API from the linksc Go SDK (github.com/getlinksc/linksc-go) and get clean content back without running a browser. This guide shows both, honestly.

Go is a great fit for scraping. The concurrency model is built for fanning out hundreds of requests, the binaries are small, and the standard library ships with a capable HTTP client. The catch is the same one every language hits eventually: a lot of the modern web does not return usable HTML to a plain HTTP client.

Let me walk through the from-scratch approach first, then the low-effort path, and be clear about when each one is the right call.

The from-scratch approach: net/http + goquery

For static pages that return real HTML, you do not need anything exotic. The standard library fetches the bytes and goquery (a jQuery-like API over Go's HTML parser) selects the nodes you want.

Install the one dependency:

go get github.com/PuerkitoBio/goquery

Then a minimal scraper looks like this:

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/PuerkitoBio/goquery"
)

func main() {
	client := &http.Client{Timeout: 15 * time.Second}

	req, err := http.NewRequest("GET", "https://example.com", nil)
	if err != nil {
		log.Fatal(err)
	}
	// Set a real User-Agent. The default Go one gets filtered often.
	req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; MyScraper/1.0)")

	resp, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		log.Fatalf("unexpected status: %d", resp.StatusCode)
	}

	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		log.Fatal(err)
	}

	// Pull every link on the page.
	doc.Find("a").Each(func(i int, s *goquery.Selection) {
		href, exists := s.Attr("href")
		if exists {
			fmt.Printf("%s -> %s\n", s.Text(), href)
		}
	})
}

A few things worth knowing here. Always set a Timeout on the client, because the zero value is no timeout and a slow server will hang your goroutine forever. Always set a User-Agent header, because Go's default identifies you as a bot and gets filtered. And always check resp.StatusCode before parsing, so you do not try to read HTML out of a 403 page.

Concurrency with goroutines

This is where Go earns its reputation. To scrape many URLs at once, fan them out over goroutines but cap the concurrency so you do not hammer a server. A buffered channel makes a clean semaphore.

package main

import (
	"fmt"
	"net/http"
	"sync"
	"time"
)

func fetch(client *http.Client, url string) (int, error) {
	req, _ := http.NewRequest("GET", url, nil)
	req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; MyScraper/1.0)")
	resp, err := client.Do(req)
	if err != nil {
		return 0, err
	}
	defer resp.Body.Close()
	return resp.StatusCode, nil
}

func main() {
	urls := []string{
		"https://example.com",
		"https://example.org",
		"https://example.net",
	}

	client := &http.Client{Timeout: 15 * time.Second}
	sem := make(chan struct{}, 5) // max 5 in flight
	var wg sync.WaitGroup

	for _, u := range urls {
		wg.Add(1)
		go func(url string) {
			defer wg.Done()
			sem <- struct{}{}        // acquire
			defer func() { <-sem }() // release

			status, err := fetch(client, url)
			if err != nil {
				fmt.Printf("%s error: %v\n", url, err)
				return
			}
			fmt.Printf("%s -> %d\n", url, status)
		}(u)
	}

	wg.Wait()
}

The sem channel caps you at five concurrent requests. Raise it for your own infrastructure, lower it to stay polite to the target. Being polite is not optional: respect robots.txt, keep rate limits reasonable, and only pull public data. If a site is telling you to slow down with 429 responses, slow down.

Where the from-scratch approach breaks

Two walls show up fast. The first is JavaScript rendering. net/http fetches the initial HTML and nothing else, so a React or Vue app that assembles its content client-side hands you an empty shell. To get the rendered DOM you need a headless browser (chromedp is the common Go choice), which means shipping a Chromium binary, managing its memory, and eating the latency.

The second wall is blocking. Datacenter IPs from your cloud provider get flagged by reputation systems, and you land on CAPTCHAs or 403s that no header tweak fixes. Rotating residential proxies help, but now you are running proxy infrastructure on top of a browser pool. We wrote up the tradeoffs in curl vs headless vs stealth browser if you want the full picture.

At that point you are maintaining a scraping platform, not scraping. That is the moment the API path pays off.

The low-effort path: the linksc Go SDK

When you would rather not run browsers and proxies, a fetch API does the rendering, proxying, and HTML-to-markdown conversion server-side and hands you clean content. The linksc Go SDK wraps this with a blocking, context.Context-aware client that fits normal Go code.

go get github.com/getlinksc/linksc-go

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	linksc "github.com/getlinksc/linksc-go"
)

func main() {
	// Falls back to the LINKSC_API_KEY env var if you pass "".
	client := linksc.New("lsc_your_api_key")

	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	// Fetch any URL as clean markdown, JS rendered server-side.
	content, err := client.Fetch(ctx, "https://example.com", &linksc.FetchOptions{
		Format: "markdown",
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(content)
}

The context.Context is the important detail for Go people. It threads cancellation and deadlines through the call the way the rest of your codebase already expects, so this drops into an HTTP handler or a worker pool without special handling.

Search works the same way, and returns full page content rather than just snippets:

serp, err := client.Search(ctx, "golang web scraping", &linksc.SearchOptions{
	Engine: "google",
})
if err != nil {
	log.Fatal(err)
}
fmt.Printf("%+v\n", serp)

If you prefer no SDK at all, the API is plain HTTP and works from net/http directly. Note the auth header is x-api-key, not Authorization: Bearer:

body := strings.NewReader(`{"url":"https://example.com","format":"markdown"}`)
req, _ := http.NewRequest("POST", "https://api.link.sc/v1/fetch", body)
req.Header.Set("x-api-key", "lsc_your_api_key")
req.Header.Set("Content-Type", "application/json")
// client.Do(req) ... decode {"content":"..."} from the response

Which should you use?

Factor	net/http + goquery	linksc Go SDK
Static HTML pages	Great fit, zero cost	Works, slight overkill
JavaScript-rendered pages	Needs chromedp + browser	Handled server-side
Blocked / anti-bot sites	You run proxies	Handled for you
Concurrency	Your goroutines	Your goroutines, plus their capacity
Ongoing maintenance	You own it all	Mostly the API's problem
Cost	Compute only	Credit-based, free tier to start

My honest take: if you are scraping a handful of well-behaved static sites, the standard library plus goquery is the right tool and you should not add a dependency you do not need. The moment you are fighting rendering or blocks across many sources, the time you spend babysitting browsers is worth more than the API credits. For a broader framing of that decision, see web scraping vs API: which to use.

You can start on the free tier (500 credits per month) and check the numbers at link.sc/pricing before you commit.

---

Want clean web data in your Go services without running a browser fleet? Get a free link.sc API key and call fetch or search in a few lines.