Go Web Scraping: net/http, goquery, Colly, and chromedp

Go is an unusually good fit for scraping, and for one structural reason: a goroutine costs a couple of kilobytes, so fetching a thousand pages concurrently is a language feature rather than an architecture. Add a static binary you can drop on any box with no runtime to install, and a standard library whose HTTP client is production-grade on its own, and you get pipelines that run on far less hardware than the Python equivalents.

This guide covers the whole Go scraping stack as it stands in 2026: net/http for fetching (headers, cookies, compression, timeouts), goquery for parsing, Colly when you need a crawler rather than a fetcher, chromedp and rod for JavaScript-rendered pages, and the goroutine patterns that keep all of it fast without getting you banned. Examples target Go 1.23+ and Colly v2.

Key Takeaways

• net/http + goquery is the default stack. Reach for Colly when you need crawling — a queue, depth limits, per-domain rate limiting, and callbacks — not just fetching
• Go transparently gzip-decompresses only if you don't set Accept-Encoding yourself. Setting a browser-like br, zstd header hands you raw compressed bytes, and the standard library decodes neither
• Always give http.Client a Timeout. The zero value means wait forever, and one hung server stalls a worker permanently
• Colly cannot execute JavaScript, and no amount of configuration changes that. Render with chromedp/rod (or an API) and parse the result
• errgroup.Group with SetLimit(n) is the whole bounded-concurrency pattern — resist unbounded go func() over a URL list
• Perfect headers still get blocked: Go's TLS handshake has a distinctive fingerprint that servers match on before they ever read your User-Agent

Why Go for web scraping

Scraping is IO-bound work with a CPU-bound tail — waiting on hundreds of sockets, then parsing HTML. Go's runtime multiplexes those waits onto a handful of OS threads automatically, so the natural way to write a Go scraper is also the fast way. There's no async/await coloring, no GIL, and no event loop to avoid blocking.

The practical consequences show up in deployment more than in benchmarks. A Go scraper compiles to one static binary, so the container is a FROM scratch image with no interpreter and no dependency resolution at deploy time. Memory stays flat and predictable under concurrency, which matters when you're running a crawler for days. And the standard library covers HTTP/1.1, HTTP/2, TLS, cookie jars, and connection pooling without a single third-party package.

What Go doesn't give you is the ecosystem depth of Python. There is no Scrapy-class framework with a plugin ecosystem, no BeautifulSoup-style forgiving API, and browser automation is a second-class citizen compared to Playwright's Node bindings. Go is the right choice when throughput and operational simplicity matter more than library breadth — the same trade-off that makes Rust worth considering for the same job.

Setting up

mkdir scraper && cd scraper
go mod init example.com/scraper

go get github.com/PuerkitoBio/goquery      # HTML parsing, jQuery-style
go get github.com/gocolly/colly/v2         # crawling framework
go get github.com/chromedp/chromedp        # headless Chrome, when needed
go get golang.org/x/sync/errgroup          # bounded concurrency

Note the /v2 on Colly — it's a Go module major version, not a directory, and importing github.com/gocolly/colly without it silently pulls the abandoned v1 API.

Fetching with net/http

The standard library client is what everything else is built on, including Colly. Learning it directly pays off, because every "how do I do X in Colly" question resolves to an http.Client setting underneath.

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "time"
)

func main() {
    client := &http.Client{Timeout: 30 * time.Second}

    req, err := http.NewRequest("GET", "https://example.com", nil)
    if err != nil {
        log.Fatal(err)
    }

    resp, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        log.Fatalf("unexpected status %d", resp.StatusCode)
    }

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%d bytes\n", len(body))
}

Two rules that cover most Go HTTP bugs in scrapers:

• Always defer resp.Body.Close(), and always read the body to completion. An unread, unclosed body leaks the connection instead of returning it to the pool, and a long-running crawler will exhaust its file descriptors.
• Reuse one http.Client. It holds the connection pool; a client per request throws away keep-alive and re-does a TLS handshake every time. http.Client is safe for concurrent use by multiple goroutines, so a single package-level client is the normal pattern.

Custom headers and user agents

Go's default User-Agent is Go-http-client/2.0, which is an instant block on any site that looks. Set headers on the request:

req, _ := http.NewRequest("GET", url, nil)

req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "+
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36")
req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
req.Header.Set("Accept-Language", "en-US,en;q=0.9")
req.Header.Set("Referer", "https://www.google.com/")
req.Header.Set("Upgrade-Insecure-Requests", "1")

Set replaces any existing value; Add appends, which is what you want for headers that legitimately repeat. To remove Go's automatic User-Agent entirely rather than replace it, assign the empty string: req.Header.Set("User-Agent", "").

For headers on every request without repeating yourself, wrap the transport:

type headerTransport struct {
    base    http.RoundTripper
    headers map[string]string
}

func (t headerTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    req = req.Clone(req.Context()) // RoundTrippers must not mutate the original
    for k, v := range t.headers {
        req.Header.Set(k, v)
    }
    return t.base.RoundTrip(req)
}

client := &http.Client{
    Timeout: 30 * time.Second,
    Transport: headerTransport{
        base: http.DefaultTransport,
        headers: map[string]string{
            "User-Agent":      chromeUA,
            "Accept-Language": "en-US,en;q=0.9",
        },
    },
}

This RoundTripper pattern is worth internalizing — it's also how you add retries, logging, caching, and (later in this guide) a rendering backend, without touching the code that makes requests.

Rotating user agents helps against naive rate limiting, and math/rand no longer needs seeding (it self-seeds as of Go 1.20, and rand.Seed is deprecated):

var userAgents = []string{chromeMacUA, chromeWinUA, firefoxLinuxUA}

req.Header.Set("User-Agent", userAgents[rand.IntN(len(userAgents))]) // math/rand/v2

The honest caveat: header spoofing has a ceiling. Go's TLS ClientHello has a distinct fingerprint (JA3/JA4), as does its HTTP/2 settings frame, and anti-bot vendors match on both before parsing a single header. A request claiming to be Chrome while handshaking like Go is a louder signal than a missing User-Agent. The mitigation is utls, which mimics browser handshakes — or offloading the fetch entirely. Our guide to HTTP headers for web scraping covers which headers actually matter, and user agent rotation covers when rotation helps and when it doesn't.

Sessions and cookies

For anything behind a login, or any site that hands out a session cookie on first visit, attach a cookie jar. The client then stores and replays cookies automatically across requests — which is all a "session" is:

import (
    "net/http"
    "net/http/cookiejar"

    "golang.org/x/net/publicsuffix"
)

jar, err := cookiejar.New(&cookiejar.Options{
    PublicSuffixList: publicsuffix.List,
})
if err != nil {
    log.Fatal(err)
}

client := &http.Client{Jar: jar, Timeout: 30 * time.Second}

// Log in — the Set-Cookie response is captured by the jar
resp, err := client.PostForm("https://example.com/login", url.Values{
    "username": {"user"},
    "password": {"pass"},
})
resp.Body.Close()

// Subsequent requests replay the session cookie automatically
resp, err = client.Get("https://example.com/dashboard")

Passing PublicSuffixList is not optional decoration. Without it, the jar has no way to reject a cookie scoped to a public suffix like .co.uk, so a hostile site can set cookies that leak across unrelated domains. The list lives in golang.org/x/net/publicsuffix.

To pre-seed a session you captured from a browser, push cookies into the jar directly:

u, _ := url.Parse("https://example.com")
jar.SetCookies(u, []*http.Cookie{
    {Name: "session_id", Value: "abc123", Path: "/"},
    {Name: "csrf_token", Value: "xyz789", Path: "/"},
})

for _, c := range jar.Cookies(u) {   // inspect what will be sent
    fmt.Printf("%s=%s\n", c.Name, c.Value)
}

Per-request cookies without a jar work too — req.AddCookie(&http.Cookie{Name: "a", Value: "b"}) — but you lose automatic capture of Set-Cookie, so the jar is almost always the right call. Note that the standard cookiejar is in-memory only: to survive a restart, serialize jar.Cookies(u) yourself and reload it with SetCookies.

Compressed responses, and the trap

This is the single most common silent bug in Go scrapers. The transport adds Accept-Encoding: gzip on its own and transparently decompresses the response — but only when you haven't set the header yourself. The moment you add a realistic browser value, Go steps back and hands you the raw compressed bytes:

// Works: transport requests gzip, decompresses transparently.
req.Header.Set("User-Agent", chromeUA)
body, _ := io.ReadAll(resp.Body)   // readable HTML

// Broken: you own the encoding now.
req.Header.Set("Accept-Encoding", "gzip, deflate, br, zstd")
body, _ := io.ReadAll(resp.Body)   // binary garbage

The symptom is a body full of unprintable bytes and a parser that finds no elements. resp.Uncompressed tells you which path you got — it's true only when the transport did the work for you. Note that when it does, Go strips Content-Encoding and Content-Length from the response headers, so checking Content-Encoding to decide whether to decompress gives you the right answer for the wrong reason.

If you do set the header — and browser-mimicking scrapers usually should — decompress explicitly. The standard library covers gzip and deflate; it has no brotli or zstd decoder, which is a problem because every real browser advertises br and Chrome advertises zstd:

import (
    "compress/gzip"
    "compress/zlib"
    "io"

    "github.com/andybalholm/brotli"
    "github.com/klauspost/compress/zstd"
)

func decompress(resp *http.Response) (io.ReadCloser, error) {
    switch strings.ToLower(resp.Header.Get("Content-Encoding")) {
    case "gzip":
        return gzip.NewReader(resp.Body)
    case "deflate":
        return zlib.NewReader(resp.Body)
    case "br":
        return io.NopCloser(brotli.NewReader(resp.Body)), nil
    case "zstd":
        d, err := zstd.NewReader(resp.Body)
        if err != nil {
            return nil, err
        }
        return d.IOReadCloser(), nil
    default:
        return resp.Body, nil
    }
}

Two details worth knowing: deflate in the wild is usually zlib-wrapped, so compress/zlib is the right reader more often than compress/flate despite the name, and both the decompressing reader and resp.Body need closing.

The simplest fix, if you don't need br/zstd in your header at all, is to only advertise what you can decode: req.Header.Set("Accept-Encoding", "gzip, deflate").

Timeouts

http.Client's zero value has no timeout, so a server that accepts your connection and then goes silent holds a goroutine forever. In a concurrent scraper that's how you lose workers one at a time until throughput quietly hits zero.

Timeout on the client covers everything — connect, TLS, headers, and body read — and is the one setting you must not skip:

client := &http.Client{Timeout: 30 * time.Second}

For finer control, split the budget across the transport, so a slow DNS lookup and a slow body download fail differently:

transport := &http.Transport{
    DialContext: (&net.Dialer{
        Timeout:   5 * time.Second,   // TCP connect
        KeepAlive: 30 * time.Second,
    }).DialContext,
    TLSHandshakeTimeout:   5 * time.Second,
    ResponseHeaderTimeout: 15 * time.Second, // time to first byte
    IdleConnTimeout:       90 * time.Second,
    MaxIdleConns:          200,
    MaxIdleConnsPerHost:   20,   // default is 2 — raise it for concurrent scraping
    ForceAttemptHTTP2:     true,
}

client := &http.Client{Transport: transport, Timeout: 60 * time.Second}

MaxIdleConnsPerHost defaults to 2, which is the setting people most often miss: hammering one domain with 50 goroutines against a default transport means 48 of them open and discard fresh connections on every request. Raising it to roughly your per-host concurrency makes handshakes rare.

Per-request deadlines go through the context, which also lets a cancelled parent job tear down in-flight fetches:

ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()

req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
resp, err := client.Do(req)

Detect timeouts with errors.Is(err, context.DeadlineExceeded) for context expiry, and os.IsTimeout(err) for the client-level Timeout. They are different errors from the same symptom, and code that only checks one will misclassify half its failures.

Parsing HTML with goquery

goquery is the parsing layer almost every Go scraper uses. It wraps the official golang.org/x/net/html tokenizer in a jQuery-shaped API and uses cascadia for CSS selectors, so doc.Find(".product h2") behaves the way it does in a browser console.

package main

import (
    "fmt"
    "log"
    "net/http"
    "strings"

    "github.com/PuerkitoBio/goquery"
)

func main() {
    resp, err := http.Get("https://example.com/products")
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    doc.Find(".product").Each(func(i int, s *goquery.Selection) {
        name := strings.TrimSpace(s.Find("h2").Text())
        price := strings.TrimSpace(s.Find(".price").Text())
        link, _ := s.Find("a").Attr("href")

        fmt.Printf("%s — %s (%s)\n", name, price, link)
    })
}

NewDocumentFromReader takes any io.Reader, so it parses straight from a response body without buffering the whole page into a string first.

The traversal API is small and you'll use most of it:

sel.Text()                       // concatenated text of the selection and descendants
sel.Attr("href")                 // (value, exists)
sel.AttrOr("href", "")           // value or fallback — usually what you want
sel.Html()                       // inner HTML as a string
sel.First() / sel.Eq(2)          // narrow to one element
sel.Length()                     // how many matched — check this before assuming
sel.Filter("[data-id]")          // narrow by another selector
sel.Parent() / sel.Children()    // walk the tree
sel.Map(func(i int, s *goquery.Selection) string { return s.Text() })

Tables are the case worth having a snippet for, since positional cell access is the usual approach:

doc.Find("table#prices tbody tr").Each(func(i int, row *goquery.Selection) {
    cells := row.Find("td")
    if cells.Length() < 3 {
        return // skip header or spacer rows
    }
    fmt.Println(
        strings.TrimSpace(cells.Eq(0).Text()),
        strings.TrimSpace(cells.Eq(1).Text()),
        strings.TrimSpace(cells.Eq(2).Text()),
    )
})

Three goquery gotchas that cost people an afternoon each:

• Relative URLs stay relative. goquery does no resolution. Keep the base URL around and resolve with net/url: abs, err := base.Parse(href). If you build the document with goquery.NewDocumentFromResponse, doc.Url is populated and doc.Url.Parse(href) works; with NewDocumentFromReader it's nil and will panic if you assume otherwise.
• Find on an empty selection returns an empty selection, not an error. Chains fail silently and .Text() returns "". Check .Length() when a missing element should be an error rather than an empty string.
• No XPath. cascadia is CSS-only. If you're porting XPath expressions, either convert them (our XPath cheat sheet has the equivalences) or use htmlquery, which does support XPath over the same net/html tree.

For the rare case where you don't want the dependency, golang.org/x/net/html alone works — but you're hand-walking a linked list of nodes:

doc, err := html.Parse(resp.Body)
// then recurse over n.FirstChild / n.NextSibling, checking
// n.Type == html.ElementNode && n.Data == "a", and scanning n.Attr yourself

That's roughly thirty lines to reimplement what doc.Find("a[href]") does in one. Use it when you need streaming tokenization of a huge document (html.NewTokenizer) or want zero dependencies; otherwise take goquery, which sits on top of the same parser anyway.

Colly: when you need a crawler

goquery parses one page. Colly crawls a site — it manages a request queue, deduplicates URLs, enforces depth limits and per-domain rate limits, handles cookies and redirects, retries, and caches responses to disk. If your job is "fetch this URL and extract three fields," Colly is overhead. If it's "walk every category page, then every product on them, politely, without revisiting," Colly is the thing that already solved it.

The model is callbacks registered on a collector:

package main

import (
    "fmt"
    "log"
    "time"

    "github.com/gocolly/colly/v2"
)

func main() {
    c := colly.NewCollector(
        colly.AllowedDomains("example.com", "www.example.com"),
        colly.MaxDepth(2),
        colly.UserAgent("Mozilla/5.0 (compatible; MyScraper/1.0)"),
        colly.CacheDir("./cache"),
    )

    c.Limit(&colly.LimitRule{
        DomainGlob:  "*",
        Parallelism: 4,
        RandomDelay: 2 * time.Second,
    })

    c.OnRequest(func(r *colly.Request) {
        fmt.Println("→", r.URL)
    })

    c.OnHTML(".product", func(e *colly.HTMLElement) {
        fmt.Printf("%s — %s\n", e.ChildText("h2"), e.ChildText(".price"))
    })

    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        e.Request.Visit(e.Attr("href")) // resolved against the current page
    })

    c.OnError(func(r *colly.Response, err error) {
        log.Printf("%s failed: %v", r.Request.URL, err)
    })

    c.Visit("https://example.com/catalog")
}

The callbacks fire in a fixed order per request — OnRequest, then OnResponse, then OnHTML/OnXML for each match, then OnScraped — with OnError replacing everything after OnRequest on failure. A few things this snippet gets for free that you'd otherwise write yourself:

• AllowedDomains keeps a crawl from wandering onto the whole internet via one stray link
• MaxDepth bounds how far link-following goes
• CacheDir writes successful responses to disk, so re-running during development doesn't re-hit the site — genuinely the best reason to use Colly while iterating on selectors
• e.Request.Visit(href) resolves relative URLs and skips already-visited ones, which is two bugs you don't get to write

e.DOM is a *goquery.Selection, so anything goquery can do is available inside a Colly callback — the two libraries compose rather than compete:

c.OnHTML("table#prices", func(e *colly.HTMLElement) {
    e.DOM.Find("tbody tr").Each(func(i int, row *goquery.Selection) {
        // ...
    })
})

For a two-stage crawl (listing pages, then detail pages) use c.Clone() so the detail collector gets its own callbacks and its own visited set:

detail := c.Clone()

detail.OnHTML("#description", func(e *colly.HTMLElement) {
    fmt.Println(e.Text)
})

c.OnHTML(".product a", func(e *colly.HTMLElement) {
    detail.Visit(e.Request.AbsoluteURL(e.Attr("href")))
})

Rate limiting and concurrency in Colly

LimitRule is Colly's politeness control. Parallelism caps simultaneous requests per matching domain; Delay waits a fixed interval between them and RandomDelay waits a random one up to the value given — prefer RandomDelay, since a scraper firing on an exact one-second cadence is trivially recognizable.

c := colly.NewCollector(colly.Async(true))

c.Limit(&colly.LimitRule{
    DomainGlob:  "*.example.com",
    Parallelism: 8,
    RandomDelay: 1 * time.Second,
})

for _, u := range urls {
    c.Visit(u)
}
c.Wait() // required with Async — otherwise main exits mid-crawl

colly.Async(true) is what makes Parallelism mean anything; without it requests run one at a time regardless of the limit rule. The matching cost is that Visit returns immediately, so you must call c.Wait() or the program exits before the crawl finishes. Rules are matched by DomainGlob, so you can be aggressive on your own infrastructure and gentle on a third party by registering two.

For very large crawls, the queue package persists pending URLs instead of holding them all in memory:

import "github.com/gocolly/colly/v2/queue"

q, _ := queue.New(8, &queue.InMemoryQueueStorage{MaxSize: 100000})
for _, u := range urls {
    q.AddURL(u)
}
q.Run(c)

Redirects in Colly

Colly follows redirects automatically, up to 10 hops, and re-applies the collector's headers to each hop — which stock net/http does not do, and which matters when a login flow bounces you across hosts.

To change the limit or inspect the chain, replace the handler. via holds the requests already made, so its length is your hop count:

c.SetRedirectHandler(func(req *http.Request, via []*http.Request) error {
    if len(via) >= 3 {
        return fmt.Errorf("too many redirects for %s", req.URL)
    }
    log.Printf("redirect %d → %s", len(via), req.URL)
    return nil
})

To stop following and inspect the 3xx yourself — the usual choice when a redirect's Set-Cookie is the thing you actually want, or when you're auditing redirect chains for SEO:

c.SetRedirectHandler(func(req *http.Request, via []*http.Request) error {
    return http.ErrUseLastResponse
})

c.OnResponse(func(r *colly.Response) {
    if r.StatusCode >= 300 && r.StatusCode < 400 {
        fmt.Printf("%d → %s\n", r.StatusCode, r.Headers.Get("Location"))
        return
    }
    fmt.Printf("final: %d, %d bytes\n", r.StatusCode, len(r.Body))
})

http.ErrUseLastResponse is a sentinel from net/http, not a real error — returning it tells the client to hand back the 3xx response instead of following it.

The redirect gotcha that catches everyone: AllowedDomains is enforced on redirect targets too. A site that redirects example.com → www.example.com will fail with a "forbidden domain" error if your allow-list only has the bare host. List every host in the chain, or drop AllowedDomains and filter in OnRequest instead. URLFilters behaves the same way.

Proxies and extensions

import (
    "github.com/gocolly/colly/v2/extensions"
    "github.com/gocolly/colly/v2/proxy"
)

extensions.RandomUserAgent(c) // rotate UA per request
extensions.Referer(c)         // set Referer from the previous page

rp, err := proxy.RoundRobinProxySwitcher(
    "http://user:pass@proxy1.example.com:8000",
    "socks5://proxy2.example.com:1080",
)
if err == nil {
    c.SetProxyFunc(rp)
}

RoundRobinProxySwitcher rotates a fixed list per request. It doesn't retire proxies that start failing, so for anything long-running you'll want a custom ProxyFunc that tracks health — or a provider whose single endpoint rotates for you. Our proxy provider comparison covers the pools worth pointing it at, and types of proxies covers when datacenter IPs are enough.

JavaScript-heavy pages

Colly cannot render JavaScript. Neither can goquery or net/http. They are HTTP clients: they fetch the HTML the server sent and parse it, and if that HTML is <div id="root"></div> with a bundle that populates it, there is nothing in the response to select. No user agent, header, or Colly option changes this — the framework has no JavaScript engine, and none can be configured in.

You have three real options, in ascending order of cost.

First, check whether you need a browser at all. Most single-page apps get their data from a JSON endpoint you can call directly. Open DevTools, filter the Network tab to Fetch/XHR, reload, and look at what comes back. Hitting that API with net/http is faster, more stable, and gives you structured data instead of HTML you have to parse. This is worth ten minutes of investigation before reaching for a browser — it frequently saves the entire rendering layer.

Second, drive a real browser. chromedp speaks the Chrome DevTools Protocol directly, with no WebDriver in the middle:

package main

import (
    "context"
    "log"
    "time"

    "github.com/chromedp/chromedp"
)

func main() {
    ctx, cancel := chromedp.NewContext(context.Background())
    defer cancel()

    ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
    defer cancel()

    var renderedHTML string
    err := chromedp.Run(ctx,
        chromedp.Navigate("https://example.com/app"),
        chromedp.WaitVisible("#product-list", chromedp.ByID),
        chromedp.OuterHTML("html", &renderedHTML),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Hand the rendered HTML to goquery
    doc, err := goquery.NewDocumentFromReader(strings.NewReader(renderedHTML))
    // ...
}

That last step is the pattern: render with chromedp, parse with goquery. Don't try to feed rendered HTML back into a Colly collector — Colly's request lifecycle expects to do its own fetching, and the contortions required aren't worth it when goquery parses a string in one line. (If you want Colly's crawling machinery and rendering, the clean way is a custom transport — see below.)

Always WaitVisible on a selector that only exists after render. chromedp.Sleep is a guess that's either too short on a slow day or wasted time on a fast one.

For production, control the browser process explicitly:

opts := append(chromedp.DefaultExecAllocatorOptions[:],
    chromedp.DisableGPU,
    chromedp.NoSandbox,                    // required in most containers
    chromedp.ProxyServer("http://proxy.example.com:8000"),
    chromedp.UserAgent(chromeUA),
)

allocCtx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
defer cancel()

browserCtx, cancel := chromedp.NewContext(allocCtx)
defer cancel()

// Each page gets a tab off the shared browser, not a new Chrome process
for _, u := range urls {
    tabCtx, tabCancel := chromedp.NewContext(browserCtx)
    // ... chromedp.Run(tabCtx, ...)
    tabCancel()
}

Reusing one browser across pages is the difference between a workable scraper and one that spawns a Chrome process per URL. Every context must be cancelled — a leaked one leaves a browser process running, and on a long crawl that's how you exhaust a machine.

rod is the main alternative, with a friendlier API built around Must* methods that panic instead of returning errors — pleasant for scripts, less so for services (rod provides non-Must variants for those):

browser := rod.New().MustConnect()
defer browser.MustClose()

page := browser.MustPage("https://example.com/app")
page.MustWaitStable()      // waits for the DOM to stop changing
html := page.MustHTML()

rod also has MustWaitStable, built-in retry semantics, and a nicer story for input simulation. chromedp is more widely deployed and closer to the raw protocol. Either is a defensible choice; neither changes the fundamental cost of running Chrome.

Third, offload rendering. A headless browser needs 300–700 MB of RAM per instance, Chrome and its system libraries in your image, and a supervisor to restart crashed processes — which erases most of Go's deployment advantage. Our headless browser guide covers the operational side; the alternative is to keep the Go side as plain HTTP, covered below.

Choosing a library

Two libraries you'll find in older articles and should skip: pholcus is unmaintained and its documentation is largely untranslated, and surf hasn't kept pace — its stateful-browsing niche is covered by a cookie jar plus goquery in less code.

On performance, the honest version is that the ranking is structural rather than benchmarkable. net/http + goquery and Colly are in the same class — Colly is net/http underneath, so its overhead is bookkeeping, and which one is faster on your workload depends on your concurrency settings, not the library. The real gap is against browser automation, and it isn't close: chromedp and rod run an entire browser engine per page — parsing CSS, executing JavaScript, building a render tree — where Colly parses HTML into a node tree and stops. That's hundreds of megabytes versus a few, and it holds regardless of tuning.

Which is the actual decision rule: use a browser only for pages that need one. A crawler that renders every page because a few require it costs an order of magnitude more than one that fetches normally and escalates selectively.

Concurrency patterns

The naive version — go fetch(url) in a loop — opens ten thousand simultaneous connections, exhausts your file descriptors, and gets you blocked in that order. You want bounded concurrency, and errgroup is the cleanest expression of it:

import "golang.org/x/sync/errgroup"

func scrapeAll(ctx context.Context, urls []string) ([]Product, error) {
    results := make([]Product, len(urls))

    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(20) // at most 20 in flight

    for i, u := range urls {
        i, u := i, u
        g.Go(func() error {
            p, err := scrapeOne(ctx, u)
            if err != nil {
                return fmt.Errorf("%s: %w", u, err)
            }
            results[i] = p // distinct index per goroutine — no mutex needed
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        return nil, err
    }
    return results, nil
}

Two things make this work. SetLimit blocks g.Go once the limit is reached, so the loop paces itself against completions instead of queueing everything up front. And errgroup.WithContext cancels the derived context on the first error, so in-flight requests tear down instead of running to completion for results you're going to discard.

Writing to results[i] from many goroutines is safe here specifically because each writes a distinct index. Any shared map or slice append needs a mutex or a channel — and go build -race during development is how you find out you forgot.

When you want to keep going after individual failures rather than aborting — usually right for scraping, where some URLs are always going to 404 — collect errors instead of returning them:

g.Go(func() error {
    p, err := scrapeOne(ctx, u)
    if err != nil {
        mu.Lock()
        failures = append(failures, fmt.Errorf("%s: %w", u, err))
        mu.Unlock()
        return nil // don't cancel siblings
    }
    results[i] = p
    return nil
})

Concurrency limit and politeness are different knobs. Twenty goroutines against twenty different hosts is unremarkable; twenty against one host is a small denial-of-service. For single-host crawls, add a rate limiter on top of the concurrency limit:

import "golang.org/x/time/rate"

limiter := rate.NewLimiter(rate.Limit(5), 1) // 5 requests/second, burst 1

g.Go(func() error {
    if err := limiter.Wait(ctx); err != nil {
        return err
    }
    return scrapeOne(ctx, u)
})

Errors and retries

Go's HTTP client returns an error for transport failures only. A 404 or a 503 is a successful request with an unhappy status code, so err == nil says nothing about whether you got the page:

resp, err := client.Do(req)
if err != nil {
    return nil, err // DNS, TCP, TLS, timeout
}
defer resp.Body.Close()

if resp.StatusCode != http.StatusOK {
    io.Copy(io.Discard, resp.Body) // drain so the connection is reusable
    return nil, fmt.Errorf("http %d", resp.StatusCode)
}

Retry transient failures only — 429, 5xx, and connection errors — and never a 404 or a 403, which will fail identically forever. Exponential backoff with jitter keeps a fleet of workers from resynchronizing into a thundering herd after an outage:

func fetchWithRetry(ctx context.Context, client *http.Client, url string) (*http.Response, error) {
    backoff := 500 * time.Millisecond

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
        if err != nil {
            return nil, err
        }

        resp, err := client.Do(req)
        switch {
        case err == nil && resp.StatusCode < 500 && resp.StatusCode != 429:
            return resp, nil // success, or a permanent failure — either way, stop
        case err == nil:
            io.Copy(io.Discard, resp.Body)
            resp.Body.Close()
        }

        jitter := time.Duration(rand.Int64N(int64(backoff / 2)))
        select {
        case <-time.After(backoff + jitter):
        case <-ctx.Done():
            return nil, ctx.Err()
        }
        backoff *= 2
    }
    return nil, fmt.Errorf("giving up on %s", url)
}

Sleeping in a select against ctx.Done() rather than calling time.Sleep is what makes the retry loop cancellable — a time.Sleep in a worker ignores shutdown for its full duration.

Honor Retry-After when a 429 sends one; it's the server telling you exactly how long to wait, and ignoring it is how a soft rate limit becomes a hard ban. If you'd rather not maintain this, hashicorp/go-retryablehttp returns a standard *http.Client with retries already wired in, so it drops into existing code — including as Colly's transport via c.WithTransport. This is the same gap every systems-language HTTP client leaves open; the equivalent in the Rust ecosystem is covered in our reqwest guide.

Rendering and proxies without the browser

The two walls every Go scraper eventually hits are the two things Go can't fix with better code: pages that need JavaScript, and IP addresses that get blocked. Both are infrastructure problems, and both can be moved behind an HTTP call so the Go side stays plain net/http.

WebScraping.AI renders pages in Chromium behind a rotating proxy pool and returns the result over a normal API:

type Product struct {
    Name  string `json:"name"`
    Price string `json:"price"`
}

func extract(ctx context.Context, client *http.Client, target string) (Product, error) {
    api, _ := url.Parse("https://api.webscraping.ai/ai/fields")
    q := api.Query()
    q.Set("api_key", os.Getenv("WEBSCRAPING_AI_KEY"))
    q.Set("url", target)
    q.Set("fields[name]", "Product name")
    q.Set("fields[price]", "Price including currency symbol")
    api.RawQuery = q.Encode()

    req, err := http.NewRequestWithContext(ctx, "GET", api.String(), nil)
    if err != nil {
        return Product{}, err
    }

    resp, err := client.Do(req)
    if err != nil {
        return Product{}, err
    }
    defer resp.Body.Close()

    var p Product
    return p, json.NewDecoder(resp.Body).Decode(&p)
}

/ai/fields returns structured JSON straight into a struct, which skips the selector layer entirely — useful when page markup changes often enough that maintaining CSS selectors is the real cost. /html returns the rendered HTML for goquery to parse when you'd rather keep your own extraction logic, and /text returns visible text for feeding an LLM or a RAG knowledge base. The relevant knobs are js (rendering on or off), wait_for (a CSS selector to wait for), proxy (datacenter, residential, or stealth), country, and device.

Because it's just HTTP, it slots into the RoundTripper pattern from earlier — which is also the clean way to give Colly rendering, since the collector keeps its crawling, dedup, and rate limiting while every fetch transparently goes through the API:

type renderTransport struct {
    apiKey string
    base   http.RoundTripper
}

func (t renderTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    api, _ := url.Parse("https://api.webscraping.ai/html")
    q := api.Query()
    q.Set("api_key", t.apiKey)
    q.Set("url", req.URL.String())
    q.Set("js", "true")
    q.Set("proxy", "residential")
    api.RawQuery = q.Encode()

    proxied, err := http.NewRequestWithContext(req.Context(), "GET", api.String(), nil)
    if err != nil {
        return nil, err
    }
    return t.base.RoundTrip(proxied)
}

c := colly.NewCollector(colly.AllowedDomains("example.com"))
c.WithTransport(renderTransport{apiKey: os.Getenv("WEBSCRAPING_AI_KEY"), base: http.DefaultTransport})

Every OnHTML callback now sees fully rendered markup, and nothing else about the crawler changes. The shape most continuous Go pipelines end up with is exactly this: an errgroup fan-out over the API, results decoded into structs, whether that's price monitoring or B2B lead generation.

Before you scale any of this up, it's worth knowing where the lines are — is web scraping legal covers what public data, terms of service, and rate limits actually mean in practice.

Frequently asked questions

Is Go good for web scraping? Yes, particularly for high-volume pipelines. Goroutines make concurrent fetching cheap and simple, memory stays flat under load, and the result is a single static binary with no runtime to install. The trade-off is a thinner ecosystem than Python's — no Scrapy-class framework, and browser automation is less polished than Playwright's Node bindings. Choose Go when throughput and deployment simplicity outweigh library breadth.

Colly or goquery — which should I use? They solve different problems and are frequently used together. goquery parses HTML with CSS selectors; Colly is a crawler that manages queuing, deduplication, depth limits, rate limiting, and caching, and hands each page to your callbacks. For a known list of URLs, net/http + goquery is less machinery. For following links across a site, Colly. Inside a Colly callback, e.DOM is a goquery selection, so you're using both anyway.

Can Colly scrape JavaScript-rendered pages? No. Colly is an HTTP client with no JavaScript engine, so it only sees the HTML the server returned. The options are to call the underlying JSON API the page itself uses (check DevTools → Network → Fetch/XHR first, it's often there), render with chromedp or rod and parse the output with goquery, or route Colly's transport through a rendering API so the crawler keeps working unchanged.

Why is my Go scraper getting binary garbage instead of HTML? You set Accept-Encoding yourself. Go's transport decompresses gzip transparently only when it added the header, so setting a browser-like gzip, deflate, br, zstd makes decompression your job — and the standard library has no brotli or zstd decoder. Either advertise only gzip, deflate, or decompress explicitly based on Content-Encoding using andybalholm/brotli and klauspost/compress/zstd. resp.Uncompressed tells you which path a given response took.

How do I keep a session across requests in Go? Give your http.Client a cookiejar created with PublicSuffixList: publicsuffix.List. The client then captures Set-Cookie responses and replays them on subsequent requests to matching domains, which is all a session is. To reuse cookies captured from a browser, push them in with jar.SetCookies. The standard jar is in-memory only, so persist it yourself if a restart needs to keep the session.

How many goroutines should I use for scraping? Bound them with errgroup.SetLimit, and set the number by target rather than by hardware — 10–20 in flight across many hosts is unremarkable, while the same number against one host is abusive. For single-host crawls, add golang.org/x/time/rate on top of the concurrency limit and raise Transport.MaxIdleConnsPerHost from its default of 2, or most of your requests will pay for a fresh TLS handshake.

chromedp or rod? Both drive Chrome over the DevTools Protocol and both work. chromedp is more widely deployed and sits closer to the raw protocol; rod has a friendlier API, MustWaitStable for settling dynamic pages, and better input simulation. Pick either — the meaningful decision is whether you need a browser at all, since both cost hundreds of megabytes per instance versus a few for Colly.

Does setting a browser User-Agent stop me getting blocked? It clears the lowest bar and nothing above it. Anti-bot services fingerprint the TLS ClientHello and HTTP/2 settings frame, both of which are distinctive for Go's standard library and are checked before any header is read — so a request claiming to be Chrome while handshaking like Go is more suspicious than one that doesn't lie. Mitigations are utls for browser-like handshakes, residential proxies for IP reputation, or moving the fetch behind a service that handles both.