Go Web Scraping: The Complete Guide

I reach for Go when a scrape has to run fast and concurrent: tens of thousands of pages, a static binary I can drop on any box, and goroutines instead of a thread-pool framework. This guide is the map for that path. Which library does what, a full Colly walkthrough you can compile, how to parse and paginate, how to set rate limits and parallelism, and how to keep a crawler alive once a site pushes back. The code uses the documented API for each library at its current 2026 version, pointed at books.toscrape.com, a sandbox built for scraping practice.

Is Go good for web scraping?

Yes, Go is a strong choice for web scraping, and it pulls ahead when the job is large and concurrent. The standard library ships a production HTTP client, goroutines make thousands of parallel fetches cheap, and go build produces one static binary with no interpreter to install on the target host. For parsing and crawling, the ecosystem centers on one mature framework, Colly, so you rarely assemble pieces by hand.

Three things make Go pull its weight here:

• Concurrency is the language, not a library. A colly.Async() collector plus a LimitRule crawls many pages at once with a few lines, and goroutines keep memory flat where a thread-per-request model would balloon.
• Deployment is a single binary. Cross-compile for Linux, copy one file, run it. No virtualenv, no JVM, no node_modules on the server.
• The HTTP client is built in. net/http handles TLS, redirects, and cookies out of the box, so even a no-framework scraper is short.

Python has more scraping tutorials and a deeper bench of one-off scripts, which I cover in the Python guide. Go wins when throughput, low memory, and easy deployment matter more than the size of the tutorial pool.

What are the main Go web scraping libraries?

Four tools cover the whole field, and the right one depends on whether you need a full crawler and whether the page runs JavaScript. Static HTML with crawling needs Colly. Parsing alone needs goquery. JavaScript-rendered pages need a browser, which is chromedp. Raw API calls need only the standard library.

A few notes from using all four:

• Colly is the one you reach for first, and the one most of this guide uses. It both fetches and parses, follows links, manages cookies, and rate-limits, so for static crawling it is usually the only third-party dependency you need.
• goquery is the parser underneath Colly. You import it directly only when you already have HTML in hand (from a file, an API, or a custom client) and just want CSS-selector queries over it.
• chromedp speaks the Chrome DevTools Protocol to a real headless Chrome, so it renders anything a user sees. The cost is a Chrome binary on the host plus more time and memory per page.
• net/http ships with Go. When the data lives behind a JSON API, skip HTML parsing entirely, decode the JSON into a struct, and you are done.

How do you install Colly?

One go get inside a module, and the import path carries the major version. Colly’s current line is v2, so the module path ends in /v2. Initialize a module first if you have not:

go mod init bookscraper
go get github.com/gocolly/colly/v2

The matching import is the full v2 path. Dropping the /v2 pulls the old v1 line, which is a common first mistake:

import "github.com/gocolly/colly/v2"

That is the whole install. go get resolves goquery and the other transitive dependencies into your go.mod automatically, so there is nothing else to add. Any Go toolchain with modules enabled (Go 1.16+ defaults to modules) is enough.

How do you scrape a website with Go and Colly?

The loop is three steps: create a collector, register an OnHTML callback for the element you want, then call Visit. Colly fetches the page, runs your callback once per matching element, and hands you a *colly.HTMLElement to read from.

This program fetches the first catalogue page and prints the title and price for every book on it:

package main

import (
	"fmt"
	"log"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector(
		colly.UserAgent("Mozilla/5.0 (compatible; BookScraper/1.0)"),
	)

	// Runs once per matching element on the page.
	c.OnHTML("article.product_pod", func(e *colly.HTMLElement) {
		title := e.ChildAttr("h3 a", "title")
		price := e.ChildText(".price_color")
		fmt.Printf("%-8s %s\n", price, title)
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.OnError(func(r *colly.Response, err error) {
		log.Printf("error on %s: %v", r.Request.URL, err)
	})

	if err := c.Visit("https://books.toscrape.com/catalogue/page-1.html"); err != nil {
		log.Fatal(err)
	}
}

Each part of that maps to one job:

• colly.NewCollector(...) builds the collector. Options like colly.UserAgent(...) go here. By default Colly sends its own agent string, which many sites block, so set a browser-like one on every real scrape.
• c.OnHTML(selector, fn) registers a callback keyed to a CSS selector. Colly runs it once for each element that matches, so the body handles a single book card.
• e.ChildAttr("h3 a", "title") reads an attribute from a descendant of the matched element, here the full title this page stores in the anchor’s title attribute. e.ChildText(".price_color") reads the stripped text of a descendant.
• c.OnRequest and c.OnError are lifecycle hooks. The first fires before every request, the second when a request fails, which keeps one bad page from being silent.
• c.Visit(url) starts the crawl. It returns an error, so check it.

Run it from the module directory:

go run .

This is the shape I would ship for a static page. The selectors are the only part that changes per site, so the pattern transfers directly to any target whose data is in the initial HTML.

How do you parse HTML with goquery selectors in Colly?

Read fields off the *colly.HTMLElement Colly hands each callback, because that element wraps a goquery selection. Colly is built on goquery, so inside OnHTML you have jQuery-style helpers without importing goquery yourself. The element methods cover almost everything.

When you need the raw goquery selection (for example to read a sibling or a parent), it is on e.DOM. For structured output, marshal each record into a struct as you go:

type Book struct {
	Title string
	Price string
	Stock string
}

func main() {
	var books []Book

	c := colly.NewCollector()

	c.OnHTML("article.product_pod", func(e *colly.HTMLElement) {
		books = append(books, Book{
			Title: e.ChildAttr("h3 a", "title"),
			Price: e.ChildText(".price_color"),
			Stock: e.ChildText(".availability"),
		})
	})

	// Fires once after all OnHTML callbacks for a page finish.
	c.OnScraped(func(r *colly.Response) {
		fmt.Printf("collected %d books from %s\n", len(books), r.Request.URL)
	})

	c.Visit("https://books.toscrape.com/catalogue/page-1.html")
}

Two things worth noting:

• OnScraped runs after every OnHTML callback for a page has fired, so it is the place to count results, write a file, or push to a channel.
• Appending to a shared slice is fine for a synchronous collector. The moment you turn on async crawling (next section), that slice is touched by multiple goroutines, so guard it with a sync.Mutex or send records over a channel instead.

How do you follow links and handle pagination in Go?

Call e.Request.Visit on the next-page link from inside an OnHTML callback. Colly tracks visited URLs and queues new ones, so paginated crawling is one extra callback that fires when it finds the “next” anchor and follows it. Colly resolves the relative href against the current page for you.

package main

import (
	"fmt"
	"log"

	"github.com/gocolly/colly/v2"
)

func main() {
	pages := 0

	c := colly.NewCollector(
		colly.AllowedDomains("books.toscrape.com"),
	)

	// Count each listing page as it loads.
	c.OnHTML("body", func(e *colly.HTMLElement) {
		pages++
	})

	// Follow the next-page link until it disappears.
	c.OnHTML("li.next a", func(e *colly.HTMLElement) {
		next := e.Attr("href")
		fmt.Println("following", next)
		e.Request.Visit(e.Request.AbsoluteURL(next))
	})

	c.OnError(func(r *colly.Response, err error) {
		log.Printf("error on %s: %v", r.Request.URL, err)
	})

	c.Visit("https://books.toscrape.com/catalogue/page-1.html")

	fmt.Println("pages crawled:", pages)
}

Two details make this robust on a real target:

• e.Request.AbsoluteURL(next) turns the relative href into a full URL against the page it came from. e.Request.Visit(...) queues it, and Colly skips URLs it has already seen, so the crawl terminates when the next link points nowhere new.
• colly.AllowedDomains(...) fences the crawl to one host. Without it, an off-site link in the markup could send your crawler wandering across the web.

Against the full books.toscrape.com catalogue this walks all 50 listing pages. The same “match the next anchor, resolve it, visit it” pattern handles any next-button pagination.

How do you set rate limits and concurrency in Colly?

Use one colly.LimitRule, and turn on colly.Async() when you want parallel fetches. A synchronous collector already paces itself to one request at a time, so a LimitRule matters most once you go async, where it caps how many requests run together and how long Colly waits between them.

The LimitRule fields that do the work:

A polite async crawl looks like this. Async() makes Visit non-blocking, so c.Wait() at the end holds the program open until the queue drains:

package main

import (
	"fmt"
	"log"
	"time"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector(
		colly.AllowedDomains("books.toscrape.com"),
		colly.Async(true),
	)

	// At most 2 requests at once, 1s + up to 1s random between them.
	c.Limit(&colly.LimitRule{
		DomainGlob:  "*books.toscrape.com*",
		Parallelism: 2,
		Delay:       1 * time.Second,
		RandomDelay: 1 * time.Second,
	})

	c.OnHTML("article.product_pod", func(e *colly.HTMLElement) {
		log.Println(e.ChildText(".price_color"), e.ChildAttr("h3 a", "title"))
	})

	for i := 1; i <= 50; i++ {
		c.Visit(fmt.Sprintf("https://books.toscrape.com/catalogue/page-%d.html", i))
	}

	c.Wait() // block until every queued request finishes
}

The two pieces that trip people up:

• colly.Async(true) plus c.Wait() go together. Async makes Visit return immediately and run in a goroutine, so without c.Wait() the program exits before any page loads.
• Parallelism is per matching domain, not global. Two is gentle on a single host. The Delay plus RandomDelay combination spaces requests so the traffic does not arrive in a detectable, evenly-timed burst.

How do you scrape JavaScript-heavy sites in Go?

Drive a real browser with chromedp, because Colly cannot run JavaScript. When a page loads its data with JavaScript after the initial response, that content is absent from the HTML Colly downloads, so your OnHTML callbacks never fire. chromedp controls headless Chrome over the DevTools Protocol, so it renders the page exactly as a user sees it, and then you can parse the result.

go get github.com/chromedp/chromedp

You navigate, wait for the rendered content, pull the outer HTML, then hand it to goquery so your parsing code looks the same as before:

package main

import (
	"context"
	"log"
	"strings"
	"time"

	"github.com/PuerkitoBio/goquery"
	"github.com/chromedp/chromedp"
)

func main() {
	ctx, cancel := chromedp.NewContext(context.Background())
	defer cancel()

	ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
	defer cancel()

	var html string
	err := chromedp.Run(ctx,
		chromedp.Navigate("https://example.com"),
		chromedp.WaitVisible("body", chromedp.ByQuery),
		chromedp.OuterHTML("html", &html, chromedp.ByQuery),
	)
	if err != nil {
		log.Fatal(err)
	}

	doc, err := goquery.NewDocumentFromReader(strings.NewReader(html))
	if err != nil {
		log.Fatal(err)
	}

	doc.Find(".product").Each(func(i int, s *goquery.Selection) {
		log.Println(s.Text())
	})
}

What matters here:

• chromedp.NewContext(...) starts a headless Chrome and ties its lifetime to the context. The context.WithTimeout wrapper guarantees the run cannot hang forever, and both cancel calls free the browser.
• chromedp.WaitVisible("body", ...) blocks until the element is rendered, which is how you wait for JavaScript-loaded content before reading it. For a specific widget, wait on its selector instead of body.
• chromedp.OuterHTML("html", &html, ...) captures the fully rendered DOM into a string, which goquery parses with the same selector calls you would use anywhere.

The quick test for which path you need: fetch a page with Colly and check whether a value you can see in the browser shows up. When it is missing, the page is JavaScript-rendered and you move to chromedp or a rendering scraper API.

How do you avoid getting blocked when scraping in Go?

Look like a normal browser, slow down, and spread requests across IP addresses. A site flags a scraper on three signals: requests that arrive too fast, headers that look automated, and too much traffic from one IP. Each has a countermeasure that works whether you are on Colly or chromedp.

These measures carry a small project a long way. The trouble starts at scale: rotating a large IP pool, solving CAPTCHAs, and patching per-site quirks becomes its own engineering job, and it pulls focus from the data you actually want. That is where a scraper API like ChocoData fits the category, with a universal endpoint plus 453 dedicated endpoints that return the page so your Go code keeps working unchanged while the blocking is handled upstream. You point c.Visit at the API URL instead of the target, parse the response with the same OnHTML callbacks, and skip the proxy and CAPTCHA plumbing. I cover that full tradeoff, and the legal side, in the web scraping guide and in how to scrape without getting blocked.

For the scraping itself, the stack in this guide will carry you a long way. Take the async pagination crawler, point it at a site you care about, adjust the selectors until the fields come out clean, and add the anti-block measures as the target pushes back.