Colly is an open-source, idiomatic web scraping framework for Go (Golang) designed to be elegant and versatile. It simplifies the process of building web scrapers and crawlers by providing a number of useful features that are commonly required in these applications. Below are the main features of Colly:
1. Clean API
Colly provides a clean and intuitive API that makes it easy for developers to start scraping websites without having to deal with low-level details like managing HTTP requests and parsing HTML.
2. Fast HTTP Engine
Colly is built on top of Go's standard HTTP library and can perform multiple requests in parallel. It can be fine-tuned to optimize the scraping speed and handle large volumes of data efficiently.
3. CSS Selector Support
It supports CSS selectors to locate and extract data from HTML documents, similar to jQuery. This makes it easy to pinpoint the exact pieces of information you want to scrape from a webpage.
4. XPath Selector Support
In addition to CSS selectors, Colly also supports XPath queries. This provides another powerful way to navigate and select nodes within XML/HTML documents.
5. Automatic Cookie and Session Handling
Colly can automatically manage cookies and sessions. This means you don't have to manually handle the storage and sending of cookies between requests.
6. Caching
To avoid downloading the same page multiple times, Colly supports caching. You can cache the responses in memory or use any backend that satisfies the http.RoundTripper interface, allowing for persistent caching.
7. Asynchronous Jobs
Colly can perform scraping jobs asynchronously, which is beneficial for parallelizing tasks and improving the overall efficiency of the scraping process.
8. Rate Limiting
It has built-in support for rate limiting to ensure that your scraper does not hit the target website too frequently, which could lead to your IP getting banned.
9. Proxy Switcher
Colly allows you to use a set of proxy servers and switch between them either randomly or using a custom logic to avoid detection.
10. Distributed Scraping
With Colly, you can create distributed scraping solutions that can run on multiple machines to scale up the scraping tasks.
11. Extensions
Colly has a variety of extensions that can be used to extend its functionality, such as for working with forms, logging, and even integrating with storage systems like MongoDB or Amazon S3.
12. Robust Error Handling
It has robust error handling mechanisms that allow you to gracefully handle any issues that arise during the scraping process.
13. Debugging Support
Colly comes with built-in support for debugging which makes it easier to troubleshoot issues with your scrapers.
14. Binary Data Handling
Colly can handle binary data, which is useful when you need to download images, videos, or any other type of binary content.
Example Usage of Colly in Go:
Here's a simple example of how to use Colly to scrape links from a website:
package main
import (
"fmt"
"github.com/gocolly/colly"
)
func main() {
// Create a new collector
c := colly.NewCollector()
// On every <a> element which has href attribute call callback
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
link := e.Attr("href")
fmt.Printf("Link found: %q -> %s\n", e.Text, link)
})
// Before making a request print "Visiting ..."
c.OnRequest(func(r *colly.Request) {
fmt.Println("Visiting", r.URL.String())
})
// Start scraping on https://hackerspaces.org
c.Visit("https://hackerspaces.org/")
}
In this example, Colly is used to find all links on the hackerspaces.org website and print them to the console. Notice how the OnHTML method is used to define a callback that processes each a element with an href attribute.
Remember that when using Colly or any web scraping tool, it's important to respect the website's robots.txt rules and terms of service. Always use web scraping responsibly to avoid legal issues and to prevent overloading the website's servers.
