How do I customize the User-Agent string in Colly?

In Colly, the popular Go package for web scraping, you can customize the User-Agent string by setting the UserAgent attribute on the Collector object before you start making requests. The User-Agent string is an HTTP header that servers use to identify the client making the request. Customizing it can be useful for mimicking different browsers or devices or for avoiding detection as a bot when scraping websites.

Here's a simple example of how to set a custom User-Agent in Colly:

package main

import (
    "fmt"
    "log"

    "github.com/gocolly/colly"
)

func main() {
    // Create a new collector
    c := colly.NewCollector(
        // Optionally, you can set other options such as AllowedDomains, etc.
    )

    // Set custom User-Agent
    c.UserAgent = "MyCustomUserAgentString/1.0"

    // Start scraping on http://httpbin.org
    c.OnHTML("pre", func(e *colly.HTMLElement) {
        fmt.Println("Response body:", e.Text)
    })

    // Handle request errors
    c.OnError(func(r *colly.Response, err error) {
        log.Println("Request URL:", r.Request.URL, "failed with response:", r, "\nError:", err)
    })

    // Start scraping
    err := c.Visit("http://httpbin.org/user-agent")
    if err != nil {
        log.Fatal(err)
    }
}

In this example, we set the User-Agent to "MyCustomUserAgentString/1.0" before making a request to http://httpbin.org/user-agent, which is a simple service for testing HTTP requests including headers. The code sets up an HTML element handler for the <pre> tag, which contains the JSON response from httpbin.org showing the User-Agent string received by the server.

Remember that while changing the User-Agent can help in web scraping, it's important to respect the terms of service of the website and to scrape responsibly. Excessive requests or attempting to circumvent anti-scraping measures may lead to your IP being blocked or other legal actions.

To ensure that the custom User-Agent is set for every request, including those made within callbacks or after redirects, make sure to set the UserAgent attribute before initiating any requests with your Collector instance.

If you need to rotate User-Agents or have more complex requirements, you could also set the OnRequest callback to modify the request header dynamically:

c.OnRequest(func(r *colly.Request) {
    r.Headers.Set("User-Agent", "MyDynamicUserAgentString/2.0")
})

This will set the User-Agent header to "MyDynamicUserAgentString/2.0" for each request made during the scraping session. You can customize the string or choose it from a list of User-Agents according to your logic.