What are some common user-agent strings I can use in Go for web scraping?

When writing a web scraper in Go, it's often necessary to set a custom User-Agent header to emulate a real web browser. This can help in avoiding detection as a bot and allow your scraper to access web pages more reliably. Here are several common User-Agent strings that you can use in your Go web scraping projects:

Common User-Agent Strings

• Google Chrome on Windows 10:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36

• Mozilla Firefox on Windows 10:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0

• Apple Safari on macOS:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15

• Microsoft Edge on Windows 10:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.705.50 Safari/537.36 Edg/88.0.705.50

• Googlebot:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• iPhone Safari (iOS 14):

Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Mobile/15E148 Safari/604.1

• Android Browser on Samsung Galaxy:

Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-G981U) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/13.0 Chrome/83.0.4103.106 Mobile Safari/537.36

Setting the User-Agent in Go

When you perform web scraping in Go, you can use the net/http package to create a custom HTTP client and set the User-Agent header. Below is an example of how to set a User-Agent in a Go HTTP request:

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {
    // Define the URL to scrape
    url := "https://example.com"

    // Create a new HTTP request
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        panic(err)
    }

    // Set the User-Agent header
    req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36")

    // Create an HTTP client and send the request
    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    // Read and print the response body
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(body))
}

In this example, we're setting the User-Agent to that of a Chrome browser on Windows 10.

Notes on Web Scraping

• Always respect the website's robots.txt file and terms of service when scraping.
• It's good practice to not send too many requests in a short period to avoid overloading the server.
• Some websites may still detect and block scraping activities, even with a common User-Agent, so consider other strategies like rotating User-Agent strings, using proxies, and implementing proper timing between requests.
• Always check for and handle possible errors when making HTTP requests.

Remember that web scraping can have legal and ethical implications, so use these techniques responsibly and consider the impact of your actions on the target website.