Table of contents

How do I set custom headers and user agents in Go HTTP requests?

Setting custom headers and user agents in Go HTTP requests is essential for web scraping, API interactions, and mimicking browser behavior. Go provides several methods through the http.Request.Header field to customize HTTP headers effectively.

Basic Header Setting

The most straightforward approach uses http.NewRequest and the Header.Set() method:

package main

import (
    "fmt"
    "io"
    "net/http"
)

func main() {
    client := &http.Client{}

    // Create a new HTTP request
    req, err := http.NewRequest("GET", "https://httpbin.org/headers", nil)
    if err != nil {
        fmt.Printf("Error creating request: %v\n", err)
        return
    }

    // Set custom User-Agent
    req.Header.Set("User-Agent", "MyGoScraper/1.0 (https://example.com)")

    // Set additional custom headers
    req.Header.Set("Accept", "text/html,application/xhtml+xml")
    req.Header.Set("Accept-Language", "en-US,en;q=0.9")
    req.Header.Set("Cache-Control", "no-cache")

    // Perform the request
    resp, err := client.Do(req)
    if err != nil {
        fmt.Printf("Error making request: %v\n", err)
        return
    }
    defer resp.Body.Close()

    // Read and print response
    body, _ := io.ReadAll(resp.Body)
    fmt.Printf("Response: %s\n", body)
}

Header Methods Comparison

Go provides different methods for setting headers, each with specific use cases:

package main

import (
    "fmt"
    "net/http"
)

func demonstrateHeaderMethods() {
    req, _ := http.NewRequest("GET", "https://example.com", nil)

    // Set() - Replaces any existing values
    req.Header.Set("Authorization", "Bearer token123")

    // Add() - Appends to existing values (useful for multiple values)
    req.Header.Add("Accept", "application/json")
    req.Header.Add("Accept", "text/html") // Now has both values

    // Direct assignment (less common)
    req.Header["X-Custom-Header"] = []string{"value1", "value2"}

    // Get header values
    fmt.Printf("Accept headers: %v\n", req.Header["Accept"])
    fmt.Printf("User-Agent: %s\n", req.Header.Get("User-Agent"))
}

Common User-Agent Examples

Here are realistic user-agent strings for different scenarios:

package main

import (
    "fmt"
    "net/http"
)

func setCommonUserAgents() {
    userAgents := map[string]string{
        "chrome_windows": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
        "firefox_mac":    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:122.0) Gecko/20100101 Firefox/122.0",
        "safari_ios":     "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1",
        "bot_friendly":   "GoBot/1.0 (+https://example.com/bot)",
    }

    for name, ua := range userAgents {
        req, _ := http.NewRequest("GET", "https://example.com", nil)
        req.Header.Set("User-Agent", ua)
        fmt.Printf("%s: %s\n", name, ua)
    }
}

Advanced Header Configuration

For more complex scenarios, you can create reusable header configurations:

package main

import (
    "net/http"
    "time"
)

type HeaderConfig struct {
    UserAgent    string
    Accept       string
    Language     string
    Encoding     string
    CacheControl string
    Custom       map[string]string
}

func createRequestWithConfig(url string, config HeaderConfig) (*http.Request, error) {
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        return nil, err
    }

    // Set standard headers
    if config.UserAgent != "" {
        req.Header.Set("User-Agent", config.UserAgent)
    }
    if config.Accept != "" {
        req.Header.Set("Accept", config.Accept)
    }
    if config.Language != "" {
        req.Header.Set("Accept-Language", config.Language)
    }
    if config.Encoding != "" {
        req.Header.Set("Accept-Encoding", config.Encoding)
    }
    if config.CacheControl != "" {
        req.Header.Set("Cache-Control", config.CacheControl)
    }

    // Set custom headers
    for key, value := range config.Custom {
        req.Header.Set(key, value)
    }

    return req, nil
}

func main() {
    config := HeaderConfig{
        UserAgent:    "Mozilla/5.0 (compatible; GoScraper/1.0)",
        Accept:       "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        Language:     "en-US,en;q=0.5",
        Encoding:     "gzip, deflate",
        CacheControl: "no-cache",
        Custom: map[string]string{
            "X-Requested-With": "XMLHttpRequest",
            "Referer":          "https://google.com",
        },
    }

    req, err := createRequestWithConfig("https://example.com", config)
    if err != nil {
        panic(err)
    }

    client := &http.Client{Timeout: 10 * time.Second}
    resp, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
}

Web Scraping Best Practices

When setting headers for web scraping, follow these guidelines:

package main

import (
    "math/rand"
    "net/http"
    "time"
)

// Rotate user agents to avoid detection
var userAgents = []string{
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
}

func createScrapingRequest(url string) (*http.Request, error) {
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        return nil, err
    }

    // Randomize user agent
    rand.Seed(time.Now().UnixNano())
    ua := userAgents[rand.Intn(len(userAgents))]
    req.Header.Set("User-Agent", ua)

    // Set browser-like headers
    req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
    req.Header.Set("Accept-Language", "en-US,en;q=0.5")
    req.Header.Set("Accept-Encoding", "gzip, deflate")
    req.Header.Set("Connection", "keep-alive")
    req.Header.Set("Upgrade-Insecure-Requests", "1")

    // Optional: Set referer to appear more legitimate
    req.Header.Set("Referer", "https://www.google.com/")

    return req, nil
}

Header Validation and Debugging

Always validate your headers and debug requests when needed:

package main

import (
    "fmt"
    "net/http"
    "strings"
)

func debugHeaders(req *http.Request) {
    fmt.Println("Request Headers:")
    for name, values := range req.Header {
        fmt.Printf("  %s: %s\n", name, strings.Join(values, ", "))
    }
}

func validateHeaders(req *http.Request) error {
    requiredHeaders := []string{"User-Agent", "Accept"}

    for _, header := range requiredHeaders {
        if req.Header.Get(header) == "" {
            return fmt.Errorf("missing required header: %s", header)
        }
    }
    return nil
}

Key Points

  • Use Header.Set() for single-value headers like User-Agent
  • Use Header.Add() for multi-value headers like Accept
  • Always set realistic User-Agent strings to avoid being blocked
  • Include common browser headers for better compatibility
  • Respect robots.txt and rate limits when scraping
  • Test headers with services like httpbin.org to verify they're sent correctly

Setting appropriate headers is crucial for successful HTTP requests in Go, especially for web scraping applications where mimicking browser behavior can prevent blocking and ensure reliable data extraction.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I process compressed HTTP responses in Go?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon