How to Set Request Timeouts in Colly

Setting appropriate request timeouts is crucial for building robust web scrapers with Colly. Timeouts prevent your scraper from hanging indefinitely when websites are slow or unresponsive, ensuring your application remains performant and reliable.

Understanding Timeouts in Colly

Colly provides several ways to configure timeouts for HTTP requests. The primary method is through the SetRequestTimeout() function, which sets a global timeout for all requests made by a collector instance.

Basic Timeout Configuration

Setting a Global Timeout

The simplest way to set a timeout in Colly is using the SetRequestTimeout() method:

package main

import (
    "fmt"
    "time"

    "github.com/gocolly/colly/v2"
)

func main() {
    c := colly.NewCollector()

    // Set a 30-second timeout for all requests
    c.SetRequestTimeout(30 * time.Second)

    c.OnHTML("title", func(e *colly.HTMLElement) {
        fmt.Println("Title:", e.Text)
    })

    c.OnError(func(r *colly.Response, err error) {
        fmt.Printf("Request failed: %s\n", err.Error())
    })

    c.Visit("https://example.com")
    c.Wait()
}

Using Transport Configuration

For more granular control, you can configure timeouts through the HTTP transport:

package main

import (
    "net/http"
    "time"

    "github.com/gocolly/colly/v2"
)

func main() {
    c := colly.NewCollector()

    // Configure custom HTTP transport with specific timeouts
    transport := &http.Transport{
        DialTimeout:           5 * time.Second,  // Connection timeout
        TLSHandshakeTimeout:   5 * time.Second,  // TLS handshake timeout
        ResponseHeaderTimeout: 10 * time.Second, // Response header timeout
        IdleConnTimeout:       30 * time.Second, // Idle connection timeout
    }

    client := &http.Client{
        Transport: transport,
        Timeout:   30 * time.Second, // Total request timeout
    }

    c.OnRequest(func(r *colly.Request) {
        r.Ctx.Put("client", client)
    })

    // Use the custom client
    c.OnResponse(func(r *colly.Response) {
        fmt.Printf("Response received from %s\n", r.Request.URL)
    })

    c.Visit("https://httpbin.org/delay/2")
    c.Wait()
}

Advanced Timeout Strategies

Per-Request Timeouts

You can set different timeouts for specific requests using context:

package main

import (
    "context"
    "fmt"
    "time"

    "github.com/gocolly/colly/v2"
)

func main() {
    c := colly.NewCollector()

    c.OnRequest(func(r *colly.Request) {
        // Set a shorter timeout for specific URLs
        if r.URL.Host == "slow-website.com" {
            ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
            defer cancel()
            r.Ctx = colly.NewContext()
            r.Ctx.Put("context", ctx)
        }
    })

    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        link := e.Attr("href")
        e.Request.Visit(link)
    })

    c.OnError(func(r *colly.Response, err error) {
        if err.Error() == "context deadline exceeded" {
            fmt.Printf("Request to %s timed out\n", r.Request.URL)
        }
    })

    c.Visit("https://example.com")
    c.Wait()
}

Timeout with Retry Logic

Combining timeouts with retry mechanisms provides better reliability:

package main

import (
    "fmt"
    "strings"
    "time"

    "github.com/gocolly/colly/v2"
    "github.com/gocolly/colly/v2/debug"
)

func main() {
    c := colly.NewCollector(
        colly.Debugger(&debug.LogDebugger{}),
    )

    // Set a reasonable timeout
    c.SetRequestTimeout(15 * time.Second)

    // Enable retry with exponential backoff
    c.OnError(func(r *colly.Response, err error) {
        if strings.Contains(err.Error(), "timeout") {
            fmt.Printf("Timeout for %s, retrying...\n", r.Request.URL)

            // Retry with increased timeout
            retryCollector := colly.NewCollector()
            retryCollector.SetRequestTimeout(30 * time.Second)
            retryCollector.Visit(r.Request.URL.String())
        }
    })

    c.OnHTML("title", func(e *colly.HTMLElement) {
        fmt.Printf("Successfully scraped: %s\n", e.Text)
    })

    c.Visit("https://httpbin.org/delay/5")
    c.Wait()
}

Best Practices for Timeout Configuration

1. Choose Appropriate Timeout Values

Different types of requests require different timeout values:

func setupTimeouts(c *colly.Collector, requestType string) {
    switch requestType {
    case "api":
        c.SetRequestTimeout(10 * time.Second)
    case "heavy_page":
        c.SetRequestTimeout(60 * time.Second)
    case "image_download":
        c.SetRequestTimeout(120 * time.Second)
    default:
        c.SetRequestTimeout(30 * time.Second)
    }
}

2. Implement Graceful Error Handling

Always handle timeout errors gracefully to maintain scraper stability:

c.OnError(func(r *colly.Response, err error) {
    switch {
    case strings.Contains(err.Error(), "timeout"):
        fmt.Printf("Timeout error for %s: %v\n", r.Request.URL, err)
        // Log timeout for monitoring
        logTimeout(r.Request.URL.String())

    case strings.Contains(err.Error(), "connection refused"):
        fmt.Printf("Connection refused for %s\n", r.Request.URL)

    default:
        fmt.Printf("Unexpected error: %v\n", err)
    }
})

3. Monitor Timeout Patterns

Track timeout occurrences to optimize your timeout settings:

type TimeoutStats struct {
    TotalRequests int
    Timeouts      int
    AverageTime   time.Duration
}

var stats TimeoutStats

c.OnRequest(func(r *colly.Request) {
    r.Ctx.Put("start_time", time.Now())
    stats.TotalRequests++
})

c.OnResponse(func(r *colly.Response) {
    startTime := r.Ctx.GetAny("start_time").(time.Time)
    duration := time.Since(startTime)
    stats.AverageTime = (stats.AverageTime + duration) / 2
})

c.OnError(func(r *colly.Response, err error) {
    if strings.Contains(err.Error(), "timeout") {
        stats.Timeouts++
    }
})

Handling Different Network Conditions

When scraping websites with varying response times, consider implementing adaptive timeouts. Similar to how to handle timeouts in Puppeteer, you can adjust timeout values based on historical performance:

type AdaptiveTimeout struct {
    baseTimeout    time.Duration
    maxTimeout     time.Duration
    successTimes   []time.Duration
    timeoutCount   int
    requestCount   int
}

func (at *AdaptiveTimeout) GetTimeout() time.Duration {
    if len(at.successTimes) == 0 {
        return at.baseTimeout
    }

    // Calculate average response time
    var total time.Duration
    for _, t := range at.successTimes {
        total += t
    }
    avgTime := total / time.Duration(len(at.successTimes))

    // Add buffer based on timeout rate
    timeoutRate := float64(at.timeoutCount) / float64(at.requestCount)
    buffer := time.Duration(float64(avgTime) * (1 + timeoutRate))

    timeout := avgTime + buffer
    if timeout > at.maxTimeout {
        timeout = at.maxTimeout
    }

    return timeout
}

Common Timeout Scenarios

Slow Loading Pages

For websites that consistently load slowly, increase the base timeout and implement progressive timeout strategies:

c.SetRequestTimeout(60 * time.Second)

c.OnRequest(func(r *colly.Request) {
    // Increase timeout for known slow domains
    if strings.Contains(r.URL.Host, "slow-site.com") {
        // Create new collector with extended timeout
        slowCollector := colly.NewCollector()
        slowCollector.SetRequestTimeout(120 * time.Second)
        slowCollector.Visit(r.URL.String())
        return
    }
})

API Endpoints

When scraping API endpoints, use shorter timeouts with proper retry logic:

func scrapeAPI(endpoint string) {
    c := colly.NewCollector()
    c.SetRequestTimeout(5 * time.Second)

    retryCount := 0
    maxRetries := 3

    c.OnError(func(r *colly.Response, err error) {
        if strings.Contains(err.Error(), "timeout") && retryCount < maxRetries {
            retryCount++
            time.Sleep(time.Duration(retryCount) * time.Second)
            c.Visit(endpoint)
        }
    })

    c.OnResponse(func(r *colly.Response) {
        // Process API response
        fmt.Printf("API Response: %s\n", string(r.Body))
    })

    c.Visit(endpoint)
    c.Wait()
}

Integration with Error Handling

Effective timeout management works hand-in-hand with proper error handling. For comprehensive error management strategies in Colly, refer to our guide on implementing retry logic for failed requests, which covers how to handle various types of errors including timeouts.

Performance Considerations

When setting timeouts, consider the impact on overall scraper performance. Too short timeouts can cause unnecessary failures, while too long timeouts can severely impact throughput. For detailed performance optimization strategies, check our article on performance considerations when using Colly.

Conclusion

Setting appropriate request timeouts in Colly is essential for building reliable web scrapers. By using SetRequestTimeout() for basic scenarios and implementing more sophisticated timeout strategies for complex requirements, you can ensure your scraper handles various network conditions gracefully.

Remember to: - Set reasonable default timeouts (typically 15-30 seconds) - Implement proper error handling for timeout scenarios - Monitor timeout patterns to optimize settings - Use adaptive timeouts for websites with variable response times - Consider different timeout values for different types of content

When dealing with JavaScript-heavy sites that require longer load times, you might also want to explore browser automation tools that offer more sophisticated timeout handling for dynamic content loading.

Table of contents