How do I manage relative and absolute URLs when web scraping with GoQuery?

When web scraping with GoQuery in Go (Golang), managing relative and absolute URLs is essential for following links, downloading resources, and maintaining the correct context of the scraped content. GoQuery is a library that provides jQuery-like syntax for manipulating HTML documents, which makes it ideal for scraping tasks.

Here's how you can manage relative and absolute URLs when using GoQuery:

Handling Relative URLs

Relative URLs are URLs that are relative to the current page's URL. They generally omit the protocol (e.g., http:// or https://) and the domain name. To resolve a relative URL to an absolute one, you can use the url package from the Go standard library to parse the current page's URL and then resolve the relative path against it.

Here's an example of how to do this:

package main

import (
    "fmt"
    "net/http"
    "net/url"
    "github.com/PuerkitoBio/goquery"
    "log"
)

func resolveURL(baseURL, relativeURL string) (string, error) {
    base, err := url.Parse(baseURL)
    if err != nil {
        return "", err
    }
    rel, err := url.Parse(relativeURL)
    if err != nil {
        return "", err
    }
    return base.ResolveReference(rel).String(), nil
}

func main() {
    // Assume this is the URL of the page you are scraping
    pageURL := "https://example.com/path/page.html"

    // Fetch the page
    resp, err := http.Get(pageURL)
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    // Create a goquery document from the HTTP response
    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    // Find all links and resolve their URLs
    doc.Find("a").Each(func(i int, s *goquery.Selection) {
        // Get the href attribute of the link
        href, exists := s.Attr("href")
        if exists {
            // Resolve the relative URL
            absoluteURL, err := resolveURL(pageURL, href)
            if err != nil {
                log.Println("Error resolving URL:", err)
                return
            }
            fmt.Println(absoluteURL)
        }
    })
}

In this example, the resolveURL function takes a base URL (the URL of the current page) and a relative URL (found in the href attribute of a link) and resolves them to an absolute URL.

Handling Absolute URLs

Absolute URLs contain the full path, including the protocol and the domain name. When you encounter an absolute URL, there's no need to resolve it against the base URL since it can be used directly.

When scraping, you can check if a URL is absolute by parsing it and examining the Scheme and Host fields of the resulting url.URL struct. If those fields are not empty, the URL is considered absolute.

Here's a quick function to check if a URL is absolute:

func isAbsoluteURL(u string) bool {
    parsedURL, err := url.Parse(u)
    if err != nil {
        return false // or handle error according to your needs
    }
    return parsedURL.Scheme != "" && parsedURL.Host != ""
}

You can use this function in your scraping code to determine whether to resolve the URL or use it as is.

Remember that when scraping websites, you should always respect the robots.txt rules and any additional terms of service the website may have regarding automated access. Also, be polite and avoid making excessive requests that could overload the website's servers.