Pholcus is a high-concurrency, distributed, web crawler software written in the Go programming language. It's designed to aid in the task of web scraping, which involves programmatically downloading web pages and extracting useful information from them. Pholcus is known for its simplicity and flexibility, making it suitable for both personal and business purposes.
Features of Pholcus:
- High Concurrency: Utilizes Go's goroutines for concurrent operations, which allows it to perform multiple tasks simultaneously.
- Distributed: Can be deployed across different machines to scale the web crawling process.
- Rich Colly Support: Pholcus builds on the features provided by Colly, another web scraping framework in Go, which provides a lot of functionality out-of-the-box for scraping tasks.
- User-Friendly: It has a simple-to-use GUI for those who prefer not to work directly with code.
- Flexible: Offers support for both fixed and random User Agent strings, as well as proxy rotation to avoid detection by anti-scraping mechanisms.
- Pluggable: You can extend its functionality by writing plugins.
How Pholcus is Used in Web Scraping:
Pholcus can be used as a standalone software or as a library within your Go projects. As a standalone tool, it can be used through its GUI or command line interface. When used as a library, it allows developers to write custom spiders tailored to the specific needs of their web scraping tasks.
Below is a simple example of how you might use Pholcus in a Go project to perform web scraping. Please note that this is a simplified example for illustration purposes:
package main
import (
"github.com/henrylee2cn/pholcus/exec"
_ "github.com/henrylee2cn/pholcus_lib" // This is where you'd import your spiders
"github.com/henrylee2cn/pholcus/common/goquery" // Required for the query functionality
)
func main() {
// Set up the crawler
exec.DefaultRun("web")
// Define your spider logic
spider := exec.NewSpider(
// Your spider name and description
"ExampleSpider", "This is an example spider for demonstration purposes.",
)
// Add a request to the spider
spider.Request(
"GET", // The HTTP method
"https://example.com", // The URL you want to crawl
nil, // The HTTP body, if necessary
nil, // The HTTP header, if necessary
func(ctx *exec.Context) { // The parsing function
// Use goquery to parse the page
doc, err := goquery.NewDocumentFromReader(ctx.GetResponse().Body)
if err != nil {
return
}
// Extract data using goquery selectors
doc.Find("a").Each(func(i int, s *goquery.Selection) {
// For each <a> tag found, print its href attribute
href, _ := s.Attr("href")
println(href)
})
},
)
// Run the spider
exec.Run()
}
To run the code above, you would need to install Pholcus and its dependencies, which can be done with the go get command:
go get -u github.com/henrylee2cn/pholcus
Before using Pholcus for web scraping, always make sure to comply with the website's robots.txt file and terms of service. Additionally, be respectful of the site's resources and do not overload their servers with too many requests in a short period of time.
