Is WebMagic an open-source web scraping framework?

Yes, WebMagic is an open-source web scraping framework. It's a scalable crawler framework for Java, which provides a simple way to extract information from the web and save the data you collect in the way you want. WebMagic is inspired by Scrapy, a web scraping framework for Python, but it's designed specifically for the Java ecosystem.

The WebMagic framework includes several core components that make it easy to customize and extend, such as:

• Downloader: The module that performs HTTP requests and fetches the web pages.
• PageProcessor: The component where you define how to extract information from the pages you have downloaded.
• Scheduler: The part that manages the URLs to crawl.
• Pipeline: The module where you process the results after extraction and typically save them to a database, file, or perform other actions.

WebMagic is particularly suitable for those who are familiar with Java and prefer to work within the Java ecosystem for their web scraping tasks. It can be used for a variety of purposes, from data mining and monitoring to automated testing.

Here is a simple example of how you might use WebMagic in a Java project:

import us.codecraft.webmagic.Page;
import us.codecraft.webmagic.Site;
import us.codecraft.webmagic.Spider;
import us.codecraft.webmagic.processor.PageProcessor;

public class MyProcessor implements PageProcessor {

    // Configure the site settings like retry times, sleep time, user agent, etc.
    private Site site = Site.me().setRetryTimes(3).setSleepTime(1000);

    @Override
    public void process(Page page) {
        // Define how to extract information and where to save it
        page.putField("title", page.getHtml().xpath("//title/text()").toString());
        // You can also use CSS selectors, regex, and more
    }

    @Override
    public Site getSite() {
        return site;
    }

    public static void main(String[] args) {
        // Start the crawler
        Spider.create(new MyProcessor())
            // Starting URL
            .addUrl("http://example.com")
            // Open 5 threads
            .thread(5)
            // Launch the crawler
            .run();
    }
}

To include WebMagic in your project, you would typically add the dependency to your pom.xml if you're using Maven:

<dependencies>
    <dependency>
        <groupId>us.codecraft</groupId>
        <artifactId>webmagic-core</artifactId>
        <version>0.7.3</version>
    </dependency>
</dependencies>

Or if you're using Gradle, add this to your build.gradle:

dependencies {
    implementation 'us.codecraft:webmagic-core:0.7.3'
}

As with any web scraping tool, it's important to use WebMagic responsibly. Respect the robots.txt file of websites, comply with their terms of service, and don't overload their servers with too many requests in a short period.