Is it possible to use WebMagic with a proxy server?

Yes, it is possible to use WebMagic with a proxy server. WebMagic is a flexible and extensible web crawling framework for Java that supports various features including proxy rotation.

To use a proxy server with WebMagic, you can configure it through the Site object, which holds various settings for the crawler. You can set a single proxy or a list of proxies to be used during the web crawling process.

Here is an example of how to set up a proxy server with WebMagic:

import us.codecraft.webmagic.Site;
import us.codecraft.webmagic.Spider;
import us.codecraft.webmagic.processor.PageProcessor;
import us.codecraft.webmagic.proxy.Proxy;
import us.codecraft.webmagic.proxy.SimpleProxyProvider;

public class MyPageProcessor implements PageProcessor {
    // Define your Site with proxy details
    private Site site = Site.me()
        .setRetryTimes(3)
        .setSleepTime(1000)
        .setTimeOut(10000)
        .setUseGzip(true)
        .setHttpProxy(new Proxy("your.proxy.host", 8080)); // Set your proxy host and port here

    @Override
    public void process(us.codecraft.webmagic.Page page) {
        // Your scraping logic here
    }

    @Override
    public Site getSite() {
        return site;
    }

    public static void main(String[] args) {
        Spider.create(new MyPageProcessor())
            // Define your starting URL here
            .addUrl("http://example.com")
            // Start the spider
            .run();
    }
}

In the above example, we create a Site object and set the proxy using the setHttpProxy method, which accepts a Proxy object that holds the proxy host and port. You can also set credentials if the proxy requires authentication.

If you have a list of proxy servers and want to rotate them, you can use the SimpleProxyProvider class to manage the proxies:

import us.codecraft.webmagic.proxy.Proxy;
import us.codecraft.webmagic.proxy.SimpleProxyProvider;

// ...

SimpleProxyProvider proxyProvider = SimpleProxyProvider.from(
    new Proxy("proxy1.server.com", 8080),
    new Proxy("proxy2.server.com", 8080),
    // Add more proxies as needed
);

site.setHttpProxyPool(proxyProvider.getProxyList());

In this case, SimpleProxyProvider.from takes a list of Proxy objects. The setHttpProxyPool method of Site is used to set the list of proxies to the proxy pool for rotation.

Remember that when using a proxy, the target website might still detect and block your requests if the proxy is known to be used for scraping or if there's suspicious activity coming from the proxy IP. Always respect the website's robots.txt and terms of service when scraping and ensure that you are not violating any laws or regulations.