Is there a way to customize the User-Agent in WebMagic?

WebMagic is an open source Java framework used for web scraping. When performing web scraping tasks, it's common to customize the User-Agent string in the HTTP request headers to mimic a real web browser, as some websites may block requests that appear to come from bots or automated scripts.

In WebMagic, you can customize the User-Agent and other request headers using the Site class, which allows you to set various parameters for your web scraping bot, including the User-Agent.

Here's an example of how to customize the User-Agent string in WebMagic:

import us.codecraft.webmagic.Site;
import us.codecraft.webmagic.Spider;
import us.codecraft.webmagic.processor.PageProcessor;
import us.codecraft.webmagic.Page;
import us.codecraft.webmagic.selector.PlainText;

public class CustomUserAgentProcessor implements PageProcessor {

    // Define your custom User-Agent string
    private Site site = Site.me()
            .setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3");

    @Override
    public void process(Page page) {
        // Your scraping logic here
        // For example, extract the title of the web page
        String title = page.getHtml().xpath("//title/text()").toString();
        page.putField("title", title);
    }

    @Override
    public Site getSite() {
        return site;
    }

    public static void main(String[] args) {
        Spider.create(new CustomUserAgentProcessor())
                .addUrl("http://example.com") // Replace with your target URL
                .thread(5)
                .run();
    }
}

In the above code snippet, the setUserAgent method of the Site class is used to set a custom User-Agent string. You can replace the string with any User-Agent that suits your scraping task. Then, you create a Spider instance with the CustomUserAgentProcessor and start it with the run method.

Remember to follow the website's robots.txt file rules and terms of service to avoid violating any usage policies. Some websites may have strict rules about scraping, and setting a custom User-Agent that mimics a web browser does not give you permission to scrape without regard for these rules.