Is it possible to scrape content behind authentication using HtmlUnit and how?

Yes, it's possible to scrape content behind authentication using HtmlUnit. HtmlUnit is a "GUI-Less browser for Java programs" that models HTML documents and provides an API to invoke pages, fill out forms, click links, etc., just like you do in a regular browser. It's typically used for testing web pages from a Java application but can also be used to perform web scraping, including content that requires authentication.

To scrape content behind authentication, you need to simulate the login process programmatically using HtmlUnit. This involves sending a POST request with the necessary credentials (username and password) to the login form's action URL. After a successful login, you can then navigate to the pages that are accessible only after authentication.

Here's a general example of how you could use HtmlUnit to log in to a website and scrape content:

import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;
import com.gargoylesoftware.htmlunit.html.HtmlTextInput;

public class HtmlUnitScraper {

    public static void main(String[] args) {
        // Create and configure WebClient
        try (final WebClient webClient = new WebClient()) {
            // Disable JavaScript if it's not needed
            webClient.getOptions().setJavaScriptEnabled(false);

            // Go to login page
            final HtmlPage loginPage = webClient.getPage("http://example.com/login");

            // Get the form that we are dealing with and within that form, 
            // find the submit button and the fields that we want to submit
            // The form name will vary based on the website
            final HtmlForm form = loginPage.getFormByName("myform");

            final HtmlTextInput usernameField = form.getInputByName("username");
            final HtmlTextInput passwordField = form.getInputByName("password");
            final HtmlSubmitInput button = form.getInputByName("submit");

            // Fill in the login form
            usernameField.setValueAttribute("myUsername");
            passwordField.setValueAttribute("myPassword");

            // Click the submit button
            final HtmlPage pageAfterLogin = button.click();

            // Verify that login was successful if necessary
            // You can check for certain text or elements on the page to verify

            // Now you can scrape content from the page after login
            final String content = pageAfterLogin.asXml();

            // Process the content as needed
            System.out.println(content);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Important Considerations:

1. Handling Cookies: HtmlUnit automatically handles cookies, just like browsers do. After logging in, it will store and send cookies as needed for maintaining the session.

2. SSL/TLS Issues: If the website uses HTTPS and either has an invalid certificate or does SSL in a non-standard way, you might need to configure HtmlUnit to ignore these issues, although this is not recommended for production use due to security concerns.

3. JavaScript: If the website relies on JavaScript to render the login form or content, you need to enable JavaScript in the WebClient options.

4. Redirection: Some websites might redirect you after login. HtmlUnit's HtmlPage object will follow redirects by default, just like a normal browser.

Remember that web scraping content behind authentication is subject to the terms of service and privacy policy of the website you are accessing. Always ensure you have permission to scrape the content and that you are not violating any laws or agreements.