How do I handle redirects when using IronWebScraper?

IronWebScraper is a C# library designed to make web scraping simple. When working with web pages that employ redirects, handling them properly is crucial to ensure you're scraping the intended content. Fortunately, IronWebScraper automatically handles HTTP redirects (status codes 300-399) by following them until the final resource is reached.

However, if you need more control over how redirects are handled, you can customize this behavior by overriding the Request method in your WebScraper subclass or by handling the OnBeforeRequest event.

Here's an example of how you can handle redirects by overriding the Request method:

using IronWebScraper;

public class RedirectHandlingScraper : WebScraper
{
    public override void Init()
    {
        this.Request("http://example.com", Parse);
    }

    public override void Parse(Response response)
    {
        if (response.StatusCode == System.Net.HttpStatusCode.Redirect || 
            response.StatusCode == System.Net.HttpStatusCode.MovedPermanently)
        {
            // Handle the redirect manually if needed
            var redirectUrl = response.Headers["Location"];
            this.Request(redirectUrl, Parse);
        }
        else
        {
            // Normal parsing
        }
    }
}

public class Program
{
    public static void Main()
    {
        var scraper = new RedirectHandlingScraper();
        scraper.Start();
    }
}

If you want to handle redirects globally for all requests, you can attach an event handler to the OnBeforeRequest event. This event is triggered before any HTTP request is made. Here's an example of how you can use this event to inspect and modify requests:

using IronWebScraper;

public class GlobalRedirectScraper : WebScraper
{
    public override void Init()
    {
        this.OnBeforeRequest += HandleRedirects;
        this.Request("http://example.com", Parse);
    }

    private void HandleRedirects(object sender, RequestEventArgs e)
    {
        e.Request.AllowAutoRedirect = false; // Disable automatic redirects
        // You can now handle redirects manually by inspecting the response in the Parse method
    }

    public override void Parse(Response response)
    {
        if (response.StatusCode == System.Net.HttpStatusCode.Redirect || 
            response.StatusCode == System.Net.HttpStatusCode.MovedPermanently)
        {
            // Handle the redirect manually
            var redirectUrl = response.Headers["Location"];
            this.Request(redirectUrl, Parse);
        }
        else
        {
            // Normal parsing
        }
    }
}

public class Program
{
    public static void Main()
    {
        var scraper = new GlobalRedirectScraper();
        scraper.Start();
    }
}

In the examples above, we check the response status code to determine if it's a redirect. If so, we manually handle the redirect by issuing a new request to the URL found in the Location header. Note that AllowAutoRedirect is set to false to prevent automatic redirection and to give you full control over the redirect process.

Remember to replace http://example.com with the URL you want to scrape and customize the Parse method to handle the content of the final page after all redirects have been followed.