How can I handle pagination with IronWebScraper?

IronWebScraper is a C# web scraping library that provides an easy-to-use API for scraping websites. When dealing with pagination during web scraping, it's important to create a logic that can navigate through pages either by following 'next' button links or by iterating through a series of predictable URLs.

The following example demonstrates how to handle pagination with IronWebScraper:

using IronWebScraper;

public class PaginatedScraper : WebScraper
{
    public override void Init()
    {
        // Start by requesting the first page
        this.Request("http://example.com/page/1", Parse);
    }

    public override void Parse(Response response)
    {
        // Parse the page, extract data, etc.
        foreach (var item in response.Css("div.item"))
        {
            string title = item.Css("h2.title").TextContentClean;
            // Process or save your data here
            Console.WriteLine(title);
        }

        // Look for the 'next' page link and navigate to it
        var nextPageLink = response.Css("a.next").FirstOrDefault()?.Attributes["href"];
        if (nextPageLink != null)
        {
            // Request the 'next' page by calling the same Parse method
            this.Request(nextPageLink, Parse);
        }
    }
}

public class Program
{
    static void Main(string[] args)
    {
        var scraper = new PaginatedScraper();
        scraper.Start();
    }
}

In this example:

1. We create a class PaginatedScraper that inherits from WebScraper.
2. In the Init method, we start by requesting the first page of the pagination sequence.
3. The Parse method is responsible for processing the content of each page. It's where you would extract the data you need using CSS selectors.
4. After processing the page, we look for a 'next' link by using the response.Css("a.next") selector. If a 'next' link is found, we request that page by calling this.Request(nextPageLink, Parse), which effectively creates a recursive loop to navigate through pagination.
5. Finally, in the Main method, we instantiate PaginatedScraper and call its Start method to begin scraping.

Please note that the exact CSS selectors (div.item, h2.title, a.next) are placeholders and should be replaced with the actual selectors that match the content of the website you are scraping. Also, you should always ensure that your web scraping activities comply with the website's terms of service and legal regulations.

IronWebScraper handles the asynchronous nature of web requests behind the scenes, making it straightforward to implement pagination without worrying about the underlying asynchronous code.

Remember to add a reference to IronWebScraper in your project, which you can do by installing it via NuGet:

Install-Package IronWebScraper

Or by using the .NET CLI:

dotnet add package IronWebScraper

This example demonstrates a simple pagination handling scenario. Depending on the structure of the website you're scraping, there may be variations, such as pagination via URL parameters (?page=2), which would require slightly different logic to construct the URLs for each page.