Table of contents

What are the debugging options available in Puppeteer-Sharp?

Puppeteer-Sharp provides extensive debugging capabilities that help developers troubleshoot automation scripts, identify issues, and monitor browser behavior. Understanding these debugging options is crucial for building robust web scraping and automation solutions.

Console Output and Logging

Enabling DevTools Mode

The most fundamental debugging approach in Puppeteer-Sharp is running the browser in non-headless mode with DevTools enabled:

using PuppeteerSharp;

var launchOptions = new LaunchOptions
{
    Headless = false,
    DevTools = true,
    SlowMo = 250 // Slow down operations by 250ms
};

await using var browser = await Puppeteer.LaunchAsync(launchOptions);
var page = await browser.NewPageAsync();
await page.GoToAsync("https://example.com");

This configuration opens a visible browser window with DevTools automatically opened, allowing you to inspect elements, monitor network requests, and debug JavaScript execution in real-time.

Console Message Monitoring

Puppeteer-Sharp can capture and log all console messages from the target page:

page.Console += (sender, e) => 
{
    Console.WriteLine($"[{e.Message.Type}] {e.Message.Text}");

    // Log arguments if any
    foreach (var arg in e.Message.Args)
    {
        Console.WriteLine($"  Arg: {arg}");
    }
};

// Navigate and interact with the page
await page.GoToAsync("https://example.com");

Exception and Error Handling

Implement comprehensive error handling to catch and debug various types of exceptions:

page.PageError += (sender, e) => 
{
    Console.WriteLine($"Page Error: {e.Error.Message}");
    Console.WriteLine($"Stack Trace: {e.Error.StackTrace}");
};

page.RequestFailed += (sender, e) => 
{
    Console.WriteLine($"Request Failed: {e.Request.Url}");
    Console.WriteLine($"Failure Text: {e.Request.FailureText}");
};

try
{
    await page.GoToAsync("https://example.com");
    await page.ClickAsync("#submit-button");
}
catch (PuppeteerException ex)
{
    Console.WriteLine($"Puppeteer Exception: {ex.Message}");
    Console.WriteLine($"Stack Trace: {ex.StackTrace}");
}

Visual Debugging Options

Screenshot Debugging

Take screenshots at different stages of your automation to visually debug the process:

// Take full page screenshot
await page.ScreenshotAsync("/path/to/debug-screenshot.png", new ScreenshotOptions
{
    FullPage = true,
    Type = ScreenshotType.Png
});

// Take screenshot of specific element
var element = await page.QuerySelectorAsync("#specific-element");
await element.ScreenshotAsync("/path/to/element-screenshot.png");

// Conditional screenshot based on conditions
if (await page.QuerySelectorAsync(".error-message") != null)
{
    await page.ScreenshotAsync("/path/to/error-state.png");
}

Video Recording

For complex debugging scenarios, enable video recording to capture the entire automation session:

var launchOptions = new LaunchOptions
{
    Headless = false,
    Args = new[] { "--enable-logging", "--v=1" }
};

var browser = await Puppeteer.LaunchAsync(launchOptions);
var page = await browser.NewPageAsync();

// Start recording (requires additional setup with screen recording tools)
await page.GoToAsync("https://example.com");
// Perform your automation steps
await page.ClickAsync("#button");
await page.TypeAsync("#input", "test data");

Network Request Debugging

Request and Response Monitoring

Monitor all network requests to debug API calls, resource loading, and response handling:

page.Request += async (sender, e) => 
{
    Console.WriteLine($"Request: {e.Request.Method} {e.Request.Url}");
    Console.WriteLine($"Headers: {string.Join(", ", e.Request.Headers)}");

    if (e.Request.PostData != null)
    {
        Console.WriteLine($"Post Data: {e.Request.PostData}");
    }
};

page.Response += async (sender, e) => 
{
    Console.WriteLine($"Response: {e.Response.Status} {e.Response.Url}");
    Console.WriteLine($"Headers: {string.Join(", ", e.Response.Headers)}");

    // Log response body for specific requests
    if (e.Response.Url.Contains("api/data"))
    {
        var responseText = await e.Response.TextAsync();
        Console.WriteLine($"Response Body: {responseText}");
    }
};

await page.GoToAsync("https://example.com");

Request Interception for Debugging

Intercept and modify requests to test different scenarios:

await page.SetRequestInterceptionAsync(true);

page.Request += async (sender, e) => 
{
    var request = e.Request;

    // Log and optionally modify requests
    Console.WriteLine($"Intercepting: {request.Url}");

    if (request.Url.Contains("debug-endpoint"))
    {
        // Continue with modified headers
        await request.ContinueAsync(new Payload
        {
            Headers = new Dictionary<string, object>
            {
                ["X-Debug"] = "true",
                ["Authorization"] = "Bearer debug-token"
            }
        });
    }
    else
    {
        await request.ContinueAsync();
    }
};

Advanced Debugging Techniques

Performance Monitoring

Monitor page performance metrics to identify bottlenecks:

await page.GoToAsync("https://example.com");

// Get performance metrics
var metrics = await page.MetricsAsync();
foreach (var metric in metrics)
{
    Console.WriteLine($"{metric.Key}: {metric.Value}");
}

// Monitor specific performance events
await page.EvaluateExpressionAsync(@"
    performance.mark('automation-start');
    // Your automation code here
    performance.mark('automation-end');
    performance.measure('automation-duration', 'automation-start', 'automation-end');
");

var performanceEntries = await page.EvaluateExpressionAsync<object[]>(@"
    JSON.stringify(performance.getEntriesByType('measure'))
");

Memory and Resource Monitoring

Track memory usage and resource consumption during automation:

// Enable runtime domain for memory monitoring
var client = await page.Target.CreateCDPSessionAsync();
await client.SendAsync("Runtime.enable");
await client.SendAsync("HeapProfiler.enable");

// Take heap snapshot for memory analysis
await client.SendAsync("HeapProfiler.takeHeapSnapshot");

// Monitor JavaScript execution context
page.Console += (sender, e) => 
{
    if (e.Message.Type == ConsoleType.Error)
    {
        Console.WriteLine($"JS Error: {e.Message.Text}");
    }
};

Debugging Element Interactions

Debug element selection and interaction issues with detailed logging:

public async Task<ElementHandle> DebugQuerySelector(IPage page, string selector)
{
    Console.WriteLine($"Attempting to find element: {selector}");

    var element = await page.QuerySelectorAsync(selector);

    if (element == null)
    {
        Console.WriteLine($"Element not found: {selector}");

        // Take screenshot for debugging
        await page.ScreenshotAsync($"debug-element-not-found-{DateTime.Now:yyyyMMdd-HHmmss}.png");

        // List all similar elements
        var allElements = await page.QuerySelectorAllAsync(selector.Split(' ')[0]);
        Console.WriteLine($"Found {allElements.Length} elements with similar selector");

        return null;
    }

    // Verify element is visible and clickable
    var boundingBox = await element.BoundingBoxAsync();
    var isVisible = await element.IsIntersectingViewportAsync();

    Console.WriteLine($"Element found - Visible: {isVisible}, BoundingBox: {boundingBox}");

    return element;
}

// Usage
var button = await DebugQuerySelector(page, "#submit-button");
if (button != null)
{
    await button.ClickAsync();
}

Browser and Context Debugging

Browser Process Monitoring

Monitor the underlying Chromium process for debugging browser-level issues:

var launchOptions = new LaunchOptions
{
    Headless = false,
    Args = new[]
    {
        "--enable-logging",
        "--log-level=0",
        "--remote-debugging-port=9222"
    }
};

var browser = await Puppeteer.LaunchAsync(launchOptions);

// Access browser process information
var process = browser.Process;
if (process != null)
{
    Console.WriteLine($"Browser Process ID: {process.Id}");
    Console.WriteLine($"Browser Executable: {browser.Process?.StartInfo?.FileName}");
}

// Monitor browser disconnection
browser.Disconnected += (sender, e) => 
{
    Console.WriteLine("Browser disconnected");
};

browser.TargetCreated += (sender, e) => 
{
    Console.WriteLine($"New target created: {e.Target.Url}");
};

Context Isolation Debugging

When working with browser contexts, implement debugging for context-specific issues:

var context = await browser.CreateIncognitoBrowserContextAsync();

context.TargetCreated += (sender, e) => 
{
    Console.WriteLine($"Target created in context: {e.Target.Url}");
};

context.Page += (sender, e) => 
{
    Console.WriteLine($"New page in context: {e.Page.Url}");

    // Add debugging to new pages in this context
    e.Page.Console += (pageSender, pageEvent) => 
    {
        Console.WriteLine($"[Context Page] {pageEvent.Message.Text}");
    };
};

var page = await context.NewPageAsync();
await page.GoToAsync("https://example.com");

Integration with Development Tools

Debugging with Wait Strategies

Implement robust wait strategies with debugging information, similar to handling timeouts in Puppeteer:

public async Task<bool> WaitForElementWithDebug(IPage page, string selector, int timeoutMs = 30000)
{
    var stopwatch = System.Diagnostics.Stopwatch.StartNew();

    try
    {
        Console.WriteLine($"Waiting for element: {selector}");
        await page.WaitForSelectorAsync(selector, new WaitForSelectorOptions
        {
            Timeout = timeoutMs
        });

        Console.WriteLine($"Element found after {stopwatch.ElapsedMilliseconds}ms");
        return true;
    }
    catch (WaitTaskTimeoutException)
    {
        Console.WriteLine($"Timeout waiting for element: {selector} after {stopwatch.ElapsedMilliseconds}ms");

        // Take screenshot for debugging
        await page.ScreenshotAsync($"timeout-debug-{DateTime.Now:yyyyMMdd-HHmmss}.png");

        return false;
    }
}

Debugging Network Issues

For debugging network-related problems, implement comprehensive network request monitoring:

var networkDebugger = new NetworkDebugger();
await networkDebugger.EnableFor(page);

public class NetworkDebugger
{
    private readonly List<string> _failedRequests = new();
    private readonly Dictionary<string, DateTime> _requestTiming = new();

    public async Task EnableFor(IPage page)
    {
        page.Request += LogRequest;
        page.Response += LogResponse;
        page.RequestFailed += LogFailedRequest;
    }

    private async void LogRequest(object sender, RequestEventArgs e)
    {
        _requestTiming[e.Request.Url] = DateTime.Now;
        Console.WriteLine($"[REQUEST] {e.Request.Method} {e.Request.Url}");
    }

    private async void LogResponse(object sender, ResponseCreatedEventArgs e)
    {
        var duration = DateTime.Now - _requestTiming.GetValueOrDefault(e.Response.Url, DateTime.Now);
        Console.WriteLine($"[RESPONSE] {e.Response.Status} {e.Response.Url} ({duration.TotalMilliseconds}ms)");
    }

    private async void LogFailedRequest(object sender, RequestEventArgs e)
    {
        _failedRequests.Add(e.Request.Url);
        Console.WriteLine($"[FAILED] {e.Request.Url} - {e.Request.FailureText}");
    }

    public void PrintSummary()
    {
        Console.WriteLine($"Total failed requests: {_failedRequests.Count}");
        foreach (var failed in _failedRequests)
        {
            Console.WriteLine($"  - {failed}");
        }
    }
}

Best Practices for Debugging

When debugging Puppeteer-Sharp applications, implement these best practices:

  1. Always use try-catch blocks around critical operations
  2. Enable comprehensive logging for network requests, console messages, and page errors
  3. Take screenshots at key points in your automation flow
  4. Use meaningful selector strategies and validate element presence before interaction
  5. Monitor performance metrics to identify bottlenecks
  6. Implement timeout handling with proper error messages

For more complex scenarios involving browser session management or handling authentication, combine these debugging techniques with your specific use case requirements.

By leveraging these comprehensive debugging options in Puppeteer-Sharp, you can efficiently identify and resolve issues in your web automation and scraping projects, ensuring robust and reliable applications.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I handle SSL certificate errors in Puppeteer-Sharp?
What is the difference between Page.WaitForSelector and Page.QuerySelector in Puppeteer-Sharp?
How can I simulate different network conditions with Puppeteer-Sharp?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon