Table of contents

How to handle page redirections in Puppeteer?

Page redirections are common in web applications and essential to handle properly in Puppeteer automation. This guide covers different types of redirections and best practices for handling them.

Understanding Page Redirections

Redirections occur when: - Clicking links that navigate to new pages - Submitting forms that redirect after processing - Automatic redirects (HTTP 3xx status codes) - JavaScript-triggered navigation

Basic Redirection Handling

Use waitForNavigation() to wait for page redirections to complete:

const puppeteer = require('puppeteer');

async function handleBasicRedirect() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.goto('https://example.com');

    // Set up navigation promise BEFORE triggering redirect
    const navigationPromise = page.waitForNavigation();

    // Trigger the redirect
    await page.click('a[href="/redirect-page"]');

    // Wait for navigation to complete
    await navigationPromise;

    console.log('Redirected to:', page.url());
    await browser.close();
}

waitUntil Options

The waitUntil parameter determines when navigation is considered complete:

// Wait for DOM content to load (fastest)
await page.waitForNavigation({ waitUntil: 'domcontentloaded' });

// Wait for load event (default)
await page.waitForNavigation({ waitUntil: 'load' });

// Wait until no network requests for 500ms
await page.waitForNavigation({ waitUntil: 'networkidle0' });

// Wait until ≤2 network requests for 500ms
await page.waitForNavigation({ waitUntil: 'networkidle2' });

Form Submission Redirects

Handle form submissions that redirect to success pages:

async function handleFormRedirect() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.goto('https://example.com/login');

    // Fill form fields
    await page.type('#username', 'user@example.com');
    await page.type('#password', 'password123');

    // Set up navigation promise before submission
    const navigationPromise = page.waitForNavigation({
        waitUntil: 'networkidle0'
    });

    // Submit form
    await page.click('button[type="submit"]');

    // Wait for redirect to dashboard
    await navigationPromise;

    console.log('Redirected after login:', page.url());
    await browser.close();
}

Multiple Redirects

Handle chains of redirects:

async function handleMultipleRedirects() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.goto('https://example.com');

    // Handle first redirect
    let navigationPromise = page.waitForNavigation();
    await page.click('#first-link');
    await navigationPromise;

    console.log('First redirect:', page.url());

    // Handle second redirect
    navigationPromise = page.waitForNavigation();
    await page.click('#second-link');
    await navigationPromise;

    console.log('Final destination:', page.url());
    await browser.close();
}

Error Handling and Timeouts

Add robust error handling for failed redirections:

async function handleRedirectWithErrorHandling() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    try {
        await page.goto('https://example.com');

        const navigationPromise = page.waitForNavigation({
            waitUntil: 'networkidle0',
            timeout: 10000 // 10 second timeout
        });

        await page.click('#redirect-link');
        await navigationPromise;

        console.log('Successfully redirected to:', page.url());

    } catch (error) {
        if (error.name === 'TimeoutError') {
            console.log('Navigation timeout - page may not have redirected');
        } else {
            console.log('Navigation error:', error.message);
        }
    } finally {
        await browser.close();
    }
}

Detecting Redirects

Monitor redirect responses to understand the redirect chain:

async function trackRedirects() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    // Track response events
    page.on('response', response => {
        if (response.status() >= 300 && response.status() < 400) {
            console.log(`Redirect: ${response.status()} from ${response.url()}`);
        }
    });

    await page.goto('https://httpbin.org/redirect/3');
    console.log('Final URL:', page.url());

    await browser.close();
}

Promise.all Pattern

Handle simultaneous navigation and actions:

async function simultaneousNavigationAndAction() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.goto('https://example.com');

    // Execute click and wait for navigation simultaneously
    const [response] = await Promise.all([
        page.waitForNavigation({ waitUntil: 'networkidle0' }),
        page.click('#redirect-button')
    ]);

    console.log('Navigation completed with status:', response.status());
    await browser.close();
}

Best Practices

  1. Always set up waitForNavigation() before triggering the redirect
  2. Choose appropriate waitUntil values based on page complexity
  3. Use timeouts to prevent hanging on failed redirections
  4. Handle errors gracefully with try-catch blocks
  5. Monitor network activity for complex redirect chains

Common Pitfalls

  • Calling waitForNavigation() after the action that triggers navigation
  • Using wrong waitUntil option for your use case
  • Not handling timeout errors
  • Assuming all redirects complete successfully

Understanding these patterns will help you build robust Puppeteer scripts that handle various redirection scenarios effectively.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon