How do I use the 'waitFor' function in Puppeteer?

Puppeteer provides several wait functions to pause script execution until specific conditions are met. These functions are essential for reliable web scraping and automated testing, ensuring elements are loaded before interaction.

Important Note: Deprecated page.waitFor()

The generic page.waitFor() method has been deprecated in modern Puppeteer versions. Use the specific wait methods instead:

Available Wait Functions

1. page.waitForSelector(selector, options)

Waits for a CSS selector to appear in the DOM. This is the most commonly used wait function.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  try {
    await page.goto('https://example.com');

    // Wait for element to appear
    await page.waitForSelector('div.content', { 
      visible: true,    // Wait until element is visible
      timeout: 5000     // 5 second timeout
    });

    // Element is now ready for interaction
    const element = await page.$('div.content');
    console.log(await element.textContent());

  } catch (error) {
    console.error('Element not found:', error.message);
  } finally {
    await browser.close();
  }
})();

Options: - visible: Wait until element is visible (default: false) - hidden: Wait until element is hidden - timeout: Maximum wait time in milliseconds (default: 30000)

2. page.waitForXPath(xpath, options)

Waits for an element matching an XPath expression:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  try {
    await page.goto('https://example.com');

    // Wait for XPath element
    const element = await page.waitForXPath('//button[contains(text(), "Submit")]', {
      visible: true,
      timeout: 10000
    });

    // Click the found element
    await element.click();

  } catch (error) {
    console.error('XPath element not found:', error.message);
  } finally {
    await browser.close();
  }
})();

3. page.waitForFunction(pageFunction, options, ...args)

Waits for a JavaScript function to return a truthy value:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  try {
    await page.goto('https://example.com');

    // Wait for custom condition
    await page.waitForFunction(
      () => {
        const loadingSpinner = document.querySelector('.loading');
        return !loadingSpinner || loadingSpinner.style.display === 'none';
      },
      { timeout: 15000, polling: 'raf' }
    );

    // Wait for specific text content
    await page.waitForFunction(
      (text) => document.body.innerText.includes(text),
      { timeout: 10000 },
      'Data loaded successfully'
    );

    console.log('Page is fully loaded');

  } catch (error) {
    console.error('Condition not met:', error.message);
  } finally {
    await browser.close();
  }
})();

Options: - polling: How often to poll ('raf', 'mutation', or number in ms) - timeout: Maximum wait time in milliseconds

4. page.waitForTimeout(timeout)

Waits for a specific amount of time. Use sparingly as it's less reliable than condition-based waits:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://example.com');

  // Wait for 2 seconds (not recommended for production)
  await page.waitForTimeout(2000);

  // Better approach: wait for specific condition
  await page.waitForSelector('.dynamic-content');

  await browser.close();
})();

Advanced Wait Strategies

Waiting for Network Activity

// Wait for page to be idle (no network requests for 500ms)
await page.goto('https://example.com', { 
  waitUntil: 'networkidle0' 
});

// Wait for navigation to complete
await Promise.all([
  page.waitForNavigation({ waitUntil: 'networkidle2' }),
  page.click('a[href="/next-page"]')
]);

Multiple Wait Conditions

// Wait for either element to appear
const elementHandle = await Promise.race([
  page.waitForSelector('.success-message'),
  page.waitForSelector('.error-message')
]);

// Wait for element to disappear
await page.waitForSelector('.loading-spinner', { 
  hidden: true 
});

Error Handling and Timeouts

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  try {
    await page.goto('https://example.com');

    // Set custom timeout for specific wait
    await page.waitForSelector('.slow-loading-element', {
      timeout: 30000
    });

  } catch (error) {
    if (error.name === 'TimeoutError') {
      console.log('Element did not appear within timeout period');
    } else {
      console.error('Unexpected error:', error.message);
    }
  } finally {
    await browser.close();
  }
})();

Best Practices

1. Use specific wait methods instead of deprecated page.waitFor()
2. Set appropriate timeouts based on expected load times
3. Wait for visibility when you need to interact with elements
4. Avoid waitForTimeout() unless absolutely necessary
5. Handle TimeoutError exceptions gracefully
6. Use waitForFunction() for complex custom conditions
7. Combine with waitUntil options in page.goto() for better reliability

These wait functions ensure your Puppeteer scripts are robust and handle dynamic content loading effectively.