Table of contents

How to Use Puppeteer with Proxy Servers

Using proxy servers with Puppeteer is essential for web scraping projects that require IP rotation, geographic location changes, or bypassing rate limits. This comprehensive guide covers everything you need to know about integrating proxy servers with Puppeteer effectively.

Why Use Proxy Servers with Puppeteer?

Proxy servers serve several critical purposes in web scraping:

  • IP Rotation: Avoid getting blocked by distributing requests across multiple IP addresses
  • Geographic Targeting: Access region-specific content by using proxies from different locations
  • Rate Limit Bypass: Distribute requests to avoid hitting API or server rate limits
  • Anonymity: Hide your real IP address and location
  • Load Distribution: Spread traffic across multiple proxy servers for better performance

Basic Proxy Configuration

HTTP Proxy Setup

The most straightforward way to configure a proxy with Puppeteer is through the browser launch options:

const puppeteer = require('puppeteer');

async function launchWithProxy() {
  const browser = await puppeteer.launch({
    args: [
      '--proxy-server=http://proxy-server.com:8080'
    ],
    headless: true
  });

  const page = await browser.newPage();
  await page.goto('https://httpbin.org/ip');
  const content = await page.content();
  console.log(content);

  await browser.close();
}

launchWithProxy();

SOCKS Proxy Configuration

For SOCKS proxies, use the following format:

const browser = await puppeteer.launch({
  args: [
    '--proxy-server=socks5://proxy-server.com:1080'
  ],
  headless: true
});

Proxy Authentication

Many proxy services require authentication. Here's how to handle authenticated proxies:

Basic Authentication

const puppeteer = require('puppeteer');

async function launchWithAuthenticatedProxy() {
  const browser = await puppeteer.launch({
    args: [
      '--proxy-server=http://proxy-server.com:8080'
    ],
    headless: true
  });

  const page = await browser.newPage();

  // Set proxy authentication
  await page.authenticate({
    username: 'your-username',
    password: 'your-password'
  });

  await page.goto('https://httpbin.org/ip');
  const content = await page.content();
  console.log(content);

  await browser.close();
}

launchWithAuthenticatedProxy();

Advanced Authentication with Request Interception

For more complex authentication scenarios, use request interception:

const puppeteer = require('puppeteer');

async function advancedProxyAuth() {
  const browser = await puppeteer.launch({
    args: [
      '--proxy-server=http://proxy-server.com:8080'
    ],
    headless: true
  });

  const page = await browser.newPage();

  // Enable request interception
  await page.setRequestInterception(true);

  page.on('request', (request) => {
    const headers = {
      ...request.headers(),
      'Proxy-Authorization': 'Basic ' + Buffer.from('username:password').toString('base64')
    };

    request.continue({ headers });
  });

  await page.goto('https://httpbin.org/ip');
  const content = await page.content();
  console.log(content);

  await browser.close();
}

advancedProxyAuth();

Proxy Rotation Implementation

Implementing proxy rotation helps distribute requests and avoid detection:

const puppeteer = require('puppeteer');

class ProxyRotator {
  constructor(proxies) {
    this.proxies = proxies;
    this.currentIndex = 0;
  }

  getNextProxy() {
    const proxy = this.proxies[this.currentIndex];
    this.currentIndex = (this.currentIndex + 1) % this.proxies.length;
    return proxy;
  }

  async createBrowserWithProxy() {
    const proxy = this.getNextProxy();

    const browser = await puppeteer.launch({
      args: [
        `--proxy-server=${proxy.protocol}://${proxy.host}:${proxy.port}`
      ],
      headless: true
    });

    const page = await browser.newPage();

    if (proxy.username && proxy.password) {
      await page.authenticate({
        username: proxy.username,
        password: proxy.password
      });
    }

    return { browser, page };
  }
}

// Usage example
const proxies = [
  { protocol: 'http', host: 'proxy1.com', port: 8080, username: 'user1', password: 'pass1' },
  { protocol: 'http', host: 'proxy2.com', port: 8080, username: 'user2', password: 'pass2' },
  { protocol: 'socks5', host: 'proxy3.com', port: 1080, username: 'user3', password: 'pass3' }
];

const rotator = new ProxyRotator(proxies);

async function scrapeWithRotation() {
  const urls = ['https://httpbin.org/ip', 'https://httpbin.org/headers', 'https://httpbin.org/user-agent'];

  for (const url of urls) {
    const { browser, page } = await rotator.createBrowserWithProxy();

    try {
      await page.goto(url);
      const content = await page.content();
      console.log(`Content from ${url}:`, content.substring(0, 200));
    } catch (error) {
      console.error(`Error accessing ${url}:`, error.message);
    } finally {
      await browser.close();
    }
  }
}

scrapeWithRotation();

Handling Proxy Failures

Robust proxy handling includes error management and fallback mechanisms:

const puppeteer = require('puppeteer');

class RobustProxyManager {
  constructor(proxies) {
    this.proxies = proxies;
    this.failedProxies = new Set();
  }

  getWorkingProxy() {
    const workingProxies = this.proxies.filter(proxy => 
      !this.failedProxies.has(`${proxy.host}:${proxy.port}`)
    );

    if (workingProxies.length === 0) {
      throw new Error('No working proxies available');
    }

    return workingProxies[Math.floor(Math.random() * workingProxies.length)];
  }

  markProxyAsFailed(proxy) {
    this.failedProxies.add(`${proxy.host}:${proxy.port}`);
  }

  async createBrowserWithRetry(maxRetries = 3) {
    for (let attempt = 0; attempt < maxRetries; attempt++) {
      try {
        const proxy = this.getWorkingProxy();

        const browser = await puppeteer.launch({
          args: [
            `--proxy-server=${proxy.protocol}://${proxy.host}:${proxy.port}`,
            '--no-sandbox',
            '--disable-setuid-sandbox'
          ],
          headless: true
        });

        const page = await browser.newPage();

        if (proxy.username && proxy.password) {
          await page.authenticate({
            username: proxy.username,
            password: proxy.password
          });
        }

        // Test the proxy
        await page.goto('https://httpbin.org/ip', { timeout: 10000 });

        return { browser, page, proxy };
      } catch (error) {
        console.error(`Proxy attempt ${attempt + 1} failed:`, error.message);
        if (attempt === maxRetries - 1) {
          throw error;
        }
      }
    }
  }
}

Testing Proxy Connections

Always test your proxy connections before using them in production:

async function testProxy(proxy) {
  const browser = await puppeteer.launch({
    args: [
      `--proxy-server=${proxy.protocol}://${proxy.host}:${proxy.port}`
    ],
    headless: true
  });

  const page = await browser.newPage();

  try {
    if (proxy.username && proxy.password) {
      await page.authenticate({
        username: proxy.username,
        password: proxy.password
      });
    }

    // Test IP detection
    await page.goto('https://httpbin.org/ip', { timeout: 10000 });
    const ipResponse = await page.evaluate(() => {
      return JSON.parse(document.body.innerText);
    });

    console.log(`Proxy ${proxy.host}:${proxy.port} - IP: ${ipResponse.origin}`);

    // Test speed
    const startTime = Date.now();
    await page.goto('https://httpbin.org/delay/1');
    const endTime = Date.now();

    console.log(`Proxy ${proxy.host}:${proxy.port} - Response time: ${endTime - startTime}ms`);

    return { success: true, ip: ipResponse.origin, responseTime: endTime - startTime };
  } catch (error) {
    console.error(`Proxy ${proxy.host}:${proxy.port} failed:`, error.message);
    return { success: false, error: error.message };
  } finally {
    await browser.close();
  }
}

Python Implementation with Pyppeteer

For Python developers, here's how to use proxies with Pyppeteer:

import asyncio
from pyppeteer import launch

async def launch_with_proxy():
    browser = await launch({
        'args': [
            '--proxy-server=http://proxy-server.com:8080'
        ],
        'headless': True
    })

    page = await browser.newPage()

    # Set proxy authentication
    await page.authenticate({
        'username': 'your-username',
        'password': 'your-password'
    })

    await page.goto('https://httpbin.org/ip')
    content = await page.content()
    print(content)

    await browser.close()

asyncio.run(launch_with_proxy())

Best Practices for Proxy Usage

1. User Agent Rotation

Combine proxy rotation with user agent rotation for better anonymity:

const userAgents = [
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
  'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
];

async function scrapeWithUserAgentRotation() {
  const { browser, page } = await rotator.createBrowserWithProxy();

  const randomUserAgent = userAgents[Math.floor(Math.random() * userAgents.length)];
  await page.setUserAgent(randomUserAgent);

  await page.goto('https://httpbin.org/headers');
  const content = await page.content();
  console.log(content);

  await browser.close();
}

2. Request Delays

Add delays between requests to avoid overwhelming servers:

async function scrapeWithDelay(urls, minDelay = 1000, maxDelay = 3000) {
  for (const url of urls) {
    const { browser, page } = await rotator.createBrowserWithProxy();

    try {
      await page.goto(url);
      const content = await page.content();
      console.log(`Scraped: ${url}`);
    } catch (error) {
      console.error(`Error scraping ${url}:`, error.message);
    } finally {
      await browser.close();
    }

    // Random delay between requests
    const delay = Math.random() * (maxDelay - minDelay) + minDelay;
    await new Promise(resolve => setTimeout(resolve, delay));
  }
}

3. Monitor Proxy Health

class ProxyHealthMonitor {
  constructor(proxies) {
    this.proxies = proxies;
    this.healthStats = new Map();
  }

  async checkProxyHealth(proxy) {
    const startTime = Date.now();

    try {
      const browser = await puppeteer.launch({
        args: [`--proxy-server=${proxy.protocol}://${proxy.host}:${proxy.port}`],
        headless: true
      });

      const page = await browser.newPage();

      if (proxy.username && proxy.password) {
        await page.authenticate({
          username: proxy.username,
          password: proxy.password
        });
      }

      await page.goto('https://httpbin.org/ip', { timeout: 10000 });
      const responseTime = Date.now() - startTime;

      await browser.close();

      this.healthStats.set(proxy.host, {
        isHealthy: true,
        responseTime,
        lastChecked: new Date()
      });

      return true;
    } catch (error) {
      this.healthStats.set(proxy.host, {
        isHealthy: false,
        error: error.message,
        lastChecked: new Date()
      });

      return false;
    }
  }

  getHealthyProxies() {
    return this.proxies.filter(proxy => {
      const health = this.healthStats.get(proxy.host);
      return health && health.isHealthy;
    });
  }
}

Troubleshooting Common Issues

Connection Timeouts

const browser = await puppeteer.launch({
  args: [
    '--proxy-server=http://proxy-server.com:8080',
    '--proxy-bypass-list=localhost,127.0.0.1',
    '--disable-web-security'
  ],
  headless: true
});

const page = await browser.newPage();
page.setDefaultTimeout(30000); // 30 second timeout

DNS Resolution Issues

const browser = await puppeteer.launch({
  args: [
    '--proxy-server=http://proxy-server.com:8080',
    '--host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE proxy-server.com"'
  ],
  headless: true
});

Handling SSL Certificate Issues

const browser = await puppeteer.launch({
  args: [
    '--proxy-server=http://proxy-server.com:8080',
    '--ignore-certificate-errors',
    '--ignore-ssl-errors',
    '--ignore-certificate-errors-spki-list'
  ],
  headless: true
});

Performance Optimization

Connection Pooling

class ProxyConnectionPool {
  constructor(proxies, maxConnections = 10) {
    this.proxies = proxies;
    this.maxConnections = maxConnections;
    this.activeConnections = new Map();
    this.connectionQueue = [];
  }

  async getConnection() {
    if (this.activeConnections.size < this.maxConnections) {
      const proxy = this.getRandomProxy();
      const connection = await this.createConnection(proxy);
      this.activeConnections.set(connection.id, connection);
      return connection;
    }

    return new Promise((resolve) => {
      this.connectionQueue.push(resolve);
    });
  }

  async createConnection(proxy) {
    const browser = await puppeteer.launch({
      args: [`--proxy-server=${proxy.protocol}://${proxy.host}:${proxy.port}`],
      headless: true
    });

    const page = await browser.newPage();

    if (proxy.username && proxy.password) {
      await page.authenticate({
        username: proxy.username,
        password: proxy.password
      });
    }

    return {
      id: Math.random().toString(36).substr(2, 9),
      browser,
      page,
      proxy
    };
  }

  releaseConnection(connection) {
    this.activeConnections.delete(connection.id);

    if (this.connectionQueue.length > 0) {
      const resolve = this.connectionQueue.shift();
      this.getConnection().then(resolve);
    }
  }

  getRandomProxy() {
    return this.proxies[Math.floor(Math.random() * this.proxies.length)];
  }
}

Integration with WebScraping.AI

For production web scraping needs, consider using WebScraping.AI's proxy-enabled API instead of managing proxies manually. The service provides automatic proxy rotation, CAPTCHA solving, and optimized browser automation for better performance with built-in proxy support.

Using WebScraping.AI API

const fetch = require('node-fetch');

async function scrapeWithWebScrapingAI() {
  const response = await fetch('https://api.webscraping.ai/html', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer YOUR_API_KEY'
    },
    body: JSON.stringify({
      url: 'https://example.com',
      proxy: 'residential',
      country: 'US',
      device: 'desktop',
      js: true
    })
  });

  const data = await response.json();
  console.log(data.html);
}

Conclusion

Using Puppeteer with proxy servers requires careful configuration and error handling. Key takeaways include:

  • Always test proxy connections before production use
  • Implement proper rotation and fallback mechanisms
  • Combine proxy rotation with user agent rotation for better anonymity
  • Add appropriate delays between requests
  • Handle authentication properly for proxy services
  • Monitor proxy performance and replace failed proxies
  • Use connection pooling for better performance
  • Consider managed solutions like WebScraping.AI for production use

By following these practices, you can create robust web scraping solutions that effectively utilize proxy servers while maintaining reliability and avoiding detection.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How to handle infinite scroll pages in Puppeteer?
How to extract text content from specific elements?
How to handle date and time pickers in Puppeteer?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon