Table of contents

How do I configure a proxy with Symfony Panther for web scraping?

Configuring a proxy with Symfony Panther allows you to route your web scraping requests through an intermediate server, providing anonymity, geographic location switching, and rate limiting bypass capabilities.

Installation

First, ensure Symfony Panther is installed in your project:

composer require symfony/panther

Basic Proxy Configuration

HTTP/HTTPS Proxy Setup

Configure proxy settings through WebDriver capabilities when creating the Panther client:

<?php

use Symfony\Component\Panther\Client;
use Facebook\WebDriver\WebDriverCapabilityType;

// Basic proxy configuration
$capabilities = [
    WebDriverCapabilityType::PROXY => [
        'proxyType' => 'manual',
        'httpProxy' => 'proxy.example.com:8080',
        'sslProxy' => 'proxy.example.com:8080',
    ],
];

$client = Client::createChromeClient(null, null, [], null, [
    'capabilities' => $capabilities,
]);

// Perform web scraping
$crawler = $client->request('GET', 'https://example.com');
$title = $crawler->filter('title')->text();

$client->quit();

SOCKS Proxy Configuration

For SOCKS proxies, use the socksProxy capability:

<?php

use Symfony\Component\Panther\Client;
use Facebook\WebDriver\WebDriverCapabilityType;

$capabilities = [
    WebDriverCapabilityType::PROXY => [
        'proxyType' => 'manual',
        'socksProxy' => 'socks5-proxy.example.com:1080',
        'socksVersion' => 5, // SOCKS version (4 or 5)
    ],
];

$client = Client::createChromeClient(null, null, [], null, [
    'capabilities' => $capabilities,
]);

Authenticated Proxy Configuration

For proxies requiring authentication, include credentials in the proxy URL:

<?php

use Symfony\Component\Panther\Client;
use Facebook\WebDriver\WebDriverCapabilityType;

$proxyHost = 'authenticated-proxy.example.com';
$proxyPort = 8080;
$username = 'your_username';
$password = 'your_password';

$capabilities = [
    WebDriverCapabilityType::PROXY => [
        'proxyType' => 'manual',
        'httpProxy' => "$username:$password@$proxyHost:$proxyPort",
        'sslProxy' => "$username:$password@$proxyHost:$proxyPort",
    ],
];

$client = Client::createChromeClient(null, null, [], null, [
    'capabilities' => $capabilities,
]);

Advanced Configuration Examples

Complete Web Scraping Class with Proxy

<?php

use Symfony\Component\Panther\Client;
use Facebook\WebDriver\WebDriverCapabilityType;

class ProxyWebScraper
{
    private $client;

    public function __construct(string $proxyHost, int $proxyPort, string $username = null, string $password = null)
    {
        $proxyUrl = $proxyHost . ':' . $proxyPort;

        if ($username && $password) {
            $proxyUrl = "$username:$password@$proxyUrl";
        }

        $capabilities = [
            WebDriverCapabilityType::PROXY => [
                'proxyType' => 'manual',
                'httpProxy' => $proxyUrl,
                'sslProxy' => $proxyUrl,
            ],
        ];

        // Additional Chrome options for better scraping
        $chromeOptions = [
            '--disable-blink-features=AutomationControlled',
            '--disable-web-security',
            '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
        ];

        $this->client = Client::createChromeClient(null, $chromeOptions, [], null, [
            'capabilities' => $capabilities,
        ]);
    }

    public function scrapeUrl(string $url): array
    {
        $crawler = $this->client->request('GET', $url);

        return [
            'title' => $crawler->filter('title')->count() > 0 ? $crawler->filter('title')->text() : null,
            'description' => $crawler->filter('meta[name="description"]')->count() > 0 
                ? $crawler->filter('meta[name="description"]')->attr('content') : null,
            'h1' => $crawler->filter('h1')->count() > 0 ? $crawler->filter('h1')->text() : null,
        ];
    }

    public function close(): void
    {
        $this->client->quit();
    }
}

// Usage example
$scraper = new ProxyWebScraper('proxy.example.com', 8080, 'username', 'password');
$data = $scraper->scrapeUrl('https://example.com');
$scraper->close();

Firefox WebDriver with Proxy

<?php

use Symfony\Component\Panther\Client;
use Facebook\WebDriver\WebDriverCapabilityType;

$capabilities = [
    WebDriverCapabilityType::PROXY => [
        'proxyType' => 'manual',
        'httpProxy' => 'proxy.example.com:8080',
        'sslProxy' => 'proxy.example.com:8080',
    ],
];

$client = Client::createFirefoxClient(null, null, [], null, [
    'capabilities' => $capabilities,
]);

Proxy Types and Configuration Options

| Proxy Type | Configuration Key | Example Value | |------------|------------------|---------------| | HTTP | httpProxy | proxy.example.com:8080 | | HTTPS | sslProxy | proxy.example.com:8080 | | SOCKS4 | socksProxy + socksVersion: 4 | socks.example.com:1080 | | SOCKS5 | socksProxy + socksVersion: 5 | socks.example.com:1080 | | FTP | ftpProxy | ftp-proxy.example.com:8080 |

Error Handling and Troubleshooting

<?php

try {
    $client = Client::createChromeClient(null, null, [], null, [
        'capabilities' => $capabilities,
    ]);

    $crawler = $client->request('GET', 'https://httpbin.org/ip');
    $response = $crawler->text();

    // Verify proxy is working by checking IP
    $ipData = json_decode($response, true);
    echo "Current IP: " . $ipData['origin'] . "\n";

} catch (\Exception $e) {
    echo "Proxy configuration error: " . $e->getMessage() . "\n";
} finally {
    if (isset($client)) {
        $client->quit();
    }
}

Best Practices

  1. Always close the client after use to free resources
  2. Test proxy connectivity before starting large scraping operations
  3. Rotate proxies for high-volume scraping to avoid detection
  4. Handle proxy failures gracefully with fallback mechanisms
  5. Respect rate limits even when using proxies
  6. Verify proxy anonymity by checking your IP address

Legal Considerations

When using proxies for web scraping: - Check the target website's robots.txt and terms of service - Ensure proxy usage complies with your proxy provider's terms - Respect website rate limits and implement delays between requests - Consider the legal implications of circumventing geographic restrictions

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon