Table of contents

How do I set custom HTTP headers when making requests with Symfony Panther?

Setting custom HTTP headers in Symfony Panther is essential for web scraping and browser testing scenarios where you need to authenticate, specify user agents, or send custom metadata with your requests.

Basic Header Configuration

Symfony Panther uses the setServerParameters() method to configure HTTP headers. Headers must be prefixed with HTTP_ and use uppercase names with underscores instead of hyphens.

Installation

First, install Symfony Panther via Composer:

composer require symfony/panther

Basic Example

use Symfony\Component\Panther\PantherTestCase;

class CustomHeadersTest extends PantherTestCase
{
    public function testWithCustomHeaders()
    {
        $client = static::createPantherClient();

        // Set custom HTTP headers
        $client->setServerParameters([
            'HTTP_USER_AGENT' => 'CustomBot/1.0',
            'HTTP_AUTHORIZATION' => 'Bearer your-api-token',
            'HTTP_ACCEPT' => 'application/json',
            'HTTP_CONTENT_TYPE' => 'application/json'
        ]);

        $crawler = $client->request('GET', 'https://api.example.com/data');

        // Your test assertions here
        $this->assertSelectorTextContains('body', 'Expected content');
    }
}

Common Header Examples

Authentication Headers

// Bearer token authentication
$client->setServerParameters([
    'HTTP_AUTHORIZATION' => 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...'
]);

// Basic authentication
$credentials = base64_encode('username:password');
$client->setServerParameters([
    'HTTP_AUTHORIZATION' => 'Basic ' . $credentials
]);

// API key authentication
$client->setServerParameters([
    'HTTP_X_API_KEY' => 'your-api-key-here'
]);

User Agent Customization

$client->setServerParameters([
    'HTTP_USER_AGENT' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
]);

// Mobile user agent
$client->setServerParameters([
    'HTTP_USER_AGENT' => 'Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X) AppleWebKit/605.1.15'
]);

Content and Cache Headers

$client->setServerParameters([
    'HTTP_ACCEPT' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'HTTP_ACCEPT_LANGUAGE' => 'en-US,en;q=0.5',
    'HTTP_ACCEPT_ENCODING' => 'gzip, deflate',
    'HTTP_CACHE_CONTROL' => 'no-cache',
    'HTTP_PRAGMA' => 'no-cache'
]);

Advanced Usage

Dynamic Header Updates

class DynamicHeadersTest extends PantherTestCase
{
    public function testDynamicHeaders()
    {
        $client = static::createPantherClient();

        // Initial headers
        $client->setServerParameters([
            'HTTP_USER_AGENT' => 'InitialBot/1.0'
        ]);

        $client->request('GET', 'https://example.com/page1');

        // Update headers for subsequent requests
        $client->setServerParameters([
            'HTTP_USER_AGENT' => 'UpdatedBot/2.0',
            'HTTP_X_SESSION_TOKEN' => 'session-token-123'
        ]);

        $client->request('GET', 'https://example.com/page2');
    }
}

Header Validation Testing

public function testHeaderValidation()
{
    $client = static::createPantherClient();

    $client->setServerParameters([
        'HTTP_X_CUSTOM_HEADER' => 'test-value'
    ]);

    $crawler = $client->request('GET', 'https://httpbin.org/headers');

    // Verify the header was sent correctly
    $response = $crawler->text();
    $this->assertStringContains('X-Custom-Header', $response);
    $this->assertStringContains('test-value', $response);
}

Header Naming Conventions

When setting headers in Symfony Panther, follow these conversion rules:

| Original Header | Panther Format | |----------------|----------------| | User-Agent | HTTP_USER_AGENT | | Content-Type | HTTP_CONTENT_TYPE | | Authorization | HTTP_AUTHORIZATION | | X-API-Key | HTTP_X_API_KEY | | Accept-Language | HTTP_ACCEPT_LANGUAGE |

Non-Test Environment Usage

For standalone applications (not extending PantherTestCase):

use Symfony\Component\Panther\Client;

// Create client without test case
$client = Client::createChromeClient();

$client->setServerParameters([
    'HTTP_USER_AGENT' => 'StandaloneApp/1.0',
    'HTTP_AUTHORIZATION' => 'Bearer token-here'
]);

$crawler = $client->request('GET', 'https://api.example.com');

Best Practices

  1. Set headers before making requests - Headers must be configured before calling request()
  2. Use realistic user agents - Avoid detection by using common browser user agents
  3. Handle authentication properly - Store sensitive tokens securely, not in code
  4. Test header transmission - Use services like httpbin.org to verify headers are sent correctly
  5. Consider rate limiting - Add delays between requests when scraping to avoid being blocked

Alternatives

For simpler HTTP requests without browser overhead, consider:

  • Symfony HttpClient: Lightweight HTTP client with easier header management
  • Guzzle: Popular PHP HTTP client library
  • cURL: Direct PHP cURL extension for basic requests

Symfony Panther is best when you need full browser functionality with JavaScript execution.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon