How do I set a referer or user-agent for Guzzle requests?

Setting custom headers like Referer and User-Agent in Guzzle HTTP requests is essential for web scraping and API integration. These headers help identify your application and can be required by many websites.

Setting Headers Per Request

To set headers for individual requests, pass them in the options array:

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;

$client = new Client();

$response = $client->request('GET', 'https://httpbin.org/headers', [
    'headers' => [
        'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
        'Referer' => 'https://www.google.com'
    ]
]);

echo $response->getBody();

Setting Default Headers for All Requests

Configure headers globally when creating the client:

<?php
$client = new Client([
    'base_uri' => 'https://api.example.com',
    'headers' => [
        'User-Agent' => 'MyApp/2.0 (https://myapp.com)',
        'Referer' => 'https://myapp.com'
    ]
]);

// These headers will be included in all requests
$response = $client->get('/users');
$response2 = $client->post('/data', ['json' => ['key' => 'value']]);

Common User-Agent Examples

Different User-Agent strings for various use cases:

<?php
// Browser-like User-Agent
$browserUA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36';

// Custom application User-Agent
$appUA = 'MyWebScraper/1.0 (+https://myapp.com/bot)';

// Mobile User-Agent
$mobileUA = 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1';

$client = new Client();
$response = $client->get('https://example.com', [
    'headers' => ['User-Agent' => $browserUA]
]);

Overriding Default Headers

You can override default headers for specific requests:

<?php
$client = new Client([
    'headers' => [
        'User-Agent' => 'DefaultAgent/1.0'
    ]
]);

// This request will use a different User-Agent
$response = $client->get('https://example.com', [
    'headers' => [
        'User-Agent' => 'SpecialAgent/2.0',
        'Referer' => 'https://special-referrer.com'
    ]
]);

Multiple HTTP Methods

Setting headers works with all HTTP methods:

<?php
$headers = [
    'User-Agent' => 'MyApp/1.0',
    'Referer' => 'https://myapp.com'
];

// GET request
$getResponse = $client->get('https://example.com', ['headers' => $headers]);

// POST request
$postResponse = $client->post('https://example.com/api', [
    'headers' => $headers,
    'json' => ['data' => 'value']
]);

// PUT request
$putResponse = $client->put('https://example.com/resource/1', [
    'headers' => $headers,
    'json' => ['updated' => 'data']
]);

Best Practices

• Use realistic User-Agent strings: Avoid generic ones that might be blocked
• Respect robots.txt: Check website policies before scraping
• Rotate User-Agents: For large-scale scraping, consider rotating between different agents
• Include contact information: Add your website/email in custom User-Agent strings
• Set appropriate Referer: Use realistic referrer URLs that match your use case

Common Issues and Solutions

Issue: Some websites block requests with missing or suspicious User-Agent headers.

Solution: Use browser-like User-Agent strings and ensure the Referer header matches expected patterns.

Issue: Default headers not being applied to all requests.

Solution: Verify the client configuration and ensure you're using the same client instance for all requests.