Table of contents

Can I customize the User-Agent string in urllib3 requests?

Yes, you can easily customize the User-Agent string in urllib3 requests by setting custom headers. The User-Agent header is crucial for web scraping as it identifies your client to the server and can help avoid blocks or access restrictions that some websites impose on default urllib3 user agents.

Understanding User-Agent Headers

The User-Agent header is an HTTP header that contains information about the client making the request. Many websites use this header to:

  • Identify different browsers and devices
  • Serve different content based on the client type
  • Block automated requests with suspicious or default user agents
  • Gather analytics about their users

urllib3's default User-Agent string typically looks like urllib3/1.26.x which can easily be identified as an automated request.

Setting Custom User-Agent in urllib3

Basic User-Agent Customization

Here's how to set a custom User-Agent header in urllib3:

import urllib3

# Create a PoolManager instance
http = urllib3.PoolManager()

# Define custom headers with User-Agent
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}

# Make a request with custom User-Agent
response = http.request('GET', 'https://httpbin.org/headers', headers=headers)
print(response.data.decode('utf-8'))

Using Realistic Browser User-Agents

For web scraping, it's often beneficial to use realistic browser User-Agent strings:

import urllib3
import json

# Common browser User-Agent strings
user_agents = {
    'chrome': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
    'firefox': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/121.0',
    'safari': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15',
    'edge': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0'
}

http = urllib3.PoolManager()

# Use Chrome User-Agent
headers = {'User-Agent': user_agents['chrome']}
response = http.request('GET', 'https://httpbin.org/user-agent', headers=headers)

print("Server received User-Agent:")
print(json.loads(response.data.decode('utf-8'))['user-agent'])

Setting Default Headers for All Requests

You can set default headers that will be used for all requests made with a PoolManager:

import urllib3

# Create PoolManager with default headers
http = urllib3.PoolManager(
    headers={
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
    }
)

# All requests will now use the custom User-Agent
response1 = http.request('GET', 'https://httpbin.org/user-agent')
response2 = http.request('GET', 'https://httpbin.org/headers')

print("Response 1:", response1.data.decode('utf-8'))
print("Response 2:", response2.data.decode('utf-8'))

Advanced User-Agent Strategies

Rotating User-Agents

For large-scale scraping, rotating User-Agent strings can help avoid detection:

import urllib3
import random
import time

class UserAgentRotator:
    def __init__(self):
        self.user_agents = [
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/121.0',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15',
            'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
        ]
        self.http = urllib3.PoolManager()

    def get_random_user_agent(self):
        return random.choice(self.user_agents)

    def make_request(self, url, method='GET'):
        headers = {'User-Agent': self.get_random_user_agent()}
        return self.http.request(method, url, headers=headers)

# Usage example
scraper = UserAgentRotator()

urls = [
    'https://httpbin.org/user-agent',
    'https://httpbin.org/headers',
    'https://httpbin.org/ip'
]

for url in urls:
    response = scraper.make_request(url)
    print(f"Request to {url}")
    print(f"Status: {response.status}")
    print(f"Response: {response.data.decode('utf-8')[:100]}...")
    print("-" * 50)
    time.sleep(1)  # Be respectful with delays

Mobile User-Agents

Sometimes you need to simulate mobile devices:

import urllib3

mobile_user_agents = {
    'iphone': 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Mobile/15E148 Safari/604.1',
    'android': 'Mozilla/5.0 (Linux; Android 13; SM-G991B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Mobile Safari/537.36',
    'ipad': 'Mozilla/5.0 (iPad; CPU OS 17_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Mobile/15E148 Safari/604.1'
}

http = urllib3.PoolManager()

# Simulate iPhone request
headers = {'User-Agent': mobile_user_agents['iphone']}
response = http.request('GET', 'https://httpbin.org/headers', headers=headers)

print("Mobile request headers:")
print(response.data.decode('utf-8'))

Combining with Other Headers

User-Agent works best when combined with other realistic browser headers:

import urllib3

def create_realistic_headers(user_agent_type='chrome'):
    user_agents = {
        'chrome': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
        'firefox': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/121.0'
    }

    base_headers = {
        'User-Agent': user_agents.get(user_agent_type, user_agents['chrome']),
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        'Connection': 'keep-alive',
        'Upgrade-Insecure-Requests': '1',
    }

    if user_agent_type == 'chrome':
        base_headers.update({
            'sec-ch-ua': '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"',
            'sec-ch-ua-mobile': '?0',
            'sec-ch-ua-platform': '"Windows"',
            'Sec-Fetch-Dest': 'document',
            'Sec-Fetch-Mode': 'navigate',
            'Sec-Fetch-Site': 'none',
            'Sec-Fetch-User': '?1'
        })

    return base_headers

http = urllib3.PoolManager()

# Create realistic Chrome headers
headers = create_realistic_headers('chrome')
response = http.request('GET', 'https://httpbin.org/headers', headers=headers)

print("Realistic browser headers:")
print(response.data.decode('utf-8'))

Testing User-Agent Configuration

You can verify that your User-Agent is being sent correctly:

import urllib3
import json

def test_user_agent(custom_ua):
    http = urllib3.PoolManager()
    headers = {'User-Agent': custom_ua}

    # Test with httpbin.org which echoes back headers
    response = http.request('GET', 'https://httpbin.org/user-agent', headers=headers)

    if response.status == 200:
        data = json.loads(response.data.decode('utf-8'))
        print(f"Sent: {custom_ua}")
        print(f"Received: {data['user-agent']}")
        print(f"Match: {custom_ua == data['user-agent']}")
    else:
        print(f"Request failed with status: {response.status}")

# Test different User-Agents
test_cases = [
    'MyCustomBot/1.0',
    'Mozilla/5.0 (compatible; CustomScraper/1.0)',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
]

for ua in test_cases:
    print(f"\nTesting User-Agent: {ua}")
    test_user_agent(ua)
    print("-" * 60)

JavaScript Example with Node.js

While urllib3 is Python-specific, here's how you might achieve similar functionality with Node.js for comparison:

const https = require('https');

// Function to make request with custom User-Agent
function makeRequestWithUserAgent(url, userAgent) {
    const options = {
        hostname: 'httpbin.org',
        path: '/user-agent',
        method: 'GET',
        headers: {
            'User-Agent': userAgent
        }
    };

    return new Promise((resolve, reject) => {
        const req = https.request(options, (res) => {
            let data = '';
            res.on('data', (chunk) => {
                data += chunk;
            });
            res.on('end', () => {
                resolve(JSON.parse(data));
            });
        });

        req.on('error', (error) => {
            reject(error);
        });

        req.end();
    });
}

// Usage
const customUserAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36';
makeRequestWithUserAgent('https://httpbin.org/user-agent', customUserAgent)
    .then(response => {
        console.log('Server received User-Agent:', response['user-agent']);
    })
    .catch(error => {
        console.error('Error:', error);
    });

Best Practices for User-Agent Customization

1. Use Realistic User-Agents

Always use realistic, current browser User-Agent strings rather than obviously fake ones:

# Good - realistic browser User-Agent
good_ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'

# Avoid - obviously fake or outdated
avoid_ua = 'MyBot/1.0' or 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)'

2. Match User-Agent with Other Headers

Ensure your other headers are consistent with your chosen User-Agent. Different browsers send different header combinations.

3. Rotate User-Agents Responsibly

If rotating User-Agents, use a reasonable pool of current, realistic options and don't change too frequently.

4. Respect robots.txt and Rate Limits

Customizing User-Agent doesn't exempt you from following website policies and being respectful with request rates.

Error Handling

Always implement proper error handling when making requests:

import urllib3
import json

def safe_request_with_custom_ua(url, user_agent):
    http = urllib3.PoolManager()
    headers = {'User-Agent': user_agent}

    try:
        response = http.request('GET', url, headers=headers, timeout=10)
        return {
            'status': response.status,
            'data': response.data.decode('utf-8'),
            'success': True
        }
    except urllib3.exceptions.TimeoutError:
        return {'success': False, 'error': 'Request timed out'}
    except urllib3.exceptions.HTTPError as e:
        return {'success': False, 'error': f'HTTP error: {str(e)}'}
    except Exception as e:
        return {'success': False, 'error': f'Unexpected error: {str(e)}'}

# Example usage
result = safe_request_with_custom_ua(
    'https://httpbin.org/user-agent',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
)

if result['success']:
    print(f"Status: {result['status']}")
    print(f"Response: {result['data']}")
else:
    print(f"Error: {result['error']}")

Command Line Examples

You can also test User-Agent behavior using curl from the command line:

# Test default curl User-Agent
curl -s https://httpbin.org/user-agent

# Test with custom User-Agent
curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
     -s https://httpbin.org/user-agent

# Compare responses
curl -H "User-Agent: urllib3/1.26.12" -s https://httpbin.org/user-agent
curl -H "User-Agent: Mozilla/5.0 (compatible; MyBot/1.0)" -s https://httpbin.org/user-agent

Integration with WebScraping.AI

When building web scraping applications, you might want to combine urllib3's flexibility with more advanced scraping capabilities. For complex scenarios involving JavaScript-heavy sites or sophisticated anti-bot measures, consider using specialized tools like Puppeteer for handling browser sessions or services that can handle authentication automatically.

Common User-Agent Patterns

Here are some commonly used User-Agent patterns for different use cases:

import urllib3

# Desktop browsers
desktop_agents = {
    'chrome_windows': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
    'chrome_mac': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
    'firefox_windows': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/121.0',
    'safari_mac': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15'
}

# Mobile browsers
mobile_agents = {
    'chrome_android': 'Mozilla/5.0 (Linux; Android 13; SM-G991B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Mobile Safari/537.36',
    'safari_ios': 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Mobile/15E148 Safari/604.1',
    'chrome_ios': 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/120.0.0.0 Mobile/15E148 Safari/604.1'
}

# Bot-friendly User-Agents (for APIs that allow bots)
bot_agents = {
    'polite_bot': 'Mozilla/5.0 (compatible; MyBot/1.0; +http://www.example.com/bot)',
    'research_bot': 'Mozilla/5.0 (compatible; ResearchBot/1.0; Educational purposes)',
    'monitoring_bot': 'Mozilla/5.0 (compatible; SiteMonitor/1.0; Health check)'
}

Conclusion

Customizing the User-Agent string in urllib3 is straightforward and essential for effective web scraping. By setting appropriate headers, rotating User-Agents when necessary, and following best practices, you can create more reliable and respectful scraping applications. Remember to always test your User-Agent configuration and implement proper error handling to ensure robust operation.

The key is to balance effectiveness with responsibility - use realistic User-Agents that help your scraping succeed while respecting website policies and server resources. Whether you're building a simple data collection script or a complex scraping system, proper User-Agent management is a fundamental skill that will improve your success rate and help maintain good relationships with the websites you're accessing.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I implement rate limiting when using urllib3?
What are the security considerations when using urllib3 for web scraping?
How do I handle HTTP status codes effectively with urllib3?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon