Table of contents

How do I perform basic HTTP authentication with Requests?

HTTP Basic Authentication is one of the most common authentication methods used by web APIs and protected web resources. The Python Requests library provides several convenient ways to handle basic authentication, making it straightforward to access protected endpoints that require username and password credentials.

Understanding HTTP Basic Authentication

HTTP Basic Authentication sends credentials (username and password) encoded in Base64 format within the Authorization header. The format follows the pattern: Authorization: Basic <base64-encoded-credentials>. While simple to implement, it's important to note that Basic Authentication should only be used over HTTPS connections to ensure credential security.

Basic Authentication Methods with Requests

Method 1: Using the auth Parameter (Recommended)

The most straightforward approach is using the auth parameter with a tuple containing your username and password:

import requests

# Basic authentication using auth parameter
response = requests.get(
    'https://api.example.com/protected-endpoint',
    auth=('username', 'password')
)

if response.status_code == 200:
    data = response.json()
    print("Authentication successful!")
    print(data)
elif response.status_code == 401:
    print("Authentication failed - check credentials")
else:
    print(f"Request failed with status code: {response.status_code}")

Method 2: Using HTTPBasicAuth Class

For more explicit control, you can use the HTTPBasicAuth class from requests.auth:

from requests.auth import HTTPBasicAuth
import requests

# Using HTTPBasicAuth class
auth = HTTPBasicAuth('username', 'password')
response = requests.get(
    'https://api.example.com/protected-endpoint',
    auth=auth
)

print(f"Status Code: {response.status_code}")
print(f"Response: {response.text}")

Method 3: Manual Header Construction

You can also manually construct the Authorization header, though this is generally not recommended as it's more error-prone:

import requests
import base64

# Manual header construction (not recommended)
username = 'your_username'
password = 'your_password'

# Create the base64 encoded credentials
credentials = f"{username}:{password}"
encoded_credentials = base64.b64encode(credentials.encode()).decode()

headers = {
    'Authorization': f'Basic {encoded_credentials}',
    'Content-Type': 'application/json'
}

response = requests.get(
    'https://api.example.com/protected-endpoint',
    headers=headers
)

Practical Examples and Use Cases

Example 1: API Access with Error Handling

Here's a robust example that includes proper error handling and response validation:

import requests
from requests.exceptions import RequestException, HTTPError

def make_authenticated_request(url, username, password):
    """
    Make an authenticated HTTP request with comprehensive error handling
    """
    try:
        response = requests.get(
            url,
            auth=(username, password),
            timeout=30,  # Add timeout for reliability
            verify=True  # Ensure SSL certificate verification
        )

        # Raise an exception for HTTP error status codes
        response.raise_for_status()

        return response.json()

    except HTTPError as http_err:
        if response.status_code == 401:
            print("Authentication failed: Invalid credentials")
        elif response.status_code == 403:
            print("Access forbidden: Insufficient permissions")
        else:
            print(f"HTTP error occurred: {http_err}")
    except RequestException as req_err:
        print(f"Request error occurred: {req_err}")
    except ValueError as json_err:
        print(f"JSON decode error: {json_err}")

    return None

# Usage example
api_data = make_authenticated_request(
    'https://api.example.com/data',
    'your_username',
    'your_password'
)

if api_data:
    print("Data retrieved successfully:", api_data)

Example 2: Session-Based Authentication

For multiple requests to the same API, using a session is more efficient:

import requests

# Create a session with persistent authentication
session = requests.Session()
session.auth = ('username', 'password')

# Add common headers
session.headers.update({
    'User-Agent': 'MyApp/1.0',
    'Accept': 'application/json'
})

# Make multiple requests using the same session
try:
    # First request
    response1 = session.get('https://api.example.com/endpoint1')
    print(f"First request status: {response1.status_code}")

    # Second request - authentication is automatically included
    response2 = session.get('https://api.example.com/endpoint2')
    print(f"Second request status: {response2.status_code}")

    # POST request with data
    post_data = {'key': 'value'}
    response3 = session.post('https://api.example.com/submit', json=post_data)
    print(f"POST request status: {response3.status_code}")

finally:
    # Always close the session
    session.close()

Security Best Practices

Environment Variables for Credentials

Never hardcode credentials in your source code. Use environment variables instead:

import os
import requests

# Get credentials from environment variables
username = os.getenv('API_USERNAME')
password = os.getenv('API_PASSWORD')

if not username or not password:
    raise ValueError("API credentials not found in environment variables")

response = requests.get(
    'https://api.example.com/protected',
    auth=(username, password)
)

Set environment variables in your shell:

export API_USERNAME="your_username"
export API_PASSWORD="your_password"
python your_script.py

Using Configuration Files

For development environments, you can use configuration files (ensure they're in .gitignore):

import json
import requests

# Load credentials from config file
with open('config.json', 'r') as config_file:
    config = json.load(config_file)

auth_credentials = (config['username'], config['password'])

response = requests.get(
    config['api_url'],
    auth=auth_credentials
)

Advanced Authentication Scenarios

Conditional Authentication

Sometimes you need to handle both authenticated and non-authenticated requests:

import requests

def api_request(url, username=None, password=None):
    """
    Make API request with optional authentication
    """
    kwargs = {'timeout': 30}

    if username and password:
        kwargs['auth'] = (username, password)

    response = requests.get(url, **kwargs)
    return response

# Public endpoint (no auth required)
public_data = api_request('https://api.example.com/public')

# Protected endpoint (auth required)
private_data = api_request(
    'https://api.example.com/private',
    username='user',
    password='pass'
)

Handling Authentication with Proxies

When working with proxies, you might need both proxy authentication and basic authentication:

import requests

# Authentication for both proxy and target server
proxies = {
    'http': 'http://proxy_user:proxy_pass@proxy.example.com:8080',
    'https': 'https://proxy_user:proxy_pass@proxy.example.com:8080'
}

response = requests.get(
    'https://api.example.com/protected',
    auth=('api_username', 'api_password'),  # API authentication
    proxies=proxies,  # Proxy with authentication
    verify=True
)

Integration with Other Tools

While basic authentication with Requests works well for API endpoints, some authentication flows require browser interaction. For complex authentication scenarios like OAuth2 flows or JavaScript-heavy login processes, you might need to combine Requests with browser automation tools. Tools like Puppeteer can handle authentication flows that require user interaction or JavaScript execution before making API calls with Requests.

Common Issues and Troubleshooting

Issue 1: 401 Unauthorized Errors

import requests

response = requests.get(
    'https://api.example.com/protected',
    auth=('username', 'password')
)

if response.status_code == 401:
    print("Possible issues:")
    print("1. Incorrect username or password")
    print("2. Account may be locked or disabled")
    print("3. API requires different authentication method")
    print("4. Check if credentials need URL encoding")

Issue 2: SSL Certificate Problems

import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning

# Disable SSL warnings (not recommended for production)
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# For development/testing only - disable SSL verification
response = requests.get(
    'https://api.example.com/protected',
    auth=('username', 'password'),
    verify=False  # Only for development!
)

# Better approach: specify custom CA bundle
response = requests.get(
    'https://api.example.com/protected',
    auth=('username', 'password'),
    verify='/path/to/ca-bundle.crt'
)

Issue 3: Special Characters in Credentials

If your username or password contains special characters, ensure proper handling:

import requests
from urllib.parse import quote

# URL encode credentials if they contain special characters
username = quote('user@domain.com')
password = quote('p@ssw0rd!')

response = requests.get(
    'https://api.example.com/protected',
    auth=(username, password)
)

Testing Authentication

Unit Testing with Mock

import requests
from unittest.mock import patch, Mock

def test_authenticated_request():
    with patch('requests.get') as mock_get:
        # Mock successful authentication
        mock_response = Mock()
        mock_response.status_code = 200
        mock_response.json.return_value = {'data': 'success'}
        mock_get.return_value = mock_response

        # Your function that makes authenticated requests
        result = make_authenticated_request(
            'https://api.example.com/test',
            'test_user',
            'test_pass'
        )

        # Verify the request was made with correct auth
        mock_get.assert_called_once_with(
            'https://api.example.com/test',
            auth=('test_user', 'test_pass'),
            timeout=30,
            verify=True
        )

        assert result == {'data': 'success'}

Performance Considerations

For high-frequency API calls, consider these optimizations:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Configure session with connection pooling and retries
session = requests.Session()
session.auth = ('username', 'password')

# Configure retry strategy
retry_strategy = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504]
)

# Mount adapter with retry strategy
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("http://", adapter)
session.mount("https://", adapter)

# Set connection pooling
session.mount("https://", HTTPAdapter(pool_connections=20, pool_maxsize=20))

Real-World API Examples

GitHub API Authentication

import requests

# GitHub API requires basic auth for private repositories
username = 'your_github_username'
token = 'your_personal_access_token'  # Use token instead of password

response = requests.get(
    'https://api.github.com/user/repos',
    auth=(username, token),
    headers={'Accept': 'application/vnd.github.v3+json'}
)

if response.status_code == 200:
    repos = response.json()
    for repo in repos:
        print(f"Repository: {repo['name']}")

REST API with Basic Auth

import requests
import os

def fetch_customer_data(customer_id):
    """
    Fetch customer data from CRM API
    """
    api_base = 'https://api.crm-system.com'
    username = os.getenv('CRM_USERNAME')
    password = os.getenv('CRM_PASSWORD')

    response = requests.get(
        f'{api_base}/customers/{customer_id}',
        auth=(username, password),
        headers={
            'Accept': 'application/json',
            'User-Agent': 'MyApp/1.0'
        },
        timeout=10
    )

    if response.status_code == 200:
        return response.json()
    elif response.status_code == 404:
        return None
    else:
        response.raise_for_status()

# Usage
customer = fetch_customer_data('12345')
if customer:
    print(f"Customer: {customer['name']}")

Working with Multiple Endpoints

When dealing with multiple APIs that require different authentication, organize your code for maintainability:

import requests
from typing import Dict, Optional

class AuthenticatedAPIClient:
    """
    A reusable client for API endpoints requiring basic authentication
    """

    def __init__(self, base_url: str, username: str, password: str):
        self.base_url = base_url.rstrip('/')
        self.session = requests.Session()
        self.session.auth = (username, password)
        self.session.headers.update({
            'Accept': 'application/json',
            'Content-Type': 'application/json'
        })

    def get(self, endpoint: str, params: Optional[Dict] = None) -> requests.Response:
        """Make GET request to endpoint"""
        url = f"{self.base_url}/{endpoint.lstrip('/')}"
        return self.session.get(url, params=params)

    def post(self, endpoint: str, data: Optional[Dict] = None) -> requests.Response:
        """Make POST request to endpoint"""
        url = f"{self.base_url}/{endpoint.lstrip('/')}"
        return self.session.post(url, json=data)

    def close(self):
        """Close the session"""
        self.session.close()

# Usage example
client = AuthenticatedAPIClient(
    'https://api.example.com',
    'username',
    'password'
)

try:
    response = client.get('/users/profile')
    if response.status_code == 200:
        profile = response.json()
        print(f"User: {profile['name']}")
finally:
    client.close()

Command Line Tools

For testing authentication from the command line, you can create utility scripts:

#!/usr/bin/env python3
"""
Command line tool for testing API authentication
Usage: python auth_test.py <url> <username> <password>
"""

import sys
import requests
from requests.auth import HTTPBasicAuth

def test_auth(url, username, password):
    """Test basic authentication against a URL"""
    try:
        response = requests.get(
            url,
            auth=HTTPBasicAuth(username, password),
            timeout=10
        )

        print(f"Status Code: {response.status_code}")
        print(f"Response Headers: {dict(response.headers)}")

        if response.status_code == 200:
            print("✓ Authentication successful")
            return True
        elif response.status_code == 401:
            print("✗ Authentication failed - check credentials")
            return False
        else:
            print(f"✗ Unexpected status code: {response.status_code}")
            return False

    except requests.exceptions.RequestException as e:
        print(f"✗ Request failed: {e}")
        return False

if __name__ == "__main__":
    if len(sys.argv) != 4:
        print("Usage: python auth_test.py <url> <username> <password>")
        sys.exit(1)

    url, username, password = sys.argv[1:4]
    success = test_auth(url, username, password)
    sys.exit(0 if success else 1)

Browser Session Integration

Sometimes you need to extract session data from browser automation tools and use it with Requests. This approach is useful when initial authentication requires browser interaction, but subsequent API calls can use Requests for better performance:

import requests

def use_browser_session_with_requests(cookies_from_browser):
    """
    Use cookies obtained from browser automation for API requests
    """
    session = requests.Session()

    # Add cookies from browser session
    for cookie in cookies_from_browser:
        session.cookies.set(cookie['name'], cookie['value'])

    # Now make API calls with the authenticated session
    response = session.get('https://api.example.com/protected-data')
    return response.json()

This pattern works well when combined with browser automation tools for handling complex authentication that establish the initial session.

Conclusion

HTTP Basic Authentication with Python Requests is straightforward and secure when implemented correctly. Key takeaways:

  1. Use the auth parameter - It's the most convenient and reliable method
  2. Always use HTTPS - Basic authentication sends credentials in an easily decoded format
  3. Store credentials securely - Use environment variables or secure configuration management
  4. Implement proper error handling - Check status codes and handle network exceptions
  5. Use sessions for multiple requests - More efficient for repeated API calls
  6. Test thoroughly - Verify authentication works across different scenarios

The auth parameter is the recommended approach for most use cases, while sessions are ideal for multiple requests to the same API. For more complex scenarios involving browser automation or JavaScript-heavy authentication flows, consider combining Requests with browser automation tools.

Remember to test your authentication thoroughly and monitor for changes in the API's authentication requirements, as these can evolve over time.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I handle OAuth authentication using Requests?
How do I make asynchronous requests with Requests?
How do I retry failed requests automatically with Requests?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon