Table of contents

How do I Mock HTTP Requests for Testing with Requests?

Mocking HTTP requests is essential for creating reliable, fast, and isolated unit tests when working with the Python Requests library. Instead of making actual network calls during testing, mocking allows you to simulate API responses, test error conditions, and ensure your tests run consistently without depending on external services.

Why Mock HTTP Requests?

Testing code that makes HTTP requests presents several challenges:

  • External dependencies: Tests shouldn't rely on external APIs being available
  • Speed: Network requests slow down test execution
  • Reliability: External services may be unreliable or rate-limited
  • Cost: Some APIs charge per request
  • Testing edge cases: It's difficult to simulate error conditions with real APIs

Popular Libraries for Mocking HTTP Requests

1. responses Library

The responses library is the most popular choice for mocking HTTP requests in Python. It's specifically designed to work with the Requests library.

Installation

pip install responses

Basic Usage

import requests
import responses
import pytest

@responses.activate
def test_api_call():
    # Mock the API response
    responses.add(
        responses.GET,
        'https://api.example.com/users/1',
        json={'id': 1, 'name': 'John Doe', 'email': 'john@example.com'},
        status=200
    )

    # Make the request
    response = requests.get('https://api.example.com/users/1')

    # Assert the response
    assert response.status_code == 200
    assert response.json()['name'] == 'John Doe'

Advanced responses Usage

import responses
import requests

class UserService:
    def __init__(self, base_url):
        self.base_url = base_url

    def get_user(self, user_id):
        response = requests.get(f"{self.base_url}/users/{user_id}")
        response.raise_for_status()
        return response.json()

    def create_user(self, user_data):
        response = requests.post(f"{self.base_url}/users", json=user_data)
        response.raise_for_status()
        return response.json()

class TestUserService:

    @responses.activate
    def test_get_user_success(self):
        # Mock successful response
        responses.add(
            responses.GET,
            'https://api.example.com/users/123',
            json={'id': 123, 'name': 'Jane Doe'},
            status=200
        )

        service = UserService('https://api.example.com')
        user = service.get_user(123)

        assert user['id'] == 123
        assert user['name'] == 'Jane Doe'

    @responses.activate
    def test_get_user_not_found(self):
        # Mock 404 error
        responses.add(
            responses.GET,
            'https://api.example.com/users/999',
            json={'error': 'User not found'},
            status=404
        )

        service = UserService('https://api.example.com')

        with pytest.raises(requests.exceptions.HTTPError):
            service.get_user(999)

    @responses.activate
    def test_create_user(self):
        # Mock POST request
        responses.add(
            responses.POST,
            'https://api.example.com/users',
            json={'id': 456, 'name': 'New User', 'email': 'new@example.com'},
            status=201
        )

        service = UserService('https://api.example.com')
        user_data = {'name': 'New User', 'email': 'new@example.com'}
        created_user = service.create_user(user_data)

        assert created_user['id'] == 456
        assert created_user['name'] == 'New User'

        # Verify the request was made correctly
        assert len(responses.calls) == 1
        assert responses.calls[0].request.url == 'https://api.example.com/users'

Dynamic Responses with Callbacks

import json
import responses

def request_callback(request):
    # Parse request data
    if request.body:
        data = json.loads(request.body)
        # Return different responses based on request data
        if data.get('email') == 'admin@example.com':
            return (403, {}, json.dumps({'error': 'Admin creation not allowed'}))

    # Default successful response
    return (201, {}, json.dumps({'id': 789, 'created': True}))

@responses.activate
def test_dynamic_response():
    responses.add_callback(
        responses.POST,
        'https://api.example.com/users',
        callback=request_callback,
        content_type='application/json'
    )

    # Test successful creation
    response = requests.post(
        'https://api.example.com/users',
        json={'name': 'Regular User', 'email': 'user@example.com'}
    )
    assert response.status_code == 201

    # Test admin restriction
    response = requests.post(
        'https://api.example.com/users',
        json={'name': 'Admin User', 'email': 'admin@example.com'}
    )
    assert response.status_code == 403

2. requests-mock Library

requests-mock provides a pytest fixture and context manager approach to mocking.

Installation

pip install requests-mock

Usage with Pytest Fixture

import requests
import requests_mock

def test_with_requests_mock(requests_mock):
    # Register mock
    requests_mock.get(
        'https://api.example.com/data',
        json={'result': 'success'},
        status_code=200
    )

    # Make request
    response = requests.get('https://api.example.com/data')

    assert response.json()['result'] == 'success'

def test_with_context_manager():
    with requests_mock.Mocker() as m:
        m.get('https://api.example.com/data', text='mock response')

        response = requests.get('https://api.example.com/data')
        assert response.text == 'mock response'

Advanced requests-mock Features

import requests_mock

def test_request_matching():
    with requests_mock.Mocker() as m:
        # Match requests with query parameters
        m.get(
            'https://api.example.com/search',
            additional_matcher=lambda req: 'python' in req.query,
            json={'results': ['Python result 1', 'Python result 2']}
        )

        response = requests.get('https://api.example.com/search?q=python')
        assert len(response.json()['results']) == 2

def test_request_history():
    with requests_mock.Mocker() as m:
        m.post('https://api.example.com/data', status_code=201)

        requests.post('https://api.example.com/data', json={'key': 'value'})

        # Check request history
        assert len(m.request_history) == 1
        assert m.request_history[0].method == 'POST'
        assert 'key' in m.request_history[0].text

3. unittest.mock with Requests

Using Python's built-in unittest.mock provides more granular control but requires more setup.

import unittest.mock
import requests

def fetch_user_data(user_id):
    response = requests.get(f'https://api.example.com/users/{user_id}')
    response.raise_for_status()
    return response.json()

def test_fetch_user_with_mock():
    with unittest.mock.patch('requests.get') as mock_get:
        # Configure mock response
        mock_response = unittest.mock.Mock()
        mock_response.status_code = 200
        mock_response.json.return_value = {'id': 1, 'name': 'Test User'}
        mock_response.raise_for_status.return_value = None
        mock_get.return_value = mock_response

        # Call function
        result = fetch_user_data(1)

        # Assertions
        assert result['name'] == 'Test User'
        mock_get.assert_called_once_with('https://api.example.com/users/1')

Testing Different HTTP Methods

Mocking Different Request Types

import responses

@responses.activate
def test_all_http_methods():
    # GET request
    responses.add(
        responses.GET,
        'https://api.example.com/items',
        json=[{'id': 1, 'name': 'Item 1'}]
    )

    # POST request
    responses.add(
        responses.POST,
        'https://api.example.com/items',
        json={'id': 2, 'name': 'New Item'},
        status=201
    )

    # PUT request
    responses.add(
        responses.PUT,
        'https://api.example.com/items/1',
        json={'id': 1, 'name': 'Updated Item'}
    )

    # DELETE request
    responses.add(
        responses.DELETE,
        'https://api.example.com/items/1',
        status=204
    )

    # Test GET
    get_response = requests.get('https://api.example.com/items')
    assert len(get_response.json()) == 1

    # Test POST
    post_response = requests.post(
        'https://api.example.com/items',
        json={'name': 'New Item'}
    )
    assert post_response.status_code == 201

    # Test PUT
    put_response = requests.put(
        'https://api.example.com/items/1',
        json={'name': 'Updated Item'}
    )
    assert put_response.json()['name'] == 'Updated Item'

    # Test DELETE
    delete_response = requests.delete('https://api.example.com/items/1')
    assert delete_response.status_code == 204

Testing Error Conditions

import responses
import requests

@responses.activate
def test_network_errors():
    # Simulate connection error
    responses.add(
        responses.GET,
        'https://api.example.com/timeout',
        body=requests.exceptions.ConnectTimeout('Connection timeout')
    )

    with pytest.raises(requests.exceptions.ConnectTimeout):
        requests.get('https://api.example.com/timeout')

@responses.activate
def test_http_errors():
    # Simulate server error
    responses.add(
        responses.GET,
        'https://api.example.com/server-error',
        json={'error': 'Internal server error'},
        status=500
    )

    response = requests.get('https://api.example.com/server-error')
    assert response.status_code == 500
    assert 'error' in response.json()

Integration with pytest

Creating Reusable Fixtures

import pytest
import responses

@pytest.fixture
def mock_api():
    with responses.RequestsMock() as rsps:
        # Add common mock responses
        rsps.add(
            responses.GET,
            'https://api.example.com/health',
            json={'status': 'healthy'},
            status=200
        )
        yield rsps

def test_api_health(mock_api):
    response = requests.get('https://api.example.com/health')
    assert response.json()['status'] == 'healthy'

def test_additional_mocks(mock_api):
    # Add more mocks to the existing fixture
    mock_api.add(
        responses.GET,
        'https://api.example.com/users',
        json=[{'id': 1, 'name': 'User 1'}]
    )

    response = requests.get('https://api.example.com/users')
    assert len(response.json()) == 1

JavaScript Testing with Mock Service Worker

For JavaScript projects using fetch or axios, Mock Service Worker (MSW) provides similar functionality:

// setupTests.js
import { setupServer } from 'msw/node'
import { rest } from 'msw'

const server = setupServer(
  rest.get('https://api.example.com/users/:id', (req, res, ctx) => {
    const { id } = req.params
    return res(
      ctx.json({
        id: parseInt(id),
        name: 'John Doe',
        email: 'john@example.com'
      })
    )
  })
)

beforeAll(() => server.listen())
afterEach(() => server.resetHandlers())
afterAll(() => server.close())

// userService.test.js
import { fetchUser } from './userService'

test('fetches user data', async () => {
  const user = await fetchUser(1)
  expect(user.name).toBe('John Doe')
})

test('handles error response', async () => {
  server.use(
    rest.get('https://api.example.com/users/:id', (req, res, ctx) => {
      return res(ctx.status(404), ctx.json({ error: 'User not found' }))
    })
  )

  await expect(fetchUser(999)).rejects.toThrow('User not found')
})

Best Practices for Mocking HTTP Requests

1. Keep Mocks Realistic

Ensure your mock responses closely match real API responses in structure and content:

# Good: Realistic response structure
responses.add(
    responses.GET,
    'https://api.github.com/users/octocat',
    json={
        'login': 'octocat',
        'id': 1,
        'avatar_url': 'https://github.com/images/error/octocat_happy.gif',
        'type': 'User',
        'created_at': '2011-01-25T18:44:36Z'
    }
)

# Bad: Oversimplified response
responses.add(
    responses.GET,
    'https://api.github.com/users/octocat',
    json={'name': 'octocat'}
)

2. Test Both Success and Failure Cases

@pytest.mark.parametrize('status_code,expected_exception', [
    (400, requests.exceptions.HTTPError),
    (401, requests.exceptions.HTTPError),
    (403, requests.exceptions.HTTPError),
    (404, requests.exceptions.HTTPError),
    (500, requests.exceptions.HTTPError),
])
@responses.activate
def test_error_handling(status_code, expected_exception):
    responses.add(
        responses.GET,
        'https://api.example.com/data',
        status=status_code
    )

    with pytest.raises(expected_exception):
        response = requests.get('https://api.example.com/data')
        response.raise_for_status()

3. Verify Request Details

@responses.activate
def test_request_verification():
    responses.add(
        responses.POST,
        'https://api.example.com/users',
        json={'id': 1, 'created': True}
    )

    requests.post(
        'https://api.example.com/users',
        json={'name': 'John', 'email': 'john@example.com'},
        headers={'Authorization': 'Bearer token123'}
    )

    # Verify the request was made correctly
    assert len(responses.calls) == 1
    request = responses.calls[0].request
    assert request.headers['Authorization'] == 'Bearer token123'
    assert '"name": "John"' in request.body

4. Mock External Dependencies

When testing web scraping code, mock external APIs to ensure consistent test results:

import responses
import requests
from bs4 import BeautifulSoup

def scrape_website_data(url):
    response = requests.get(url)
    response.raise_for_status()
    soup = BeautifulSoup(response.content, 'html.parser')
    return [link.get('href') for link in soup.find_all('a')]

@responses.activate
def test_scrape_website_data():
    html_content = '''
    <html>
        <body>
            <a href="/page1">Page 1</a>
            <a href="/page2">Page 2</a>
        </body>
    </html>
    '''

    responses.add(
        responses.GET,
        'https://example.com',
        body=html_content,
        status=200,
        content_type='text/html'
    )

    links = scrape_website_data('https://example.com')
    assert '/page1' in links
    assert '/page2' in links

Performance Considerations

Parallel Test Execution

When running tests in parallel, ensure mocks are properly isolated:

import pytest
import responses

@pytest.fixture(autouse=True)
def reset_responses():
    """Reset responses after each test to prevent cross-test pollution."""
    yield
    responses.reset()
    responses.stop()

@responses.activate
def test_isolated_mock_1():
    responses.add(responses.GET, 'https://api.example.com/test1', json={'test': 1})
    response = requests.get('https://api.example.com/test1')
    assert response.json()['test'] == 1

@responses.activate
def test_isolated_mock_2():
    responses.add(responses.GET, 'https://api.example.com/test2', json={'test': 2})
    response = requests.get('https://api.example.com/test2')
    assert response.json()['test'] == 2

Conclusion

Mocking HTTP requests is crucial for creating robust, fast, and reliable tests. The responses library provides the most straightforward approach for mocking Requests calls, while requests-mock offers excellent pytest integration. Choose the approach that best fits your testing framework and requirements.

When building web scraping applications that need to handle dynamic content loading or complex authentication flows, proper testing with mocked requests ensures your scraping logic works correctly without depending on external services. Similarly, if you're monitoring network requests during development, having comprehensive test coverage with mocked responses helps validate your request handling logic.

Remember to keep your mocks realistic, test both success and failure scenarios, and verify that your code makes the expected requests with the correct parameters and headers.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I handle request and response encoding issues?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon