How to Mock HTTParty Requests for Testing Purposes

When building Ruby applications that consume external APIs using HTTParty, it's essential to mock HTTP requests during testing to ensure your tests are fast, reliable, and don't depend on external services. This comprehensive guide covers various approaches to mocking HTTParty requests, from simple stubs to sophisticated recording and playback systems.

Why Mock HTTParty Requests?

Mocking HTTP requests in tests provides several critical benefits:

• Speed: Eliminates network latency and external API response times
• Reliability: Tests don't fail due to network issues or external service downtime
• Predictability: Control exact responses and test edge cases
• Cost efficiency: Avoid charges from paid APIs during testing
• Isolation: Test your code logic independently of external dependencies

Setting Up WebMock for HTTParty Testing

WebMock is the most popular gem for stubbing HTTP requests in Ruby applications. It integrates seamlessly with HTTParty and provides a clean API for mocking requests.

Installation and Configuration

Add WebMock to your Gemfile's test group:

# Gemfile
group :test do
  gem 'webmock'
  gem 'rspec'
end

Configure WebMock in your test setup:

# spec/spec_helper.rb
require 'webmock/rspec'

RSpec.configure do |config|
  # Disable real HTTP requests during tests
  config.before(:suite) do
    WebMock.disable_net_connect!(allow_localhost: true)
  end
end

Basic HTTParty Request Mocking

Let's start with a simple example of an HTTParty service class and how to mock its requests:

# app/services/api_client.rb
class ApiClient
  include HTTParty
  base_uri 'https://api.example.com'

  def get_user(id)
    self.class.get("/users/#{id}")
  end

  def create_user(user_data)
    self.class.post('/users', {
      body: user_data.to_json,
      headers: { 'Content-Type' => 'application/json' }
    })
  end
end

Mocking GET Requests

Here's how to mock a simple GET request using WebMock:

# spec/services/api_client_spec.rb
require 'rails_helper'

RSpec.describe ApiClient do
  let(:client) { ApiClient.new }

  describe '#get_user' do
    it 'returns user data from the API' do
      user_response = {
        id: 123,
        name: 'John Doe',
        email: 'john@example.com'
      }

      # Stub the HTTP request
      stub_request(:get, 'https://api.example.com/users/123')
        .to_return(
          status: 200,
          body: user_response.to_json,
          headers: { 'Content-Type' => 'application/json' }
        )

      response = client.get_user(123)

      expect(response.code).to eq(200)
      expect(response.parsed_response['name']).to eq('John Doe')
    end
  end
end

Mocking POST Requests with Request Body Validation

For POST requests, you can also validate the request body:

describe '#create_user' do
  it 'creates a new user via API' do
    user_data = { name: 'Jane Doe', email: 'jane@example.com' }

    stub_request(:post, 'https://api.example.com/users')
      .with(
        body: user_data.to_json,
        headers: { 'Content-Type' => 'application/json' }
      )
      .to_return(
        status: 201,
        body: { id: 456, **user_data }.to_json,
        headers: { 'Content-Type' => 'application/json' }
      )

    response = client.create_user(user_data)

    expect(response.code).to eq(201)
    expect(response.parsed_response['id']).to eq(456)
  end
end

Advanced Mocking Techniques

Dynamic Response Generation

You can create dynamic responses based on request parameters:

stub_request(:get, /https:\/\/api\.example\.com\/users\/\d+/)
  .to_return do |request|
    user_id = request.uri.path.split('/').last
    {
      status: 200,
      body: { id: user_id.to_i, name: "User #{user_id}" }.to_json,
      headers: { 'Content-Type' => 'application/json' }
    }
  end

Simulating Network Errors

Test how your application handles various failure scenarios:

describe 'error handling' do
  it 'handles timeout errors gracefully' do
    stub_request(:get, 'https://api.example.com/users/123')
      .to_timeout

    expect { client.get_user(123) }.to raise_error(Net::TimeoutError)
  end

  it 'handles server errors' do
    stub_request(:get, 'https://api.example.com/users/123')
      .to_return(status: 500, body: 'Internal Server Error')

    response = client.get_user(123)
    expect(response.code).to eq(500)
  end
end

Request Verification

Verify that requests were made with correct parameters:

it 'makes the correct API call' do
  stub_request(:get, 'https://api.example.com/users/123')
    .to_return(status: 200, body: '{}')

  client.get_user(123)

  expect(WebMock).to have_requested(:get, 'https://api.example.com/users/123')
    .with(headers: { 'User-Agent' => /HTTParty/ })
end

Using VCR for Record and Playback Testing

VCR (Video Cassette Recorder) is another powerful tool that records real HTTP interactions and replays them during tests. This approach is particularly useful when you want to use real API responses but still maintain test isolation.

VCR Setup

Add VCR to your Gemfile:

# Gemfile
group :test do
  gem 'vcr'
  gem 'webmock'
end

Configure VCR:

# spec/spec_helper.rb
require 'vcr'

VCR.configure do |config|
  config.cassette_library_dir = 'spec/fixtures/vcr_cassettes'
  config.hook_into :webmock
  config.configure_rspec_metadata!
  config.allow_http_connections_when_no_cassette = false
end

Using VCR in Tests

describe '#get_user', :vcr do
  it 'returns user data from the API' do
    response = client.get_user(123)

    expect(response.code).to eq(200)
    expect(response.parsed_response).to have_key('name')
  end
end

The first time this test runs, VCR will record the actual HTTP request. Subsequent runs will use the recorded response, making tests fast and deterministic.

Testing HTTParty with RSpec Shared Examples

Create reusable test patterns for common HTTParty scenarios:

# spec/support/shared_examples/api_client.rb
RSpec.shared_examples 'an API client' do |endpoint, method|
  it 'handles successful responses' do
    stub_request(method, endpoint)
      .to_return(status: 200, body: '{"success": true}')

    response = subject.public_send(method.to_s.downcase, endpoint)
    expect(response.code).to eq(200)
  end

  it 'handles error responses' do
    stub_request(method, endpoint)
      .to_return(status: 404, body: '{"error": "Not found"}')

    response = subject.public_send(method.to_s.downcase, endpoint)
    expect(response.code).to eq(404)
  end
end

Use the shared examples in your tests:

describe ApiClient do
  subject { described_class.new }

  it_behaves_like 'an API client', 'https://api.example.com/users', :get
end

Mocking Authentication and Headers

When working with APIs that require authentication, mock the authentication flow. For instance, when testing web scraping scenarios that require handling authentication in web scraping applications, you'll want to ensure your HTTP client properly manages session tokens:

class AuthenticatedApiClient
  include HTTParty
  base_uri 'https://api.example.com'

  def initialize(api_key)
    @api_key = api_key
  end

  def get_protected_resource
    self.class.get('/protected', headers: auth_headers)
  end

  private

  def auth_headers
    { 'Authorization' => "Bearer #{@api_key}" }
  end
end

Test with authentication headers:

describe AuthenticatedApiClient do
  let(:client) { AuthenticatedApiClient.new('test-api-key') }

  it 'includes authentication headers' do
    stub_request(:get, 'https://api.example.com/protected')
      .with(headers: { 'Authorization' => 'Bearer test-api-key' })
      .to_return(status: 200, body: '{"data": "secret"}')

    response = client.get_protected_resource
    expect(response.code).to eq(200)
  end
end

Best Practices for HTTParty Testing

1. Use Factory Methods for Consistent Test Data

# spec/support/factories/api_responses.rb
module ApiResponseFactories
  def user_response(id: 123, name: 'Test User')
    {
      id: id,
      name: name,
      email: "#{name.downcase.gsub(' ', '.')}@example.com",
      created_at: Time.current.iso8601
    }
  end

  def error_response(message: 'Something went wrong')
    { error: message, timestamp: Time.current.iso8601 }
  end
end

RSpec.configure do |config|
  config.include ApiResponseFactories
end

2. Test Both Success and Failure Scenarios

Always test how your application handles various response codes and error conditions. This is especially important when implementing retry logic for failed API requests:

describe 'error handling' do
  context 'when API returns 401 Unauthorized' do
    it 'raises an authentication error' do
      stub_request(:get, 'https://api.example.com/users/123')
        .to_return(status: 401, body: error_response('Invalid token').to_json)

      expect { client.get_user(123) }.to raise_error(AuthenticationError)
    end
  end

  context 'when API returns 429 Rate Limited' do
    it 'implements retry logic' do
      stub_request(:get, 'https://api.example.com/users/123')
        .to_return(status: 429)
        .then
        .to_return(status: 200, body: user_response.to_json)

      response = client.get_user_with_retry(123)
      expect(response.code).to eq(200)
    end
  end
end

3. Organize Tests with Contexts

Use RSpec contexts to group related test scenarios:

describe ApiClient do
  describe '#get_user' do
    context 'when user exists' do
      before do
        stub_request(:get, 'https://api.example.com/users/123')
          .to_return(status: 200, body: user_response.to_json)
      end

      it 'returns the user data' do
        # test implementation
      end
    end

    context 'when user does not exist' do
      before do
        stub_request(:get, 'https://api.example.com/users/999')
          .to_return(status: 404, body: error_response('User not found').to_json)
      end

      it 'returns 404 status' do
        # test implementation
      end
    end
  end
end

4. Mock Concurrent Requests

When testing applications that make concurrent requests for faster scraping, ensure your mocks handle multiple simultaneous requests:

describe 'concurrent requests' do
  it 'handles multiple requests simultaneously' do
    # Mock multiple user endpoints
    (1..5).each do |id|
      stub_request(:get, "https://api.example.com/users/#{id}")
        .to_return(status: 200, body: user_response(id: id).to_json)
    end

    # Test concurrent fetching
    responses = (1..5).map { |id| Thread.new { client.get_user(id) } }
                     .map(&:value)

    expect(responses).to all(have_attributes(code: 200))
  end
end

Integration with CI/CD

Ensure your mocked tests run consistently in continuous integration environments:

# .github/workflows/test.yml
name: Test
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Setup Ruby
        uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - name: Run tests
        run: bundle exec rspec
        env:
          RAILS_ENV: test

Testing Rate Limiting and Retry Logic

When implementing rate limiting protection, test both the detection and response mechanisms:

describe 'rate limiting' do
  it 'respects rate limits' do
    # First request succeeds
    stub_request(:get, 'https://api.example.com/users/1')
      .to_return(status: 200, body: user_response.to_json)

    # Second request hits rate limit
    stub_request(:get, 'https://api.example.com/users/2')
      .to_return(
        status: 429,
        headers: { 'Retry-After' => '60' },
        body: { error: 'Rate limit exceeded' }.to_json
      )

    first_response = client.get_user(1)
    expect(first_response.code).to eq(200)

    second_response = client.get_user(2)
    expect(second_response.code).to eq(429)
    expect(second_response.headers['retry-after']).to eq('60')
  end
end

Conclusion

Mocking HTTParty requests is essential for creating reliable, fast, and maintainable test suites. Whether you choose WebMock for fine-grained control or VCR for recording real interactions, the key is to test both successful scenarios and error conditions comprehensively.

By implementing proper mocking strategies, you ensure that your application gracefully handles various API responses and network conditions. This approach leads to more robust applications and gives you confidence when deploying changes that interact with external services.

Remember to keep your mocks realistic and update them when the external API changes. Regular integration tests against the real API (perhaps in a staging environment) can help ensure your mocks remain accurate and your application continues to work correctly with the actual service.