Table of contents

How can I log HTTParty requests and responses for debugging purposes?

Debugging HTTParty requests is essential for troubleshooting API integrations and web scraping applications. HTTParty provides several built-in mechanisms for logging requests and responses, plus you can implement custom logging solutions for more detailed debugging information.

Built-in Debug Option

The simplest way to enable logging in HTTParty is using the built-in debug_output option:

require 'httparty'

class ApiClient
  include HTTParty

  # Enable debug output to STDOUT
  debug_output $stdout

  base_uri 'https://api.example.com'
end

# Make a request with debug output
response = ApiClient.get('/users/1')

This will output detailed information about the request and response:

opening connection to api.example.com:443...
opened
starting SSL for api.example.com:443...
SSL established
<- "GET /users/1 HTTP/1.1\r\nConnection: close\r\nHost: api.example.com\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Content-Type: application/json\r\n"
-> "Content-Length: 156\r\n"
-> "\r\n"
reading 156 bytes...
-> "{\"id\":1,\"name\":\"John Doe\",\"email\":\"john@example.com\"}"
read 156 bytes
connection closed

Logging to a File

You can redirect debug output to a file for persistent logging:

require 'httparty'

class ApiClient
  include HTTParty

  # Log to a file
  debug_output File.open('httparty_debug.log', 'w+')

  base_uri 'https://api.example.com'
end

response = ApiClient.get('/users')

Custom Logger Implementation

For more control over logging format and content, implement a custom logger:

require 'httparty'
require 'logger'

class CustomHTTPartyLogger
  def initialize(logger)
    @logger = logger
  end

  def <<(data)
    @logger.info(data)
  end
end

class ApiClient
  include HTTParty

  logger = Logger.new(STDOUT)
  logger.level = Logger::DEBUG

  debug_output CustomHTTPartyLogger.new(logger)

  base_uri 'https://api.example.com'
end

Advanced Logging with Callbacks

HTTParty supports callbacks that allow you to log specific events during the request lifecycle:

require 'httparty'
require 'logger'

class ApiClient
  include HTTParty

  base_uri 'https://api.example.com'

  # Set up logger
  @@logger = Logger.new('api_requests.log')

  # Custom parser with logging
  def self.perform_request(http_method, path, options, &block)
    start_time = Time.now

    # Log request details
    @@logger.info("Starting #{http_method.upcase} request to #{path}")
    @@logger.debug("Request options: #{options.inspect}")

    begin
      response = super(http_method, path, options, &block)

      # Log response details
      duration = ((Time.now - start_time) * 1000).round(2)
      @@logger.info("Request completed in #{duration}ms with status #{response.code}")
      @@logger.debug("Response headers: #{response.headers.inspect}")
      @@logger.debug("Response body: #{response.body}")

      response
    rescue => e
      @@logger.error("Request failed: #{e.message}")
      raise
    end
  end
end

# Usage
response = ApiClient.get('/users/1')

Logging Request and Response Bodies Separately

When you need to log only specific parts of the HTTP exchange:

require 'httparty'
require 'json'

class ApiClient
  include HTTParty

  base_uri 'https://api.example.com'

  def self.logged_request(method, path, options = {})
    # Log request
    puts "=== REQUEST ==="
    puts "#{method.upcase} #{path}"
    puts "Headers: #{options[:headers]}" if options[:headers]
    puts "Body: #{options[:body]}" if options[:body]
    puts

    # Make request
    response = send(method, path, options)

    # Log response
    puts "=== RESPONSE ==="
    puts "Status: #{response.code} #{response.message}"
    puts "Headers: #{response.headers}"
    puts "Body: #{response.body}"
    puts "=" * 50

    response
  end
end

# Usage
response = ApiClient.logged_request(:get, '/users/1')

Structured Logging with JSON

For applications that require structured logging:

require 'httparty'
require 'json'
require 'logger'

class StructuredLogger
  def initialize(logger)
    @logger = logger
  end

  def log_request(method, url, options)
    log_entry = {
      timestamp: Time.now.iso8601,
      type: 'http_request',
      method: method.to_s.upcase,
      url: url,
      headers: options[:headers] || {},
      body: options[:body]
    }

    @logger.info(JSON.generate(log_entry))
  end

  def log_response(response, duration)
    log_entry = {
      timestamp: Time.now.iso8601,
      type: 'http_response',
      status_code: response.code,
      duration_ms: duration,
      headers: response.headers.to_h,
      body_size: response.body&.length || 0
    }

    @logger.info(JSON.generate(log_entry))
  end
end

class ApiClient
  include HTTParty

  base_uri 'https://api.example.com'

  @@structured_logger = StructuredLogger.new(Logger.new('api_structured.log'))

  def self.perform_request(http_method, path, options, &block)
    start_time = Time.now
    full_url = "#{base_uri}#{path}"

    @@structured_logger.log_request(http_method, full_url, options)

    response = super(http_method, path, options, &block)
    duration = ((Time.now - start_time) * 1000).round(2)

    @@structured_logger.log_response(response, duration)

    response
  end
end

Conditional Logging

Enable logging only in specific environments or conditions:

require 'httparty'

class ApiClient
  include HTTParty

  base_uri 'https://api.example.com'

  # Enable debug output only in development/test environments
  if ENV['RAILS_ENV'] == 'development' || ENV['DEBUG_HTTPARTY']
    debug_output $stdout
  end

  # Or use a more sophisticated approach
  if defined?(Rails) && Rails.env.development?
    debug_output Rails.logger
  end
end

Logging with Middleware Pattern

Create reusable logging middleware for complex applications:

require 'httparty'

module HTTPartyLoggingMiddleware
  def self.included(base)
    base.extend(ClassMethods)
  end

  module ClassMethods
    def with_logging(logger = nil)
      @logger = logger || Logger.new(STDOUT)

      define_singleton_method :perform_request do |http_method, path, options, &block|
        request_id = SecureRandom.hex(8)

        @logger.info("[#{request_id}] → #{http_method.upcase} #{path}")
        @logger.debug("[#{request_id}] Request options: #{options}")

        start_time = Time.now
        response = super(http_method, path, options, &block)
        duration = Time.now - start_time

        @logger.info("[#{request_id}] ← #{response.code} (#{(duration * 1000).round}ms)")
        @logger.debug("[#{request_id}] Response: #{response.body}")

        response
      end

      self
    end
  end
end

class ApiClient
  include HTTParty
  include HTTPartyLoggingMiddleware

  base_uri 'https://api.example.com'
end

# Usage
client = ApiClient.with_logging(Logger.new('api.log'))
response = client.get('/users/1')

Integration with Rails Logger

When working within a Rails application, integrate with Rails' logging system:

class ApiClient
  include HTTParty

  base_uri 'https://api.example.com'

  if defined?(Rails)
    debug_output Rails.logger

    # Custom logging method that respects Rails log levels
    def self.rails_logged_request(method, path, options = {})
      Rails.logger.debug("HTTParty #{method.upcase} #{base_uri}#{path}")
      Rails.logger.debug("Request options: #{options}") if options.any?

      response = send(method, path, options)

      Rails.logger.info("HTTParty response: #{response.code}")
      Rails.logger.debug("Response body: #{response.body}")

      response
    end
  end
end

Performance Considerations

When implementing logging for production environments, consider these performance factors:

class PerformantApiClient
  include HTTParty

  base_uri 'https://api.example.com'

  # Only log response bodies for errors or when explicitly requested
  def self.conditional_log_request(method, path, options = {})
    log_response_body = options.delete(:log_response_body) || false

    response = send(method, path, options)

    # Always log basic request info
    logger.info("#{method.upcase} #{path} → #{response.code}")

    # Log detailed info only for errors or when requested
    if response.code >= 400 || log_response_body
      logger.error("Error response: #{response.body}")
    end

    response
  end

  private

  def self.logger
    @logger ||= Logger.new('api_errors.log')
  end
end

Best Practices

  1. Security: Never log sensitive data like API keys, passwords, or personal information
  2. Performance: In production, avoid logging large response bodies unless necessary
  3. Rotation: Use log rotation to prevent log files from growing too large
  4. Filtering: Implement filtering to exclude sensitive headers or parameters
  5. Structured Format: Use structured logging (JSON) for better parsing and analysis

When debugging HTTParty requests becomes complex, consider using specialized tools for monitoring network requests in Puppeteer for more advanced scenarios or implementing custom error handling strategies for robust applications.

By implementing these logging strategies, you'll have comprehensive visibility into your HTTParty requests and responses, making debugging and monitoring much more effective.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

What are the best practices for handling HTTP errors and exceptions in HTTParty?
How do I implement custom middleware or callbacks with HTTParty?
Can HTTParty be used with WebSocket connections for real-time scraping?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon