What is the best way to handle rate limiting with urllib3?

Rate limiting is an important aspect to consider when performing web scraping or API calls to prevent overwhelming the server with too many requests in a short period. When using urllib3 in Python, it does not provide built-in rate limiting functionality. However, you can handle rate limiting by implementing a delay between requests or using a more sophisticated approach such as token bucket algorithms.

Here's a basic example of how you might implement a simple delay between requests using urllib3 along with Python's time.sleep function:

import urllib3
import time

http = urllib3.PoolManager()

urls = [
    'http://example.com/page1',
    'http://example.com/page2',
    # Add more URLs as needed
]

rate_limit_seconds = 1  # Set the desired time between requests

for url in urls:
    response = http.request('GET', url)
    # Process the response
    print(response.status)
    time.sleep(rate_limit_seconds)  # Wait for the rate limit interval before the next request

For more sophisticated rate limiting, you might implement a token bucket algorithm. Here's a simple token bucket rate limiter example:

import urllib3
import time

class TokenBucket:
    def __init__(self, tokens, fill_rate):
        """
        tokens is the total tokens in the bucket.
        fill_rate is the rate in tokens/second that the bucket will be refilled.
        """
        self.capacity = tokens
        self._tokens = tokens
        self.fill_rate = fill_rate
        self.timestamp = time.time()

    def consume(self, tokens):
        if tokens <= self.get_tokens():
            self._tokens -= tokens
            return True
        return False

    def get_tokens(self):
        now = time.time()
        delta = self.fill_rate * (now - self.timestamp)
        self._tokens = min(self.capacity, self._tokens + delta)
        self.timestamp = now
        return self._tokens

# Initialize the token bucket (e.g., 5 tokens and refill 1 token per second)
bucket = TokenBucket(5, 1)

http = urllib3.PoolManager()

urls = [
    'http://example.com/page1',
    'http://example.com/page2',
    # Add more URLs as needed
]

for url in urls:
    # Assume each request consumes 1 token
    if not bucket.consume(1):
        # Calculate the time to wait for the next token
        sleep_time = 1 - (bucket.get_tokens() - 1)
        time.sleep(sleep_time)
    response = http.request('GET', url)
    # Process the response
    print(response.status)

In this example, the TokenBucket class is used to control the rate of HTTP requests. It will only allow a request to be made if there is a token available, and it refills the tokens at a specified rate. If there are no tokens available when a request is attempted, the code calculates the time until the next token is available and sleeps for that duration.

When implementing rate limiting, it is also crucial to respect the Retry-After HTTP header if the server sends it after responding with a 429 (Too Many Requests) status code. This header indicates how long the client should wait before making a new request.

Remember to always review the terms of service for the website you are scraping, and ensure that your web scraping activities are compliant with their policies and legal requirements.