Does MechanicalSoup provide any rate-limiting features?

MechanicalSoup is a Python library for automating interaction with websites. It provides a simple API for navigating and interacting with pages, similar to what you would do in a web browser. MechanicalSoup is built on top of two well-known Python libraries: requests for handling HTTP requests and BeautifulSoup for parsing HTML.

MechanicalSoup itself does not have built-in rate-limiting features. Rate limiting is a technique used to control the amount of requests a user can make to a server in a given amount of time to prevent abuse or to comply with the terms of service of web APIs and websites.

However, you can implement rate limiting in your MechanicalSoup scripts by using Python's standard libraries or third-party packages. Here are a few methods to implement rate limiting:

Using time.sleep()

You can use the time.sleep() function from the Python standard library to add delays between your requests:

import time
import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser()

urls = ['http://example.com/page1', 'http://example.com/page2', 'http://example.com/page3']

for url in urls:
    browser.open(url)
    # Process the page...
    time.sleep(1)  # Sleep for 1 second between requests

Using ratelimit Library

You can use the ratelimit package, which is a third-party library that provides a decorator to limit the rate of function calls:

from ratelimit import limits, sleep_and_retry
import mechanicalsoup

ONE_MINUTE = 60

@sleep_and_retry
@limits(calls=10, period=ONE_MINUTE)
def fetch_url(url):
    browser.open(url)
    # Process the page...

browser = mechanicalsoup.StatefulBrowser()

urls = ['http://example.com/page1', 'http://example.com/page2', 'http://example.com/page3']

for url in urls:
    fetch_url(url)

In the example above, the fetch_url function is limited to 10 calls per minute. If the limit is exceeded, the sleep_and_retry decorator will put the function to sleep until it's allowed to proceed.

Using requests with requests_toolbelt

Since MechanicalSoup is built on top of requests, you can use the requests_toolbelt library's HTTPAdapter to add rate limiting:

from requests_toolbelt.adapters import HTTPAdapter
import mechanicalsoup

class RateLimitAdapter(HTTPAdapter):
    def __init__(self, delay):
        super(RateLimitAdapter, self).__init__()
        self.delay = delay

    def send(self, request, **kwargs):
        response = super(RateLimitAdapter, self).send(request, **kwargs)
        time.sleep(self.delay)
        return response

browser = mechanicalsoup.StatefulBrowser()
adapter = RateLimitAdapter(delay=1)  # 1 second delay
browser.session.mount('http://', adapter)
browser.session.mount('https://', adapter)

urls = ['http://example.com/page1', 'http://example.com/page2', 'http://example.com/page3']

for url in urls:
    browser.open(url)
    # Process the page...

In this example, the RateLimitAdapter adds a delay after each request is sent.

Remember to use web scraping and rate limiting responsibly. Always check a website's robots.txt file and terms of service to understand the rules and limitations of scraping their content, and respect any rate limits they specify to avoid legal issues or being banned from the site.