How do I implement rate limiting in Scrapy?
Rate limiting is crucial for responsible web scraping to prevent overwhelming target servers and avoid getting blocked. Scrapy provides several built-in mechanisms and customization options to control the rate of requests sent to websites. This guide covers various approaches from basic delays to advanced throttling strategies.
Understanding Rate Limiting in Scrapy
Rate limiting controls how frequently your spider sends requests to a website. Without proper rate limiting, you risk:
- Getting blocked by anti-bot systems
- Overloading the target server
- Receiving HTTP 429 (Too Many Requests) errors
- Poor server performance for other users
Method 1: Basic Download Delay
The simplest approach is setting a fixed delay between requests using the DOWNLOAD_DELAY setting.
Global Download Delay
Add this to your settings.py file:
# settings.py
DOWNLOAD_DELAY = 2 # 2 seconds delay between requests
RANDOMIZE_DOWNLOAD_DELAY = 0.5 # Randomize delay (0.5 * to 1.5 * DOWNLOAD_DELAY)
Per-Spider Download Delay
You can also set delays specific to individual spiders:
# my_spider.py
import scrapy
class MySpider(scrapy.Spider):
name = 'rate_limited_spider'
download_delay = 3 # 3 seconds delay for this spider
randomize_download_delay = 0.5
def start_requests(self):
urls = ['https://example.com/page1', 'https://example.com/page2']
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
# Extract data here
yield {'title': response.css('title::text').get()}
Method 2: AutoThrottle Extension (Recommended)
AutoThrottle is Scrapy's intelligent rate limiting system that automatically adjusts delays based on server response times and latency.
Enabling AutoThrottle
Add these settings to your settings.py:
# settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1 # Initial delay
AUTOTHROTTLE_MAX_DELAY = 60 # Maximum delay
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Average concurrent requests
AUTOTHROTTLE_DEBUG = True # Enable to see throttling stats
# Optional: Custom concurrency settings
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AutoThrottle Configuration Options
- AUTOTHROTTLE_START_DELAY: Initial delay (in seconds)
- AUTOTHROTTLE_MAX_DELAY: Maximum delay cap
- AUTOTHROTTLE_TARGET_CONCURRENCY: Target number of concurrent requests
- AUTOTHROTTLE_DEBUG: Shows throttling statistics in logs
Advanced AutoThrottle Configuration
# settings.py - Advanced AutoThrottle setup
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 0.5
AUTOTHROTTLE_MAX_DELAY = 30
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
AUTOTHROTTLE_DEBUG = True
# Fine-tune concurrency
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_TIMEOUT = 180
Method 3: Custom Rate Limiting with Middleware
For more sophisticated rate limiting, create custom middleware:
# middlewares.py
import time
from scrapy.downloadermiddlewares.retry import RetryMiddleware
from scrapy.utils.response import response_status_message
class CustomRateLimitMiddleware:
def __init__(self, delay=1.0, randomize=False):
self.delay = delay
self.randomize = randomize
self.last_request_time = {}
@classmethod
def from_crawler(cls, crawler):
settings = crawler.settings
delay = settings.getfloat('CUSTOM_DELAY', 1.0)
randomize = settings.getbool('CUSTOM_RANDOMIZE', False)
return cls(delay, randomize)
def process_request(self, request, spider):
domain = request.url.split('/')[2]
current_time = time.time()
if domain in self.last_request_time:
elapsed = current_time - self.last_request_time[domain]
if elapsed < self.delay:
sleep_time = self.delay - elapsed
if self.randomize:
import random
sleep_time *= random.uniform(0.5, 1.5)
time.sleep(sleep_time)
self.last_request_time[domain] = time.time()
return None
Enable the middleware in settings:
# settings.py
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.CustomRateLimitMiddleware': 585,
}
CUSTOM_DELAY = 2.0
CUSTOM_RANDOMIZE = True
Method 4: Dynamic Rate Limiting Based on Response
Implement adaptive rate limiting that responds to server behavior:
# middlewares.py
class AdaptiveRateLimitMiddleware:
def __init__(self):
self.delays = {}
self.default_delay = 1.0
self.max_delay = 10.0
self.min_delay = 0.1
def process_response(self, request, response, spider):
domain = self._get_domain(request.url)
if response.status == 429: # Too Many Requests
# Increase delay for this domain
current_delay = self.delays.get(domain, self.default_delay)
new_delay = min(current_delay * 2, self.max_delay)
self.delays[domain] = new_delay
spider.logger.info(f"Rate limited! Increasing delay to {new_delay}s for {domain}")
elif response.status == 200:
# Gradually decrease delay on successful requests
current_delay = self.delays.get(domain, self.default_delay)
new_delay = max(current_delay * 0.9, self.min_delay)
self.delays[domain] = new_delay
return response
def process_request(self, request, spider):
domain = self._get_domain(request.url)
delay = self.delays.get(domain, self.default_delay)
if delay > self.min_delay:
time.sleep(delay)
return None
def _get_domain(self, url):
return url.split('/')[2]
Method 5: Per-Domain Rate Limiting
Different websites may require different rate limiting strategies:
# spiders/multi_domain_spider.py
import scrapy
class MultiDomainSpider(scrapy.Spider):
name = 'multi_domain'
custom_settings = {
'AUTOTHROTTLE_ENABLED': True,
'AUTOTHROTTLE_START_DELAY': 1,
'AUTOTHROTTLE_MAX_DELAY': 10,
'AUTOTHROTTLE_TARGET_CONCURRENCY': 1.0,
}
def start_requests(self):
# Different delays for different domains
domains = {
'https://fast-site.com': 0.5,
'https://slow-site.com': 3.0,
'https://strict-site.com': 5.0,
}
for base_url, delay in domains.items():
meta = {'download_delay': delay}
yield scrapy.Request(
url=f"{base_url}/page1",
callback=self.parse,
meta=meta
)
def parse(self, response):
# Process response
yield {'url': response.url, 'title': response.css('title::text').get()}
Method 6: Time-Based Rate Limiting
Implement time-window-based rate limiting (e.g., max 10 requests per minute):
# middlewares.py
import time
from collections import defaultdict, deque
class TimeWindowRateLimitMiddleware:
def __init__(self, max_requests=10, time_window=60):
self.max_requests = max_requests
self.time_window = time_window
self.request_times = defaultdict(deque)
def process_request(self, request, spider):
domain = request.url.split('/')[2]
current_time = time.time()
domain_requests = self.request_times[domain]
# Remove old requests outside the time window
while domain_requests and domain_requests[0] <= current_time - self.time_window:
domain_requests.popleft()
# Check if we've exceeded the rate limit
if len(domain_requests) >= self.max_requests:
# Calculate how long to wait
oldest_request = domain_requests[0]
wait_time = self.time_window - (current_time - oldest_request)
spider.logger.info(f"Rate limit reached for {domain}. Waiting {wait_time:.2f}s")
time.sleep(wait_time)
# Record this request
domain_requests.append(current_time)
return None
Best Practices for Rate Limiting
1. Start Conservative
Begin with longer delays and gradually optimize:
# Conservative starting point
DOWNLOAD_DELAY = 3
AUTOTHROTTLE_START_DELAY = 2
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
2. Monitor Server Response
Check server response times and adjust accordingly:
# Enable detailed logging
AUTOTHROTTLE_DEBUG = True
LOG_LEVEL = 'INFO'
# Monitor response times in your spider
def parse(self, response):
download_latency = response.meta.get('download_latency', 0)
if download_latency > 5: # If response took more than 5 seconds
self.logger.warning(f"Slow response: {download_latency}s for {response.url}")
3. Respect robots.txt
Many websites specify crawl delays in their robots.txt:
# settings.py
ROBOTSTXT_OBEY = True # This will respect Crawl-delay directive
4. Handle Rate Limit Responses
Properly handle 429 and other rate limiting responses:
# settings.py
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]
RETRY_TIMES = 3
# Custom retry middleware for 429 responses
class RateLimitRetryMiddleware(RetryMiddleware):
def process_response(self, request, response, spider):
if response.status == 429:
retry_after = response.headers.get('Retry-After')
if retry_after:
delay = int(retry_after)
spider.logger.info(f"Rate limited. Retrying after {delay} seconds")
time.sleep(delay)
return super().process_response(request, response, spider)
Testing Rate Limiting
Test your rate limiting configuration:
# Run with debug info
scrapy crawl myspider -s AUTOTHROTTLE_DEBUG=True -L INFO
# Monitor request frequency
scrapy crawl myspider -s LOG_LEVEL=DEBUG | grep "download_delay"
Common Rate Limiting Scenarios
E-commerce Sites
DOWNLOAD_DELAY = 2
RANDOMIZE_DOWNLOAD_DELAY = 0.5
CONCURRENT_REQUESTS_PER_DOMAIN = 1
News Websites
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
APIs with Rate Limits
# For APIs with explicit rate limits (e.g., 100 requests/hour)
DOWNLOAD_DELAY = 36 # 3600 seconds / 100 requests = 36 seconds
Rate limiting is essential for sustainable web scraping. While tools like Puppeteer handle authentication flows differently, Scrapy's built-in throttling mechanisms provide robust solutions for controlling request rates. Combined with proper error handling strategies, these techniques ensure reliable and respectful web scraping operations.
Remember that effective rate limiting balances scraping efficiency with server respect, helping maintain long-term access to target websites while avoiding detection and blocks.
