What are Scrapy middlewares and how do I use them?

Scrapy middlewares are a way of extending Scrapy functionality through custom processing. They are essentially hooks that allow you to insert custom processing or handle different aspects of the request/response processing.

Middlewares are organized in a stack and the order in which they are processed is determined by their order in this stack. They can process requests before they are sent to the downloader, responses before they are sent to the spider, and also the items that a spider returns.

Middlewares are specified in the DOWNLOADER_MIDDLEWARES setting, which is a dictionary where keys are the middleware paths and the values are the middleware orders.

There are three types of middlewares in Scrapy:

1. Downloader Middleware: It is used for processing requests and responses; it sits between Scrapy’s request/response processing.

2. Spider Middleware: It is used for processing spider input (responses) and output (items and requests).

3. Extension Middleware: It extends Scrapy with custom functionality.

How to Use Scrapy Middleware

To create a custom Scrapy middleware, you need to define a class and implement methods that will handle requests, responses, and exceptions.

Here is an example of a custom downloader middleware in Python:

class CustomDownloaderMiddleware:
    def process_request(self, request, spider):
        request.meta['proxy'] = "http://PROXY:PORT"
        return None

    def process_response(self, request, response, spider):
        if response.status != 200:
            return request
        return response

    def process_exception(self, request, exception, spider):
        pass

In the process_request method, we set a proxy for the request. In the process_response method, we check if the response status is different from 200 (HTTP OK), and if it is, we return the request to be rescheduled. The process_exception method is called when an exception occurs while downloading a request.

After creating your middleware, you have to add it to the DOWNLOADER_MIDDLEWARES setting in your project’s settings:

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.CustomDownloaderMiddleware': 543,
}

The number 543 is the order in which the middleware is processed. Middlewares with lower numbers are processed first.

Conclusion

Scrapy middleware allows you to add custom processing to your Scrapy project. They can handle requests before they are sent to the downloader, responses before they are sent to the spider, and items before they are sent to the item pipeline. By understanding how to use middlewares, you can extend Scrapy to handle a wide variety of tasks.