How can I extend Scrapy?

You can extend Scrapy by adding your own functionality through extensions, middlewares, pipelines and spiders. Each of these components has its own specific purpose and place in the Scrapy architecture.

1. Extensions:

Extensions are a way to extend Scrapy functionality by hooking into Scrapy's signals and implementing certain methods. They are instantiated and configured via settings.

Here is an example of a simple extension that counts the number of crawled pages:

from scrapy import signals

class CountPagesExtension:
    def __init__(self, stats):
        self.stats = stats

    @classmethod
    def from_crawler(cls, crawler):
        extension = cls(crawler.stats)
        crawler.signals.connect(extension.page_scraped, signal=signals.response_received)
        return extension

    def page_scraped(self, response):
        self.stats.inc_value('pages_crawled')

The extension is enabled by adding it to settings.py:

EXTENSIONS = {
    'myproject.extensions.CountPagesExtension': 500,
}

2. Middlewares:

Middlewares are a way to process Requests and Responses that pass through the Scrapy engine. You can create your own middleware to modify Requests, Responses or exceptions that pass through it.

Here is an example of a simple middleware that adds a custom header to every Request:

class CustomHeaderMiddleware:
    def process_request(self, request, spider):
        request.headers['Custom-Header'] = 'CustomValue'

The middleware is enabled by adding it to settings.py:

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.CustomHeaderMiddleware': 500,
}

3. Pipelines:

Pipelines are a way to process the items that are returned by the spiders. They are typically used to clean, validate and persist data.

Here is an example of a simple pipeline that drops items with missing fields:

class DropMissingFieldsPipeline:
    def process_item(self, item, spider):
        if item.get('missing_field') is None:
            raise DropItem("Missing field!")
        return item

The pipeline is enabled by adding it to settings.py:

ITEM_PIPELINES = {
    'myproject.pipelines.DropMissingFieldsPipeline': 500,
}

4. Spiders:

Spiders are a way to define how a site or a group of sites should be scraped, including how to perform the crawl and how to extract the data. You can extend the base Spider class to define your own spiders.

Here is an example of a simple spider that scrapes quotes from http://quotes.toscrape.com:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('span small::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

This spider can be run with the scrapy crawl command:

scrapy crawl quotes