---
title: "Scrapy Web Scraping: The Complete Guide"
description: "How to scrape with Scrapy: spiders, selectors, items, pipelines, the settings that matter, middlewares, proxies, JavaScript rendering, and deployment."
url: https://webscraping.ai/blog/scrapy-web-scraping
markdown_index: https://webscraping.ai/llms.txt
---
# Scrapy Web Scraping: The Complete Guide

Scrapy is the framework you graduate to when a scraping script stops being a script. It gives you an asynchronous crawl engine, a scheduler, deduplication, retries, throttling, and a data pipeline — features you would otherwise write badly by hand somewhere around page five thousand. This guide covers the whole workflow: project layout, spiders, selectors, items and pipelines, the settings that actually change outcomes, middlewares, proxies, JavaScript rendering, scaling, and deployment.

## Key Takeaways

- Scrapy is a crawling _framework_, not a parsing library — it competes with a whole hand-rolled crawler, not with Beautiful Soup
- `scrapy startproject` gives you the layout; `scrapy genspider` gives you a spider; `scrapy crawl name -O out.json` gives you data
- Requests are deduplicated by fingerprint automatically, retried automatically, and throttled by AutoThrottle — the three things people most often rebuild by accident
- `CONCURRENT_REQUESTS` and `AUTOTHROTTLE_ENABLED` are the two settings that decide whether a crawl is fast, blocked, or rude
- Scrapy does not run JavaScript; `scrapy-playwright` adds a real browser per request, and finding the site's own JSON API is usually better than either
- Anti-bot walls and proxy fleets are infrastructure problems, not spider-code problems

## What is Scrapy?

Scrapy is an open-source web crawling and scraping framework for Python, maintained by Zyte. Current releases require Python 3.10 or newer. Underneath it runs on Twisted, an asynchronous networking engine, which is why a single Scrapy process can keep dozens of requests in flight without you writing any concurrency code.

What you get out of the box, and what each piece would otherwise cost you:

- **A scheduler with a duplicate filter** — every request is fingerprinted and re-requests are dropped
- **A retry middleware** — transient failures and 5xx responses are retried with no code from you
- **AutoThrottle** — concurrency adapts to how fast the target server is actually responding
- **Selectors** — CSS and XPath over `parsel`/`lxml`, on every response object
- **Item pipelines** — a chain of processing stages between "parsed" and "stored"
- **Feed exports** — JSON, JSON Lines, CSV, and XML to a file, S3, or GCS with one flag
- **`scrapy shell`** — an interactive REPL against a live response, which is the fastest selector debugger in Python

The tradeoff is that Scrapy has opinions. You write callbacks and yield objects rather than calling functions in a loop, and the framework decides when things run. For a five-page scrape that structure is overhead. For a hundred-thousand-page crawl it is the reason the job finishes.

## Scrapy vs requests + Beautiful Soup vs Playwright

These tools are usually presented as rivals. They are better understood as three different jobs.

| | Scrapy | requests + Beautiful Soup | Playwright / Selenium |
| --- | --- | --- | --- |
| What it is | Crawling framework | HTTP client + HTML parser | Browser automation |
| Runs JavaScript | No (add-on required) | No | Yes |
| Concurrency | Async, built in | Manual (threads/asyncio) | Heavy — one browser per worker |
| Retries, dedup, throttling | Built in | You write them | You write them |
| Speed per page | Very fast | Very fast | 10–50× slower |
| RAM per concurrent page | A few MB | A few MB | 100–400 MB |
| Learning curve | Steep | Flat | Moderate |
| Best for | Large crawls, recurring jobs | One-off scripts, a few hundred pages | JS-rendered pages, interaction |

The honest decision rule:

- **A handful of pages, run once?** Use `requests` and Beautiful Soup. See our [Python requests guide](https://webscraping.ai/blog/python-requests-guide) and [Beautiful Soup guide](https://webscraping.ai/blog/beautiful-soup-python).
- **Thousands of pages, link-following, or a job that runs on a schedule?** Use Scrapy. The framework overhead pays for itself the first time a crawl dies halfway through and you need to resume it.
- **Data only appears after scripts run?** You need a browser — either standalone [Playwright](https://webscraping.ai/blog/playwright-web-scraping), or Playwright bolted into Scrapy (covered below).

They also combine. Scrapy for fetching and crawl management, Beautiful Soup for a parsing step you already have working, Playwright for the handful of pages that need rendering. Nothing forces you to pick one. Our [Python web scraping libraries](https://webscraping.ai/blog/python-web-scraping-libraries) comparison covers the wider ecosystem, and [web scraping with Python](https://webscraping.ai/blog/web-scraping-with-python) walks the fundamentals if you are earlier in the journey.

## Installing Scrapy

Install into a virtual environment — Scrapy pulls in Twisted, lxml, and cryptography, and you do not want those fighting with your system Python:

```bash
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install scrapy
scrapy version
```

On Linux, if the wheels fall back to a source build, install the headers first:

```bash
sudo apt-get install python3-dev build-essential libxml2-dev libxslt1-dev libssl-dev libffi-dev
```

Conda users can take the conda-forge build instead, which avoids compilation entirely:

```bash
conda install -c conda-forge scrapy
```

Windows generally installs from wheels without incident; if a dependency does try to compile, install the Microsoft C++ Build Tools or switch to conda. Pin the version in `requirements.txt` (`scrapy==2.19.0` or whatever you tested against) so a minor release never surprises a production crawl.

## Creating a project

Scrapy projects have a fixed layout, and the framework relies on it:

```bash
scrapy startproject bookstore
cd bookstore
scrapy genspider books books.toscrape.com
```

```text
bookstore/
├── scrapy.cfg # deployment config — marks the project root
└── bookstore/
    ├── items.py # data schemas
    ├── middlewares.py # request/response hooks
    ├── pipelines.py # post-parse processing
    ├── settings.py # everything configurable
    └── spiders/
        └── books.py # your spiders
```

You can run a spider without a project (a single `.py` file with `scrapy runspider file.py`), which is handy for experiments. Anything you intend to keep should be a project — pipelines, settings, and deployment all assume it.

## Your first spider

A spider defines where to start, and what to do with each response:

```python
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css("p.price_color::text").get(),
                "in_stock": "In stock" in book.css("p.availability::text").get(""),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)
```

Run it and write the results:

```bash
scrapy crawl books -O books.json # -O overwrites, -o appends
scrapy crawl books -O books.jsonl # JSON Lines — better for large crawls
scrapy crawl books -O books.csv
```

Three things in that spider carry most of the framework's weight. `yield` on a dict hands an item to the pipeline. `response.follow()` resolves relative URLs against the current page and produces a new request. And returning that request rather than fetching it means the scheduler — not your code — decides when it runs, which is what makes the crawl concurrent.

### `start_urls`, `start()`, and the old `start_requests()`

`start_urls` is the shortcut. When you need headers, cookies, POST bodies, or a different callback for the first requests, override the async `start()` hook instead:

```python
class BooksSpider(scrapy.Spider):
    name = "books"

    async def start(self):
        for page in range(1, 51):
            yield scrapy.Request(
                f"https://books.toscrape.com/catalogue/page-{page}.html",
                callback=self.parse,
                headers={"Accept-Language": "en-US"},
                meta={"page": page},
            )
```

Scrapy 2.13 introduced `start()` and deprecated `start_requests()`; Scrapy 2.16 removed `start_requests()` entirely, so on current releases a spider that only defines `start_requests()` falls back to the default `start()`, which just reads `start_urls` — and crawls nothing if that is empty. You will still meet the old form in most existing code and tutorials — rename it to `async def start(self)`, or define both if you must support Scrapy older than 2.13. Because `start()` is async, you can `await` inside it, which helps when seed URLs come from a database or an API:

```python
async def start(self):
    async for url in self.seed_urls():
        yield scrapy.Request(url, callback=self.parse)
```

## Selectors

Every response carries `.css()` and `.xpath()`, and they return selector lists you can chain:

```python
response.css("h1::text").get() # first match, or None
response.css("h1::text").get(default="") # first match with a fallback
response.css(".price::text").getall() # every match, as a list
response.css("a::attr(href)").getall() # attribute values
response.css("a").attrib["href"] # attributes of the first match

response.xpath("//h1/text()").get()
response.xpath("//div[@class='product']//span[@class='price']/text()").getall()

# Chaining — scope a sub-selector to a parent
for card in response.css("div.product"):
    name = card.css("h2::text").get()
    price = card.xpath(".//span[@class='price']/text()").get() # note the leading dot
```

Two habits worth forming early. Use `.get()` rather than `[0]` — a missing element gives you `None` instead of an `IndexError` that kills the callback. And when chaining XPath inside a loop, start the expression with `.` — a bare `//` searches the whole document again and will silently return the first product's price for every card.

`re()` and `re_first()` pull values out of matched text without a separate parsing step:

```python
response.css("p.stock::text").re_first(r"(\d+) available")
```

For anything non-trivial, develop selectors in the shell rather than by re-running the spider:

```bash
scrapy shell "https://books.toscrape.com/"
>>> response.css("article.product_pod h3 a::attr(title)").getall()
>>> view(response) # opens Scrapy's copy of the page in a browser
```

`view(response)` is the honest one — it shows you what Scrapy received, not what your browser renders. If the elements you want are missing there, you have a JavaScript problem, not a selector problem. Our [XPath cheat sheet](https://webscraping.ai/blog/xpath-cheat-sheet) is a useful companion when CSS runs out of expressiveness.

## Spider vs CrawlSpider

`scrapy.Spider` gives you explicit control: you decide which links to follow, in code. `CrawlSpider` follows links declaratively from a rule set, which suits broad site crawls where the pattern is regular:

```python
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class BooksCrawlSpider(CrawlSpider):
    name = "books_crawl"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    rules = (
        # Follow category and pagination links, don't parse them
        Rule(LinkExtractor(allow=r"catalogue/category/")),
        # Parse product pages, don't follow links out of them
        Rule(LinkExtractor(allow=r"catalogue/[\w-]+_\d+/index\.html"),
             callback="parse_book", follow=False),
    )

    def parse_book(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "price": response.css("p.price_color::text").get(),
            "upc": response.css("table tr:nth-child(1) td::text").get(),
        }
```

Older tutorials warn never to name a `CrawlSpider` callback `parse`, because early Scrapy versions applied the rules inside `parse` itself. Current releases apply them in an internal `_parse` method, so that warning is obsolete; the rule Scrapy's docs still stress is to set an explicit `callback` on any extra request you yield from a `CrawlSpider`.

Reach for `CrawlSpider` when URL patterns describe the site well. Reach for a plain `Spider` when navigation depends on page content — pagination tokens, "load more" cursors, or a search form you have to submit.

## Items and pipelines

Yielding dicts is fine to start. Items give you a declared schema, so a typo becomes an error instead of a silently missing column:

```python
# items.py
import scrapy

class BookItem(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()
    in_stock = scrapy.Field()
    scraped_at = scrapy.Field()
```

Pipelines process every item after parsing, in the order given by their priority number (lower runs first):

```python
# pipelines.py
from datetime import datetime, timezone

import psycopg
from scrapy.exceptions import DropItem

class CleanPricePipeline:
    def process_item(self, item):
        raw = item.get("price")
        if not raw:
            raise DropItem("Missing price")
        item["price"] = float(raw.replace("£", "").strip())
        item["scraped_at"] = datetime.now(timezone.utc).isoformat()
        return item

class PostgresPipeline:
    def __init__ (self, dsn):
        self.dsn = dsn

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings["DATABASE_URL"])

    def open_spider(self):
        self.conn = psycopg.connect(self.dsn)

    def process_item(self, item):
        with self.conn.cursor() as cur:
            cur.execute(
                "INSERT INTO books (title, price) VALUES (%s, %s) "
                "ON CONFLICT (title) DO UPDATE SET price = EXCLUDED.price",
                (item["title"], item["price"]),
            )
        self.conn.commit()
        return item

    def close_spider(self):
        self.conn.close()
```

```python
# settings.py
ITEM_PIPELINES = {
    "bookstore.pipelines.CleanPricePipeline": 300,
    "bookstore.pipelines.PostgresPipeline": 800,
}
```

`open_spider` and `close_spider` are where connections belong — opening a database connection inside `process_item` gives you one connection per item. Note the signatures: since Scrapy 2.14 (and 2.18 for `open_spider`/`close_spider`) the `spider` argument these methods used to take is deprecated; read settings in `from_crawler` instead. Raising `DropItem` discards an item without stopping the crawl, which is the right way to enforce validation.

### Writing into Django models

If the scraped data belongs to an existing Django app, you can use its ORM from a pipeline by setting up Django before Scrapy touches the models:

```python
# settings.py (Scrapy)
import os, sys, django

sys.path.insert(0, "/path/to/django_project")
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "django_project.settings")
django.setup()
```

```python
# pipelines.py
from myapp.models import Book

class DjangoPipeline:
    def process_item(self, item):
        Book.objects.update_or_create(
            title=item["title"],
            defaults={"price": item["price"]},
        )
        return item
```

`update_or_create` rather than `create` is deliberate: a recurring crawl will re-encounter the same records, and it keeps the pipeline idempotent. The `django.setup()` call must run before any model import, which is why it lives in Scrapy's `settings.py` rather than in the pipeline module.

## The settings that actually matter

`settings.py` has dozens of knobs. These are the ones that change outcomes:

```python
# Politeness and speed
CONCURRENT_REQUESTS = 16 # total in-flight requests
CONCURRENT_REQUESTS_PER_DOMAIN = 8 # per-domain cap — the one that matters for politeness
DOWNLOAD_DELAY = 0.5 # seconds between requests to a domain
DOWNLOAD_DELAY_JITTER = 0.5 # ± 50% random variation (the default)

# AutoThrottle — adaptive concurrency
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
AUTOTHROTTLE_DEBUG = False # True prints the delay it picks per response

# Reliability
DOWNLOAD_TIMEOUT = 30
RETRY_ENABLED = True
RETRY_TIMES = 3

# Manners and identity
ROBOTSTXT_OBEY = True
USER_AGENT = "mybot (+https://example.com/about-our-crawler)"

# Caching during development — never re-fetch while iterating on selectors
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 3600
```

`AUTOTHROTTLE_ENABLED` deserves a note, because it is the setting that most often stops a crawl from getting blocked. Rather than a fixed delay, Scrapy measures the target's response latency and adjusts its delay to hold roughly `AUTOTHROTTLE_TARGET_CONCURRENCY` requests in flight. When the server slows down — the first symptom of you being too aggressive — the crawler backs off automatically. It is slower than a flat-out crawl by design, and that is the point.

`DOWNLOAD_DELAY` and AutoThrottle coexist: `DOWNLOAD_DELAY` becomes the floor AutoThrottle will not go below. `HTTPCACHE_ENABLED` is a development setting worth switching on the moment you start iterating on selectors — it turns a fifty-page crawl into a local replay and takes the load off the site you are working against.

Any of these can be overridden per spider:

```python
class BooksSpider(scrapy.Spider):
    name = "books"
    custom_settings = {
        "DOWNLOAD_DELAY": 2.0,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2,
    }
```

Or on the command line for a one-off run: `scrapy crawl books -s DOWNLOAD_DELAY=2`.

## Duplicate filtering

Scrapy deduplicates requests for you. Every request gets a fingerprint derived from its method, canonicalized URL, and body, and the scheduler drops any request whose fingerprint it has already seen. This is why a crawl that follows "next page" links in both directions does not loop forever.

Three things you will eventually need to control.

**Bypassing the filter for a single request.** Login pages, polling endpoints, and pagination cursors that reuse a URL need `dont_filter=True`:

```python
yield scrapy.Request(url, callback=self.parse, dont_filter=True)
```

**Seeing what got filtered.** Set `DUPEFILTER_DEBUG = True` and every dropped request is logged rather than counted silently. The `dupefilter/filtered` stat at the end of a run is the summary number:

```python
def closed(self, reason):
    self.logger.info("Filtered %s duplicates",
                     self.crawler.stats.get_value("dupefilter/filtered", 0))
```

**Persisting the filter across runs.** By default the fingerprint set lives in memory and dies with the process. `JOBDIR` writes the scheduler queue and the seen-fingerprints set to disk, which both survives a restart and lets you resume an interrupted crawl:

```bash
scrapy crawl books -s JOBDIR=crawls/books-1
```

Stop that run with a single Ctrl-C (a second one kills it uncleanly) and the same command picks up where it left off.

If the default fingerprint is wrong for your target — say the site appends tracking parameters that do not change the content — subclass the fingerprinter rather than the whole filter:

```python
# dupefilters.py
from w3lib.url import url_query_cleaner
from scrapy.utils.request import RequestFingerprinter

class CleanURLFingerprinter(RequestFingerprinter):
    def fingerprint(self, request):
        cleaned = url_query_cleaner(
            request.url, ["utm_source", "utm_medium", "utm_campaign"], remove=True
        )
        return super().fingerprint(request.replace(url=cleaned))
```

```python
# settings.py
REQUEST_FINGERPRINTER_CLASS = "bookstore.dupefilters.CleanURLFingerprinter"
```

Older tutorials call `scrapy.utils.request.request_fingerprint()` directly. That function is gone from current Scrapy; the fingerprinter component above replaced it, and inside a spider or middleware you reach it as `self.crawler.request_fingerprinter.fingerprint(request)`.

**Request dedup is not item dedup.** Two different URLs can yield the same product. That is a pipeline's job:

```python
class DedupePipeline:
    def open_spider(self):
        self.seen = set()

    def process_item(self, item):
        key = item["upc"]
        if key in self.seen:
            raise DropItem(f"Duplicate item: {key}")
        self.seen.add(key)
        return item
```

An in-memory set is fine up to a few million items. Beyond that, use the database's unique constraint (as in the `ON CONFLICT` pipeline above) or a Redis set, and let the storage layer be the source of truth.

## Retries, errors, and timeouts

`RetryMiddleware` is enabled by default and handles the common cases before you write a line:

```python
# settings.py
RETRY_ENABLED = True
RETRY_TIMES = 3 # retries after the first attempt
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429]
```

Connection errors, DNS failures, and timeouts are retried too — those are matched by exception type, not status code. Note what is _not_ in the list: 400, 401, 403, and 404 are permanent, and retrying them wastes your rate budget while making you look more like a bot.

For anything a status code cannot express, subclass the middleware. The most common real case is a server that returns HTTP 200 with a challenge or "too many requests" page, which no status-code list can catch:

```python
# middlewares.py
from scrapy.downloadermiddlewares.retry import RetryMiddleware, get_retry_request

class SmartRetryMiddleware(RetryMiddleware):
    def process_response(self, request, response):
        if b"Just a moment" in response.body or b"rate limit" in response.body.lower():
            return get_retry_request(request, spider=self.crawler.spider,
                                     reason="soft-block") or response
        return super().process_response(request, response)
```

```python
# settings.py — replace the default at its own priority slot
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": None,
    "bookstore.middlewares.SmartRetryMiddleware": 550,
}
```

`get_retry_request()` is the supported helper: it increments the retry counter, respects `RETRY_TIMES`, updates the stats, and returns `None` once the budget is exhausted.

One gap to know about: Scrapy does not honour the `Retry-After` header automatically. A 429 is retried like any other listed status, on the normal delay schedule, ignoring however long the server asked you to wait. The practical answer is AutoThrottle plus a `DOWNLOAD_DELAY` floor that keeps you out of 429 territory in the first place, rather than a middleware that tries to sleep — blocking inside a middleware stalls the whole reactor, not just that request.

### Handling failures in the spider

Retries eventually run out. `errback` catches what is left, so one dead URL does not vanish without trace:

```python
from scrapy.spidermiddlewares.httperror import HttpError
from twisted.internet.error import TimeoutError, DNSLookupError

class BooksSpider(scrapy.Spider):
    name = "books"
    handle_httpstatus_list = [404] # let 404s reach parse() instead of being dropped

    async def start(self):
        for url in self.urls:
            yield scrapy.Request(url, callback=self.parse, errback=self.on_error)

    def on_error(self, failure):
        request = failure.request
        if failure.check(HttpError):
            self.logger.error("HTTP %s on %s", failure.value.response.status, request.url)
        elif failure.check(DNSLookupError):
            self.logger.error("DNS failure on %s", request.url)
        elif failure.check(TimeoutError):
            self.logger.error("Timeout on %s", request.url)
        else:
            self.logger.error("Unhandled %r on %s", failure.value, request.url)
```

Exceptions raised inside `parse()` are logged and the response is dropped — they do not stop the crawl, which is convenient right up until you notice thousands of them scrolling past. Check `spider_exceptions` in the end-of-run stats before you trust a crawl's output.

### Timeouts

`DOWNLOAD_TIMEOUT` (default 180 seconds) is the ceiling for a whole download, and it is set at three levels:

```python
# Globally
DOWNLOAD_TIMEOUT = 30

# Per spider
class BooksSpider(scrapy.Spider):
    custom_settings = {"DOWNLOAD_TIMEOUT": 15}

# Per request — the meta key wins over both
yield scrapy.Request(url, meta={"download_timeout": 10})
```

The subtlety worth knowing: this is a timeout on the complete response, not on time-to-first-byte. A server that dribbles out a large page over 40 seconds will trip a 30-second timeout even though it never actually stalled. If large downloads are timing out but small ones are fine, raise the value rather than assuming the target is slow. Timeouts surface as `twisted.internet.error.TimeoutError`, are retried by `RetryMiddleware` like any other transient failure, and reach your `errback` once retries are exhausted.

## Middlewares

Middlewares are how you modify requests and responses globally. There are two kinds, and mixing them up is a common first-week mistake:

- **Downloader middlewares** sit between the engine and the network. Proxies, headers, user agents, retries, caching, and browser integrations all live here.
- **Spider middlewares** sit between the engine and your spider callbacks. Filtering items, handling exceptions from callbacks, and processing the output of `parse()` live here.

A downloader middleware has three hooks. Returning `None` from `process_request` or `process_exception` passes control to the next middleware; returning a `Response` or `Request` short-circuits the chain. `process_response` must always return a `Response` or `Request` (or raise):

```python
# middlewares.py
class HeaderMiddleware:
    def process_request(self, request):
        request.headers["Accept-Language"] = "en-US,en;q=0.9"

    def process_response(self, request, response):
        return response # or a Request to reschedule, or raise IgnoreRequest

    def process_exception(self, request, exception):
        return None
```

Since Scrapy 2.14, declaring the old trailing `spider` argument on these hooks is deprecated and triggers a warning — if you need the spider, keep a reference to the crawler in `from_crawler` and use `self.crawler.spider`.

The numbers in `DOWNLOADER_MIDDLEWARES` are ordering, not identity. Lower numbers process requests earlier and responses _later_ — the chain is walked outward on the way in and back inward on the way out. Setting a built-in middleware to `None` disables it, which is how you replace one with your own.

## Proxies and user agents

Scrapy reads the standard `http_proxy`/`https_proxy` environment variables through `HttpProxyMiddleware`. For per-request control, set the `proxy` meta key:

```python
yield scrapy.Request(url, meta={"proxy": "http://user:pass@proxy.example.com:8080"})
```

Rotation is a small middleware:

```python
# middlewares.py
import random

class ProxyRotationMiddleware:
    def __init__ (self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_LIST"))

    def process_request(self, request):
        if "proxy" not in request.meta:
            request.meta["proxy"] = random.choice(self.proxies)

    def process_exception(self, request, exception):
        # Retry through a different proxy on connection failure
        new_request = request.copy()
        new_request.meta["proxy"] = random.choice(self.proxies)
        new_request.dont_filter = True
        return new_request
```

User-agent rotation works the same way, though it matters less than it used to — modern anti-bot systems fingerprint TLS handshakes and header _ordering_, not just the UA string. A rotating user agent paired with a static datacenter IP and a Python TLS fingerprint fools very little. See [user agent rotation](https://webscraping.ai/blog/user-agent-rotation-for-web-scraping) for what it does and does not buy, and [types of proxies](https://webscraping.ai/blog/types-of-proxies-for-web-scraping) for the residential-versus-datacenter tradeoff.

`from_crawler` is the standard constructor hook for every Scrapy component — middlewares, pipelines, extensions — and it is how you read settings without importing them. Older code uses `from_settings`; it was deprecated in Scrapy 2.12 and is no longer called since 2.14, so port it to `from_crawler`.

## JavaScript-rendered pages

Scrapy speaks HTTP. It does not run JavaScript, so a React storefront that renders its catalogue client-side gives you an empty shell. `scrapy shell <url>` plus `view(response)` tells you within seconds whether this is your problem.

There are three answers, in ascending order of cost.

### Find the underlying API first

Most JavaScript-rendered pages fetch their data from a JSON endpoint. Open the browser's Network tab, filter to XHR/Fetch, and look for the request that carries the data. When you find it, you get clean structured data with none of the rendering cost:

```python
import json

def parse(self, response):
    data = response.json()
    for product in data["products"]:
        yield {"name": product["title"], "price": product["price"]["amount"]}

    if data.get("next_cursor"):
        yield scrapy.Request(
            f"https://example.com/api/products?cursor={data['next_cursor']}",
            callback=self.parse,
        )
```

This is worth ten minutes of investigation on every JS-heavy target. A JSON endpoint is faster, more stable across redesigns, and often paginates in larger chunks than the visible UI.

### scrapy-playwright

When there is no usable API, `scrapy-playwright` runs a real Chromium per request and hands the rendered HTML back to your normal callbacks:

```bash
pip install scrapy-playwright
playwright install chromium
```

```python
# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAYWRIGHT_BROWSER_TYPE = "chromium"
PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": True}
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 30_000
```

```python
from scrapy_playwright.page import PageMethod

class JsSpider(scrapy.Spider):
    name = "js"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/products",
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("wait_for_selector", ".product-card"),
                    PageMethod("evaluate", "window.scrollBy(0, document.body.scrollHeight)"),
                    PageMethod("wait_for_timeout", 1000),
                ],
            },
        )

    def parse(self, response):
        for card in response.css(".product-card"):
            yield {"name": card.css("h2::text").get()}
```

The asyncio reactor is required — it is the default in recent Scrapy releases, but set it explicitly if you are on an older version or you will get a confusing reactor-already-installed error. Two practical notes: turn `playwright: True` on per request rather than globally, so only the pages that need a browser pay for one; and each concurrent page is a real Chromium tab using 100–400 MB, so `CONCURRENT_REQUESTS` needs to come down hard when Playwright is in the mix. Our [Playwright guide](https://webscraping.ai/blog/playwright-web-scraping) covers the browser API itself in depth.

### scrapy-splash (legacy)

`scrapy-splash` drives Splash, a scriptable headless browser that runs as a separate service, usually in Docker. You will find it in a lot of older tutorials and existing codebases, and it still works. For new projects, prefer `scrapy-playwright`: Splash uses an older WebKit engine that struggles with modern JavaScript, its Lua scripting layer is a language boundary you have to learn, and it needs a service to operate alongside your crawler. If you have inherited a working Splash setup, there is no urgency to migrate — just do not start there.

## Cookies, sessions, and login

Cookies are handled automatically. `CookiesMiddleware` keeps a jar per crawl and sends cookies back the way a browser would, so a login persists across subsequent requests with no code:

```python
COOKIES_ENABLED = True # default
COOKIES_DEBUG = True # logs every cookie sent and received
```

A form login is a single helper call. `FormRequest.from_response()` is deprecated since Scrapy 2.16; the replacement Scrapy's docs point to is the [form2request](https://pypi.org/project/form2request/) library (`pip install form2request`):

```python
from form2request import form2request

def parse_login_page(self, response):
    request_data = form2request(
        response.css("form"),
        data={"username": "user", "password": "pass"},
    )
    yield request_data.to_scrapy(callback=self.after_login)

def after_login(self, response):
    if "authentication failed" in response.text.lower():
        self.logger.error("Login failed")
        return
    yield response.follow("/dashboard", callback=self.parse_dashboard)
```

`form2request` reads the form out of the page, which means hidden CSRF tokens are carried over automatically — the reason it is worth using over a hand-built POST.

For concurrent independent sessions, `cookiejar` in meta keeps them isolated:

```python
yield scrapy.Request(url, meta={"cookiejar": user_id})
```

If a site sets cookies from JavaScript rather than `Set-Cookie` headers, Scrapy will not see them — that is a rendering problem, and the answer is the previous section.

## Parallelism and scaling

Scrapy is already concurrent inside one process. The engine keeps up to `CONCURRENT_REQUESTS` in flight, and because Twisted is asynchronous rather than threaded, that costs you very little memory. Before reaching for anything more elaborate, tune what you have:

```python
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 16 # raise only if you own the target
CONCURRENT_ITEMS = 100 # pipeline concurrency
REACTOR_THREADPOOL_MAXSIZE = 20 # DNS resolution and other blocking calls
```

For a single-domain crawl this is almost always the limit that matters, and it is a politeness limit, not a technical one. A crawl that is too slow because of `CONCURRENT_REQUESTS_PER_DOMAIN` is usually a crawl that should stay slow.

**Multiple domains in one process** scale naturally — Scrapy queues per domain, so a crawl across a thousand sites at 8 concurrent requests each keeps the engine saturated without hammering anyone.

**Multiple processes** are the next step, and the simplest split is by URL range. Each process needs its own `JOBDIR`, or they will fight over the same queue files:

```bash
scrapy crawl books -a start=1 -a end=1000 -s JOBDIR=crawls/shard-1 &
scrapy crawl books -a start=1001 -a end=2000 -s JOBDIR=crawls/shard-2 &
wait
```

**Truly distributed crawling** — many machines sharing one frontier — needs a shared queue and a shared duplicate filter. `scrapy-redis` is the established option:

```python
# settings.py
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_PERSIST = True
REDIS_URL = "redis://localhost:6379"
```

```python
from scrapy_redis.spiders import RedisSpider

class DistributedSpider(RedisSpider):
    name = "distributed"
    redis_key = "distributed:start_urls"
```

Every worker pulls from the same Redis queue and checks the same fingerprint set, so you can add and remove machines mid-crawl. Seed it with `redis-cli lpush distributed:start_urls "https://example.com"`.

Two scaling failure modes worth naming in advance. Memory grows with the scheduler queue, not with pages crawled — a broad crawl that discovers links faster than it consumes them will exhaust RAM, and `MEMUSAGE_LIMIT_MB` will stop the spider before the OOM killer does. And the bottleneck often turns out to be a pipeline, not the network: a synchronous database write in `process_item` blocks the whole reactor. Batch the writes or move them off the reactor thread.

## Files and PDFs

`FilesPipeline` downloads linked files, deduplicates them by content hash, and records the results on the item:

```python
# settings.py
ITEM_PIPELINES = {"scrapy.pipelines.files.FilesPipeline": 1}
FILES_STORE = "downloads"
```

```python
class ReportItem(scrapy.Item):
    file_urls = scrapy.Field() # the pipeline reads this field
    files = scrapy.Field() # and writes results here

def parse(self, response):
    yield ReportItem(file_urls=response.css("a[href$='.pdf']::attr(href)").getall())
```

The field names are fixed unless you override `FILES_URLS_FIELD` and `FILES_RESULT_FIELD`. `ImagesPipeline` works the same way with thumbnailing on top.

Text extraction is a separate step, and Scrapy has no opinion about it — do it in a pipeline with PyMuPDF (fast, good with tables), `pypdf` (pure Python, no build step), or `pdfplumber` (best on complex layouts):

```python
import fitz # PyMuPDF

class PdfTextPipeline:
    def process_item(self, item):
        for f in item.get("files", []):
            with fitz.open(f"downloads/{f['path']}") as doc:
                item["text"] = "\n".join(page.get_text() for page in doc)
        return item
```

Large PDFs will exhaust memory if you hold several open at once — keep pipeline concurrency low for these items.

## Deployment

**Docker** is the baseline. Scrapy containers are unremarkable — the only real requirement is unbuffered output, or your logs will appear in unhelpful bursts:

```dockerfile
FROM python:3.12-slim

ENV PYTHONUNBUFFERED=1
WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
      build-essential libxml2-dev libxslt1-dev \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
RUN useradd -m scraper && chown -R scraper /app
USER scraper

CMD ["scrapy", "crawl", "books", "-O", "/data/books.jsonl"]
```

Mount a volume for output (`-v $(pwd)/data:/data`), or write to a database and skip the volume. If you are running Playwright in the container, start from `mcr.microsoft.com/playwright/python` instead — it ships the browsers and their system libraries, which is a long list to assemble yourself.

**Scheduling** is usually the actual requirement, and it rarely needs Scrapy-specific tooling. A container plus cron, a Kubernetes `CronJob`, an ECS scheduled task, or a Cloud Run job all work, and they give you the retry and alerting behaviour your platform already has. Serverless functions are a poor fit for anything but the smallest crawls — Lambda's 15-minute ceiling fights the framework rather than helping it.

**Scrapyd** is the official self-hosted option: a daemon you deploy spider eggs to, with a JSON API to schedule runs and a basic web UI. Useful if you want spider management without building it, and dated — no authentication by default, so keep it off the public internet. **Zyte Scrapy Cloud** is the managed equivalent from Scrapy's maintainers, deployed with `shub deploy`: the least operational work, if you are happy to pay for it and to keep crawl scheduling in a vendor's product.

Whatever runs the spider, monitor the end-of-run stats rather than just the exit code. A spider that returns zero after scraping nothing looks identical to a successful run from the outside:

```python
def closed(self, reason):
    stats = self.crawler.stats.get_stats()
    if stats.get("item_scraped_count", 0) < self.expected_minimum:
        self.logger.error("Only %s items — site layout may have changed",
                          stats.get("item_scraped_count", 0))
```

Silent breakage from a redesigned page is the most common way a production crawler fails, and an item-count floor catches almost all of it.

## When Scrapy isn't enough

Scrapy solves crawl orchestration extremely well. It does not solve the adversarial part of scraping:

- **Anti-bot systems.** Cloudflare, DataDome, and PerimeterX fingerprint TLS handshakes and header ordering. Scrapy's Python TLS stack does not look like Chrome, and no combination of user agents fixes that.
- **Proxy operations.** Real crawls need residential rotation, geo-targeting, IP health tracking, and ban detection — infrastructure work that has nothing to do with your extraction logic.
- **Browser fleets.** Once `scrapy-playwright` is in the loop, you are operating a Chromium cluster with all the memory and crash-recovery work that implies.

If you would rather keep writing spiders and hand off the rendering and proxy layer, a scraping API slots into a Scrapy project as an ordinary request. [WebScraping.AI](https://webscraping.ai/) renders pages in headless Chrome behind rotating proxies (datacenter by default, `proxy=residential` or `proxy=stealth` when a target needs it):

```python
import scrapy
from urllib.parse import urlencode

API = "https://api.webscraping.ai/html"

class ApiSpider(scrapy.Spider):
    name = "api"

    async def start(self):
        for url in self.target_urls:
            params = {"api_key": self.settings["WSAI_KEY"], "url": url, "js": "true"}
            yield scrapy.Request(f"{API}?{urlencode(params)}",
                                 callback=self.parse, meta={"source_url": url})

    def parse(self, response):
        # response.body is the fully rendered HTML — selectors work normally
        yield {
            "url": response.meta["source_url"],
            "title": response.css("h1::text").get(),
        }
```

Everything downstream — selectors, items, pipelines, feed exports, `JOBDIR` resume — keeps working unchanged, because to Scrapy this is just another HTTP response. Set `CONCURRENT_REQUESTS` to match your plan's concurrency and turn `ROBOTSTXT_OBEY` off, since Scrapy would otherwise fetch the API host's `robots.txt` rather than the target's.

The [`/ai/fields` endpoint](https://webscraping.ai/docs) goes a step further and returns extracted values rather than HTML, which removes the selector-maintenance problem that breaks crawlers after a site redesign:

```python
params = {
    "api_key": KEY,
    "url": "https://example.com/product/42",
    "fields[price]": "Product price, numbers only",
    "fields[stock]": "Is the product in stock? yes or no",
}
```

See the [AI web scraping overview](https://webscraping.ai/ai-web-scraping) for the extraction endpoints, or the [headless browser guide](https://webscraping.ai/blog/headless-browser-guide) if you would rather run the browsers yourself.

## Frequently asked questions

**Is Scrapy better than Beautiful Soup?** They are not the same kind of tool. Beautiful Soup parses HTML you already fetched; Scrapy fetches, schedules, retries, deduplicates, and pipes data — and has its own selectors built in. For one page, Beautiful Soup wins on simplicity. For ten thousand pages with link following, Scrapy wins on everything. You can also use both: Scrapy for the crawl, Beautiful Soup inside a callback if you already have parsing code that works.

**Can Scrapy handle JavaScript?** Not on its own — it makes HTTP requests and never executes scripts. `scrapy-playwright` adds a real browser per request, and `scrapy-splash` is the older alternative. Before either, check whether the page loads its data from a JSON API you can call directly; that is faster and more stable than rendering.

**How do I stop my Scrapy spider from getting blocked?** In order of effect: enable AutoThrottle, keep `CONCURRENT_REQUESTS_PER_DOMAIN` low, set a real user agent, respect `robots.txt`, and back off properly on 429 responses. Beyond that it becomes an infrastructure question — rotating residential proxies and anti-bot handling, not spider settings. Also make sure you are not retrying 403s, which is the fastest way to confirm to a target that you are automated.

**How do I resume an interrupted crawl?** Run with `-s JOBDIR=crawls/name`, which persists the scheduler queue and the duplicate filter to disk. Stop with a single Ctrl-C so the spider shuts down cleanly, then run the same command to resume. A second Ctrl-C force-kills and can leave the state files inconsistent.

**Why is my spider only scraping the first page?** Usually one of three things: the pagination link is rendered by JavaScript and is not in the HTML Scrapy received (check `view(response)` in `scrapy shell`); the next-page URL is relative and was passed to `scrapy.Request()` instead of `response.follow()`; or `allowed_domains` does not cover the host the pagination links point to, so the offsite middleware drops them — check the `offsite/filtered` stat.

**Is Scrapy still maintained?** Yes. It is actively developed under Zyte's stewardship — 2.19.0 shipped in September 2026 — and recent releases added an asyncio-native `start()` hook, coroutine-based component APIs, and experimental support for running without the Twisted reactor. It remains the default answer for large-scale Python crawling.

**Is scraping with Scrapy legal?** Using a crawler is legal; what matters is what you collect and how you use it — public versus gated data, terms of service, the load you place on the target, and privacy law where personal data is involved. Our [guide to web scraping legality](https://webscraping.ai/blog/is-web-scraping-legal) covers the current state of the case law.