Scrapy Web Scraping: The Complete Guide

Scrapy is the framework you graduate to when a scraping script stops being a script. It gives you an asynchronous crawl engine, a scheduler, deduplication, retries, throttling, and a data pipeline — features you would otherwise write badly by hand somewhere around page five thousand. This guide covers the whole workflow: project layout, spiders, selectors, items and pipelines, the settings that actually change outcomes, middlewares, proxies, JavaScript rendering, scaling, and deployment.

Key Takeaways

• Scrapy is a crawling framework, not a parsing library — it competes with a whole hand-rolled crawler, not with Beautiful Soup
• scrapy startproject gives you the layout; scrapy genspider gives you a spider; scrapy crawl name -O out.json gives you data
• Requests are deduplicated by fingerprint automatically, retried automatically, and throttled by AutoThrottle — the three things people most often rebuild by accident
• CONCURRENT_REQUESTS and AUTOTHROTTLE_ENABLED are the two settings that decide whether a crawl is fast, blocked, or rude
• Scrapy does not run JavaScript; scrapy-playwright adds a real browser per request, and finding the site's own JSON API is usually better than either
• Anti-bot walls and proxy fleets are infrastructure problems, not spider-code problems

What is Scrapy?

Scrapy is an open-source web crawling and scraping framework for Python, maintained by Zyte. Current releases require Python 3.10 or newer. Underneath it runs on Twisted, an asynchronous networking engine, which is why a single Scrapy process can keep dozens of requests in flight without you writing any concurrency code.

What you get out of the box, and what each piece would otherwise cost you:

• A scheduler with a duplicate filter — every request is fingerprinted and re-requests are dropped
• A retry middleware — transient failures and 5xx responses are retried with no code from you
• AutoThrottle — concurrency adapts to how fast the target server is actually responding
• Selectors — CSS and XPath over parsel/lxml, on every response object
• Item pipelines — a chain of processing stages between "parsed" and "stored"
• Feed exports — JSON, JSON Lines, CSV, and XML to a file, S3, or GCS with one flag
• scrapy shell — an interactive REPL against a live response, which is the fastest selector debugger in Python

The tradeoff is that Scrapy has opinions. You write callbacks and yield objects rather than calling functions in a loop, and the framework decides when things run. For a five-page scrape that structure is overhead. For a hundred-thousand-page crawl it is the reason the job finishes.

Scrapy vs requests + Beautiful Soup vs Playwright

These tools are usually presented as rivals. They are better understood as three different jobs.

The honest decision rule:

• A handful of pages, run once? Use requests and Beautiful Soup. See our Python requests guide and Beautiful Soup guide.
• Thousands of pages, link-following, or a job that runs on a schedule? Use Scrapy. The framework overhead pays for itself the first time a crawl dies halfway through and you need to resume it.
• Data only appears after scripts run? You need a browser — either standalone Playwright, or Playwright bolted into Scrapy (covered below).

They also combine. Scrapy for fetching and crawl management, Beautiful Soup for a parsing step you already have working, Playwright for the handful of pages that need rendering. Nothing forces you to pick one. Our Python web scraping libraries comparison covers the wider ecosystem, and web scraping with Python walks the fundamentals if you are earlier in the journey.

Installing Scrapy

Install into a virtual environment — Scrapy pulls in Twisted, lxml, and cryptography, and you do not want those fighting with your system Python:

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install scrapy
scrapy version

On Linux, if the wheels fall back to a source build, install the headers first:

sudo apt-get install python3-dev build-essential libxml2-dev libxslt1-dev libssl-dev libffi-dev

Conda users can take the conda-forge build instead, which avoids compilation entirely:

conda install -c conda-forge scrapy

Windows generally installs from wheels without incident; if a dependency does try to compile, install the Microsoft C++ Build Tools or switch to conda. Pin the version in requirements.txt (scrapy==2.13.3 or whatever you tested against) so a minor release never surprises a production crawl.

Creating a project

Scrapy projects have a fixed layout, and the framework relies on it:

scrapy startproject bookstore
cd bookstore
scrapy genspider books books.toscrape.com

bookstore/
├── scrapy.cfg              # deployment config — marks the project root
└── bookstore/
    ├── items.py            # data schemas
    ├── middlewares.py      # request/response hooks
    ├── pipelines.py        # post-parse processing
    ├── settings.py         # everything configurable
    └── spiders/
        └── books.py        # your spiders

You can run a spider without a project (a single .py file with scrapy runspider file.py), which is handy for experiments. Anything you intend to keep should be a project — pipelines, settings, and deployment all assume it.

Your first spider

A spider defines where to start, and what to do with each response:

import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css("p.price_color::text").get(),
                "in_stock": "In stock" in book.css("p.availability::text").get(""),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Run it and write the results:

scrapy crawl books -O books.json      # -O overwrites, -o appends
scrapy crawl books -O books.jsonl     # JSON Lines — better for large crawls
scrapy crawl books -O books.csv

Three things in that spider carry most of the framework's weight. yield on a dict hands an item to the pipeline. response.follow() resolves relative URLs against the current page and produces a new request. And returning that request rather than fetching it means the scheduler — not your code — decides when it runs, which is what makes the crawl concurrent.

start_urls, start_requests(), and start()

start_urls is the shortcut. When you need headers, cookies, POST bodies, or a different callback for the first requests, override the start hook instead:

class BooksSpider(scrapy.Spider):
    name = "books"

    def start_requests(self):
        for page in range(1, 51):
            yield scrapy.Request(
                f"https://books.toscrape.com/catalogue/page-{page}.html",
                callback=self.parse,
                headers={"Accept-Language": "en-US"},
                meta={"page": page},
            )

Scrapy 2.13 introduced an asynchronous start() method that supersedes start_requests() and lets you await inside the start logic — useful when your seed URLs come from a database or an API. start_requests() still works and remains the form you will meet in most existing code and tutorials:

async def start(self):
    async for url in self.seed_urls():
        yield scrapy.Request(url, callback=self.parse)

Selectors

Every response carries .css() and .xpath(), and they return selector lists you can chain:

response.css("h1::text").get()             # first match, or None
response.css("h1::text").get(default="")   # first match with a fallback
response.css(".price::text").getall()      # every match, as a list
response.css("a::attr(href)").getall()     # attribute values
response.css("a").attrib["href"]           # attributes of the first match

response.xpath("//h1/text()").get()
response.xpath("//div[@class='product']//span[@class='price']/text()").getall()

# Chaining — scope a sub-selector to a parent
for card in response.css("div.product"):
    name = card.css("h2::text").get()
    price = card.xpath(".//span[@class='price']/text()").get()   # note the leading dot

Two habits worth forming early. Use .get() rather than [0] — a missing element gives you None instead of an IndexError that kills the callback. And when chaining XPath inside a loop, start the expression with . — a bare // searches the whole document again and will silently return the first product's price for every card.

re() and re_first() pull values out of matched text without a separate parsing step:

response.css("p.stock::text").re_first(r"(\d+) available")

For anything non-trivial, develop selectors in the shell rather than by re-running the spider:

scrapy shell "https://books.toscrape.com/"
>>> response.css("article.product_pod h3 a::attr(title)").getall()
>>> view(response)     # opens Scrapy's copy of the page in a browser

view(response) is the honest one — it shows you what Scrapy received, not what your browser renders. If the elements you want are missing there, you have a JavaScript problem, not a selector problem. Our XPath cheat sheet is a useful companion when CSS runs out of expressiveness.

Spider vs CrawlSpider

scrapy.Spider gives you explicit control: you decide which links to follow, in code. CrawlSpider follows links declaratively from a rule set, which suits broad site crawls where the pattern is regular:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class BooksCrawlSpider(CrawlSpider):
    name = "books_crawl"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    rules = (
        # Follow category and pagination links, don't parse them
        Rule(LinkExtractor(allow=r"catalogue/category/")),
        # Parse product pages, don't follow links out of them
        Rule(LinkExtractor(allow=r"catalogue/[\w-]+_\d+/index\.html"),
             callback="parse_book", follow=False),
    )

    def parse_book(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "price": response.css("p.price_color::text").get(),
            "upc": response.css("table tr:nth-child(1) td::text").get(),
        }

The one rule that trips everyone: do not name a CrawlSpider callback parse. CrawlSpider implements parse itself to apply the rules, and overriding it silently disables all link following.

Reach for CrawlSpider when URL patterns describe the site well. Reach for a plain Spider when navigation depends on page content — pagination tokens, "load more" cursors, or a search form you have to submit.

Items and pipelines

Yielding dicts is fine to start. Items give you a declared schema, so a typo becomes an error instead of a silently missing column:

# items.py
import scrapy

class BookItem(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()
    in_stock = scrapy.Field()
    scraped_at = scrapy.Field()

Pipelines process every item after parsing, in the order given by their priority number (lower runs first):

# pipelines.py
from datetime import datetime, timezone
from scrapy.exceptions import DropItem

class CleanPricePipeline:
    def process_item(self, item, spider):
        raw = item.get("price")
        if not raw:
            raise DropItem("Missing price")
        item["price"] = float(raw.replace("£", "").strip())
        item["scraped_at"] = datetime.now(timezone.utc).isoformat()
        return item

class PostgresPipeline:
    def open_spider(self, spider):
        self.conn = psycopg.connect(spider.settings["DATABASE_URL"])

    def process_item(self, item, spider):
        with self.conn.cursor() as cur:
            cur.execute(
                "INSERT INTO books (title, price) VALUES (%s, %s) "
                "ON CONFLICT (title) DO UPDATE SET price = EXCLUDED.price",
                (item["title"], item["price"]),
            )
        self.conn.commit()
        return item

    def close_spider(self, spider):
        self.conn.close()

# settings.py
ITEM_PIPELINES = {
    "bookstore.pipelines.CleanPricePipeline": 300,
    "bookstore.pipelines.PostgresPipeline": 800,
}

open_spider and close_spider are where connections belong — opening a database connection inside process_item gives you one connection per item. Raising DropItem discards an item without stopping the crawl, which is the right way to enforce validation.

Writing into Django models

If the scraped data belongs to an existing Django app, you can use its ORM from a pipeline by setting up Django before Scrapy touches the models:

# settings.py (Scrapy)
import os, sys, django

sys.path.insert(0, "/path/to/django_project")
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "django_project.settings")
django.setup()

# pipelines.py
from myapp.models import Book

class DjangoPipeline:
    def process_item(self, item, spider):
        Book.objects.update_or_create(
            title=item["title"],
            defaults={"price": item["price"]},
        )
        return item

update_or_create rather than create is deliberate: a recurring crawl will re-encounter the same records, and it keeps the pipeline idempotent. The django.setup() call must run before any model import, which is why it lives in Scrapy's settings.py rather than in the pipeline module.

The settings that actually matter

settings.py has dozens of knobs. These are the ones that change outcomes:

# Politeness and speed
CONCURRENT_REQUESTS = 16              # total in-flight requests
CONCURRENT_REQUESTS_PER_DOMAIN = 8    # per-domain cap — the one that matters for politeness
DOWNLOAD_DELAY = 0.5                  # seconds between requests to a domain (± 50% jitter)
RANDOMIZE_DOWNLOAD_DELAY = True       # on by default

# AutoThrottle — adaptive concurrency
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
AUTOTHROTTLE_DEBUG = False            # True prints the delay it picks per response

# Reliability
DOWNLOAD_TIMEOUT = 30
RETRY_ENABLED = True
RETRY_TIMES = 3

# Manners and identity
ROBOTSTXT_OBEY = True
USER_AGENT = "mybot (+https://example.com/about-our-crawler)"

# Caching during development — never re-fetch while iterating on selectors
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 3600

AUTOTHROTTLE_ENABLED deserves a note, because it is the setting that most often stops a crawl from getting blocked. Rather than a fixed delay, Scrapy measures the target's response latency and adjusts its delay to hold roughly AUTOTHROTTLE_TARGET_CONCURRENCY requests in flight. When the server slows down — the first symptom of you being too aggressive — the crawler backs off automatically. It is slower than a flat-out crawl by design, and that is the point.

DOWNLOAD_DELAY and AutoThrottle coexist: DOWNLOAD_DELAY becomes the floor AutoThrottle will not go below. HTTPCACHE_ENABLED is a development setting worth switching on the moment you start iterating on selectors — it turns a fifty-page crawl into a local replay and takes the load off the site you are working against.

Any of these can be overridden per spider:

class BooksSpider(scrapy.Spider):
    name = "books"
    custom_settings = {
        "DOWNLOAD_DELAY": 2.0,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2,
    }

Or on the command line for a one-off run: scrapy crawl books -s DOWNLOAD_DELAY=2.

Duplicate filtering

Scrapy deduplicates requests for you. Every request gets a fingerprint derived from its method, canonicalized URL, and body, and the scheduler drops any request whose fingerprint it has already seen. This is why a crawl that follows "next page" links in both directions does not loop forever.

Three things you will eventually need to control.

Bypassing the filter for a single request. Login pages, polling endpoints, and pagination cursors that reuse a URL need dont_filter=True:

yield scrapy.Request(url, callback=self.parse, dont_filter=True)

Seeing what got filtered. Set DUPEFILTER_DEBUG = True and every dropped request is logged rather than counted silently. The dupefilter/filtered stat at the end of a run is the summary number:

def closed(self, reason):
    self.logger.info("Filtered %s duplicates",
                     self.crawler.stats.get_value("dupefilter/filtered", 0))

Persisting the filter across runs. By default the fingerprint set lives in memory and dies with the process. JOBDIR writes the scheduler queue and the seen-fingerprints set to disk, which both survives a restart and lets you resume an interrupted crawl:

scrapy crawl books -s JOBDIR=crawls/books-1

Stop that run with a single Ctrl-C (a second one kills it uncleanly) and the same command picks up where it left off.

If the default fingerprint is wrong for your target — say the site appends tracking parameters that do not change the content — subclass the fingerprinter rather than the whole filter:

# dupefilters.py
from w3lib.url import url_query_cleaner
from scrapy.utils.request import RequestFingerprinter

class CleanURLFingerprinter(RequestFingerprinter):
    def fingerprint(self, request):
        cleaned = url_query_cleaner(
            request.url, ["utm_source", "utm_medium", "utm_campaign"], remove=True
        )
        return super().fingerprint(request.replace(url=cleaned))

# settings.py
REQUEST_FINGERPRINTER_CLASS = "bookstore.dupefilters.CleanURLFingerprinter"

Older tutorials call scrapy.utils.request.request_fingerprint() directly. That function is gone from current Scrapy; the fingerprinter component above replaced it, and inside a spider or middleware you reach it as self.crawler.request_fingerprinter.fingerprint(request).

Request dedup is not item dedup. Two different URLs can yield the same product. That is a pipeline's job:

class DedupePipeline:
    def open_spider(self, spider):
        self.seen = set()

    def process_item(self, item, spider):
        key = item["upc"]
        if key in self.seen:
            raise DropItem(f"Duplicate item: {key}")
        self.seen.add(key)
        return item

An in-memory set is fine up to a few million items. Beyond that, use the database's unique constraint (as in the ON CONFLICT pipeline above) or a Redis set, and let the storage layer be the source of truth.

Retries, errors, and timeouts

RetryMiddleware is enabled by default and handles the common cases before you write a line:

# settings.py
RETRY_ENABLED = True
RETRY_TIMES = 3                                    # retries after the first attempt
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429]

Connection errors, DNS failures, and timeouts are retried too — those are matched by exception type, not status code. Note what is not in the list: 400, 401, 403, and 404 are permanent, and retrying them wastes your rate budget while making you look more like a bot.

For anything a status code cannot express, subclass the middleware. The most common real case is a server that returns HTTP 200 with a challenge or "too many requests" page, which no status-code list can catch:

# middlewares.py
from scrapy.downloadermiddlewares.retry import RetryMiddleware, get_retry_request

class SmartRetryMiddleware(RetryMiddleware):
    def process_response(self, request, response, spider):
        if b"Just a moment" in response.body or b"rate limit" in response.body.lower():
            return get_retry_request(request, spider=spider, reason="soft-block") or response
        return super().process_response(request, response, spider)

# settings.py — replace the default at its own priority slot
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": None,
    "bookstore.middlewares.SmartRetryMiddleware": 550,
}

get_retry_request() is the supported helper: it increments the retry counter, respects RETRY_TIMES, updates the stats, and returns None once the budget is exhausted.

One gap to know about: Scrapy does not honour the Retry-After header automatically. A 429 is retried like any other listed status, on the normal delay schedule, ignoring however long the server asked you to wait. The practical answer is AutoThrottle plus a DOWNLOAD_DELAY floor that keeps you out of 429 territory in the first place, rather than a middleware that tries to sleep — blocking inside a middleware stalls the whole reactor, not just that request.

Handling failures in the spider

Retries eventually run out. errback catches what is left, so one dead URL does not vanish without trace:

from scrapy.spidermiddlewares.httperror import HttpError
from twisted.internet.error import TimeoutError, DNSLookupError

class BooksSpider(scrapy.Spider):
    name = "books"
    handle_httpstatus_list = [404]          # let 404s reach parse() instead of being dropped

    def start_requests(self):
        for url in self.urls:
            yield scrapy.Request(url, callback=self.parse, errback=self.on_error)

    def on_error(self, failure):
        request = failure.request
        if failure.check(HttpError):
            self.logger.error("HTTP %s on %s", failure.value.response.status, request.url)
        elif failure.check(DNSLookupError):
            self.logger.error("DNS failure on %s", request.url)
        elif failure.check(TimeoutError):
            self.logger.error("Timeout on %s", request.url)
        else:
            self.logger.error("Unhandled %r on %s", failure.value, request.url)

Exceptions raised inside parse() are logged and the response is dropped — they do not stop the crawl, which is convenient right up until you notice thousands of them scrolling past. Check spider_exceptions in the end-of-run stats before you trust a crawl's output.

Timeouts

DOWNLOAD_TIMEOUT (default 180 seconds) is the ceiling for a whole download, and it is set at three levels:

# Globally
DOWNLOAD_TIMEOUT = 30

# Per spider
class BooksSpider(scrapy.Spider):
    custom_settings = {"DOWNLOAD_TIMEOUT": 15}

# Per request — the meta key wins over both
yield scrapy.Request(url, meta={"download_timeout": 10})

The subtlety worth knowing: this is a timeout on the complete response, not on time-to-first-byte. A server that dribbles out a large page over 40 seconds will trip a 30-second timeout even though it never actually stalled. If large downloads are timing out but small ones are fine, raise the value rather than assuming the target is slow. Timeouts surface as twisted.internet.error.TimeoutError, are retried by RetryMiddleware like any other transient failure, and reach your errback once retries are exhausted.

Middlewares

Middlewares are how you modify requests and responses globally. There are two kinds, and mixing them up is a common first-week mistake:

• Downloader middlewares sit between the engine and the network. Proxies, headers, user agents, retries, caching, and browser integrations all live here.
• Spider middlewares sit between the engine and your spider callbacks. Filtering items, handling exceptions from callbacks, and processing the output of parse() live here.

A downloader middleware has three hooks. Returning None from any of them passes control to the next middleware; returning a Response or Request short-circuits the chain:

# middlewares.py
class HeaderMiddleware:
    def process_request(self, request, spider):
        request.headers["Accept-Language"] = "en-US,en;q=0.9"

    def process_response(self, request, response, spider):
        return response        # or a Request to reschedule, or raise IgnoreRequest

    def process_exception(self, request, exception, spider):
        return None

The numbers in DOWNLOADER_MIDDLEWARES are ordering, not identity. Lower numbers process requests earlier and responses later — the chain is walked outward on the way in and back inward on the way out. Setting a built-in middleware to None disables it, which is how you replace one with your own.

Proxies and user agents

Scrapy reads the standard http_proxy/https_proxy environment variables through HttpProxyMiddleware. For per-request control, set the proxy meta key:

yield scrapy.Request(url, meta={"proxy": "http://user:pass@proxy.example.com:8080"})

Rotation is a small middleware:

# middlewares.py
import random

class ProxyRotationMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_LIST"))

    def process_request(self, request, spider):
        if "proxy" not in request.meta:
            request.meta["proxy"] = random.choice(self.proxies)

    def process_exception(self, request, exception, spider):
        # Retry through a different proxy on connection failure
        new_request = request.copy()
        new_request.meta["proxy"] = random.choice(self.proxies)
        new_request.dont_filter = True
        return new_request

User-agent rotation works the same way, though it matters less than it used to — modern anti-bot systems fingerprint TLS handshakes and header ordering, not just the UA string. A rotating user agent paired with a static datacenter IP and a Python TLS fingerprint fools very little. See user agent rotation for what it does and does not buy, and types of proxies for the residential-versus-datacenter tradeoff.

from_crawler is the standard constructor hook for every Scrapy component — middlewares, pipelines, extensions — and it is how you read settings without importing them. Older code uses from_settings; it is deprecated in favour of from_crawler.

JavaScript-rendered pages

Scrapy speaks HTTP. It does not run JavaScript, so a React storefront that renders its catalogue client-side gives you an empty shell. scrapy shell <url> plus view(response) tells you within seconds whether this is your problem.

There are three answers, in ascending order of cost.

Find the underlying API first

Most JavaScript-rendered pages fetch their data from a JSON endpoint. Open the browser's Network tab, filter to XHR/Fetch, and look for the request that carries the data. When you find it, you get clean structured data with none of the rendering cost:

import json

def parse(self, response):
    data = response.json()
    for product in data["products"]:
        yield {"name": product["title"], "price": product["price"]["amount"]}

    if data.get("next_cursor"):
        yield scrapy.Request(
            f"https://example.com/api/products?cursor={data['next_cursor']}",
            callback=self.parse,
        )

This is worth ten minutes of investigation on every JS-heavy target. A JSON endpoint is faster, more stable across redesigns, and often paginates in larger chunks than the visible UI.

scrapy-playwright

When there is no usable API, scrapy-playwright runs a real Chromium per request and hands the rendered HTML back to your normal callbacks:

pip install scrapy-playwright
playwright install chromium

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAYWRIGHT_BROWSER_TYPE = "chromium"
PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": True}
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 30_000

from scrapy_playwright.page import PageMethod

class JsSpider(scrapy.Spider):
    name = "js"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/products",
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("wait_for_selector", ".product-card"),
                    PageMethod("evaluate", "window.scrollBy(0, document.body.scrollHeight)"),
                    PageMethod("wait_for_timeout", 1000),
                ],
            },
        )

    def parse(self, response):
        for card in response.css(".product-card"):
            yield {"name": card.css("h2::text").get()}

The asyncio reactor is required — it is the default in recent Scrapy releases, but set it explicitly if you are on an older version or you will get a confusing reactor-already-installed error. Two practical notes: turn playwright: True on per request rather than globally, so only the pages that need a browser pay for one; and each concurrent page is a real Chromium tab using 100–400 MB, so CONCURRENT_REQUESTS needs to come down hard when Playwright is in the mix. Our Playwright guide covers the browser API itself in depth.

scrapy-splash (legacy)

scrapy-splash drives Splash, a scriptable headless browser that runs as a separate service, usually in Docker. You will find it in a lot of older tutorials and existing codebases, and it still works. For new projects, prefer scrapy-playwright: Splash uses an older WebKit engine that struggles with modern JavaScript, its Lua scripting layer is a language boundary you have to learn, and it needs a service to operate alongside your crawler. If you have inherited a working Splash setup, there is no urgency to migrate — just do not start there.

Cookies, sessions, and login

Cookies are handled automatically. CookiesMiddleware keeps a jar per crawl and sends cookies back the way a browser would, so a login persists across subsequent requests with no code:

COOKIES_ENABLED = True     # default
COOKIES_DEBUG = True       # logs every cookie sent and received

A form login is a single helper call:

from scrapy.http import FormRequest

def parse_login_page(self, response):
    yield FormRequest.from_response(
        response,
        formdata={"username": "user", "password": "pass"},
        callback=self.after_login,
    )

def after_login(self, response):
    if "authentication failed" in response.text.lower():
        self.logger.error("Login failed")
        return
    yield response.follow("/dashboard", callback=self.parse_dashboard)

from_response reads the form out of the page, which means hidden CSRF tokens are carried over automatically — the reason it is worth using over a hand-built POST.

For concurrent independent sessions, cookiejar in meta keeps them isolated:

yield scrapy.Request(url, meta={"cookiejar": user_id})

If a site sets cookies from JavaScript rather than Set-Cookie headers, Scrapy will not see them — that is a rendering problem, and the answer is the previous section.

Parallelism and scaling

Scrapy is already concurrent inside one process. The engine keeps up to CONCURRENT_REQUESTS in flight, and because Twisted is asynchronous rather than threaded, that costs you very little memory. Before reaching for anything more elaborate, tune what you have:

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 16     # raise only if you own the target
CONCURRENT_ITEMS = 100                  # pipeline concurrency
REACTOR_THREADPOOL_MAXSIZE = 20         # DNS resolution and other blocking calls

For a single-domain crawl this is almost always the limit that matters, and it is a politeness limit, not a technical one. A crawl that is too slow because of CONCURRENT_REQUESTS_PER_DOMAIN is usually a crawl that should stay slow.

Multiple domains in one process scale naturally — Scrapy queues per domain, so a crawl across a thousand sites at 8 concurrent requests each keeps the engine saturated without hammering anyone.

Multiple processes are the next step, and the simplest split is by URL range. Each process needs its own JOBDIR, or they will fight over the same queue files:

scrapy crawl books -a start=1 -a end=1000 -s JOBDIR=crawls/shard-1 &
scrapy crawl books -a start=1001 -a end=2000 -s JOBDIR=crawls/shard-2 &
wait

Truly distributed crawling — many machines sharing one frontier — needs a shared queue and a shared duplicate filter. scrapy-redis is the established option:

# settings.py
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_PERSIST = True
REDIS_URL = "redis://localhost:6379"

from scrapy_redis.spiders import RedisSpider

class DistributedSpider(RedisSpider):
    name = "distributed"
    redis_key = "distributed:start_urls"

Every worker pulls from the same Redis queue and checks the same fingerprint set, so you can add and remove machines mid-crawl. Seed it with redis-cli lpush distributed:start_urls "https://example.com".

Two scaling failure modes worth naming in advance. Memory grows with the scheduler queue, not with pages crawled — a broad crawl that discovers links faster than it consumes them will exhaust RAM, and MEMUSAGE_LIMIT_MB will stop the spider before the OOM killer does. And the bottleneck often turns out to be a pipeline, not the network: a synchronous database write in process_item blocks the whole reactor. Batch the writes or move them off the reactor thread.

Files and PDFs

FilesPipeline downloads linked files, deduplicates them by content hash, and records the results on the item:

# settings.py
ITEM_PIPELINES = {"scrapy.pipelines.files.FilesPipeline": 1}
FILES_STORE = "downloads"

class ReportItem(scrapy.Item):
    file_urls = scrapy.Field()      # the pipeline reads this field
    files = scrapy.Field()          # and writes results here

def parse(self, response):
    yield ReportItem(file_urls=response.css("a[href$='.pdf']::attr(href)").getall())

The field names are fixed unless you override FILES_URLS_FIELD and FILES_RESULT_FIELD. ImagesPipeline works the same way with thumbnailing on top.

Text extraction is a separate step, and Scrapy has no opinion about it — do it in a pipeline with PyMuPDF (fast, good with tables), pypdf (pure Python, no build step), or pdfplumber (best on complex layouts):

import fitz    # PyMuPDF

class PdfTextPipeline:
    def process_item(self, item, spider):
        for f in item.get("files", []):
            with fitz.open(f"downloads/{f['path']}") as doc:
                item["text"] = "\n".join(page.get_text() for page in doc)
        return item

Large PDFs will exhaust memory if you hold several open at once — keep pipeline concurrency low for these items.

Deployment

Docker is the baseline. Scrapy containers are unremarkable — the only real requirement is unbuffered output, or your logs will appear in unhelpful bursts:

FROM python:3.12-slim

ENV PYTHONUNBUFFERED=1
WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
      build-essential libxml2-dev libxslt1-dev \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
RUN useradd -m scraper && chown -R scraper /app
USER scraper

CMD ["scrapy", "crawl", "books", "-O", "/data/books.jsonl"]

Mount a volume for output (-v $(pwd)/data:/data), or write to a database and skip the volume. If you are running Playwright in the container, start from mcr.microsoft.com/playwright/python instead — it ships the browsers and their system libraries, which is a long list to assemble yourself.

Scheduling is usually the actual requirement, and it rarely needs Scrapy-specific tooling. A container plus cron, a Kubernetes CronJob, an ECS scheduled task, or a Cloud Run job all work, and they give you the retry and alerting behaviour your platform already has. Serverless functions are a poor fit for anything but the smallest crawls — Lambda's 15-minute ceiling fights the framework rather than helping it.

Scrapyd is the official self-hosted option: a daemon you deploy spider eggs to, with a JSON API to schedule runs and a basic web UI. Useful if you want spider management without building it, and dated — no authentication by default, so keep it off the public internet. Zyte Scrapy Cloud is the managed equivalent from Scrapy's maintainers, deployed with shub deploy: the least operational work, if you are happy to pay for it and to keep crawl scheduling in a vendor's product.

Whatever runs the spider, monitor the end-of-run stats rather than just the exit code. A spider that returns zero after scraping nothing looks identical to a successful run from the outside:

def closed(self, reason):
    stats = self.crawler.stats.get_stats()
    if stats.get("item_scraped_count", 0) < self.expected_minimum:
        self.logger.error("Only %s items — site layout may have changed",
                          stats.get("item_scraped_count", 0))

Silent breakage from a redesigned page is the most common way a production crawler fails, and an item-count floor catches almost all of it.

When Scrapy isn't enough

Scrapy solves crawl orchestration extremely well. It does not solve the adversarial part of scraping:

• Anti-bot systems. Cloudflare, DataDome, and PerimeterX fingerprint TLS handshakes and header ordering. Scrapy's Python TLS stack does not look like Chrome, and no combination of user agents fixes that.
• Proxy operations. Real crawls need residential rotation, geo-targeting, IP health tracking, and ban detection — infrastructure work that has nothing to do with your extraction logic.
• Browser fleets. Once scrapy-playwright is in the loop, you are operating a Chromium cluster with all the memory and crash-recovery work that implies.

If you would rather keep writing spiders and hand off the rendering and proxy layer, a scraping API slots into a Scrapy project as an ordinary request. WebScraping.AI renders pages in headless Chrome behind rotating residential proxies:

import scrapy
from urllib.parse import urlencode

API = "https://api.webscraping.ai/html"

class ApiSpider(scrapy.Spider):
    name = "api"

    def start_requests(self):
        for url in self.target_urls:
            params = {"api_key": self.settings["WSAI_KEY"], "url": url, "js": "true"}
            yield scrapy.Request(f"{API}?{urlencode(params)}",
                                 callback=self.parse, meta={"source_url": url})

    def parse(self, response):
        # response.body is the fully rendered HTML — selectors work normally
        yield {
            "url": response.meta["source_url"],
            "title": response.css("h1::text").get(),
        }

Everything downstream — selectors, items, pipelines, feed exports, JOBDIR resume — keeps working unchanged, because to Scrapy this is just another HTTP response. Set CONCURRENT_REQUESTS to match your plan's concurrency and turn ROBOTSTXT_OBEY off for the API host (the proxy still respects the target's rules).

The /ai/fields endpoint goes a step further and returns extracted values rather than HTML, which removes the selector-maintenance problem that breaks crawlers after a site redesign:

params = {
    "api_key": KEY,
    "url": "https://example.com/product/42",
    "fields[price]": "Product price, numbers only",
    "fields[stock]": "Is the product in stock? yes or no",
}

See the AI web scraping overview for the extraction endpoints, or the headless browser guide if you would rather run the browsers yourself.

Frequently asked questions

Is Scrapy better than Beautiful Soup? They are not the same kind of tool. Beautiful Soup parses HTML you already fetched; Scrapy fetches, schedules, retries, deduplicates, and pipes data — and has its own selectors built in. For one page, Beautiful Soup wins on simplicity. For ten thousand pages with link following, Scrapy wins on everything. You can also use both: Scrapy for the crawl, Beautiful Soup inside a callback if you already have parsing code that works.

Can Scrapy handle JavaScript? Not on its own — it makes HTTP requests and never executes scripts. scrapy-playwright adds a real browser per request, and scrapy-splash is the older alternative. Before either, check whether the page loads its data from a JSON API you can call directly; that is faster and more stable than rendering.

How do I stop my Scrapy spider from getting blocked? In order of effect: enable AutoThrottle, keep CONCURRENT_REQUESTS_PER_DOMAIN low, set a real user agent, respect robots.txt, and back off properly on 429 responses. Beyond that it becomes an infrastructure question — rotating residential proxies and anti-bot handling, not spider settings. Also make sure you are not retrying 403s, which is the fastest way to confirm to a target that you are automated.

How do I resume an interrupted crawl? Run with -s JOBDIR=crawls/name, which persists the scheduler queue and the duplicate filter to disk. Stop with a single Ctrl-C so the spider shuts down cleanly, then run the same command to resume. A second Ctrl-C force-kills and can leave the state files inconsistent.

Why is my spider only scraping the first page? Usually one of three things: the pagination link is rendered by JavaScript and is not in the HTML Scrapy received (check view(response) in scrapy shell); the next-page URL is relative and was passed to scrapy.Request() instead of response.follow(); or you named a CrawlSpider callback parse, which overrides the method that applies the rules and silently disables link following.

Is Scrapy still maintained? Yes. It is actively developed under Zyte's stewardship, with regular releases through 2.13 and beyond, and it has recently added an asyncio-native start hook and broader async support. It remains the default answer for large-scale Python crawling.

Is scraping with Scrapy legal? Using a crawler is legal; what matters is what you collect and how you use it — public versus gated data, terms of service, the load you place on the target, and privacy law where personal data is involved. Our guide to web scraping legality covers the current state of the case law.