---
title: "Firecrawl Guide: Self-Hosting, Pricing, API Keys, and Limits"
description: "What Firecrawl is and how it works: getting an API key, credit pricing, self-hosting with Docker, rate limits, robots.txt, legality, and alternatives."
url: https://webscraping.ai/blog/firecrawl-guide
markdown_index: https://webscraping.ai/llms.txt
---
# Firecrawl Guide: Self-Hosting, Pricing, API Keys, and Limits

[Firecrawl](https://www.firecrawl.dev) is a web scraping API built for the LLM era: give it a URL and it returns clean, LLM-ready markdown instead of raw HTML. It can crawl entire sites, extract structured data with AI, and — unusually for this category — it's open source, so you can run it on your own hardware. This guide covers the questions people actually ask before adopting it: what it does, how to get an API key, what it costs, how to self-host it with Docker, where its limits are, and the legal ground rules that apply to it (and to every other scraping tool).

## Key Takeaways

- Firecrawl converts web pages into markdown and structured data through its v2 API: `/scrape` (one URL), `/crawl` (a whole site), `/map` (discover URLs), `/search`, and AI extraction (`/extract`, which Firecrawl now positions `/agent` as the successor to).
- You get an API key by signing up at firecrawl.dev; keys are prefixed `fc-` and belong in an environment variable, never in code.
- Pricing is credit-based — one credit per page scraped — with a free plan of 1,000 credits a month and paid plans from $16/month (billed annually) as of September 2026. Unused credits don't roll over except on annual Scale and Enterprise plans, and a fetched page bills even when it returns an error status.
- The core repository is open source (AGPL-3.0) and self-hostable with Docker Compose, but the hosted cloud has features the self-hosted stack lacks — most importantly the managed anti-bot and stealth proxy layer.
- Scraping legally is about _what_ you scrape and _how_, not the tool: respect robots.txt, don't collect personal data without a lawful basis, and honor site terms where they matter.

## What is Firecrawl?

Firecrawl started as an open-source project by Mendable and grew into a hosted API used heavily in RAG (retrieval-augmented generation) pipelines. Its pitch is "websites into LLM-ready data": where a traditional scraping API hands you rendered HTML to parse, Firecrawl hands you markdown that can go straight into a vector store or a prompt.

The main endpoints:

| Endpoint | What it does | Typical use |
| --- | --- | --- |
| `/scrape` | Fetch one URL, return markdown/HTML/screenshot | Single-page extraction |
| `/crawl` | Discover and scrape all pages under a URL | Ingesting a whole site for RAG |
| `/map` | Return a site's URLs quickly without scraping | Planning a crawl, sitemap discovery |
| `/extract` | AI extraction of structured data from pages | Turning pages into JSON without selectors |
| `/agent` | Autonomous data gathering; Firecrawl's successor to `/extract` | Research agents, no URL list needed |
| `/search` | Web search plus scraping of results | Research agents |

It ships official Python and Node SDKs, an MCP server for AI agents, and integrations for LangChain, LlamaIndex, and workflow tools. JavaScript rendering is built in, with page "actions" (click, scroll, wait) for content that needs interaction before it appears.

What it is _not_: a browser automation framework. You don't get Puppeteer-style step-by-step control of a browser session — for that, use [Playwright or Puppeteer directly](https://webscraping.ai/blog/playwright-web-scraping) and treat Firecrawl as the fetch-and-convert layer.

## Getting a Firecrawl API key

1. Sign up at [firecrawl.dev](https://www.firecrawl.dev) (email or GitHub/Google OAuth). New accounts start on the free plan (1,000 credits a month) with no card required.
2. Open the dashboard — your default API key is on the **API Keys** page, prefixed with `fc-`.
3. Store it as an environment variable:

```bash
export FIRECRAWL_API_KEY="fc-your-key-here"
```

The SDKs read that variable automatically:

```python
from firecrawl import Firecrawl # pip install firecrawl-py

firecrawl = Firecrawl() # reads FIRECRAWL_API_KEY from the environment
doc = firecrawl.scrape("https://example.com", formats=["markdown"])
print(doc.markdown)
```

Or over plain HTTP:

```bash
curl -X POST https://api.firecrawl.dev/v2/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'
```

Standard key hygiene applies: keep keys out of git (use `.env` files plus a secrets manager in production), rotate a key from the dashboard if it leaks, and use separate keys per project so usage is attributable.

## Firecrawl pricing: credits, plans, and gotchas

Firecrawl bills in **credits**. A standard `/scrape` of one page costs one credit; crawling is one credit per page crawled; JSON (LLM) extraction adds 4 credits per page, and `/agent` pricing is dynamic — a single complex query can burn a large number of credits.

As of September 2026 the shape of the plans (check [firecrawl.dev/pricing](https://www.firecrawl.dev/pricing) for current numbers — they change):

- **Free** — 1,000 credits a month, 2 concurrent browsers.
- **Hobby** — **$16/month** billed annually ($19 month-to-month) for 5,000 credits.
- **Standard / Growth** — $83 and $333/month billed annually ($99 / $399 monthly) for 100k and 500k credits, with higher concurrency.
- **Scale / Enterprise** — 1M credits from $599/month billed annually, or custom volume, SLAs, and dedicated support.

Three billing details worth knowing before you commit:

1. **Credits expire monthly.** Unused credits are forfeited at the end of each billing cycle (only annual Scale and Enterprise plans roll some over) — size your plan to actual usage, not peak usage.
2. **A fetch that returns an error status still bills.** Firecrawl charges when a page _fetches_, even if the response is an error page. On well-behaved targets this rarely matters; on protected sites where many attempts come back blocked, it adds up.
3. **Variable costs on AI features.** JSON extraction, `/extract`, and agent queries don't cost a flat one credit, so budgets for AI-heavy pipelines are harder to predict than plain scraping.

## Is Firecrawl open source? Can I self-host it?

Yes, with caveats. The core repository ([github.com/firecrawl/firecrawl](https://github.com/firecrawl/firecrawl)) is licensed **AGPL-3.0** , with the SDKs under MIT. You can clone it and run the whole stack yourself, and the project maintains an official `SELF_HOST.md` guide.

Two things to understand before choosing the self-hosted route:

- **The cloud version is ahead of the OSS version.** The managed anti-bot layer (their Fire-engine infrastructure, enhanced proxies, and some advanced features) is not part of the open-source stack. Self-hosted Firecrawl fetches with the bundled Playwright service (plus a basic fetch fallback) — fine for cooperative sites, much weaker against Cloudflare-class protection.
- **AGPL is a real obligation.** If you modify Firecrawl and offer it as a network service, AGPL-3.0 requires you to publish your modifications. For internal pipelines this usually doesn't bite; for products built on a modified Firecrawl, talk to a lawyer.

## Self-hosting Firecrawl with Docker

The supported path is Docker Compose (v2). Firecrawl doesn't publish a minimum host size, but the stack runs several services including a headless browser, so give Docker generous memory.

```bash
# 1. Clone the repo and check out a release tag (see the self-hosting docs for the current one)
git clone https://github.com/firecrawl/firecrawl.git
cd firecrawl
git checkout <release-tag>
```

Create a `.env` in the repository root (don't copy `apps/api/.env.example` — the root `.env` only feeds the Compose file). The minimum for a local instance:

```bash
# Skip the database-backed auth stack for local/self-hosted use
USE_DB_AUTHENTICATION=false
POSTGRES_USER=postgres
POSTGRES_PASSWORD=replace-with-at-least-32-random-characters
POSTGRES_DB=postgres
# Optional: enables the AI-powered extraction features
OPENAI_API_KEY=sk-...
```

Then start the stack:

```bash
docker compose up --build -d
docker compose ps --all # expect the API, workers, Playwright, Redis, RabbitMQ, PostgreSQL and supporting containers
```

Test it — no API key needed when `USE_DB_AUTHENTICATION=false`:

```bash
curl -X POST http://localhost:3002/v2/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'
```

The SDKs work against a self-hosted instance by pointing them at your URL:

```python
firecrawl = Firecrawl(api_url="http://localhost:3002", api_key="anything")
```

Operational notes from people who run it: give the Playwright service enough memory (headless Chromium is the hungriest container), put the API behind a reverse proxy with auth if it's reachable beyond localhost, and pin a release tag rather than tracking `main` — the compose file's service layout has changed between versions.

## Rate limits

The hosted API enforces per-plan rate limits on requests per minute, with concurrent-browser limits on crawls; higher tiers get higher ceilings, and the current numbers live in the docs. When you hit one you'll get HTTP `429`. The SDK already retries a few times (`max_retries=3` by default); if you need more control, back off exponentially yourself rather than hammering:

```python
import time

def scrape_with_retry(firecrawl, url, attempts=4):
    for i in range(attempts):
        try:
            return firecrawl.scrape(url, formats=["markdown"])
        except Exception as e:
            if "429" in str(e) and i < attempts - 1:
                time.sleep(2 ** i) # 1s, 2s, 4s
                continue
            raise
```

Self-hosted instances have no imposed rate limits — your ceiling is your hardware and, more importantly, the politeness of your crawling toward target sites.

## Scraping PDFs

`/scrape` handles PDF URLs: point it at a `.pdf` and it returns the extracted text as markdown, which makes it useful for ingesting documentation, papers, and reports into RAG pipelines. Parsing costs scale with document size (+1 credit per PDF page on the hosted API), so a 300-page PDF is not a one-credit operation. For scanned/image-only PDFs, extraction quality depends on the document — test on your real corpus before building a pipeline around it.

## Robots.txt, terms of service, and staying legal

This section applies to Firecrawl, to [any scraping API](https://webscraping.ai/blog/is-web-scraping-legal), and to code you write yourself — the tool doesn't change the rules.

**Robots.txt.** Firecrawl's crawler respects robots.txt directives when crawling sites; a single `/scrape` of a URL you supply is treated like a direct visit. Regardless of what the tool enforces, checking `https://target.com/robots.txt` before scraping is the baseline of good citizenship — `Disallow` rules and `Crawl-delay` tell you what the site operator considers acceptable.

**The short version of the law** (not legal advice):

- **Public data is generally scrapable.** U.S. courts (notably _hiQ v. LinkedIn_) have held that scraping publicly accessible pages doesn't violate the CFAA. The EU has similar space under database and copyright law for facts.
- **Personal data is different.** GDPR and CCPA apply to scraped personal data exactly as they do to any other collection — you need a lawful basis, and "it was public" is not automatically one.
- **Terms of service matter most when you agreed to them.** Logging in and scraping behind authentication puts you in contract territory; scraping public pages without an account is a weaker claim against you, but not zero.
- **Copyright still exists.** Scraping facts (prices, specs) is safer ground than republishing creative content wholesale.

Practical rules: scrape public pages, keep request rates polite, identify a contact in your user agent for large crawls, don't hoard personal data, and don't republish copyrighted content. Our [complete guide to web scraping legality](https://webscraping.ai/blog/is-web-scraping-legal) covers this in depth.

## Firecrawl vs. WebScraping.AI: which fits your use case?

Full disclosure: WebScraping.AI (this site) is a scraping API and competes with Firecrawl on part of this territory. The honest comparison:

| | Firecrawl | WebScraping.AI |
| --- | --- | --- |
| Whole-site crawl to markdown | **Yes** (`/crawl`) | — (single URLs) |
| Open source / self-host | **Yes** (AGPL-3.0) | — |
| LLM-ready output | Native markdown | `/text` endpoint ([try it](https://webscraping.ai/tools/html-to-markdown)) |
| AI structured extraction | `/extract` | [`/ai/fields`](https://webscraping.ai/ai-web-scraping) |
| Residential / stealth proxies | Enhanced proxy mode (no extra credits) | **Explicit tiers at published costs** |
| Failed requests | Billed on fetch, even error status | **Free — pay only for success** |
| Pricing model | Credits, dynamic costs on AI features | Published credit table, fixed cost per request type |

One thing that is mostly _not_ a difference: credits reset monthly on both services (Firecrawl rolls some over only on annual Scale and Enterprise plans), so size your plan to steady-state usage either way.

**Pick Firecrawl when** you're ingesting entire sites into a [RAG knowledge base](https://webscraping.ai/use-cases/rag-knowledge-base), you want native markdown, or you need an open-source engine you can run yourself.

**Pick WebScraping.AI when** your targets fight back. Protected sites are where explicit [residential and stealth proxy control](https://webscraping.ai/docs) and pay-only-for-success billing change the economics:

```bash
# Rendered HTML through a residential proxy — billed only if it succeeds
curl "https://api.webscraping.ai/html?api_key=YOUR_KEY&url=https://protected-site.com&js=true&proxy=residential"

# Or skip parsing entirely: structured fields from any page
curl "https://api.webscraping.ai/ai/fields?api_key=YOUR_KEY&url=https://protected-site.com/product&fields[name]=Product+name&fields[price]=Price+with+currency"
```

There's a [free trial](https://webscraping.ai/) with no card required, and a detailed feature-by-feature breakdown on the [Firecrawl alternative page](https://webscraping.ai/firecrawl-alternative).

## Frequently asked questions

**Is Firecrawl free?** There's a free plan with 1,000 credits a month (no card required) that's enough to evaluate it, and the open-source version is free to self-host on your own hardware. Sustained use of the hosted API requires a paid plan, starting at $16/month billed annually as of September 2026.

**What is Firecrawl used for?** Mostly feeding web content to LLMs: crawling documentation sites and knowledge bases into vector stores for RAG, converting pages to clean markdown for prompts, and extracting structured data with `/extract`. It's optimized for "web to LLM-ready data" rather than general-purpose browser automation.

**Can I self-host Firecrawl for free?** Yes — clone the AGPL-3.0 repository and run `docker compose up --build -d` on a host with plenty of memory. You give up the hosted platform's managed anti-bot layer and enhanced proxies, so expect lower success rates on protected sites than the cloud version delivers.

**Does Firecrawl work on sites protected by Cloudflare?** The hosted version has an enhanced proxy mode (billed at the same 1 credit per page) that handles many protected sites; the self-hosted version fetches with plain Playwright and struggles against serious anti-bot systems. If protected targets are your main workload, compare success rates against a proxy-first API like [WebScraping.AI](https://webscraping.ai/firecrawl-alternative) before committing.

**Does Firecrawl respect robots.txt?** Its site crawler honors robots.txt when discovering pages; single-URL scrapes are treated like direct visits. Either way, the responsibility for scraping a site appropriately — robots.txt, rate limits, terms — stays with you, not the tool.

**Is it legal to use Firecrawl for web scraping?** Using the tool is legal; what matters is what you scrape. Public, non-personal data at polite request rates is generally fine in the U.S. and EU. Personal data triggers GDPR/CCPA, authenticated scraping can breach terms you agreed to, and republishing copyrighted content is a separate problem. See our [web scraping legality guide](https://webscraping.ai/blog/is-web-scraping-legal) for specifics.