Firecrawl Guide: Self-Hosting, Pricing, API Keys, and Limits

Firecrawl is a web scraping API built for the LLM era: give it a URL and it returns clean, LLM-ready markdown instead of raw HTML. It can crawl entire sites, extract structured data with AI, and — unusually for this category — it's open source, so you can run it on your own hardware. This guide covers the questions people actually ask before adopting it: what it does, how to get an API key, what it costs, how to self-host it with Docker, where its limits are, and the legal ground rules that apply to it (and to every other scraping tool).

Key Takeaways

• Firecrawl converts web pages into markdown and structured data through four core endpoints: /scrape (one URL), /crawl (a whole site), /map (discover URLs), and /extract (AI-powered structured extraction).
• You get an API key by signing up at firecrawl.dev; keys are prefixed fc- and belong in an environment variable, never in code.
• Pricing is credit-based — roughly one credit per page scraped — with a free trial allotment and paid plans starting around $16/month as of mid-2026. Unused credits don't roll over, and a fetched page bills even when it returns an error status.
• The core repository is open source (AGPL-3.0) and self-hostable with Docker Compose, but the hosted cloud has features the self-hosted stack lacks — most importantly the managed anti-bot and stealth proxy layer.
• Scraping legally is about what you scrape and how, not the tool: respect robots.txt, don't collect personal data without a lawful basis, and honor site terms where they matter.

What is Firecrawl?

Firecrawl started as an open-source project by Mendable and grew into a hosted API used heavily in RAG (retrieval-augmented generation) pipelines. Its pitch is "websites into LLM-ready data": where a traditional scraping API hands you rendered HTML to parse, Firecrawl hands you markdown that can go straight into a vector store or a prompt.

The main endpoints:

It ships official Python and Node SDKs, an MCP server for AI agents, and integrations for LangChain, LlamaIndex, and workflow tools. JavaScript rendering is built in, with page "actions" (click, scroll, wait) for content that needs interaction before it appears.

What it is not: a browser automation framework. You don't get Puppeteer-style step-by-step control of a browser session — for that, use Playwright or Puppeteer directly and treat Firecrawl as the fetch-and-convert layer.

Getting a Firecrawl API key

1. Sign up at firecrawl.dev (email or GitHub/Google OAuth). New accounts get free trial credits with no card required.
2. Open the dashboard — your default API key is on the API Keys page, prefixed with fc-.
3. Store it as an environment variable:

export FIRECRAWL_API_KEY="fc-your-key-here"

The SDKs read that variable automatically:

from firecrawl import FirecrawlApp

app = FirecrawlApp()  # reads FIRECRAWL_API_KEY from the environment
result = app.scrape_url("https://example.com", formats=["markdown"])
print(result.markdown)

Or over plain HTTP:

curl -X POST https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

Standard key hygiene applies: keep keys out of git (use .env files plus a secrets manager in production), rotate a key from the dashboard if it leaks, and use separate keys per project so usage is attributable.

Firecrawl pricing: credits, plans, and gotchas

Firecrawl bills in credits. A standard /scrape of one page costs one credit; crawling is one credit per page crawled; AI-powered /extract and the agent features consume more, and the agent pricing is dynamic — a single complex query can burn a large number of credits.

As of mid-2026 the shape of the plans (check firecrawl.dev/pricing for current numbers — they change):

• Free — a trial allotment of credits (around 500) to evaluate the service.
• Hobby — entry plan from roughly $16/month for a few thousand credits.
• Standard / Growth — mid tiers in the ~$83–$333/month range for 100k–500k credits with higher rate limits.
• Enterprise — custom volume, SLAs, and dedicated support.

Three billing details worth knowing before you commit:

1. Credits expire monthly. Unused credits are forfeited at the end of each billing cycle — size your plan to actual usage, not peak usage.
2. A fetch that returns an error status still bills. Firecrawl charges when a page fetches, even if the response is an error page. On well-behaved targets this rarely matters; on protected sites where many attempts come back blocked, it adds up.
3. Dynamic costs on AI features. /extract and agent queries don't cost a flat one credit, so budgets for AI-heavy pipelines are harder to predict than plain scraping.

Is Firecrawl open source? Can I self-host it?

Yes, with caveats. The core repository (github.com/firecrawl/firecrawl) is licensed AGPL-3.0, with the SDKs under MIT. You can clone it and run the whole stack yourself, and the project maintains an official SELF_HOST.md guide.

Two things to understand before choosing the self-hosted route:

• The cloud version is ahead of the OSS version. The managed anti-bot layer (their "fire engine" infrastructure, stealth proxying, and some advanced features) is not part of the open-source stack. Self-hosted Firecrawl fetches with plain Playwright — fine for cooperative sites, much weaker against Cloudflare-class protection.
• AGPL is a real obligation. If you modify Firecrawl and offer it as a network service, AGPL-3.0 requires you to publish your modifications. For internal pipelines this usually doesn't bite; for products built on a modified Firecrawl, talk to a lawyer.

Self-hosting Firecrawl with Docker

The supported path is Docker Compose. Requirements are modest but real: Docker plus roughly 8–12 GB of RAM and ~20 GB of disk for the full stack.

# 1. Clone and enter the repo
git clone https://github.com/firecrawl/firecrawl.git
cd firecrawl

# 2. Create the environment file from the template
cp apps/api/.env.example .env

Edit .env — the minimum for a local instance:

PORT=3002
HOST=0.0.0.0
# Skip the Supabase-backed auth stack for local/self-hosted use
USE_DB_AUTHENTICATION=false
# Optional: enables the AI-powered extract features
OPENAI_API_KEY=sk-...

Then start the stack:

docker compose up -d
docker ps   # expect the API, worker, Playwright service, Redis, and supporting containers

Test it — no API key needed when USE_DB_AUTHENTICATION=false:

curl -X POST http://localhost:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

The SDKs work against a self-hosted instance by pointing them at your URL:

app = FirecrawlApp(api_url="http://localhost:3002", api_key="anything")

Operational notes from people who run it: give the Playwright service enough memory (headless Chromium is the hungriest container), put the API behind a reverse proxy with auth if it's reachable beyond localhost, and pin a release tag rather than tracking main — the compose file's service layout has changed between versions.

Rate limits

The hosted API enforces per-plan rate limits on requests per minute, with concurrent-browser limits on crawls; higher tiers get higher ceilings, and the current numbers live in the docs. When you hit one you'll get HTTP 429 — back off exponentially and retry rather than hammering:

import time

def scrape_with_retry(app, url, attempts=4):
    for i in range(attempts):
        try:
            return app.scrape_url(url, formats=["markdown"])
        except Exception as e:
            if "429" in str(e) and i < attempts - 1:
                time.sleep(2 ** i)  # 1s, 2s, 4s
                continue
            raise

Self-hosted instances have no imposed rate limits — your ceiling is your hardware and, more importantly, the politeness of your crawling toward target sites.

Scraping PDFs

/scrape handles PDF URLs: point it at a .pdf and it returns the extracted text as markdown, which makes it useful for ingesting documentation, papers, and reports into RAG pipelines. Parsing costs scale with document size (billed per PDF page on the hosted API), so a 300-page PDF is not a one-credit operation. For scanned/image-only PDFs, extraction quality depends on the document — test on your real corpus before building a pipeline around it.

Robots.txt, terms of service, and staying legal

This section applies to Firecrawl, to any scraping API, and to code you write yourself — the tool doesn't change the rules.

Robots.txt. Firecrawl's crawler respects robots.txt directives when crawling sites; a single /scrape of a URL you supply is treated like a direct visit. Regardless of what the tool enforces, checking https://target.com/robots.txt before scraping is the baseline of good citizenship — Disallow rules and Crawl-delay tell you what the site operator considers acceptable.

The short version of the law (not legal advice):

• Public data is generally scrapable. U.S. courts (notably hiQ v. LinkedIn) have held that scraping publicly accessible pages doesn't violate the CFAA. The EU has similar space under database and copyright law for facts.
• Personal data is different. GDPR and CCPA apply to scraped personal data exactly as they do to any other collection — you need a lawful basis, and "it was public" is not automatically one.
• Terms of service matter most when you agreed to them. Logging in and scraping behind authentication puts you in contract territory; scraping public pages without an account is a weaker claim against you, but not zero.
• Copyright still exists. Scraping facts (prices, specs) is safer ground than republishing creative content wholesale.

Practical rules: scrape public pages, keep request rates polite, identify a contact in your user agent for large crawls, don't hoard personal data, and don't republish copyrighted content. Our complete guide to web scraping legality covers this in depth.

Firecrawl vs. WebScraping.AI: which fits your use case?

Full disclosure: WebScraping.AI (this site) is a scraping API and competes with Firecrawl on part of this territory. The honest comparison:

Pick Firecrawl when you're ingesting entire sites into a RAG pipeline, you want native markdown, or you need an open-source engine you can run yourself.

Pick WebScraping.AI when your targets fight back. Protected sites are where explicit residential and stealth proxy control and pay-only-for-success billing change the economics:

# Rendered HTML through a residential proxy — billed only if it succeeds
curl "https://api.webscraping.ai/html?api_key=YOUR_KEY&url=https://protected-site.com&js=true&proxy=residential"

# Or skip parsing entirely: structured fields from any page
curl "https://api.webscraping.ai/ai/fields?api_key=YOUR_KEY&url=https://protected-site.com/product&fields[name]=Product+name&fields[price]=Price+with+currency"

There's a free trial with no card required, and a detailed feature-by-feature breakdown on the Firecrawl alternative page.

Frequently asked questions

Is Firecrawl free? There's a free trial allotment of credits (no card required) that's enough to evaluate it, and the open-source version is free to self-host on your own hardware. Sustained use of the hosted API requires a paid plan, starting around $16/month as of mid-2026.

What is Firecrawl used for? Mostly feeding web content to LLMs: crawling documentation sites and knowledge bases into vector stores for RAG, converting pages to clean markdown for prompts, and extracting structured data with /extract. It's optimized for "web to LLM-ready data" rather than general-purpose browser automation.

Can I self-host Firecrawl for free? Yes — clone the AGPL-3.0 repository and run docker compose up -d with ~8–12 GB of RAM available. You give up the hosted platform's managed anti-bot layer and stealth proxies, so expect lower success rates on protected sites than the cloud version delivers.

Does Firecrawl work on sites protected by Cloudflare? The hosted version has a stealth mode (at extra credit cost) that handles many protected sites; the self-hosted version fetches with plain Playwright and struggles against serious anti-bot systems. If protected targets are your main workload, compare success rates against a proxy-first API like WebScraping.AI before committing.

Does Firecrawl respect robots.txt? Its site crawler honors robots.txt when discovering pages; single-URL scrapes are treated like direct visits. Either way, the responsibility for scraping a site appropriately — robots.txt, rate limits, terms — stays with you, not the tool.

Is it legal to use Firecrawl for web scraping? Using the tool is legal; what matters is what you scrape. Public, non-personal data at polite request rates is generally fine in the U.S. and EU. Personal data triggers GDPR/CCPA, authenticated scraping can breach terms you agreed to, and republishing copyrighted content is a separate problem. See our web scraping legality guide for specifics.