---
title: "cURL Commands and Options for Web Scraping: Complete Guide"
description: "Every cURL command and option you need for web scraping, with copy-paste examples: GET/POST requests, redirects, cookies, auth, proxies, retries, and downloads."
url: https://webscraping.ai/blog/curl-commands-for-web-scraping
markdown_index: https://webscraping.ai/llms.txt
---
# cURL Commands and Options for Web Scraping: Complete Guide

cURL is the fastest way to fetch a page, test an API, or debug why your scraper gets blocked — no dependencies, no boilerplate, available on every system. But its 250+ options hide the twenty that actually matter. This guide is the complete reference for using cURL in web scraping and API work: the essential commands, every HTTP method, cookies and sessions, authentication, error handling, and parallel requests — with copy-pasteable examples throughout.

## Key Takeaways

- `curl -L -o page.html "https://example.com"` covers the basic download case: follow redirects, save to a file
- Use `--compressed`, a real `User-Agent`, and a cookie jar (`-b`/`-c`) to look like a browser instead of a bot
- `-d` sends POST data; `--data-raw` and `--data-binary` differ only in how they treat `@` and whitespace
- `--fail`, `--retry`, and `-w "%{http_code}"` turn cURL into a scriptable, error-aware client
- cURL cannot execute JavaScript — for dynamic pages you need a headless browser or a rendering API

## What is cURL?

cURL ("client URL") is a command-line tool for transferring data with URLs, built on the libcurl library that powers HTTP clients in thousands of applications. It speaks HTTP/HTTPS plus a long tail of protocols (FTP, SFTP, SMTP, WebSocket), and it's already installed almost everywhere: Windows 10 and later ship it out of the box, as do macOS and virtually every Linux distribution — on Windows just run `curl` in PowerShell or CMD (mind that PowerShell aliases `curl` to `Invoke-WebRequest` in older versions; call `curl.exe` to be explicit).

For web scraping, cURL fills three roles: a reconnaissance tool (what does this URL actually return?), a debugging tool (which header makes the difference?), and a lightweight scraper in its own right when wired into shell scripts. Browser DevTools can export any request as a cURL command ("Copy as cURL"), which makes it the lingua franca for reproducing and sharing HTTP requests. And when a command is ready to become real code, the [curl converter](https://webscraping.ai/tools/curl-converter) turns it into Python, JavaScript, PHP, Go, or Java — right in the browser.

## cURL cheat sheet: the essential options

The cheat sheet. Every option below is covered in detail later in the guide:

| Option | What it does |
| --- | --- |
| `-o file` / `-O` | Save output to a file / use the remote filename |
| `-L` (`--location`) | Follow redirects |
| `-s` (`--silent`) | Hide the progress meter |
| `-v` (`--verbose`) | Show request/response headers and TLS details |
| `-H "Name: value"` | Set a request header |
| `-A "..."` (`--user-agent`) | Set the User-Agent string |
| `-d` / `--data` | Send a POST body (form-encoded by default) |
| `-F` (`--form`) | Send `multipart/form-data` (file uploads) |
| `-X METHOD` | Override the HTTP method (PUT, PATCH, DELETE) |
| `-u user:pass` | Basic authentication |
| `-b` / `-c file` | Send cookies from / save cookies to a file |
| `--compressed` | Request and auto-decompress gzip/brotli |
| `-I` (`--head`) | Fetch headers only |
| `-k` (`--insecure`) | Skip TLS certificate verification |
| `-x` (`--proxy`) | Route the request through a proxy |
| `--fail` | Exit non-zero on HTTP errors (4xx/5xx) |
| `--retry N` | Retry transient failures |
| `-w "format"` | Print response metadata (status code, timing) |
| `--max-time` / `--connect-timeout` | Total / connection timeout in seconds |
| `-G` | Append `-d` params to the URL as a query string |

## Downloading webpages and files

The simplest scraping command sends a GET request (cURL's default method) and prints the page's HTML to stdout:

```bash
curl "https://example.com"
```

To download a file — or a page — to disk, you almost always want three more flags: follow redirects, stay quiet, save the output:

```bash
# -o: choose the filename; -O: keep the remote filename
curl -sL -o page.html "https://example.com/products"
curl -sLO "https://example.com/report.pdf"
```

Quote your URLs. Unquoted `&` characters send the rest of your command to the background, which is a classic source of "cURL only fetched half my query string" bugs.

To send query parameters cleanly instead of hand-building the URL, use `-G` with `--data-urlencode` — cURL escapes values for you:

```bash
curl -G "https://example.com/search" \
  --data-urlencode "q=web scraping api" \
  --data-urlencode "page=2"
```

### Following redirects with -L

cURL does **not** follow redirects by default — a bare request to a site that redirects `http` → `https` or apex → `www` returns a 301 page and nothing else. `-L` (long form: `--location`) makes cURL follow the `Location` header chain, and `--max-redirs` caps how far it will go:

```bash
curl -L --max-redirs 5 "http://example.com"

# See the whole redirect chain with verbose output
curl -svL -o /dev/null "http://example.com" 2>&1 | grep -iE "^(<|>) (HTTP|Location)" # -i: HTTP/2 headers are lowercase
```

If a redirect crosses to a different host, cURL strips `Authorization` headers for safety. Add `--location-trusted` only when you understand the risk of re-sending credentials to the redirect target.

## Sending data: GET, POST, and friends

### POST requests with -d

`-d` (alias `--data`) switches the request to POST and sends the payload with `Content-Type: application/x-www-form-urlencoded` — a standard HTML form submission:

```bash
curl -d "username=demo&password=secret" "https://example.com/login"
```

For JSON APIs, set the content type yourself (or use `--json`, available since cURL 7.82, which sets both `Content-Type` and `Accept`):

```bash
curl -d '{"query": "laptops", "limit": 20}' \
  -H "Content-Type: application/json" \
  "https://api.example.com/search"

# Equivalent on modern cURL
curl --json '{"query": "laptops", "limit": 20}' "https://api.example.com/search"
```

A leading `@` reads the body from a file: `-d @payload.json`.

### --data vs --data-raw vs --data-binary

The three data flags differ in small but bug-prone ways:

- `--data` / `-d` strips newlines from file input and treats a leading `@` as "read this file"
- `--data-raw` sends the argument literally — an `@` is just an `@`. Use it when your payload starts with `@` or you want zero surprises
- `--data-binary` preserves the payload byte-for-byte, including newlines — required for uploading files whose content matters (NDJSON, XML documents, images)

```bash
# Sends the five bytes "@user"
curl --data-raw "@user" "https://api.example.com/mentions"

# Uploads the file exactly as-is, newlines intact
curl --data-binary @document.xml -H "Content-Type: application/xml" \
  "https://api.example.com/import"
```

### PUT, PATCH, and DELETE

`-X` overrides the method; the data flags work the same way:

```bash
curl -X PUT -H "Content-Type: application/json" \
  -d '{"name": "Updated Product", "price": 49.99}' \
  "https://api.example.com/products/42"

curl -X PATCH -H "Content-Type: application/json" \
  -d '{"price": 39.99}' \
  "https://api.example.com/products/42"

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  "https://api.example.com/products/42"
```

Don't combine `-X POST` with `-d` — `-d` already implies POST, and an explicit `-X` breaks the automatic method switching on redirects.

### Multipart form data and file uploads

`-F` builds a `multipart/form-data` body — what browsers send for forms with file inputs. `@` attaches a file, `<` inserts a file's _content_ as a text field:

```bash
curl -F "title=Q3 Report" \
  -F "document=@report.pdf;type=application/pdf" \
  -F "notes=<notes.txt" \
  "https://example.com/upload"
```

## Looking like a browser

Sites profile the request fingerprint before deciding what to serve. cURL's defaults — `User-Agent: curl/8.x`, no `Accept-Language`, no cookies — are the easiest bot signature there is. A minimal browser-like request:

```bash
curl -sL --compressed \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" \
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
  -H "Accept-Language: en-US,en;q=0.9" \
  -e "https://www.google.com/" \
  "https://example.com/products"
```

`-e` sets the `Referer`; `--compressed` both advertises gzip/brotli support (browsers always do) and transparently decompresses the response, so you never pipe gzip bytes into your parser by accident. For sustained scraping, rotate user agents per request — see our [user-agent rotation guide](https://webscraping.ai/blog/user-agent-rotation-for-web-scraping).

Headers you set with `-H` override cURL's defaults, and an empty value (`-H "Accept:"`) removes a header entirely.

## Cookies and sessions

HTTP is stateless; sites track sessions with cookies. cURL handles them with two flags: `-c` writes received cookies to a _cookie jar_ file, `-b` sends cookies from a file (or a literal string):

```bash
# Log in once, persist the session cookie
curl -c cookies.txt -d "username=demo&password=secret" "https://example.com/login"

# Reuse the session on subsequent requests
curl -b cookies.txt "https://example.com/account/orders"

# Read and write in the same request (long-lived sessions)
curl -b cookies.txt -c cookies.txt "https://example.com/cart"

# One-off literal cookie
curl -b "sessionid=abc123; currency=USD" "https://example.com/prices"
```

The jar is a plain Netscape-format text file — you can inspect it, edit it, or seed it with cookies exported from a real browser session. Delete it to "log out."

## Authentication

### Basic auth

`-u` handles HTTP Basic authentication — cURL base64-encodes the credentials into an `Authorization` header for you:

```bash
curl -u apiuser:s3cret "https://api.example.com/data"

# Omit the password to be prompted interactively (keeps it out of shell history)
curl -u apiuser "https://api.example.com/data"
```

### Bearer tokens and API keys

Token schemes are just headers:

```bash
curl -H "Authorization: Bearer $ACCESS_TOKEN" "https://api.example.com/me"
curl -H "X-API-Key: $API_KEY" "https://api.example.com/data"
```

### OAuth 2.0

cURL doesn't implement OAuth flows, but the token exchange is a plain POST — fetch the token, then use it as a bearer:

```bash
ACCESS_TOKEN=$(curl -s -X POST "https://auth.example.com/oauth/token" \
  -d "grant_type=client_credentials" \
  -d "client_id=$CLIENT_ID" \
  -d "client_secret=$CLIENT_SECRET" | jq -r '.access_token')

curl -H "Authorization: Bearer $ACCESS_TOKEN" "https://api.example.com/reports"
```

## TLS and certificates

cURL verifies TLS certificates by default. When scraping staging environments or debugging proxies you'll hit certificate failures; `-k` (`--insecure`) makes cURL ignore SSL certificate errors — acceptable for local debugging, never for credentialed production traffic. The right fixes are pointing cURL at the correct CA bundle or client certificate:

```bash
# Trust a private/self-signed CA
curl --cacert internal-ca.pem "https://staging.example.com"

# Mutual TLS (client certificate + key)
curl --cert client.pem --key client-key.pem "https://api.example.com/secure"

# Inspect a site's certificate chain
curl -vI "https://example.com" 2>&1 | grep -A 8 "Server certificate"
```

## Handling responses

### JSON and XML

cURL outputs whatever the server sends; pair it with `jq` for JSON and `xmllint` for XML:

```bash
curl -s "https://api.example.com/products" | jq '.products[] | {name, price}'

curl -s -H "Accept: application/xml" "https://api.example.com/feed" \
  | xmllint --xpath "//item/title/text()" -
```

For scraping HTML, cURL is only the fetch layer — pipe into an HTML parser in [Python](https://webscraping.ai/blog/web-scraping-with-python) or [JavaScript](https://webscraping.ai/blog/web-scraping-with-javascript) rather than regexing raw HTML.

### Headers only, status codes, and timing with -w

`-I` sends a HEAD request when you only need headers (note: some servers answer HEAD differently than GET — `curl -sI` vs `curl -s -o /dev/null -D -` can disagree). The `-w` (write-out) flag prints response metadata after the transfer, which is the backbone of health checks and monitoring scripts:

```bash
# Just the status code
curl -s -o /dev/null -w "%{http_code}" "https://example.com"

# Status + timing breakdown
curl -s -o /dev/null -w "status=%{http_code} dns=%{time_namelookup}s connect=%{time_connect}s total=%{time_total}s\n" \
  "https://example.com"
```

### Debugging with verbose output

`-v` (`--verbose`) prints the full exchange — DNS resolution, TLS handshake, every request and response header — which answers most "why does this work in my browser but not in cURL" questions. For byte-level inspection, `--trace-ascii dump.txt` logs everything to a file:

```bash
curl -v "https://example.com" 2>&1 | grep -E "^(\*|>|<)"

# Response headers only, body discarded
curl -sD - -o /dev/null "https://example.com"
```

A minimal health-check script:

```bash
STATUS=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "https://example.com/health")
if ["$STATUS" -ne 200]; then
  echo "Health check failed: HTTP $STATUS"
  exit 1
fi
```

## Error handling, retries, and timeouts

By default cURL exits 0 even when the server returns a 404 or 500 — it successfully transferred an error page. `--fail` makes HTTP errors ≥ 400 produce exit code 22 (and suppresses the error body), which is what you want in scripts and cron jobs:

```bash
curl --fail -s -o data.json "https://api.example.com/export" || echo "Download failed"
```

cURL's exit codes distinguish failure classes: `6` couldn't resolve host, `7` connection refused, `28` timeout, `22` HTTP error (with `--fail`), `35` TLS handshake failure. Check `$?` to branch on them.

`--retry` re-attempts transient failures (timeouts, 408, 429, 500, 502, 503, 504) with exponential backoff — unless you set `--retry-delay`, which switches to a fixed wait between attempts:

```bash
curl --fail --retry 5 --retry-delay 2 --retry-max-time 120 \
  --connect-timeout 10 --max-time 60 \
  "https://api.example.com/flaky-endpoint"
```

Set both cURL timeout options on every scripted request: `--connect-timeout` bounds the TCP/TLS handshake, `--max-time` bounds the whole transfer. Without them, one hung connection stalls your whole pipeline. When you're being rate-limited, `--retry` respects a `Retry-After` header automatically.

## Multiple and parallel requests

cURL accepts multiple URLs in one invocation, and URL globbing generates ranges — useful for paginated listings:

```bash
# Pages 1-50, sequentially, over one reused connection
curl -sL -o "page_#1.html" "https://example.com/products?page=[1-50]"
```

Since 7.66, `-Z` (`--parallel`) runs transfers concurrently in a single process — `--parallel-max` caps the concurrency:

```bash
curl -ZL --parallel-max 10 -o "page_#1.html" "https://example.com/products?page=[1-100]"
```

On older systems, `xargs -P` does the same with multiple processes:

```bash
seq 1 100 | xargs -P 10 -I{} curl -sL -o "page_{}.html" "https://example.com/products?page={}"
```

Be a good citizen: cap concurrency, add `--retry`, and space batches out. Hammering a site with 100 parallel connections is the fastest way to get your IP banned — at scale you'll want [rotating proxies](https://webscraping.ai/blog/best-proxy-providers-for-web-scraping) regardless. A cron job built from exactly these pieces — globbed pagination, `--fail`, `--retry`, and `jq` — is a perfectly respectable [price monitoring](https://webscraping.ai/use-cases/price-monitoring) or [SERP monitoring](https://webscraping.ai/use-cases/serp-monitoring) pipeline.

## Using cURL with a proxy

`-x` routes requests through a proxy — HTTP, HTTPS, or SOCKS:

```bash
curl -x "http://user:pass@proxy.example.com:8080" "https://example.com"
curl -x "socks5h://127.0.0.1:9050" "https://check.torproject.org"
```

(`socks5h` resolves DNS through the proxy too — usually what you want.) The `--resolve` option is a related debugging tool: it pins a hostname to an IP without touching `/etc/hosts`, which is how you test a site on a new server before DNS cutover:

```bash
curl --resolve "example.com:443:203.0.113.7" "https://example.com"
```

## cURL vs wget

The two classic transfer tools overlap but aim at different jobs:

| | cURL | wget |
| --- | --- | --- |
| Primary job | Make HTTP requests (APIs, forms, debugging) | Download files and mirror sites |
| Recursive download | No | Yes (`-r`, `--mirror`) |
| Resume downloads | `-C -` | Automatic retry/resume by default |
| Protocols | HTTP(S), FTP, SFTP, SMTP, WS, and more | HTTP(S), FTP |
| Upload / POST / methods | Full support | Limited |
| Library form | libcurl (embeddable everywhere) | None |
| Preinstalled | Windows, macOS, Linux | Most Linux only |

For scraping work the answer is usually cURL: request control (headers, cookies, methods, proxies) is what scraping needs, and "Copy as cURL" from browser DevTools gives you a working starting point. Reach for wget when the task is "download this directory tree" or "mirror this site."

## Where cURL stops: JavaScript rendering

cURL speaks HTTP; it does not run JavaScript. If a page's content is assembled client-side — React/Vue storefronts, infinite scroll, lazy-loaded prices — cURL receives the empty application shell, and no amount of header spoofing changes that. The same goes for JavaScript-based anti-bot challenges (Cloudflare, DataDome): they're designed to require a real browser environment.

At that point you have two options: drive a headless browser yourself, or keep your cURL workflow and swap the target URL for a rendering API. [WebScraping.AI](https://webscraping.ai/) renders pages in headless Chrome with rotating proxies behind a plain HTTP endpoint, so the fix is one URL change:

```bash
# Returns the fully rendered HTML after JavaScript execution
curl -G "https://api.webscraping.ai/html" \
  --data-urlencode "url=https://example.com/spa-products" \
  --data-urlencode "api_key=$WSAI_API_KEY"

# Or ask a question about the page and get an AI-extracted answer
curl -G "https://api.webscraping.ai/ai/question" \
  --data-urlencode "url=https://example.com/product/42" \
  --data-urlencode "question=What is the price?" \
  --data-urlencode "api_key=$WSAI_API_KEY"
```

Everything in this guide — `--fail`, `--retry`, `-w`, parallel transfers, `jq` pipelines — works unchanged, because it's still just cURL. See the [API documentation](https://webscraping.ai/docs) for the `/text`, `/selected`, and AI extraction endpoints, or the broader [AI web scraping overview](https://webscraping.ai/ai-web-scraping). If the shell script is a stepping stone to something scheduled, the same endpoints are wired into an [n8n node](https://webscraping.ai/integrations/n8n) and an [MCP server](https://webscraping.ai/integrations/mcp-server) for agent workflows.

## Frequently asked questions

**Why does cURL return an empty page or different content than my browser?** Three usual suspects, in order: the site redirected and you didn't pass `-L`; the content is rendered by JavaScript (view-source in your browser to confirm — if it's not in the source, cURL can't see it either); or the site served bot-detection content based on your headers. Fix the first two with `-L` and a rendering API; the third with browser-like headers.

**How do I use cURL on Windows?** Every command in this guide works unchanged on Windows 10+ in CMD or PowerShell. The one trap: in Windows PowerShell 5.x, `curl` is an alias for `Invoke-WebRequest` — type `curl.exe` (or remove the alias) to get the real thing. In PowerShell 7+ and CMD, plain `curl` is already correct. Use double quotes around URLs; single quotes are not treated the same way by CMD.

**Is scraping with cURL legal?** Fetching publicly accessible pages is generally lawful, but terms of service, rate limits, and data-protection rules apply — see our detailed guide on [web scraping legality](https://webscraping.ai/blog/is-web-scraping-legal).

**What's the difference between `-L` and `--location`?** Nothing — every short cURL flag has a long synonym (`-A`/`--user-agent`, `-b`/`--cookie`, `-d`/`--data`). Use long forms in scripts for readability, short forms interactively.

**Can cURL scrape sites that require login?** Yes, if login is form- or token-based: POST the credentials with `-d`, capture the session with `-c cookies.txt`, then send `-b cookies.txt` on subsequent requests. Sites with JavaScript-driven or CAPTCHA-protected logins need a browser-based approach.