cURL Commands and Options for Web Scraping: Complete Guide

cURL is the fastest way to fetch a page, test an API, or debug why your scraper gets blocked — no dependencies, no boilerplate, available on every system. But its 250+ options hide the twenty that actually matter. This guide is the complete reference for using cURL in web scraping and API work: the essential commands, every HTTP method, cookies and sessions, authentication, error handling, and parallel requests — with copy-pasteable examples throughout.

Key Takeaways

• curl -L -o page.html "https://example.com" covers the basic download case: follow redirects, save to a file
• Use --compressed, a real User-Agent, and a cookie jar (-b/-c) to look like a browser instead of a bot
• -d sends POST data; --data-raw and --data-binary differ only in how they treat @ and whitespace
• --fail, --retry, and -w "%{http_code}" turn cURL into a scriptable, error-aware client
• cURL cannot execute JavaScript — for dynamic pages you need a headless browser or a rendering API

What is cURL?

cURL ("client URL") is a command-line tool for transferring data with URLs, built on the libcurl library that powers HTTP clients in thousands of applications. It speaks HTTP/HTTPS plus a long tail of protocols (FTP, SFTP, SMTP, WebSocket), and it's already installed almost everywhere: Windows 10 and later ship it out of the box, as do macOS and virtually every Linux distribution — on Windows just run curl in PowerShell or CMD (mind that PowerShell aliases curl to Invoke-WebRequest in older versions; call curl.exe to be explicit).

For web scraping, cURL fills three roles: a reconnaissance tool (what does this URL actually return?), a debugging tool (which header makes the difference?), and a lightweight scraper in its own right when wired into shell scripts. Browser DevTools can export any request as a cURL command ("Copy as cURL"), which makes it the lingua franca for reproducing and sharing HTTP requests.

cURL cheat sheet: the essential options

The cheat sheet. Every option below is covered in detail later in the guide:

Downloading webpages and files

The simplest scraping command sends a GET request (cURL's default method) and prints the page's HTML to stdout:

curl "https://example.com"

To download a file — or a page — to disk, you almost always want three more flags: follow redirects, stay quiet, save the output:

# -o: choose the filename; -O: keep the remote filename
curl -sL -o page.html "https://example.com/products"
curl -sLO "https://example.com/report.pdf"

Quote your URLs. Unquoted & characters send the rest of your command to the background, which is a classic source of "cURL only fetched half my query string" bugs.

To send query parameters cleanly instead of hand-building the URL, use -G with --data-urlencode — cURL escapes values for you:

curl -G "https://example.com/search" \
  --data-urlencode "q=web scraping api" \
  --data-urlencode "page=2"

Following redirects with -L

cURL does not follow redirects by default — a bare request to a site that redirects http → https or apex → www returns a 301 page and nothing else. -L (long form: --location) makes cURL follow the Location header chain, and --max-redirs caps how far it will go:

curl -L --max-redirs 5 "http://example.com"

# See the whole redirect chain with verbose output
curl -svL -o /dev/null "http://example.com" 2>&1 | grep -E "^(<|>) (HTTP|Location)"

If a redirect crosses to a different host, cURL strips Authorization headers for safety. Add --location-trusted only when you understand the risk of re-sending credentials to the redirect target.

Sending data: GET, POST, and friends

POST requests with -d

-d (alias --data) switches the request to POST and sends the payload with Content-Type: application/x-www-form-urlencoded — a standard HTML form submission:

curl -d "username=demo&password=secret" "https://example.com/login"

For JSON APIs, set the content type yourself (or use --json, available since cURL 7.82, which sets both Content-Type and Accept):

curl -d '{"query": "laptops", "limit": 20}' \
  -H "Content-Type: application/json" \
  "https://api.example.com/search"

# Equivalent on modern cURL
curl --json '{"query": "laptops", "limit": 20}' "https://api.example.com/search"

A leading @ reads the body from a file: -d @payload.json.

--data vs --data-raw vs --data-binary

The three data flags differ in small but bug-prone ways:

• --data / -d strips newlines from file input and treats a leading @ as "read this file"
• --data-raw sends the argument literally — an @ is just an @. Use it when your payload starts with @ or you want zero surprises
• --data-binary preserves the payload byte-for-byte, including newlines — required for uploading files whose content matters (NDJSON, XML documents, images)

# Sends the five bytes "@user"
curl --data-raw "@user" "https://api.example.com/mentions"

# Uploads the file exactly as-is, newlines intact
curl --data-binary @document.xml -H "Content-Type: application/xml" \
  "https://api.example.com/import"

PUT, PATCH, and DELETE

-X overrides the method; the data flags work the same way:

curl -X PUT -H "Content-Type: application/json" \
  -d '{"name": "Updated Product", "price": 49.99}' \
  "https://api.example.com/products/42"

curl -X PATCH -H "Content-Type: application/json" \
  -d '{"price": 39.99}' \
  "https://api.example.com/products/42"

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  "https://api.example.com/products/42"

Don't combine -X POST with -d — -d already implies POST, and an explicit -X breaks the automatic method switching on redirects.

Multipart form data and file uploads

-F builds a multipart/form-data body — what browsers send for forms with file inputs. @ attaches a file, < inserts a file's content as a text field:

curl -F "title=Q3 Report" \
  -F "document=@report.pdf;type=application/pdf" \
  -F "notes=<notes.txt" \
  "https://example.com/upload"

Looking like a browser

Sites profile the request fingerprint before deciding what to serve. cURL's defaults — User-Agent: curl/8.x, no Accept-Language, no cookies — are the easiest bot signature there is. A minimal browser-like request:

curl -sL --compressed \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" \
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
  -H "Accept-Language: en-US,en;q=0.9" \
  -e "https://www.google.com/" \
  "https://example.com/products"

-e sets the Referer; --compressed both advertises gzip/brotli support (browsers always do) and transparently decompresses the response, so you never pipe gzip bytes into your parser by accident. For sustained scraping, rotate user agents per request — see our user-agent rotation guide.

Headers you set with -H override cURL's defaults, and an empty value (-H "Accept:") removes a header entirely.

Cookies and sessions

HTTP is stateless; sites track sessions with cookies. cURL handles them with two flags: -c writes received cookies to a cookie jar file, -b sends cookies from a file (or a literal string):

# Log in once, persist the session cookie
curl -c cookies.txt -d "username=demo&password=secret" "https://example.com/login"

# Reuse the session on subsequent requests
curl -b cookies.txt "https://example.com/account/orders"

# Read and write in the same request (long-lived sessions)
curl -b cookies.txt -c cookies.txt "https://example.com/cart"

# One-off literal cookie
curl -b "sessionid=abc123; currency=USD" "https://example.com/prices"

The jar is a plain Netscape-format text file — you can inspect it, edit it, or seed it with cookies exported from a real browser session. Delete it to "log out."

Authentication

Basic auth

-u handles HTTP Basic authentication — cURL base64-encodes the credentials into an Authorization header for you:

curl -u apiuser:s3cret "https://api.example.com/data"

# Omit the password to be prompted interactively (keeps it out of shell history)
curl -u apiuser "https://api.example.com/data"

Bearer tokens and API keys

Token schemes are just headers:

curl -H "Authorization: Bearer $ACCESS_TOKEN" "https://api.example.com/me"
curl -H "X-API-Key: $API_KEY" "https://api.example.com/data"

OAuth 2.0

cURL doesn't implement OAuth flows, but the token exchange is a plain POST — fetch the token, then use it as a bearer:

ACCESS_TOKEN=$(curl -s -X POST "https://auth.example.com/oauth/token" \
  -d "grant_type=client_credentials" \
  -d "client_id=$CLIENT_ID" \
  -d "client_secret=$CLIENT_SECRET" | jq -r '.access_token')

curl -H "Authorization: Bearer $ACCESS_TOKEN" "https://api.example.com/reports"

TLS and certificates

cURL verifies TLS certificates by default. When scraping staging environments or debugging proxies you'll hit certificate failures; -k (--insecure) makes cURL ignore SSL certificate errors — acceptable for local debugging, never for credentialed production traffic. The right fixes are pointing cURL at the correct CA bundle or client certificate:

# Trust a private/self-signed CA
curl --cacert internal-ca.pem "https://staging.example.com"

# Mutual TLS (client certificate + key)
curl --cert client.pem --key client-key.pem "https://api.example.com/secure"

# Inspect a site's certificate chain
curl -vI "https://example.com" 2>&1 | grep -A 8 "Server certificate"

Handling responses

JSON and XML

cURL outputs whatever the server sends; pair it with jq for JSON and xmllint for XML:

curl -s "https://api.example.com/products" | jq '.products[] | {name, price}'

curl -s -H "Accept: application/xml" "https://api.example.com/feed" \
  | xmllint --xpath "//item/title/text()" -

For scraping HTML, cURL is only the fetch layer — pipe into an HTML parser in Python or JavaScript rather than regexing raw HTML.

Headers only, status codes, and timing with -w

-I sends a HEAD request when you only need headers (note: some servers answer HEAD differently than GET — curl -sI vs curl -s -o /dev/null -D - can disagree). The -w (write-out) flag prints response metadata after the transfer, which is the backbone of health checks and monitoring scripts:

# Just the status code
curl -s -o /dev/null -w "%{http_code}" "https://example.com"

# Status + timing breakdown
curl -s -o /dev/null -w "status=%{http_code} dns=%{time_namelookup}s connect=%{time_connect}s total=%{time_total}s\n" \
  "https://example.com"

Debugging with verbose output

-v (--verbose) prints the full exchange — DNS resolution, TLS handshake, every request and response header — which answers most "why does this work in my browser but not in cURL" questions. For byte-level inspection, --trace-ascii dump.txt logs everything to a file:

curl -v "https://example.com" 2>&1 | grep -E "^(\*|>|<)"

# Response headers only, body discarded
curl -sD - -o /dev/null "https://example.com"

A minimal health-check script:

STATUS=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "https://example.com/health")
if [ "$STATUS" -ne 200 ]; then
  echo "Health check failed: HTTP $STATUS"
  exit 1
fi

Error handling, retries, and timeouts

By default cURL exits 0 even when the server returns a 404 or 500 — it successfully transferred an error page. --fail makes HTTP errors ≥ 400 produce exit code 22 (and suppresses the error body), which is what you want in scripts and cron jobs:

curl --fail -s -o data.json "https://api.example.com/export" || echo "Download failed"

cURL's exit codes distinguish failure classes: 6 couldn't resolve host, 7 connection refused, 28 timeout, 22 HTTP error (with --fail), 35 TLS handshake failure. Check $? to branch on them.

--retry re-attempts transient failures (timeouts, 429, 5xx) with exponential backoff:

curl --fail --retry 5 --retry-delay 2 --retry-max-time 120 \
  --connect-timeout 10 --max-time 60 \
  "https://api.example.com/flaky-endpoint"

Set both cURL timeout options on every scripted request: --connect-timeout bounds the TCP/TLS handshake, --max-time bounds the whole transfer. Without them, one hung connection stalls your whole pipeline. When you're being rate-limited, --retry respects a Retry-After header automatically.

Multiple and parallel requests

cURL accepts multiple URLs in one invocation, and URL globbing generates ranges — useful for paginated listings:

# Pages 1-50, sequentially, over one reused connection
curl -sL -o "page_#1.html" "https://example.com/products?page=[1-50]"

Since 7.66, -Z (--parallel) runs transfers concurrently in a single process — --parallel-max caps the concurrency:

curl -ZL --parallel-max 10 -o "page_#1.html" "https://example.com/products?page=[1-100]"

On older systems, xargs -P does the same with multiple processes:

seq 1 100 | xargs -P 10 -I{} curl -sL -o "page_{}.html" "https://example.com/products?page={}"

Be a good citizen: cap concurrency, add --retry, and space batches out. Hammering a site with 100 parallel connections is the fastest way to get your IP banned — at scale you'll want rotating proxies regardless.

Using cURL with a proxy

-x routes requests through a proxy — HTTP, HTTPS, or SOCKS:

curl -x "http://user:pass@proxy.example.com:8080" "https://example.com"
curl -x "socks5h://127.0.0.1:9050" "https://check.torproject.org"

(socks5h resolves DNS through the proxy too — usually what you want.) The --resolve option is a related debugging tool: it pins a hostname to an IP without touching /etc/hosts, which is how you test a site on a new server before DNS cutover:

curl --resolve "example.com:443:203.0.113.7" "https://example.com"

cURL vs wget

The two classic transfer tools overlap but aim at different jobs:

For scraping work the answer is usually cURL: request control (headers, cookies, methods, proxies) is what scraping needs, and "Copy as cURL" from browser DevTools gives you a working starting point. Reach for wget when the task is "download this directory tree" or "mirror this site."

Where cURL stops: JavaScript rendering

cURL speaks HTTP; it does not run JavaScript. If a page's content is assembled client-side — React/Vue storefronts, infinite scroll, lazy-loaded prices — cURL receives the empty application shell, and no amount of header spoofing changes that. The same goes for JavaScript-based anti-bot challenges (Cloudflare, DataDome): they're designed to require a real browser environment.

At that point you have two options: drive a headless browser yourself, or keep your cURL workflow and swap the target URL for a rendering API. WebScraping.AI renders pages in headless Chrome with rotating proxies behind a plain HTTP endpoint, so the fix is one URL change:

# Returns the fully rendered HTML after JavaScript execution
curl -G "https://api.webscraping.ai/html" \
  --data-urlencode "url=https://example.com/spa-products" \
  --data-urlencode "api_key=$WSAI_API_KEY"

# Or ask a question about the page and get an AI-extracted answer
curl -G "https://api.webscraping.ai/ai/question" \
  --data-urlencode "url=https://example.com/product/42" \
  --data-urlencode "question=What is the price?" \
  --data-urlencode "api_key=$WSAI_API_KEY"

Everything in this guide — --fail, --retry, -w, parallel transfers, jq pipelines — works unchanged, because it's still just cURL. See the API documentation for the /text, /selected, and AI extraction endpoints, or the broader AI web scraping overview.

Frequently asked questions

Why does cURL return an empty page or different content than my browser? Three usual suspects, in order: the site redirected and you didn't pass -L; the content is rendered by JavaScript (view-source in your browser to confirm — if it's not in the source, cURL can't see it either); or the site served bot-detection content based on your headers. Fix the first two with -L and a rendering API; the third with browser-like headers.

How do I use cURL on Windows? Every command in this guide works unchanged on Windows 10+ in CMD or PowerShell. The one trap: in Windows PowerShell 5.x, curl is an alias for Invoke-WebRequest — type curl.exe (or remove the alias) to get the real thing. In PowerShell 7+ and CMD, plain curl is already correct. Use double quotes around URLs; single quotes are not treated the same way by CMD.

Is scraping with cURL legal? Fetching publicly accessible pages is generally lawful, but terms of service, rate limits, and data-protection rules apply — see our detailed guide on web scraping legality.

What's the difference between -L and --location? Nothing — every short cURL flag has a long synonym (-A/--user-agent, -b/--cookie, -d/--data). Use long forms in scripts for readability, short forms interactively.

Can cURL scrape sites that require login? Yes, if login is form- or token-based: POST the credentials with -d, capture the session with -c cookies.txt, then send -b cookies.txt on subsequent requests. Sites with JavaScript-driven or CAPTCHA-protected logins need a browser-based approach.