Table of contents

How do I use Curl to test API endpoints?

Curl is a powerful command-line tool that's essential for testing API endpoints during development and debugging. It allows you to send HTTP requests, inspect responses, and validate API behavior without writing code or using GUI tools. This comprehensive guide covers everything you need to know about using Curl for API testing.

What is Curl?

Curl (Client URL) is a command-line tool and library for transferring data with URLs. It supports numerous protocols including HTTP, HTTPS, FTP, and many others. For API testing, Curl's HTTP capabilities make it an indispensable tool for developers to quickly test endpoints, debug issues, and validate API responses.

Basic Curl Syntax

The basic syntax for Curl commands follows this pattern:

curl [options] [URL]

Testing Different HTTP Methods

GET Requests

GET requests are the simplest to test with Curl. By default, Curl performs GET requests:

# Basic GET request
curl https://api.example.com/users

# GET request with query parameters
curl "https://api.example.com/users?page=1&limit=10"

# GET request with verbose output
curl -v https://api.example.com/users

POST Requests

POST requests require the -X POST flag and typically include data:

# POST request with JSON data
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{"name": "John Doe", "email": "john@example.com"}'

# POST request with form data
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "name=John Doe&email=john@example.com"

# POST request with data from file
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d @user_data.json

PUT Requests

PUT requests are used for updating resources:

# PUT request to update a user
curl -X PUT https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"name": "Jane Smith", "email": "jane@example.com"}'

DELETE Requests

DELETE requests remove resources:

# DELETE request
curl -X DELETE https://api.example.com/users/123

# DELETE with authentication
curl -X DELETE https://api.example.com/users/123 \
  -H "Authorization: Bearer your-token-here"

PATCH Requests

PATCH requests perform partial updates:

# PATCH request for partial update
curl -X PATCH https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"email": "newemail@example.com"}'

Working with Headers

Headers are crucial for API communication. Here's how to set and inspect them:

# Set custom headers
curl -H "Authorization: Bearer token123" \
  -H "Content-Type: application/json" \
  -H "User-Agent: MyApp/1.0" \
  https://api.example.com/users

# Include response headers in output
curl -i https://api.example.com/users

# Show only response headers
curl -I https://api.example.com/users

# Save response headers to file
curl -D headers.txt https://api.example.com/users

Authentication Methods

Bearer Token Authentication

# Using Authorization header
curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." \
  https://api.example.com/protected-endpoint

Basic Authentication

# Using username and password
curl -u username:password https://api.example.com/protected-endpoint

# Basic auth with encoded credentials
curl -H "Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=" \
  https://api.example.com/protected-endpoint

API Key Authentication

# API key in header
curl -H "X-API-Key: your-api-key-here" \
  https://api.example.com/data

# API key as query parameter
curl "https://api.example.com/data?api_key=your-api-key-here"

Handling JSON Data

Working with JSON is common in modern APIs:

# Send JSON data
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{
    "user": {
      "name": "Alice Johnson",
      "email": "alice@example.com",
      "preferences": {
        "theme": "dark",
        "notifications": true
      }
    }
  }'

# Pretty print JSON response with jq
curl https://api.example.com/users | jq '.'

# Extract specific fields from JSON response
curl https://api.example.com/users | jq '.data[0].name'

Advanced Curl Options

Following Redirects

# Follow redirects automatically
curl -L https://api.example.com/redirect-endpoint

# Limit number of redirects
curl -L --max-redirs 5 https://api.example.com/redirect-endpoint

Timeout Configuration

# Set connection timeout (seconds)
curl --connect-timeout 10 https://api.example.com/users

# Set maximum time for entire operation
curl --max-time 30 https://api.example.com/users

# Combine both timeouts
curl --connect-timeout 10 --max-time 30 https://api.example.com/users

SSL/TLS Options

# Ignore SSL certificate errors (not recommended for production)
curl -k https://api.example.com/users

# Specify SSL certificate
curl --cert client.pem --key client.key https://api.example.com/users

# Use specific TLS version
curl --tlsv1.2 https://api.example.com/users

Cookie Handling

# Send cookies
curl -b "session_id=abc123; user_pref=dark_mode" https://api.example.com/users

# Save cookies to file
curl -c cookies.txt https://api.example.com/login

# Load cookies from file
curl -b cookies.txt https://api.example.com/protected-endpoint

Output and Response Handling

Saving Responses

# Save response to file
curl -o response.json https://api.example.com/users

# Save response with original filename
curl -O https://api.example.com/files/document.pdf

# Append timestamp to filename
curl -o "response_$(date +%Y%m%d_%H%M%S).json" https://api.example.com/users

Verbose Output and Debugging

# Verbose output showing request/response details
curl -v https://api.example.com/users

# Trace ASCII protocol exchange
curl --trace-ascii trace.txt https://api.example.com/users

# Show timing information
curl -w "Total time: %{time_total}s\nHTTP code: %{http_code}\n" \
  https://api.example.com/users

Testing File Uploads

# Upload a single file
curl -X POST https://api.example.com/upload \
  -F "file=@document.pdf" \
  -F "description=Important document"

# Upload multiple files
curl -X POST https://api.example.com/upload \
  -F "file1=@image1.jpg" \
  -F "file2=@image2.jpg" \
  -F "metadata=@info.json"

# Upload with custom filename
curl -X POST https://api.example.com/upload \
  -F "file=@local_file.txt;filename=remote_name.txt"

Error Handling and Status Codes

# Fail silently on HTTP errors
curl -f https://api.example.com/users

# Show only HTTP status code
curl -o /dev/null -s -w "%{http_code}" https://api.example.com/users

# Exit with error code on HTTP errors
curl --fail-with-body https://api.example.com/users

# Custom error handling with status code check
response=$(curl -s -w "%{http_code}" https://api.example.com/users)
http_code="${response: -3}"
if [ "$http_code" -eq 200 ]; then
    echo "Success"
else
    echo "Error: HTTP $http_code"
fi

Real-World API Testing Examples

Testing a REST API

# Test complete CRUD operations
# Create user
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{"name": "Test User", "email": "test@example.com"}'

# Read user
curl https://api.example.com/users/123

# Update user
curl -X PUT https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated User", "email": "updated@example.com"}'

# Delete user
curl -X DELETE https://api.example.com/users/123

Testing Pagination

# Test pagination parameters
curl "https://api.example.com/users?page=1&per_page=20"
curl "https://api.example.com/users?offset=0&limit=20"

Testing with Python Requests Alternative

For more complex scenarios, you might want to use Python's requests library:

import requests

# GET request
response = requests.get('https://api.example.com/users')
print(response.json())

# POST request with JSON
data = {"name": "John Doe", "email": "john@example.com"}
response = requests.post('https://api.example.com/users', json=data)
print(response.status_code)

Testing with JavaScript Fetch

In JavaScript environments, you can use fetch for similar testing:

// GET request
fetch('https://api.example.com/users')
  .then(response => response.json())
  .then(data => console.log(data));

// POST request with JSON
fetch('https://api.example.com/users', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    name: 'John Doe',
    email: 'john@example.com'
  })
})
.then(response => response.json())
.then(data => console.log(data));

Load Testing with Curl

# Simple load test with parallel requests
for i in {1..10}; do
  curl https://api.example.com/users &
done
wait

# Measure response times
curl -w "@curl-format.txt" -o /dev/null -s https://api.example.com/users

Create a curl-format.txt file with: time_namelookup: %{time_namelookup}\n time_connect: %{time_connect}\n time_appconnect: %{time_appconnect}\n time_pretransfer: %{time_pretransfer}\n time_redirect: %{time_redirect}\n time_starttransfer: %{time_starttransfer}\n ----------\n time_total: %{time_total}\n

Best Practices for API Testing with Curl

  1. Use Environment Variables: Store sensitive data like API keys in environment variables:
   export API_KEY="your-secret-key"
   curl -H "Authorization: Bearer $API_KEY" https://api.example.com/users

  1. Create Reusable Scripts: Save common curl commands in shell scripts for repeated testing.

  2. Combine with jq: Use jq for parsing and filtering JSON responses effectively.

  3. Test Error Scenarios: Always test both success and failure cases to ensure proper error handling.

  4. Use Configuration Files: For complex requests, consider using curl configuration files with the -K option.

  5. Monitor Performance: Use timing options to track response times and identify performance issues.

  6. Validate SSL Certificates: Always use proper SSL validation in production environments.

Integration with Other Tools

While Curl is excellent for quick API testing, you might also want to explore more advanced tools for complex scenarios. For web scraping tasks that require JavaScript execution, handling AJAX requests using browser automation can be more effective than simple HTTP requests. Additionally, when testing APIs that serve dynamic content, monitoring network requests in automated browsers provides deeper insights into application behavior.

Common Curl Testing Scenarios

API Health Checks

# Simple health check
curl -f https://api.example.com/health

# Health check with timeout
curl --max-time 5 -f https://api.example.com/health

# Health check with custom success criteria
curl -s https://api.example.com/health | jq -e '.status == "ok"'

Testing Rate Limits

# Test rate limiting behavior
for i in {1..100}; do
  echo "Request $i:"
  curl -w "HTTP Status: %{http_code}\n" https://api.example.com/users
  sleep 0.1
done

Webhook Testing

# Test webhook endpoint
curl -X POST https://your-app.com/webhooks/payment \
  -H "Content-Type: application/json" \
  -H "X-Webhook-Signature: sha256=your-signature" \
  -d '{
    "event": "payment.completed",
    "data": {
      "payment_id": "pay_123",
      "amount": 1000,
      "currency": "USD"
    }
  }'

Troubleshooting Common Issues

SSL Certificate Problems

# Skip SSL verification (development only)
curl -k https://self-signed.example.com/api

# Use specific CA bundle
curl --cacert /path/to/ca-bundle.crt https://api.example.com

Connection Issues

# Test with IPv4 only
curl -4 https://api.example.com/users

# Test with IPv6 only
curl -6 https://api.example.com/users

# Use specific interface
curl --interface eth0 https://api.example.com/users

Debugging Request/Response

# Full trace with timestamps
curl --trace-time --trace-ascii trace.log https://api.example.com/users

# Show detailed timing breakdown
curl -w "@curl-format.txt" https://api.example.com/users

Conclusion

Curl is an indispensable tool for API testing that every developer should master. Its versatility, speed, and ubiquity make it perfect for quick endpoint testing, debugging, and automation. From simple GET requests to complex authentication flows and file uploads, Curl provides the flexibility needed for comprehensive API testing.

By mastering these Curl techniques, you'll be able to efficiently test APIs, debug issues, and validate endpoint behavior across different environments. Remember to always respect rate limits and API terms of service when testing, and consider using dedicated testing environments when possible. Whether you're testing REST APIs, webhooks, or complex authentication flows, Curl remains one of the most reliable tools in a developer's toolkit.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

What does the --connect-timeout option do in Curl?
How do I use Curl to handle JSON responses?
What is the purpose of the --fail flag in Curl?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon