What is the Difference Between --data and --json Options in Curl?

When working with HTTP APIs and web scraping, understanding the difference between curl's --data and --json options is crucial for sending POST requests correctly. While both options are used to send data in HTTP requests, they handle content formatting, headers, and encoding differently. This guide explains these differences and provides practical examples for various use cases.

Overview of --data and --json Options

The --data option is curl's traditional method for sending POST data, while --json is a newer convenience option specifically designed for JSON payloads. Understanding when to use each option can save you debugging time and ensure your API requests work as expected.

The --data Option: Traditional Data Sending

The --data (or -d) option sends data as the request body and automatically sets the request method to POST. By default, it uses application/x-www-form-urlencoded content type.

Basic --data Syntax

curl --data "key1=value1&key2=value2" https://api.example.com/endpoint

Key Characteristics of --data

Default Content-Type: Sets Content-Type: application/x-www-form-urlencoded
Data Format: Expects URL-encoded data by default
Method: Automatically changes request method to POST
Manual Headers: Requires manual header setting for JSON content

Examples with --data

Form Data Submission:

curl --data "username=john&password=secret123" \
     https://api.example.com/login

JSON Data with Manual Headers:

curl --data '{"name": "John Doe", "email": "john@example.com"}' \
     --header "Content-Type: application/json" \
     https://api.example.com/users

Reading Data from File:

curl --data @user_data.json \
     --header "Content-Type: application/json" \
     https://api.example.com/users

The --json Option: Modern JSON Convenience

The --json option was introduced in curl 7.82.0 as a convenience feature specifically for JSON data. It automatically handles JSON formatting and sets appropriate headers.

Basic --json Syntax

curl --json '{"key1": "value1", "key2": "value2"}' https://api.example.com/endpoint

Key Characteristics of --json

Automatic Content-Type: Sets Content-Type: application/json
JSON Validation: Validates JSON syntax
Method: Automatically changes request method to POST
Accept Header: Sets Accept: application/json

Examples with --json

Simple JSON POST:

curl --json '{"name": "John Doe", "email": "john@example.com"}' \
     https://api.example.com/users

Complex JSON Structure:

curl --json '{
  "user": {
    "name": "John Doe",
    "preferences": {
      "theme": "dark",
      "notifications": true
    }
  }
}' https://api.example.com/users

JSON from File:

curl --json @user_data.json https://api.example.com/users

Side-by-Side Comparison

| Feature | --data | --json | |---------|--------|--------| | Default Content-Type | application/x-www-form-urlencoded | application/json | | Accept Header | Not set | application/json | | JSON Validation | No | Yes | | Curl Version | All versions | 7.82.0+ | | Primary Use Case | Form data, any content type | JSON APIs |

Practical Examples and Use Cases

API Authentication

Using --data for form-based login:

curl --data "username=admin&password=secret" \
     --cookie-jar cookies.txt \
     https://api.example.com/auth/login

Using --json for token-based authentication:

curl --json '{"username": "admin", "password": "secret"}' \
     https://api.example.com/auth/token

Web Scraping Scenarios

When performing web scraping tasks, you might need to interact with APIs that require different data formats. For instance, when handling authentication in Puppeteer, you might need to make preliminary API calls to obtain session tokens.

Submitting form data for session creation:

curl --data "csrf_token=abc123&user_id=456" \
     --header "X-Requested-With: XMLHttpRequest" \
     https://target-site.com/api/session

Sending JSON configuration for API endpoints:

curl --json '{
  "scraping_config": {
    "wait_for": "networkidle0",
    "viewport": {"width": 1920, "height": 1080}
  }
}' https://scraping-api.example.com/configure

Error Handling and Debugging

Verbose output with --data:

curl --data '{"test": "data"}' \
     --header "Content-Type: application/json" \
     --verbose \
     https://api.example.com/test

Verbose output with --json:

curl --json '{"test": "data"}' \
     --verbose \
     https://api.example.com/test

The --json option provides cleaner output since it automatically sets the correct headers.

Advanced Usage Patterns

Combining with Other curl Options

Using --data with custom headers:

curl --data "action=scrape&url=https://example.com" \
     --header "Authorization: Bearer token123" \
     --header "User-Agent: MyBot/1.0" \
     https://api.example.com/scrape

Using --json with authentication:

curl --json '{"url": "https://example.com", "format": "json"}' \
     --header "Authorization: Bearer token123" \
     https://api.example.com/scrape

File-based Data Submission

Reading form data from file:

echo "name=John&email=john@example.com" > form_data.txt
curl --data @form_data.txt https://api.example.com/submit

Reading JSON from file:

echo '{"name": "John", "email": "john@example.com"}' > user.json
curl --json @user.json https://api.example.com/users

Common Pitfalls and Best Practices

Content-Type Mismatches

Wrong approach - using --data without proper headers for JSON:

# This sends JSON data but with wrong Content-Type
curl --data '{"name": "John"}' https://api.example.com/users

Correct approaches:

# Option 1: Use --json (recommended for JSON)
curl --json '{"name": "John"}' https://api.example.com/users

# Option 2: Use --data with explicit headers
curl --data '{"name": "John"}' \
     --header "Content-Type: application/json" \
     https://api.example.com/users

Special Character Handling

URL encoding with --data:

curl --data "message=Hello%20World%21" https://api.example.com/send

JSON escaping with --json:

curl --json '{"message": "Hello \"World\"!"}' https://api.example.com/send

Performance Considerations

When building web scraping applications that make numerous API calls, consider using connection reuse and HTTP/2 features. This is particularly relevant when monitoring network requests in Puppeteer scenarios where you need to optimize request performance.

# Reuse connections for multiple requests
curl --json '{"batch": "request1"}' \
     --http2 \
     --keepalive-time 30 \
     https://api.example.com/batch

When to Use Each Option

Use --data when:

Working with form-based APIs
Sending non-JSON data (XML, plain text, binary)
Using older curl versions (< 7.82.0)
Need precise control over Content-Type headers
Submitting URL-encoded form data

Use --json when:

Working with modern REST APIs
Sending JSON payloads
Want automatic header management
Need JSON syntax validation
Building JSON-heavy applications

Compatibility and Version Considerations

The --json option requires curl 7.82.0 or later. Check your curl version:

curl --version

If you're using an older version, stick with --data and manual header management:

curl --data '{"key": "value"}' \
     --header "Content-Type: application/json" \
     --header "Accept: application/json" \
     https://api.example.com/endpoint

Real-World Integration Examples

Python Integration

When building Python web scrapers, you might need to test API endpoints before implementing them in your scraping code. Here's how both options translate to Python requests:

curl --data equivalent in Python:

import requests

# Form data
response = requests.post(
    'https://api.example.com/login',
    data={'username': 'john', 'password': 'secret123'}
)

# JSON data (manual headers)
response = requests.post(
    'https://api.example.com/users',
    data='{"name": "John Doe", "email": "john@example.com"}',
    headers={'Content-Type': 'application/json'}
)

curl --json equivalent in Python:

import requests

# JSON data (automatic headers)
response = requests.post(
    'https://api.example.com/users',
    json={'name': 'John Doe', 'email': 'john@example.com'}
)

JavaScript/Node.js Integration

Similar principles apply when working with browser automation tools. When handling AJAX requests using Puppeteer, understanding these curl patterns helps you intercept and modify network requests:

Using fetch API with different data formats:

// Form data equivalent to curl --data
const formResponse = await fetch('https://api.example.com/login', {
  method: 'POST',
  headers: {'Content-Type': 'application/x-www-form-urlencoded'},
  body: 'username=john&password=secret123'
});

// JSON equivalent to curl --json
const jsonResponse = await fetch('https://api.example.com/users', {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: JSON.stringify({name: 'John Doe', email: 'john@example.com'})
});

Conclusion

Both --data and --json options serve important roles in HTTP communication. The --data option provides flexibility for various content types and maintains compatibility across curl versions, while --json offers convenience and automatic header management for JSON-specific use cases. Choose the option that best fits your project requirements, API specifications, and curl version constraints.

Understanding these differences will help you build more reliable web scraping tools and API integrations, whether you're working with simple form submissions or complex JSON APIs. Remember to always test your requests and verify that the receiving API correctly processes your data format and headers.

Table of contents