How do I use Curl to send multipart form data?

Multipart form data is essential for uploading files and sending complex form submissions through HTTP requests. Curl provides powerful options for handling multipart form data, making it an excellent tool for web scraping scenarios that involve file uploads, form submissions, and API interactions requiring multipart encoding.

Understanding Multipart Form Data

Multipart form data uses the multipart/form-data content type, which allows you to send multiple pieces of data in a single HTTP request. This encoding is particularly useful for:

File uploads
Forms with mixed content types
API endpoints that accept both text and binary data
Complex form submissions with multiple fields

Basic Multipart Form Data with Curl

Using the -F Flag

The most common way to send multipart form data with curl is using the -F (or --form) flag:

curl -F "field1=value1" -F "field2=value2" https://httpbin.org/post

This command sends two form fields as multipart data. Curl automatically sets the Content-Type header to multipart/form-data when you use the -F flag.

Sending Text Fields

For simple text fields, you can specify multiple form fields:

curl -F "username=john_doe" \
     -F "email=john@example.com" \
     -F "message=Hello World" \
     https://example.com/submit

File Uploads with Multipart Data

Uploading a Single File

To upload a file, use the @ symbol followed by the file path:

curl -F "file=@/path/to/document.pdf" https://example.com/upload

Uploading Multiple Files

You can upload multiple files in a single request:

curl -F "file1=@/path/to/image.jpg" \
     -F "file2=@/path/to/document.pdf" \
     -F "description=Multiple file upload" \
     https://example.com/upload

Specifying File Content Type

By default, curl attempts to determine the file's content type. You can explicitly specify it:

curl -F "file=@document.pdf;type=application/pdf" https://example.com/upload

Custom Filename

You can specify a custom filename for the uploaded file:

curl -F "file=@/local/path/document.pdf;filename=custom_name.pdf" \
     https://example.com/upload

Advanced Multipart Configurations

Combining Files and Form Fields

Real-world scenarios often require sending both files and form data:

curl -F "title=My Document" \
     -F "category=technical" \
     -F "file=@/path/to/document.pdf;type=application/pdf" \
     -F "thumbnail=@/path/to/thumb.jpg;type=image/jpeg" \
     https://example.com/api/documents

Sending Raw Data as Form Fields

You can send raw data (including JSON) as form fields:

curl -F "metadata={\"author\":\"John Doe\",\"version\":\"1.0\"};type=application/json" \
     -F "file=@document.pdf" \
     https://example.com/upload

Using Stdin for Form Data

You can pipe data to curl for dynamic form field values:

echo "Dynamic content" | curl -F "content=<-" -F "type=text" https://example.com/submit

Practical Examples for Web Scraping

API Authentication with File Upload

Many APIs require authentication along with file uploads:

curl -H "Authorization: Bearer YOUR_TOKEN" \
     -F "file=@data.csv;type=text/csv" \
     -F "format=csv" \
     -F "delimiter=," \
     https://api.example.com/v1/data/import

Form Submission with CSRF Token

When scraping forms that require CSRF tokens:

# First, extract the CSRF token
CSRF_TOKEN=$(curl -s https://example.com/form | grep -o 'csrf_token" value="[^"]*' | cut -d'"' -f3)

# Then submit the form with the token
curl -F "csrf_token=$CSRF_TOKEN" \
     -F "username=user" \
     -F "file=@upload.txt" \
     -b cookies.txt \
     https://example.com/submit

Simulating Complex Form Submissions

For complex forms with multiple input types:

curl -F "text_field=Sample text" \
     -F "number_field=42" \
     -F "checkbox=on" \
     -F "file_upload=@image.png;type=image/png" \
     -F "hidden_field=secret_value" \
     -c cookies.txt \
     -b cookies.txt \
     https://example.com/complex-form

Headers and Debugging

Custom Headers with Multipart Data

You can add custom headers while sending multipart data:

curl -H "X-API-Key: your-api-key" \
     -H "X-Client-Version: 1.0" \
     -F "file=@data.json;type=application/json" \
     https://api.example.com/upload

Debugging Multipart Requests

Use the -v flag to see the complete request, including multipart boundaries:

curl -v -F "file=@test.txt" https://httpbin.org/post

This shows the multipart boundary and how curl structures the request.

Saving Request Details

To save the complete request for analysis:

curl --trace-ascii trace.txt \
     -F "file=@document.pdf" \
     -F "title=Test Document" \
     https://example.com/upload

Working with Authentication and Sessions

Cookie-Based Sessions

When working with web applications that use session cookies, combine multipart uploads with cookie handling in curl:

# First login and save cookies
curl -c cookies.txt -F "username=user" -F "password=pass" https://example.com/login

# Then upload file using saved session
curl -b cookies.txt -F "file=@document.pdf" -F "title=My File" https://example.com/upload

API Token Authentication

For APIs that require authentication tokens:

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
     -F "file=@data.json;type=application/json" \
     -F "metadata=Processing pipeline data" \
     https://api.example.com/v1/upload

Error Handling and Best Practices

Handling Large Files

For large file uploads, consider using progress indicators:

curl -# -F "large_file=@big_video.mp4" https://example.com/upload

The -# flag shows a progress bar instead of the default progress meter.

Setting Timeouts

For file uploads that might take time, configure appropriate timeout settings with curl:

curl --connect-timeout 30 \
     --max-time 300 \
     -F "file=@large_file.zip" \
     https://example.com/upload

Retry Logic

Implement retry logic for failed uploads:

curl --retry 3 \
     --retry-delay 5 \
     -F "file=@important.pdf" \
     https://example.com/upload

Integration with Web Scraping Workflows

Programmatic Form Submission

You can integrate curl multipart uploads into shell scripts:

#!/bin/bash
for file in *.csv; do
    echo "Uploading $file..."
    curl -F "file=@$file;type=text/csv" \
         -F "source=batch_upload" \
         https://api.example.com/data/import
    sleep 2
done

Data Processing Pipeline

Curl multipart forms work well in data processing pipelines:

# Process data and upload results
cat raw_data.txt | python process.py > processed_data.json
curl -F "processed_data=@processed_data.json;type=application/json" \
     -F "metadata=@metadata.txt" \
     https://api.example.com/results

Combining with Web Scraping Tools

When building comprehensive scraping workflows, you might extract data with one tool and submit it via multipart forms:

import subprocess
import json

# Extract data (pseudo-code)
scraped_data = {"title": "Sample", "content": "Data"}

# Save to temporary file
with open("temp_data.json", "w") as f:
    json.dump(scraped_data, f)

# Upload using curl
subprocess.run([
    "curl", 
    "-F", "data=@temp_data.json;type=application/json",
    "-F", "source=web_scraper",
    "https://api.example.com/submit"
])

Common Troubleshooting

Content-Type Issues

If the server rejects your multipart data, verify the content type:

# Explicitly set content type
curl -F "file=@data.xml;type=application/xml" https://example.com/upload

Boundary Problems

Curl automatically generates multipart boundaries, but you can inspect them:

curl -v -F "test=value" https://httpbin.org/post 2>&1 | grep boundary

Encoding Issues

For international characters in form fields:

curl -F "name=José María" -F "file=@document.pdf" https://example.com/upload

Curl handles UTF-8 encoding automatically in most cases.

Server Response Analysis

Always check server responses for debugging:

curl -w "HTTP Status: %{http_code}\nTotal Time: %{time_total}s\n" \
     -F "file=@test.pdf" \
     https://example.com/upload

Advanced Multipart Scenarios

Dynamic Field Generation

For dynamic form submissions where field names are generated programmatically:

# Using variables for dynamic field names
FIELD_NAME="upload_$(date +%s)"
curl -F "$FIELD_NAME=@document.pdf" https://example.com/dynamic-upload

Multiple File Arrays

Some APIs expect file arrays:

curl -F "files[]=@file1.txt" \
     -F "files[]=@file2.txt" \
     -F "files[]=@file3.txt" \
     https://example.com/multi-upload

Binary Data Handling

For raw binary data that's not in a file:

echo -n "binary data here" | curl -F "data=<-;type=application/octet-stream" https://example.com/binary

Conclusion

Curl's multipart form data capabilities make it an excellent choice for web scraping scenarios involving file uploads and complex form submissions. Whether you're uploading files to APIs, submitting forms with mixed content types, or integrating file uploads into automated workflows, curl provides the flexibility and reliability needed for professional web scraping applications.

Understanding these multipart techniques allows you to handle sophisticated web interactions that go beyond simple GET and POST requests, enabling more comprehensive web scraping and automation solutions. Combined with proper authentication, error handling, and integration with other scraping tools, multipart form data support makes curl a powerful component in any web scraping toolkit.

Table of contents