Table of contents

How do I handle HTTP PUT and DELETE requests with urllib3?

Urllib3 is a powerful and user-friendly HTTP client library for Python that provides robust support for all HTTP methods, including PUT and DELETE requests. These methods are essential for RESTful API interactions where you need to update (PUT) or remove (DELETE) resources on the server.

Understanding HTTP PUT and DELETE Methods

Before diving into urllib3 implementation, it's important to understand when to use these methods:

  • PUT: Used to update or create a resource at a specific URI. It's idempotent, meaning multiple identical requests should have the same effect.
  • DELETE: Used to remove a resource from the server. Also idempotent in nature.

Basic Setup and Installation

First, ensure you have urllib3 installed in your Python environment:

pip install urllib3

Import the necessary modules:

import urllib3
import json

Making HTTP PUT Requests

Simple PUT Request

Here's how to make a basic PUT request with urllib3:

import urllib3
import json

# Create a PoolManager instance
http = urllib3.PoolManager()

# Data to be sent in the PUT request
data = {
    "name": "Updated Resource",
    "description": "This resource has been updated",
    "status": "active"
}

# Convert data to JSON
json_data = json.dumps(data)

# Make PUT request
response = http.request(
    'PUT',
    'https://api.example.com/resources/123',
    body=json_data,
    headers={
        'Content-Type': 'application/json',
        'Authorization': 'Bearer your-token-here'
    }
)

print(f"Status: {response.status}")
print(f"Response: {response.data.decode('utf-8')}")

PUT Request with Form Data

For sending form-encoded data:

import urllib3

http = urllib3.PoolManager()

# Form data
fields = {
    'username': 'john_doe',
    'email': 'john@example.com',
    'status': 'active'
}

response = http.request(
    'PUT',
    'https://api.example.com/users/123',
    fields=fields,
    headers={
        'Authorization': 'Bearer your-token-here'
    }
)

if response.status == 200:
    print("Resource updated successfully")
else:
    print(f"Error: {response.status}")

PUT Request with File Upload

Urllib3 also supports file uploads in PUT requests:

import urllib3

http = urllib3.PoolManager()

# Upload a file with additional fields
with open('document.pdf', 'rb') as file:
    fields = {
        'file': ('document.pdf', file.read(), 'application/pdf'),
        'title': 'Updated Document',
        'category': 'reports'
    }

    response = http.request(
        'PUT',
        'https://api.example.com/documents/456',
        fields=fields,
        headers={
            'Authorization': 'Bearer your-token-here'
        }
    )

print(f"Upload status: {response.status}")

Making HTTP DELETE Requests

Simple DELETE Request

DELETE requests are typically simpler as they often don't require a request body:

import urllib3

http = urllib3.PoolManager()

# Make DELETE request
response = http.request(
    'DELETE',
    'https://api.example.com/resources/123',
    headers={
        'Authorization': 'Bearer your-token-here',
        'Content-Type': 'application/json'
    }
)

if response.status == 204:
    print("Resource deleted successfully")
elif response.status == 404:
    print("Resource not found")
else:
    print(f"Deletion failed with status: {response.status}")

DELETE Request with Parameters

Sometimes you need to send query parameters with DELETE requests:

import urllib3

http = urllib3.PoolManager()

# DELETE with query parameters
response = http.request(
    'DELETE',
    'https://api.example.com/resources',
    fields={
        'id': '123',
        'force': 'true',
        'cascade': 'false'
    },
    headers={
        'Authorization': 'Bearer your-token-here'
    }
)

print(f"Batch deletion status: {response.status}")

DELETE Request with Body

In some cases, DELETE requests might need a request body:

import urllib3
import json

http = urllib3.PoolManager()

# DELETE with request body
deletion_criteria = {
    "filter": {
        "created_before": "2023-01-01",
        "status": "inactive"
    },
    "confirm": True
}

response = http.request(
    'DELETE',
    'https://api.example.com/resources/bulk',
    body=json.dumps(deletion_criteria),
    headers={
        'Content-Type': 'application/json',
        'Authorization': 'Bearer your-token-here'
    }
)

print(f"Bulk deletion status: {response.status}")

Error Handling and Best Practices

Comprehensive Error Handling

Always implement proper error handling for your HTTP requests:

import urllib3
import json
from urllib3.exceptions import HTTPError, TimeoutError

http = urllib3.PoolManager()

def safe_put_request(url, data, headers=None):
    try:
        response = http.request(
            'PUT',
            url,
            body=json.dumps(data),
            headers=headers or {'Content-Type': 'application/json'},
            timeout=10.0
        )

        if response.status >= 200 and response.status < 300:
            return {
                'success': True,
                'data': json.loads(response.data.decode('utf-8')),
                'status': response.status
            }
        else:
            return {
                'success': False,
                'error': f"HTTP {response.status}",
                'response': response.data.decode('utf-8')
            }

    except TimeoutError:
        return {'success': False, 'error': 'Request timeout'}
    except HTTPError as e:
        return {'success': False, 'error': f'HTTP error: {str(e)}'}
    except Exception as e:
        return {'success': False, 'error': f'Unexpected error: {str(e)}'}

# Usage
result = safe_put_request(
    'https://api.example.com/resources/123',
    {'name': 'Updated Resource'},
    {'Authorization': 'Bearer token'}
)

if result['success']:
    print("Update successful:", result['data'])
else:
    print("Update failed:", result['error'])

Connection Pooling and Performance

For better performance, especially when making multiple requests, use connection pooling:

import urllib3

# Configure connection pool
http = urllib3.PoolManager(
    num_pools=10,
    maxsize=10,
    timeout=urllib3.Timeout(connect=5.0, read=10.0),
    retries=urllib3.Retry(
        total=3,
        backoff_factor=0.3,
        status_forcelist=[500, 502, 503, 504]
    )
)

# Make multiple requests efficiently
resources_to_update = [
    {'id': '123', 'data': {'status': 'active'}},
    {'id': '124', 'data': {'status': 'inactive'}},
    {'id': '125', 'data': {'status': 'pending'}}
]

for resource in resources_to_update:
    response = http.request(
        'PUT',
        f"https://api.example.com/resources/{resource['id']}",
        body=json.dumps(resource['data']),
        headers={'Content-Type': 'application/json'}
    )
    print(f"Updated resource {resource['id']}: {response.status}")

Working with Different Content Types

Sending XML Data

import urllib3

http = urllib3.PoolManager()

xml_data = """<?xml version="1.0" encoding="UTF-8"?>
<resource>
    <name>Updated Resource</name>
    <status>active</status>
</resource>"""

response = http.request(
    'PUT',
    'https://api.example.com/resources/123',
    body=xml_data,
    headers={
        'Content-Type': 'application/xml',
        'Authorization': 'Bearer your-token-here'
    }
)

Handling Binary Data

import urllib3

http = urllib3.PoolManager()

# Upload binary data
with open('image.jpg', 'rb') as file:
    binary_data = file.read()

response = http.request(
    'PUT',
    'https://api.example.com/images/123',
    body=binary_data,
    headers={
        'Content-Type': 'image/jpeg',
        'Authorization': 'Bearer your-token-here'
    }
)

JavaScript Equivalent Examples

For comparison, here's how similar requests would look in JavaScript using the Fetch API:

// PUT request in JavaScript
async function updateResource(id, data) {
    try {
        const response = await fetch(`https://api.example.com/resources/${id}`, {
            method: 'PUT',
            headers: {
                'Content-Type': 'application/json',
                'Authorization': 'Bearer your-token-here'
            },
            body: JSON.stringify(data)
        });

        if (response.ok) {
            const result = await response.json();
            console.log('Update successful:', result);
        } else {
            console.error('Update failed:', response.status);
        }
    } catch (error) {
        console.error('Error:', error);
    }
}

// DELETE request in JavaScript
async function deleteResource(id) {
    try {
        const response = await fetch(`https://api.example.com/resources/${id}`, {
            method: 'DELETE',
            headers: {
                'Authorization': 'Bearer your-token-here'
            }
        });

        if (response.status === 204) {
            console.log('Resource deleted successfully');
        } else {
            console.error('Deletion failed:', response.status);
        }
    } catch (error) {
        console.error('Error:', error);
    }
}

Advanced Features

Custom Retry Logic

import urllib3
from urllib3.util.retry import Retry

# Custom retry strategy
retry_strategy = Retry(
    total=5,
    status_forcelist=[429, 500, 502, 503, 504],
    method_whitelist=["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE"],
    backoff_factor=1
)

http = urllib3.PoolManager(retries=retry_strategy)

response = http.request(
    'PUT',
    'https://api.example.com/resources/123',
    body=json.dumps({'status': 'updated'}),
    headers={'Content-Type': 'application/json'}
)

SSL Certificate Verification

import urllib3
import certifi

# Secure HTTPS requests with certificate verification
http = urllib3.PoolManager(
    cert_reqs='CERT_REQUIRED',
    ca_certs=certifi.where()
)

response = http.request(
    'DELETE',
    'https://secure-api.example.com/resources/123',
    headers={'Authorization': 'Bearer your-token-here'}
)

Working with Authentication

OAuth 2.0 Bearer Token

import urllib3
import json

http = urllib3.PoolManager()

def authenticated_request(method, url, data=None, token=None):
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {token}'
    }

    if data:
        body = json.dumps(data)
    else:
        body = None

    return http.request(
        method,
        url,
        body=body,
        headers=headers
    )

# Update a resource with OAuth token
response = authenticated_request(
    'PUT',
    'https://api.example.com/users/profile',
    data={'email': 'newemail@example.com'},
    token='your-oauth-token'
)

API Key Authentication

import urllib3

http = urllib3.PoolManager()

response = http.request(
    'DELETE',
    'https://api.example.com/resources/123',
    headers={
        'X-API-Key': 'your-api-key-here',
        'Content-Type': 'application/json'
    }
)

Real-World Examples

REST API Resource Management

Here's a practical example of a class that handles CRUD operations:

import urllib3
import json
from urllib3.exceptions import HTTPError

class APIClient:
    def __init__(self, base_url, api_key=None):
        self.base_url = base_url.rstrip('/')
        self.http = urllib3.PoolManager()
        self.default_headers = {
            'Content-Type': 'application/json'
        }
        if api_key:
            self.default_headers['Authorization'] = f'Bearer {api_key}'

    def update_resource(self, resource_type, resource_id, data):
        """Update a resource using PUT request"""
        url = f"{self.base_url}/{resource_type}/{resource_id}"

        try:
            response = self.http.request(
                'PUT',
                url,
                body=json.dumps(data),
                headers=self.default_headers
            )

            if response.status == 200:
                return json.loads(response.data.decode('utf-8'))
            else:
                raise Exception(f"Update failed with status {response.status}")

        except HTTPError as e:
            raise Exception(f"HTTP error: {str(e)}")

    def delete_resource(self, resource_type, resource_id):
        """Delete a resource using DELETE request"""
        url = f"{self.base_url}/{resource_type}/{resource_id}"

        try:
            response = self.http.request(
                'DELETE',
                url,
                headers=self.default_headers
            )

            if response.status in [200, 204]:
                return True
            elif response.status == 404:
                return False  # Resource not found
            else:
                raise Exception(f"Deletion failed with status {response.status}")

        except HTTPError as e:
            raise Exception(f"HTTP error: {str(e)}")

# Usage example
client = APIClient('https://api.example.com', 'your-api-key')

# Update a user
updated_user = client.update_resource('users', '123', {
    'name': 'John Doe',
    'email': 'john.doe@example.com'
})

# Delete a post
deleted = client.delete_resource('posts', '456')
if deleted:
    print("Post deleted successfully")
else:
    print("Post not found")

Handling Response Data

Processing JSON Responses

import urllib3
import json

http = urllib3.PoolManager()

response = http.request(
    'PUT',
    'https://api.example.com/resources/123',
    body=json.dumps({'status': 'updated'}),
    headers={'Content-Type': 'application/json'}
)

# Check if request was successful
if response.status == 200:
    # Parse JSON response
    response_data = json.loads(response.data.decode('utf-8'))
    print(f"Updated resource ID: {response_data.get('id')}")
    print(f"New status: {response_data.get('status')}")
else:
    print(f"Update failed: {response.status}")
    print(f"Error message: {response.data.decode('utf-8')}")

Handling Non-JSON Responses

import urllib3

http = urllib3.PoolManager()

response = http.request(
    'DELETE',
    'https://api.example.com/files/document.pdf',
    headers={'Authorization': 'Bearer your-token'}
)

# Many DELETE requests return empty responses
if response.status == 204:
    print("File deleted successfully")
elif response.status == 404:
    print("File not found")
else:
    # Some APIs return error messages in plain text
    error_message = response.data.decode('utf-8')
    print(f"Deletion failed: {error_message}")

Best Practices and Security Considerations

Input Validation

import urllib3
import json
import re

def validate_and_update_user(user_id, user_data):
    # Validate user ID format
    if not re.match(r'^\d+$', str(user_id)):
        raise ValueError("Invalid user ID format")

    # Validate required fields
    required_fields = ['name', 'email']
    for field in required_fields:
        if field not in user_data:
            raise ValueError(f"Missing required field: {field}")

    # Validate email format
    email_pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
    if not re.match(email_pattern, user_data['email']):
        raise ValueError("Invalid email format")

    http = urllib3.PoolManager()

    response = http.request(
        'PUT',
        f'https://api.example.com/users/{user_id}',
        body=json.dumps(user_data),
        headers={'Content-Type': 'application/json'}
    )

    return response

Rate Limiting and Throttling

import urllib3
import time
from datetime import datetime, timedelta

class RateLimitedClient:
    def __init__(self, requests_per_minute=60):
        self.http = urllib3.PoolManager()
        self.requests_per_minute = requests_per_minute
        self.request_times = []

    def _enforce_rate_limit(self):
        now = datetime.now()
        # Remove requests older than 1 minute
        self.request_times = [
            req_time for req_time in self.request_times 
            if now - req_time < timedelta(minutes=1)
        ]

        if len(self.request_times) >= self.requests_per_minute:
            # Wait until we can make another request
            oldest_request = min(self.request_times)
            wait_time = 60 - (now - oldest_request).total_seconds()
            if wait_time > 0:
                time.sleep(wait_time)

    def request(self, method, url, **kwargs):
        self._enforce_rate_limit()
        self.request_times.append(datetime.now())
        return self.http.request(method, url, **kwargs)

# Usage
client = RateLimitedClient(requests_per_minute=30)
response = client.request(
    'PUT',
    'https://api.example.com/resources/123',
    body=json.dumps({'status': 'updated'}),
    headers={'Content-Type': 'application/json'}
)

Integration with Web Scraping Workflows

While urllib3 is excellent for API interactions, when dealing with dynamic web content that requires JavaScript execution, you might need to combine it with browser automation tools. For complex scenarios involving AJAX requests and dynamic content, Puppeteer can be a valuable complement to urllib3's capabilities.

For instance, you might use Puppeteer to extract authentication tokens or session cookies from a web interface, then use urllib3 to perform subsequent API operations with those credentials.

Conclusion

Urllib3 provides a robust and flexible foundation for handling PUT and DELETE requests in Python applications. Its connection pooling, retry mechanisms, and comprehensive error handling make it an excellent choice for both simple API interactions and complex web scraping workflows.

Key takeaways for effective PUT and DELETE request handling:

  • Always implement proper error handling and validation
  • Use connection pooling for better performance with multiple requests
  • Configure appropriate timeouts and retry strategies
  • Validate input data before sending requests
  • Implement rate limiting to respect API boundaries
  • Use secure authentication methods and proper headers

Whether you're building REST API clients, updating resources in bulk operations, or integrating with complex web services, urllib3's features provide the reliability and performance needed for production applications.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Can I use urllib3 with HTTP/2 protocol?
How do I parse and handle response headers with urllib3?
What are the memory usage patterns of urllib3 during large scraping operations?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon