What is the Claude API documentation for web scraping?
The Claude API documentation for web scraping is Anthropic's official developer resource that provides comprehensive guidance on using Claude's AI capabilities for extracting, parsing, and structuring data from web content. The documentation is available at https://docs.anthropic.com/ and includes detailed information about API endpoints, authentication, request/response formats, and best practices for implementing AI-powered web scraping workflows.
Unlike traditional web scraping tools that rely on rigid XPath or CSS selectors, Claude API enables intelligent data extraction using natural language prompts. This approach allows developers to describe what data they need rather than how to extract it, making scraping more resilient to website changes.
Understanding the Claude API Structure
The Claude API documentation is organized into several key sections that are particularly relevant for web scraping:
Messages API
The core endpoint for web scraping is the Messages API (/v1/messages), which accepts text input (including HTML content) and returns structured responses based on your instructions.
Basic Request Structure:
import anthropic
client = anthropic.Anthropic(api_key="your-api-key")
message = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[
{
"role": "user",
"content": "Extract product name, price, and rating from this HTML: <html>...</html>"
}
]
)
print(message.content)
JavaScript/Node.js Example:
import Anthropic from '@anthropic-ai/sdk';
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY,
});
const message = await client.messages.create({
model: 'claude-3-5-sonnet-20241022',
max_tokens: 1024,
messages: [
{
role: 'user',
content: 'Extract product details from this HTML: <html>...</html>'
}
]
});
console.log(message.content);
Available Models for Web Scraping
According to the documentation, Claude offers several models optimized for different use cases:
- Claude 3.5 Sonnet: Best balance of intelligence and speed for most scraping tasks
- Claude 3 Opus: Highest accuracy for complex data extraction
- Claude 3 Haiku: Fastest and most cost-effective for simple extraction tasks
Key Features for Web Scraping
1. Structured Output with Tool Use
The Claude API documentation describes "tool use" (function calling) as a powerful feature for getting structured JSON output from unstructured web content. This is essential for web scraping workflows.
import anthropic
import json
client = anthropic.Anthropic(api_key="your-api-key")
# Define extraction schema
tools = [
{
"name": "extract_product_data",
"description": "Extract structured product information from HTML",
"input_schema": {
"type": "object",
"properties": {
"products": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"price": {"type": "number"},
"rating": {"type": "number"},
"availability": {"type": "string"}
},
"required": ["name", "price"]
}
}
},
"required": ["products"]
}
}
]
# Send HTML content with tool definition
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=2048,
tools=tools,
messages=[
{
"role": "user",
"content": f"Extract all products from this e-commerce page: {html_content}"
}
]
)
# Parse structured output
for content in response.content:
if content.type == "tool_use":
extracted_data = content.input
print(json.dumps(extracted_data, indent=2))
2. Vision Capabilities for Screenshot Analysis
The documentation highlights Claude's vision capabilities, which allow processing screenshots alongside HTML. This is particularly useful when dealing with dynamically rendered content similar to handling AJAX requests using Puppeteer.
import Anthropic from '@anthropic-ai/sdk';
import fs from 'fs';
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY,
});
// Read screenshot as base64
const imageData = fs.readFileSync('screenshot.png').toString('base64');
const message = await client.messages.create({
model: 'claude-3-5-sonnet-20241022',
max_tokens: 2048,
messages: [
{
role: 'user',
content: [
{
type: 'image',
source: {
type: 'base64',
media_type: 'image/png',
data: imageData,
},
},
{
type: 'text',
text: 'Extract all visible product information from this e-commerce page screenshot.'
}
],
},
],
});
console.log(message.content);
3. Context Windows and Token Limits
The documentation specifies context windows for each model, which is crucial for processing large HTML documents:
- Claude 3.5 Sonnet: 200,000 tokens (~150,000 words)
- Claude 3 Opus: 200,000 tokens
- Claude 3 Haiku: 200,000 tokens
For very large web pages, you may need to chunk the HTML content or extract only the relevant sections before sending to the API.
Authentication and API Keys
According to the official documentation, authentication requires an API key obtained from the Anthropic Console:
# Set environment variable
export ANTHROPIC_API_KEY='your-api-key-here'
# Python: Using environment variable
import os
from anthropic import Anthropic
client = Anthropic(
api_key=os.environ.get("ANTHROPIC_API_KEY")
)
// JavaScript: Using environment variable
import Anthropic from '@anthropic-ai/sdk';
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY
});
Rate Limits and Pricing
The Claude API documentation outlines tier-based rate limits:
- Tier 1 (Free): 5 requests per minute
- Tier 2: 50 requests per minute
- Tier 3: 1,000 requests per minute
- Tier 4: 4,000 requests per minute
For web scraping at scale, you'll need to implement rate limiting and potentially upgrade your tier. Pricing is based on tokens processed:
- Input tokens: $3 per million tokens (Sonnet)
- Output tokens: $15 per million tokens (Sonnet)
Best Practices from the Documentation
1. Preprocessing HTML
The documentation recommends cleaning HTML before sending it to the API to reduce token usage:
from bs4 import BeautifulSoup
import anthropic
def clean_html(html_content):
"""Remove scripts, styles, and unnecessary attributes"""
soup = BeautifulSoup(html_content, 'html.parser')
# Remove script and style tags
for tag in soup(['script', 'style', 'noscript']):
tag.decompose()
# Remove comments
for comment in soup.find_all(string=lambda text: isinstance(text, str)):
if '<!--' in str(comment):
comment.extract()
return str(soup)
# Use cleaned HTML
cleaned_html = clean_html(raw_html)
client = anthropic.Anthropic(api_key="your-api-key")
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[
{
"role": "user",
"content": f"Extract product data from: {cleaned_html}"
}
]
)
2. Using System Prompts
The documentation describes system prompts as a way to set consistent behavior across multiple scraping requests:
const message = await client.messages.create({
model: 'claude-3-5-sonnet-20241022',
max_tokens: 2048,
system: 'You are a web scraping assistant. Always return data in valid JSON format. Extract only factual information present in the HTML.',
messages: [
{
role: 'user',
content: `Extract all article titles and publication dates from: ${htmlContent}`
}
]
});
3. Handling Pagination and Multiple Pages
When scraping multiple pages, the documentation suggests maintaining conversation context for consistency:
import anthropic
client = anthropic.Anthropic(api_key="your-api-key")
# Maintain conversation history
conversation = []
for page_num, html_content in enumerate(pages, 1):
conversation.append({
"role": "user",
"content": f"Extract products from page {page_num}: {html_content}"
})
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=2048,
messages=conversation
)
# Add assistant response to conversation
conversation.append({
"role": "assistant",
"content": response.content
})
print(f"Page {page_num} extracted")
Integrating with Traditional Scraping Tools
The Claude API documentation works well when combined with traditional scraping libraries. For example, you can use browser automation tools for navigating to different pages using Puppeteer and then use Claude for intelligent data extraction:
import puppeteer from 'puppeteer';
import Anthropic from '@anthropic-ai/sdk';
const anthropicClient = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY
});
async function scrapeWithClaude(url) {
// Use Puppeteer to fetch rendered HTML
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle0' });
const htmlContent = await page.content();
await browser.close();
// Use Claude to extract data
const message = await anthropicClient.messages.create({
model: 'claude-3-5-sonnet-20241022',
max_tokens: 2048,
messages: [
{
role: 'user',
content: `Extract all product information from this HTML: ${htmlContent}`
}
]
});
return message.content;
}
scrapeWithClaude('https://example.com/products').then(console.log);
Error Handling and Retry Logic
The Claude API documentation recommends implementing exponential backoff for rate limit errors:
import time
import anthropic
from anthropic import APIError, RateLimitError
def extract_with_retry(html_content, max_retries=3):
client = anthropic.Anthropic(api_key="your-api-key")
for attempt in range(max_retries):
try:
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[
{
"role": "user",
"content": f"Extract data from: {html_content}"
}
]
)
return response.content
except RateLimitError:
if attempt < max_retries - 1:
wait_time = 2 ** attempt
print(f"Rate limited. Waiting {wait_time} seconds...")
time.sleep(wait_time)
else:
raise
except APIError as e:
print(f"API error: {e}")
raise
SDK Documentation
The official documentation provides SDKs for multiple languages:
- Python:
pip install anthropic - JavaScript/TypeScript:
npm install @anthropic-ai/sdk - Java: Maven/Gradle available
- Go:
go get github.com/anthropics/anthropic-sdk-go
Each SDK includes comprehensive examples and type definitions for web scraping use cases.
Additional Resources
Beyond the core API documentation, Anthropic provides:
- API Reference: Complete endpoint specifications and parameters
- Cookbook: Practical examples including web scraping patterns
- Rate Limit Headers: Real-time information about your usage
- Streaming Responses: For processing large extractions progressively
Conclusion
The Claude API documentation provides a comprehensive foundation for building intelligent web scraping solutions. By combining Claude's natural language understanding with traditional scraping tools, developers can create robust, maintainable scrapers that adapt to website changes. The official documentation at docs.anthropic.com should be your primary reference, supplemented by the SDK-specific documentation for your chosen programming language.
For production web scraping, consider using Claude's structured output capabilities through tool use, implement proper error handling and rate limiting, and preprocess HTML to optimize token usage and costs. When dealing with complex single-page applications, similar to crawling SPAs using Puppeteer, combining browser automation with Claude's extraction capabilities provides the most robust solution.
