What are web scraping examples using ChatGPT?

ChatGPT and the OpenAI API can transform web scraping from a rigid, selector-based process into an intelligent, context-aware data extraction workflow. By leveraging large language models, you can extract structured data from HTML without writing complex parsing logic or maintaining fragile CSS selectors.

This guide explores practical examples of using ChatGPT for web scraping tasks, from simple HTML parsing to complex data extraction scenarios.

Understanding ChatGPT for Web Scraping

ChatGPT excels at understanding unstructured content and converting it into structured data. Unlike traditional web scraping that relies on XPath or CSS selectors, ChatGPT can interpret the semantic meaning of content, making it resilient to HTML structure changes.

Key Advantages

• Flexibility: Adapts to different page layouts without code changes
• Context awareness: Understands content meaning, not just structure
• Natural language instructions: Define extraction rules in plain English
• Robust to changes: Less brittle than selector-based approaches

Example 1: Extracting Product Information

Let's start with a common use case: extracting product details from an e-commerce page.

Python Implementation

import openai
import requests

# Fetch the HTML content
url = "https://example.com/product/laptop"
response = requests.get(url)
html_content = response.text

# Initialize OpenAI client
client = openai.OpenAI(api_key="your-api-key")

# Create a prompt for ChatGPT
prompt = f"""
Extract the following product information from this HTML and return it as JSON:
- Product name
- Price
- Rating (out of 5)
- Number of reviews
- Main features (as an array)
- Availability status

HTML:
{html_content[:4000]}  # Limit to avoid token limits

Return only valid JSON, no additional text.
"""

# Call ChatGPT API
completion = client.chat.completions.create(
    model="gpt-4-turbo-preview",
    messages=[
        {"role": "system", "content": "You are a web scraping assistant that extracts structured data from HTML."},
        {"role": "user", "content": prompt}
    ],
    response_format={"type": "json_object"}
)

# Parse the response
import json
product_data = json.loads(completion.choices[0].message.content)
print(json.dumps(product_data, indent=2))

Expected Output

{
  "product_name": "Dell XPS 15 Laptop",
  "price": "$1,299.99",
  "rating": 4.5,
  "number_of_reviews": 328,
  "main_features": [
    "15.6-inch 4K display",
    "Intel Core i7 processor",
    "16GB RAM",
    "512GB SSD"
  ],
  "availability_status": "In Stock"
}

Example 2: Scraping Article Metadata

Extract metadata from blog posts or news articles, including author, publication date, and tags.

JavaScript/Node.js Implementation

const OpenAI = require('openai');
const axios = require('axios');

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY
});

async function scrapeArticleMetadata(url) {
  // Fetch HTML content
  const response = await axios.get(url);
  const html = response.data;

  // Prepare the extraction prompt
  const prompt = `
Extract the following article metadata from this HTML:
- Title
- Author name
- Publication date (in ISO 8601 format)
- Reading time (in minutes)
- Tags/Categories (as array)
- Article summary (2-3 sentences)

HTML:
${html.substring(0, 5000)}

Return as JSON only.
  `;

  // Call ChatGPT
  const completion = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [
      {
        role: 'system',
        content: 'You extract structured metadata from article HTML. Always return valid JSON.'
      },
      {
        role: 'user',
        content: prompt
      }
    ],
    response_format: { type: 'json_object' }
  });

  return JSON.parse(completion.choices[0].message.content);
}

// Usage
scrapeArticleMetadata('https://example.com/blog/ai-trends-2024')
  .then(data => console.log(data))
  .catch(err => console.error(err));

Example 3: Batch Processing Multiple Pages

When scraping multiple pages, you can combine traditional HTTP requests with ChatGPT-powered data extraction for efficiency.

Python Batch Scraping

import openai
import requests
from concurrent.futures import ThreadPoolExecutor
import json

client = openai.OpenAI(api_key="your-api-key")

def extract_with_chatgpt(html_content, schema):
    """Extract data using ChatGPT based on a schema"""
    prompt = f"""
Extract data according to this schema:
{json.dumps(schema, indent=2)}

From this HTML:
{html_content[:3000]}

Return valid JSON matching the schema.
    """

    completion = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "You are a data extraction assistant."},
            {"role": "user", "content": prompt}
        ],
        response_format={"type": "json_object"}
    )

    return json.loads(completion.choices[0].message.content)

def scrape_listing_page(url):
    """Scrape a single listing page"""
    response = requests.get(url)

    schema = {
        "listings": [
            {
                "title": "string",
                "price": "number",
                "location": "string",
                "bedrooms": "number",
                "bathrooms": "number"
            }
        ]
    }

    return extract_with_chatgpt(response.text, schema)

# Scrape multiple pages in parallel
urls = [
    "https://example.com/listings?page=1",
    "https://example.com/listings?page=2",
    "https://example.com/listings?page=3"
]

with ThreadPoolExecutor(max_workers=3) as executor:
    results = list(executor.map(scrape_listing_page, urls))

# Combine all results
all_listings = []
for result in results:
    all_listings.extend(result.get('listings', []))

print(f"Scraped {len(all_listings)} total listings")

Example 4: Intelligent Table Extraction

ChatGPT excels at extracting and structuring data from HTML tables, even when they have complex layouts.

import openai
import requests

client = openai.OpenAI(api_key="your-api-key")

def scrape_table_data(url):
    """Extract table data intelligently"""
    response = requests.get(url)
    html = response.text

    prompt = f"""
Find all tables in this HTML and extract their data.
For each table:
1. Identify what the table represents
2. Extract headers
3. Extract all rows as structured data

Return as JSON with this structure:
{{
  "tables": [
    {{
      "description": "what this table shows",
      "headers": ["col1", "col2", ...],
      "rows": [
        {{"col1": "value", "col2": "value"}},
        ...
      ]
    }}
  ]
}}

HTML:
{html[:4000]}
    """

    completion = client.chat.completions.create(
        model="gpt-4-turbo-preview",
        messages=[
            {"role": "system", "content": "You are an expert at extracting tabular data from HTML."},
            {"role": "user", "content": prompt}
        ],
        response_format={"type": "json_object"}
    )

    return json.loads(completion.choices[0].message.content)

# Example usage
table_data = scrape_table_data("https://example.com/statistics")
print(json.dumps(table_data, indent=2))

Example 5: Form Data Extraction and Validation

Extract form fields and their validation rules from HTML forms.

const OpenAI = require('openai');
const axios = require('axios');

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY
});

async function extractFormSchema(url) {
  const response = await axios.get(url);

  const prompt = `
Analyze this HTML form and extract:
1. All input fields with their names and types
2. Required fields
3. Validation rules (max length, patterns, etc.)
4. Select/dropdown options
5. Form action URL and method

Return as structured JSON.

HTML:
${response.data.substring(0, 4000)}
  `;

  const completion = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [
      {
        role: 'system',
        content: 'You analyze HTML forms and extract their structure and validation rules.'
      },
      {
        role: 'user',
        content: prompt
      }
    ],
    response_format: { type: 'json_object' }
  });

  return JSON.parse(completion.choices[0].message.content);
}

// Usage
extractFormSchema('https://example.com/contact-form')
  .then(schema => {
    console.log('Form Schema:', schema);
    // Use schema to programmatically fill and submit forms
  });

Example 6: Sentiment and Content Analysis

Combine web scraping with AI-powered content analysis to extract not just data, but insights.

import openai
import requests

client = openai.OpenAI(api_key="your-api-key")

def scrape_with_analysis(url):
    """Scrape reviews with sentiment analysis"""
    response = requests.get(url)

    prompt = f"""
Extract all customer reviews from this page.
For each review, provide:
- Reviewer name
- Rating (out of 5)
- Review text
- Review date
- Sentiment (positive/negative/neutral)
- Key topics mentioned (as array)

Return as JSON with a "reviews" array.

HTML:
{response.text[:4000]}
    """

    completion = client.chat.completions.create(
        model="gpt-4-turbo-preview",
        messages=[
            {"role": "system", "content": "You extract and analyze customer reviews from HTML."},
            {"role": "user", "content": prompt}
        ],
        response_format={"type": "json_object"}
    )

    return json.loads(completion.choices[0].message.content)

# Analyze reviews
reviews_data = scrape_with_analysis("https://example.com/product/reviews")

# Calculate statistics
positive_reviews = sum(1 for r in reviews_data['reviews'] if r['sentiment'] == 'positive')
print(f"Positive reviews: {positive_reviews}/{len(reviews_data['reviews'])}")

Best Practices and Optimization

1. Token Management

ChatGPT has token limits, so preprocessing HTML is crucial:

from bs4 import BeautifulSoup

def clean_html_for_gpt(html):
    """Remove unnecessary HTML to reduce tokens"""
    soup = BeautifulSoup(html, 'html.parser')

    # Remove scripts, styles, and comments
    for tag in soup(['script', 'style', 'meta', 'link']):
        tag.decompose()

    # Get text content with minimal HTML
    return soup.get_text(separator=' ', strip=True)

2. Cost Optimization

Use GPT-3.5-turbo for simple extractions and GPT-4 for complex tasks:

def choose_model(complexity):
    """Select appropriate model based on task complexity"""
    if complexity == 'simple':
        return 'gpt-3.5-turbo'
    elif complexity == 'complex':
        return 'gpt-4-turbo-preview'
    return 'gpt-3.5-turbo'

# Use the cheaper model when possible
model = choose_model('simple')

3. Error Handling

Always implement robust error handling when integrating ChatGPT into your web scraping workflow:

import json
from openai import OpenAI, OpenAIError

def safe_extract(html_content, prompt, max_retries=3):
    """Extract data with retry logic"""
    client = OpenAI(api_key="your-api-key")

    for attempt in range(max_retries):
        try:
            completion = client.chat.completions.create(
                model="gpt-3.5-turbo",
                messages=[
                    {"role": "system", "content": "You extract structured data from HTML."},
                    {"role": "user", "content": prompt}
                ],
                response_format={"type": "json_object"}
            )

            # Validate JSON response
            data = json.loads(completion.choices[0].message.content)
            return data

        except OpenAIError as e:
            print(f"Attempt {attempt + 1} failed: {e}")
            if attempt == max_retries - 1:
                raise
        except json.JSONDecodeError as e:
            print(f"Invalid JSON response: {e}")
            if attempt == max_retries - 1:
                return None

    return None

Combining with Traditional Tools

For production systems, combine ChatGPT with traditional scraping tools:

import openai
import requests
from bs4 import BeautifulSoup

def hybrid_scraping(url):
    """Use BeautifulSoup for structure, ChatGPT for content understanding"""

    # Fetch and parse HTML
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')

    # Extract article body using selectors
    article_body = soup.find('article') or soup.find('div', class_='content')

    if article_body:
        # Use ChatGPT to understand and structure the content
        client = openai.OpenAI(api_key="your-api-key")

        prompt = f"""
Analyze this article content and extract:
- Main topic
- Key points (as bullet array)
- Mentioned entities (people, companies, places)
- Technical terms defined

Content:
{article_body.get_text()[:3000]}

Return as JSON.
        """

        completion = client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[
                {"role": "system", "content": "You analyze article content."},
                {"role": "user", "content": prompt}
            ],
            response_format={"type": "json_object"}
        )

        return json.loads(completion.choices[0].message.content)

    return None

Conclusion

ChatGPT transforms web scraping by adding intelligence and flexibility to data extraction. These examples demonstrate practical applications from simple product scraping to complex content analysis. While ChatGPT adds API costs, it significantly reduces development time and creates more maintainable scraping solutions that adapt to website changes.

For production deployments, consider combining ChatGPT with traditional tools, implementing proper error handling, and optimizing token usage to balance cost and performance. The key is choosing the right tool for each part of your scraping pipeline: traditional selectors for structure, ChatGPT for semantic understanding.