Table of contents

How do I use Deepseek for AI data extraction from unstructured content?

Deepseek is a powerful large language model (LLM) that excels at extracting structured data from unstructured content like HTML pages, text documents, and raw web data. Unlike traditional web scraping that relies on CSS selectors or XPath, Deepseek uses natural language understanding to identify and extract relevant information, making it ideal for complex or frequently changing web layouts.

Why use Deepseek for data extraction?

Deepseek offers several advantages for extracting data from unstructured content:

  • Context understanding: Deepseek can understand the semantic meaning of content, not just its structure
  • Flexibility: Works with varying HTML layouts without updating selectors
  • Cost-effective: Deepseek offers competitive pricing compared to other LLMs like GPT-4 or Claude
  • Large context window: Deepseek V3 supports up to 64K tokens, allowing you to process large web pages
  • Structured output: Can return data in JSON format for easy integration

Setting up Deepseek for data extraction

Prerequisites

Before you begin, you'll need:

  1. A Deepseek API key (obtain from platform.deepseek.com)
  2. Python or Node.js installed on your system
  3. A web scraping tool to fetch HTML content

Installation

Python:

pip install openai  # Deepseek is OpenAI-compatible
pip install requests beautifulsoup4

JavaScript:

npm install openai
npm install axios cheerio

Basic data extraction with Deepseek

Python example

Here's a complete example of extracting product information from unstructured HTML:

from openai import OpenAI
import requests
from bs4 import BeautifulSoup
import json

# Initialize Deepseek client
client = OpenAI(
    api_key="your-deepseek-api-key",
    base_url="https://api.deepseek.com"
)

# Fetch HTML content
url = "https://example.com/product-page"
response = requests.get(url)
html_content = response.text

# Clean HTML to reduce tokens (optional but recommended)
soup = BeautifulSoup(html_content, 'html.parser')
# Remove scripts, styles, and other unnecessary elements
for tag in soup(['script', 'style', 'nav', 'footer']):
    tag.decompose()
clean_html = soup.get_text(separator=' ', strip=True)

# Create extraction prompt
prompt = f"""
Extract the following product information from this HTML content:
- Product name
- Price
- Description
- Availability status
- Customer rating (if available)

Return the data as a valid JSON object with these exact keys:
product_name, price, description, availability, rating

HTML Content:
{clean_html[:4000]}  # Limit to first 4000 chars to manage token usage
"""

# Call Deepseek API
completion = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": "You are a data extraction assistant. Always return valid JSON."},
        {"role": "user", "content": prompt}
    ],
    temperature=0.0,  # Use 0 for consistent extraction
    response_format={"type": "json_object"}  # Ensure JSON output
)

# Parse the response
extracted_data = json.loads(completion.choices[0].message.content)
print(json.dumps(extracted_data, indent=2))

JavaScript example

const OpenAI = require('openai');
const axios = require('axios');
const cheerio = require('cheerio');

// Initialize Deepseek client
const client = new OpenAI({
    apiKey: 'your-deepseek-api-key',
    baseURL: 'https://api.deepseek.com'
});

async function extractProductData(url) {
    try {
        // Fetch HTML content
        const response = await axios.get(url);
        const html = response.data;

        // Clean HTML
        const $ = cheerio.load(html);
        $('script, style, nav, footer').remove();
        const cleanText = $('body').text().replace(/\s+/g, ' ').trim();

        // Create extraction prompt
        const prompt = `
Extract the following product information from this HTML content:
- Product name
- Price
- Description
- Availability status
- Customer rating (if available)

Return the data as a valid JSON object with these exact keys:
product_name, price, description, availability, rating

HTML Content:
${cleanText.substring(0, 4000)}
        `;

        // Call Deepseek API
        const completion = await client.chat.completions.create({
            model: 'deepseek-chat',
            messages: [
                { role: 'system', content: 'You are a data extraction assistant. Always return valid JSON.' },
                { role: 'user', content: prompt }
            ],
            temperature: 0.0,
            response_format: { type: 'json_object' }
        });

        // Parse and return extracted data
        const extractedData = JSON.parse(completion.choices[0].message.content);
        console.log(JSON.stringify(extractedData, null, 2));
        return extractedData;

    } catch (error) {
        console.error('Error extracting data:', error);
        throw error;
    }
}

// Usage
extractProductData('https://example.com/product-page');

Advanced extraction techniques

Batch processing multiple pages

When extracting data from multiple pages, implement batching to optimize API calls:

import asyncio
from concurrent.futures import ThreadPoolExecutor

async def extract_batch(urls, max_concurrent=5):
    """Extract data from multiple URLs concurrently"""

    def process_url(url):
        # Fetch and extract (using code from above)
        response = requests.get(url)
        # ... extraction logic ...
        return extracted_data

    with ThreadPoolExecutor(max_workers=max_concurrent) as executor:
        results = list(executor.map(process_url, urls))

    return results

# Usage
urls = [
    'https://example.com/product-1',
    'https://example.com/product-2',
    'https://example.com/product-3'
]

results = asyncio.run(extract_batch(urls))

Handling dynamic content

For JavaScript-rendered pages, you'll need to use a headless browser before passing content to Deepseek. While traditional tools require complex selector logic, Deepseek can understand the rendered content naturally:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Set up headless browser
chrome_options = Options()
chrome_options.add_argument('--headless')
driver = webdriver.Chrome(options=chrome_options)

# Load dynamic page
driver.get('https://example.com/dynamic-page')
driver.implicitly_wait(5)  # Wait for JavaScript to load

# Get rendered HTML
rendered_html = driver.page_source
driver.quit()

# Now extract with Deepseek
# ... (use the extraction code from above)

Alternatively, you can use a web scraping API that handles JavaScript rendering for you, then pass the content to Deepseek for intelligent extraction.

Schema-based extraction

For consistent extraction across multiple pages, define a clear schema:

extraction_schema = {
    "product_name": "string",
    "price": "number",
    "currency": "string (ISO code)",
    "description": "string",
    "features": "array of strings",
    "specifications": {
        "brand": "string",
        "model": "string",
        "dimensions": "string"
    },
    "availability": "boolean",
    "stock_count": "number or null",
    "images": "array of URLs",
    "rating": "number (0-5) or null",
    "review_count": "number"
}

prompt = f"""
Extract product data from the following HTML content according to this exact schema:
{json.dumps(extraction_schema, indent=2)}

Return ONLY valid JSON that matches this schema. Use null for missing values.

HTML Content:
{clean_html}
"""

Best practices for Deepseek data extraction

1. Optimize token usage

Deepseek pricing is based on tokens, so minimize unnecessary content:

def clean_html_for_extraction(html):
    """Remove unnecessary HTML elements to reduce token count"""
    soup = BeautifulSoup(html, 'html.parser')

    # Remove unnecessary tags
    for tag in soup(['script', 'style', 'nav', 'header', 'footer', 'aside', 'iframe']):
        tag.decompose()

    # Remove comments
    for comment in soup.find_all(text=lambda text: isinstance(text, Comment)):
        comment.extract()

    # Get text content with some structure preserved
    return soup.get_text(separator=' ', strip=True)

2. Use consistent temperature settings

For data extraction, always use temperature=0.0 or 0.1 to ensure consistent, deterministic results:

completion = client.chat.completions.create(
    model="deepseek-chat",
    messages=messages,
    temperature=0.0,  # Deterministic output
    max_tokens=2000   # Limit response size
)

3. Implement error handling and validation

Always validate the extracted data:

import jsonschema
from jsonschema import validate

def validate_extracted_data(data, schema):
    """Validate extracted data against schema"""
    try:
        validate(instance=data, schema=schema)
        return True
    except jsonschema.exceptions.ValidationError as err:
        print(f"Validation error: {err}")
        return False

# Define JSON schema
product_schema = {
    "type": "object",
    "required": ["product_name", "price"],
    "properties": {
        "product_name": {"type": "string"},
        "price": {"type": "number"},
        "description": {"type": "string"},
        "availability": {"type": "boolean"}
    }
}

# Validate after extraction
if validate_extracted_data(extracted_data, product_schema):
    # Process valid data
    save_to_database(extracted_data)
else:
    # Handle validation failure
    log_error(extracted_data)

4. Implement retry logic

API calls can fail, so implement robust retry mechanisms:

import time
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=4, max=10)
)
def extract_with_retry(html_content, prompt):
    """Extract data with automatic retries"""
    completion = client.chat.completions.create(
        model="deepseek-chat",
        messages=[
            {"role": "system", "content": "You are a data extraction assistant."},
            {"role": "user", "content": prompt}
        ],
        temperature=0.0,
        response_format={"type": "json_object"}
    )
    return json.loads(completion.choices[0].message.content)

Cost optimization strategies

Deepseek is cost-effective, but you can further optimize:

  1. Pre-filter HTML: Extract only relevant sections before sending to the API
  2. Cache results: Store extracted data to avoid re-processing identical pages
  3. Batch similar extractions: Combine multiple similar extractions in one prompt when appropriate
  4. Use Deepseek-Coder: For technical documentation extraction, consider using deepseek-coder model
# Example: Extract only main content before processing
def extract_main_content(html):
    """Extract only the main content area"""
    soup = BeautifulSoup(html, 'html.parser')

    # Try common main content selectors
    main_content = (
        soup.find('main') or
        soup.find('article') or
        soup.find(id='content') or
        soup.find(class_='content')
    )

    return str(main_content) if main_content else html

Handling timeouts and rate limits

When processing large volumes of content, implement proper timeout handling and respect API rate limits:

import time
from ratelimit import limits, sleep_and_retry

# Deepseek rate limits (adjust based on your plan)
REQUESTS_PER_MINUTE = 60

@sleep_and_retry
@limits(calls=REQUESTS_PER_MINUTE, period=60)
def call_deepseek_api(messages):
    """Rate-limited API call"""
    return client.chat.completions.create(
        model="deepseek-chat",
        messages=messages,
        temperature=0.0,
        timeout=30  # 30 second timeout
    )

Comparing Deepseek to traditional scraping

| Aspect | Traditional Scraping | Deepseek Extraction | |--------|---------------------|---------------------| | Setup complexity | High (selectors for each site) | Low (natural language prompts) | | Maintenance | Breaks with layout changes | Resilient to layout changes | | Context understanding | None | Excellent | | Speed | Very fast | Moderate (API latency) | | Cost | Low (compute only) | Per-token pricing | | Accuracy | 100% if selectors work | 95-99% typical |

Integration with existing scraping workflows

Deepseek works best when combined with traditional scraping tools. Here's a typical workflow:

  1. Fetch content: Use requests, axios, or a headless browser to retrieve HTML
  2. Clean and preprocess: Remove unnecessary elements to reduce token usage
  3. Extract with Deepseek: Use the LLM to intelligently extract structured data
  4. Validate and store: Verify the extracted data matches your schema before saving
def complete_extraction_pipeline(url):
    """Complete extraction pipeline with Deepseek"""

    # Step 1: Fetch
    response = requests.get(url, timeout=10)
    html = response.text

    # Step 2: Clean
    cleaned_html = clean_html_for_extraction(html)

    # Step 3: Extract
    extracted_data = extract_with_retry(cleaned_html, extraction_prompt)

    # Step 4: Validate
    if validate_extracted_data(extracted_data, product_schema):
        return extracted_data
    else:
        raise ValueError("Extracted data failed validation")

Conclusion

Deepseek provides a powerful, cost-effective solution for extracting structured data from unstructured web content. By leveraging its natural language understanding capabilities, you can build more resilient scrapers that adapt to layout changes and understand context. While it adds API costs and latency compared to traditional scraping, the reduction in maintenance overhead and improved flexibility often make it worthwhile, especially for complex extraction tasks or sites with frequently changing layouts.

For best results, combine Deepseek with traditional scraping techniques: use standard tools to fetch and render content, then leverage Deepseek's intelligence for the extraction step. This hybrid approach gives you the speed and reliability of traditional tools with the flexibility and understanding of AI-powered extraction.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

What is a good web scraping API that works with Deepseek?
How do I handle JSON extraction using Deepseek?
What is a reliable data extraction API that uses Deepseek?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon