Table of contents

How do I use LangChain for web scraping with LLMs?

LangChain is a powerful framework for building applications with Large Language Models (LLMs), and it provides excellent tools for web scraping and data extraction. By combining LangChain with LLMs, you can create intelligent scrapers that understand content semantically, extract structured data without complex selectors, and adapt to changes in website layouts.

What is LangChain?

LangChain is an open-source framework that simplifies the development of LLM-powered applications. For web scraping, it offers several key advantages:

  • Document loaders for fetching and parsing web content
  • Text splitters for handling large pages that exceed LLM token limits
  • Output parsers for extracting structured data
  • Chains for orchestrating multi-step scraping workflows
  • Integration with multiple LLM providers (OpenAI, Anthropic, Google, etc.)

Setting up LangChain for web scraping

First, install the necessary packages:

pip install langchain langchain-openai langchain-community
pip install beautifulsoup4 lxml requests

For basic setup with OpenAI:

import os
from langchain_openai import ChatOpenAI
from langchain_community.document_loaders import WebBaseLoader
from langchain.chains import create_extraction_chain

# Set up your API key
os.environ["OPENAI_API_KEY"] = "your-api-key-here"

# Initialize the LLM
llm = ChatOpenAI(
    model="gpt-4-turbo-preview",
    temperature=0  # Use 0 for deterministic extraction
)

Basic web scraping with LangChain

Loading web pages

LangChain's WebBaseLoader fetches and parses HTML content:

from langchain_community.document_loaders import WebBaseLoader

# Load a single page
loader = WebBaseLoader("https://example.com/product")
docs = loader.load()

# Access the content
print(docs[0].page_content)  # Cleaned text content
print(docs[0].metadata)       # URL and other metadata

Loading multiple pages

# Scrape multiple URLs at once
urls = [
    "https://example.com/product/1",
    "https://example.com/product/2",
    "https://example.com/product/3"
]

loader = WebBaseLoader(urls)
docs = loader.load()

for doc in docs:
    print(f"URL: {doc.metadata['source']}")
    print(f"Content length: {len(doc.page_content)}")

Extracting structured data with LLMs

Using extraction chains

LangChain's extraction chains allow you to define a schema and extract data automatically:

from langchain.chains import create_extraction_chain

# Define the schema for extraction
schema = {
    "properties": {
        "product_name": {"type": "string"},
        "price": {"type": "number"},
        "rating": {"type": "number"},
        "description": {"type": "string"},
        "in_stock": {"type": "boolean"}
    },
    "required": ["product_name", "price"]
}

# Load the webpage
loader = WebBaseLoader("https://example.com/product")
docs = loader.load()

# Create and run the extraction chain
chain = create_extraction_chain(schema, llm)
result = chain.run(docs[0].page_content)

print(result)
# Output: [{'product_name': 'Example Product', 'price': 29.99, 'rating': 4.5, ...}]

Using Pydantic models for type safety

For better type safety and validation, use Pydantic models:

from langchain.output_parsers import PydanticOutputParser
from langchain.prompts import PromptTemplate
from pydantic import BaseModel, Field
from typing import List

class Product(BaseModel):
    name: str = Field(description="Product name")
    price: float = Field(description="Product price in USD")
    rating: float = Field(description="Average customer rating")
    features: List[str] = Field(description="List of key product features")
    availability: str = Field(description="Stock availability status")

# Set up the parser
parser = PydanticOutputParser(pydantic_object=Product)

# Create a prompt template
template = """
Extract the product information from the following webpage content.

{format_instructions}

Webpage content:
{content}

Extracted data:
"""

prompt = PromptTemplate(
    template=template,
    input_variables=["content"],
    partial_variables={"format_instructions": parser.get_format_instructions()}
)

# Load and parse the webpage
loader = WebBaseLoader("https://example.com/product")
docs = loader.load()

# Create the chain
chain = prompt | llm | parser

# Run extraction
product = chain.invoke({"content": docs[0].page_content})
print(f"Product: {product.name}")
print(f"Price: ${product.price}")
print(f"Features: {', '.join(product.features)}")

Handling large pages with text splitters

When scraping pages that exceed the LLM context window, use text splitters:

from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load a large page
loader = WebBaseLoader("https://example.com/long-article")
docs = loader.load()

# Split the content into chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=4000,
    chunk_overlap=200,
    length_function=len
)

chunks = text_splitter.split_documents(docs)
print(f"Split into {len(chunks)} chunks")

# Process each chunk
results = []
for chunk in chunks:
    chain = create_extraction_chain(schema, llm)
    result = chain.run(chunk.page_content)
    results.extend(result)

# Combine results
print(f"Extracted {len(results)} items total")

Advanced scraping with custom loaders

Custom loader with JavaScript rendering

For pages requiring JavaScript execution, combine LangChain with browser automation:

from langchain.document_loaders.base import BaseLoader
from langchain.schema import Document
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from typing import List

class SeleniumLoader(BaseLoader):
    def __init__(self, urls: List[str], headless: bool = True):
        self.urls = urls
        self.headless = headless

    def load(self) -> List[Document]:
        chrome_options = Options()
        if self.headless:
            chrome_options.add_argument("--headless")

        driver = webdriver.Chrome(options=chrome_options)
        documents = []

        try:
            for url in self.urls:
                driver.get(url)
                # Wait for dynamic content to load
                driver.implicitly_wait(3)

                content = driver.page_source
                documents.append(Document(
                    page_content=content,
                    metadata={"source": url}
                ))
        finally:
            driver.quit()

        return documents

# Use the custom loader
loader = SeleniumLoader(["https://example.com/dynamic-page"])
docs = loader.load()

Using LangChain with different LLM providers

With Anthropic Claude

from langchain_anthropic import ChatAnthropic

llm = ChatAnthropic(
    model="claude-3-sonnet-20240229",
    anthropic_api_key="your-api-key"
)

# Use the same extraction patterns as before
chain = create_extraction_chain(schema, llm)
result = chain.run(docs[0].page_content)

With Google Gemini

from langchain_google_genai import ChatGoogleGenerativeAI

llm = ChatGoogleGenerativeAI(
    model="gemini-pro",
    google_api_key="your-api-key",
    temperature=0
)

chain = create_extraction_chain(schema, llm)
result = chain.run(docs[0].page_content)

Building a complete scraping pipeline

Here's a complete example that scrapes multiple product pages and saves the results:

from langchain_openai import ChatOpenAI
from langchain_community.document_loaders import WebBaseLoader
from langchain.output_parsers import PydanticOutputParser
from langchain.prompts import PromptTemplate
from pydantic import BaseModel, Field
from typing import List
import json

# Define the data model
class Product(BaseModel):
    name: str
    price: float
    category: str
    rating: float
    reviews_count: int
    description: str

# Initialize components
llm = ChatOpenAI(model="gpt-4-turbo-preview", temperature=0)
parser = PydanticOutputParser(pydantic_object=Product)

# Create prompt
prompt_template = """
Extract product information from this webpage content.
Be precise with numerical values.

{format_instructions}

Content:
{content}
"""

prompt = PromptTemplate(
    template=prompt_template,
    input_variables=["content"],
    partial_variables={"format_instructions": parser.get_format_instructions()}
)

# URLs to scrape
product_urls = [
    "https://example.com/products/laptop",
    "https://example.com/products/phone",
    "https://example.com/products/tablet"
]

# Scrape and extract
products = []
loader = WebBaseLoader(product_urls)
docs = loader.load()

for doc in docs:
    try:
        chain = prompt | llm | parser
        product = chain.invoke({"content": doc.page_content})
        products.append(product.dict())
        print(f"Extracted: {product.name}")
    except Exception as e:
        print(f"Error processing {doc.metadata['source']}: {e}")

# Save results
with open("products.json", "w") as f:
    json.dump(products, f, indent=2)

print(f"Successfully scraped {len(products)} products")

Best practices for LangChain web scraping

1. Use appropriate chunk sizes

When dealing with token limits, choose chunk sizes that fit within your model's context window while maintaining coherent content segments.

2. Implement error handling

from langchain.schema import Document
from typing import Optional

def safe_extract(doc: Document, chain, max_retries: int = 3) -> Optional[dict]:
    for attempt in range(max_retries):
        try:
            result = chain.run(doc.page_content)
            return result
        except Exception as e:
            print(f"Attempt {attempt + 1} failed: {e}")
            if attempt == max_retries - 1:
                return None
    return None

3. Optimize costs

  • Use cheaper models for simple extractions
  • Implement caching for repeated requests
  • Minimize prompt length by cleaning HTML before sending to LLM
  • Batch similar extractions together
from langchain.cache import InMemoryCache
from langchain.globals import set_llm_cache

# Enable caching
set_llm_cache(InMemoryCache())

4. Validate extracted data

from pydantic import validator

class Product(BaseModel):
    name: str
    price: float

    @validator('price')
    def price_must_be_positive(cls, v):
        if v <= 0:
            raise ValueError('Price must be positive')
        return v

    @validator('name')
    def name_must_not_be_empty(cls, v):
        if not v.strip():
            raise ValueError('Name cannot be empty')
        return v

Comparing LangChain to traditional scraping

Traditional scraping (BeautifulSoup, Scrapy): - Faster and cheaper for structured, predictable websites - Requires writing and maintaining selectors - Breaks when website structure changes - No semantic understanding of content

LangChain with LLMs: - More resilient to layout changes - Understands content semantically - Can extract complex relationships and implied data - Higher cost per request - Best for unstructured or frequently changing sites

When deciding when to use LLM-powered scraping, consider the complexity of the data, frequency of changes, and your budget.

Conclusion

LangChain provides a robust framework for building intelligent web scrapers powered by LLMs. By leveraging document loaders, extraction chains, and output parsers, you can create scrapers that understand content semantically and extract structured data without brittle selectors. While LLM-based scraping has higher costs than traditional methods, it excels at handling unstructured content, adapting to layout changes, and extracting complex relationships from web pages.

The key to successful LangChain scraping is choosing the right model for your needs, implementing proper error handling, managing token limits effectively, and validating extracted data. With these practices in place, LangChain becomes a powerful tool for modern web scraping challenges.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Are there any free AI scrapers available for web scraping?
What is the best LLM for data extraction tasks?
How do I fine-tune an LLM for web scraping tasks?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon