What is the Best Way to Scrape React or Vue.js Websites with Crawlee?
Scraping React and Vue.js websites requires special consideration because these frameworks render content dynamically using JavaScript. Unlike traditional server-side rendered pages, single-page applications (SPAs) built with React or Vue.js load content asynchronously after the initial page load, making them challenging to scrape with simple HTTP requests.
Crawlee provides powerful tools specifically designed for scraping JavaScript-heavy websites through its PlaywrightCrawler and PuppeteerCrawler classes, which use real browser automation to execute JavaScript and wait for dynamic content to load.
Understanding the Challenge with React and Vue.js Websites
React and Vue.js applications differ from traditional websites in several key ways:
- Client-side rendering: Content is generated in the browser using JavaScript rather than being sent from the server
- Asynchronous data fetching: Data is often loaded via API calls after the initial page load
- Dynamic DOM updates: The page structure changes as users interact with it
- Virtual DOM: React and Vue use a virtual DOM that requires JavaScript execution to render actual HTML
- Lazy loading: Components and data may load on-demand as users scroll or navigate
Because of these characteristics, standard HTTP-based scrapers like CheerioCrawler will only see an empty skeleton HTML page without the actual content.
Choosing the Right Crawlee Crawler
Crawlee offers three main crawler types, but for React and Vue.js applications, you need one that can execute JavaScript:
PlaywrightCrawler (Recommended)
PlaywrightCrawler is the most powerful and recommended option for scraping modern JavaScript frameworks. It uses Playwright, which supports Chromium, Firefox, and WebKit browsers.
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
console.log(`Processing: ${request.url}`);
// Wait for React/Vue app to load
await page.waitForSelector('.app-content', { timeout: 30000 });
// Extract data after JavaScript has rendered
const data = await page.evaluate(() => {
return {
title: document.querySelector('h1')?.textContent,
items: Array.from(document.querySelectorAll('.item')).map(item => ({
name: item.querySelector('.name')?.textContent,
price: item.querySelector('.price')?.textContent
}))
};
});
console.log('Extracted data:', data);
},
});
await crawler.run(['https://example-react-app.com']);
PuppeteerCrawler
PuppeteerCrawler is another excellent choice, using Puppeteer which only supports Chromium-based browsers but has a slightly simpler API.
import { PuppeteerCrawler } from 'crawlee';
const crawler = new PuppeteerCrawler({
requestHandler: async ({ page, request }) => {
// Wait for network to be idle (useful for SPAs)
await page.waitForNetworkIdle({ timeout: 30000 });
const data = await page.evaluate(() => {
// Your extraction logic here
return document.querySelector('.content')?.textContent;
});
},
});
Best Practices for Scraping React/Vue.js Websites
1. Wait for Content to Load Properly
The most critical aspect of scraping single-page applications is ensuring all content has loaded before extraction. Crawlee provides several strategies:
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Strategy 1: Wait for specific selector
await page.waitForSelector('.data-loaded-indicator', {
state: 'visible',
timeout: 30000
});
// Strategy 2: Wait for network to be idle
await page.waitForLoadState('networkidle');
// Strategy 3: Wait for specific function to be available (React)
await page.waitForFunction(() => {
return window.__REACT_READY__ === true;
});
// Strategy 4: Custom wait time (use sparingly)
await page.waitForTimeout(3000);
},
});
2. Handle Dynamic Data Loading
React and Vue.js apps often fetch data from APIs after the initial render. Monitor network requests to understand when data loading completes:
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request }) => {
// Listen for API responses
const apiDataPromise = page.waitForResponse(
response => response.url().includes('/api/products') && response.status() === 200,
{ timeout: 30000 }
);
await page.goto(request.url);
// Wait for the API call to complete
await apiDataPromise;
// Now extract the data
const products = await page.$$eval('.product-card', cards =>
cards.map(card => ({
title: card.querySelector('.title')?.textContent,
price: card.querySelector('.price')?.textContent
}))
);
},
});
3. Handle Infinite Scroll and Lazy Loading
Many React and Vue.js applications use infinite scroll to load content progressively:
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Function to scroll and wait for new content
async function autoScroll() {
await page.evaluate(async () => {
await new Promise((resolve) => {
let totalHeight = 0;
const distance = 100;
const timer = setInterval(() => {
const scrollHeight = document.body.scrollHeight;
window.scrollBy(0, distance);
totalHeight += distance;
if (totalHeight >= scrollHeight) {
clearInterval(timer);
resolve();
}
}, 100);
});
});
}
// Scroll to load all content
await autoScroll();
// Wait for final content to render
await page.waitForTimeout(2000);
// Extract all loaded data
const allData = await page.$$eval('.item', items =>
items.map(item => item.textContent)
);
},
});
4. Extract Data from React/Vue.js Component State
Sometimes it's easier to extract data directly from the framework's state rather than parsing the DOM:
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
await page.waitForLoadState('networkidle');
// Extract data from React component props/state
const reactData = await page.evaluate(() => {
// Find React root element
const rootElement = document.querySelector('#root');
if (!rootElement) return null;
// Access React internal properties (React 16+)
const reactInternalKey = Object.keys(rootElement).find(
key => key.startsWith('__reactFiber') || key.startsWith('__reactInternalInstance')
);
if (reactInternalKey) {
const fiber = rootElement[reactInternalKey];
// Navigate the fiber tree to find component state
// This is implementation-specific
return fiber?.memoizedProps?.data;
}
return null;
});
// Or extract from Vue instance
const vueData = await page.evaluate(() => {
const app = document.querySelector('#app');
// Vue 3 exposes instance via __vueParentComponent
return app?.__vueParentComponent?.ctx?.data;
});
},
});
5. Optimize Performance with Request Interception
Block unnecessary resources to speed up scraping:
const crawler = new PlaywrightCrawler({
preNavigationHooks: [
async ({ page }) => {
// Block images, fonts, and other non-essential resources
await page.route('**/*', (route) => {
const resourceType = route.request().resourceType();
if (['image', 'font', 'media'].includes(resourceType)) {
route.abort();
} else {
route.continue();
}
});
},
],
requestHandler: async ({ page, request }) => {
await page.goto(request.url);
// Your extraction logic
},
});
Complete Example: Scraping a React E-commerce Site
Here's a comprehensive example that combines all best practices:
import { PlaywrightCrawler, Dataset } from 'crawlee';
const crawler = new PlaywrightCrawler({
// Increase timeout for slow-loading SPAs
navigationTimeoutSecs: 60,
// Use headless browser
headless: true,
preNavigationHooks: [
async ({ page }) => {
// Block unnecessary resources
await page.route('**/*', (route) => {
const type = route.request().resourceType();
if (['image', 'stylesheet', 'font'].includes(type)) {
route.abort();
} else {
route.continue();
}
});
},
],
requestHandler: async ({ page, request, enqueueLinks }) => {
console.log(`Processing: ${request.url}`);
// Wait for React app to initialize
await page.waitForSelector('[data-react-root]', { timeout: 30000 });
// Wait for product grid to load
await page.waitForSelector('.product-grid', { state: 'visible' });
// Wait for API call to complete
await page.waitForResponse(
response => response.url().includes('/api/products'),
{ timeout: 30000 }
);
// Additional wait for rendering
await page.waitForTimeout(1000);
// Extract product data
const products = await page.$$eval('.product-card', cards => {
return cards.map(card => ({
title: card.querySelector('.product-title')?.textContent?.trim(),
price: card.querySelector('.product-price')?.textContent?.trim(),
rating: card.querySelector('.product-rating')?.textContent?.trim(),
url: card.querySelector('a')?.href,
inStock: !card.querySelector('.out-of-stock')
}));
});
// Save to dataset
await Dataset.pushData(products);
// Find and enqueue pagination links
await enqueueLinks({
selector: '.pagination a',
label: 'PRODUCTS',
});
console.log(`Extracted ${products.length} products`);
},
failedRequestHandler: async ({ request }) => {
console.log(`Request ${request.url} failed multiple times`);
},
});
// Start crawling
await crawler.run([
'https://example-react-shop.com/products',
]);
// Export data
const dataset = await Dataset.open();
await dataset.exportToJSON('products');
Python Example with Crawlee for Python
If you're using Python, Crawlee for Python offers similar functionality:
from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext
async def main():
crawler = PlaywrightCrawler(
max_requests_per_crawl=50,
headless=True,
)
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
page = context.page
# Wait for React/Vue app to load
await page.wait_for_selector('.app-content', timeout=30000)
# Wait for network to be idle
await page.wait_for_load_state('networkidle')
# Extract data
data = await page.evaluate("""
() => {
return Array.from(document.querySelectorAll('.item')).map(item => ({
title: item.querySelector('.title')?.textContent,
description: item.querySelector('.desc')?.textContent
}));
}
""")
# Push data to dataset
await context.push_data(data)
# Enqueue new links
await context.enqueue_links(selector='a.next-page')
await crawler.run(['https://example-vue-app.com'])
if __name__ == '__main__':
import asyncio
asyncio.run(main())
Debugging Tips
When scraping React or Vue.js applications, debugging is crucial:
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Take screenshots at different stages
await page.screenshot({ path: 'before-wait.png' });
await page.waitForSelector('.content');
await page.screenshot({ path: 'after-wait.png' });
// Log page content for inspection
const html = await page.content();
console.log('Page HTML length:', html.length);
// Check if specific elements exist
const elementExists = await page.$('.target-element') !== null;
console.log('Target element exists:', elementExists);
},
// Run in non-headless mode to watch the browser
headless: false,
});
Conclusion
Scraping React and Vue.js websites with Crawlee requires using browser automation through PlaywrightCrawler or PuppeteerCrawler. The key to success is implementing proper wait strategies to ensure JavaScript has executed and all dynamic content has loaded before extraction. By combining selector waits, network monitoring, and strategic timeouts, you can reliably extract data from even the most complex single-page applications.
Remember to always respect website terms of service, implement rate limiting, and handle errors gracefully to build robust and ethical web scrapers.
