How can I ensure the data I scrape from Trustpilot is up-to-date?

Keeping web-scraped data up-to-date, especially from a site like Trustpilot, involves regularly re-scraping the site to capture the latest reviews, ratings, and other relevant information. Here are some steps you can take to ensure the data you scrape from Trustpilot is up-to-date:

1. Determine the frequency of updates:

Understand how often new reviews are posted on Trustpilot for the businesses you're interested in. This will help you decide how frequently you need to scrape the site.

2. Automate the scraping process:

Use task schedulers or cron jobs to run your scraping scripts at the determined intervals. This way, you can automate the scraping process to occur without manual intervention.

3. Handle pagination and infinite scroll:

Trustpilot might have multiple pages of reviews or use an infinite scroll to load more content. Make sure your scraper can navigate through these to get the most recent data.

4. Implement error handling:

Your scraper should be able to handle errors and retries in case of network issues or temporary blocks by Trustpilot. This ensures that the scraper continues to function and collect data.

5. Respect Trustpilot’s Terms of Service:

Always check Trustpilot’s terms of service before scraping to ensure you are not violating their rules. Using their API, if available, could be a more reliable and legal way to obtain data.

6. Use real-time data extraction (if necessary):

For extremely time-sensitive data, you might need to implement a real-time data extraction system that scrapes Trustpilot as soon as new reviews are detected.

Example in Python using Scheduled Task:

Here's an example of a simple Python script to scrape Trustpilot reviews using BeautifulSoup, and a cron job to run it daily.

# Example Python script using BeautifulSoup to scrape Trustpilot reviews
import requests
from bs4 import BeautifulSoup

def scrape_trustpilot():
    url = ""  # Replace with the correct Trustpilot page
    response = requests.get(url)
    soup = BeautifulSoup(response.content, "html.parser")

    # Assume reviews are contained within an element with class 'review'
    reviews = soup.find_all(class_='review')
    for review in reviews:
        # Extract review data
        # You'll need to identify the correct classes or IDs for the actual data you want

if __name__ == "__main__":

Cron Job Setup:

To run the above Python script daily at 7:00 AM, you would add the following line to your crontab file:

0 7 * * * /usr/bin/python3 /path/to/your/ >> /path/to/your/logfile.log 2>&1

Example in JavaScript (Node.js) using setInterval:

If you're using Node.js, you could use setInterval to run the scraping at regular intervals.

// Example Node.js script using axios and cheerio to scrape Trustpilot reviews
const axios = require('axios');
const cheerio = require('cheerio');

const scrapeTrustpilot = async () => {
    const url = ""; // Replace with the correct Trustpilot page
    const response = await axios.get(url);
    const $ = cheerio.load(;

    // Assume reviews are contained within an element with class 'review'
    $('.review').each((index, element) => {
        // Extract review data
        // You'll need to identify the correct classes or IDs for the actual data you want

// Run the scrape function every 24 hours
setInterval(scrapeTrustpilot, 24 * 60 * 60 * 1000);

// Run it immediately on start

Final Notes:

Remember, web scraping can be a legally grey area and should be done responsibly and ethically. Comply with Trustpilot’s robots.txt file and terms of service to avoid any legal issues. If you need data at scale and with high reliability, consider reaching out to Trustpilot for official access through an API or data partnership, if available.

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping