How can I monitor Trustpilot for new reviews using scraping?

Monitoring Trustpilot for new reviews using scraping involves periodically fetching the webpage where the reviews are displayed and extracting the relevant information. Before proceeding, it's crucial to note that scraping Trustpilot or any other website should be done in compliance with their terms of service, and you should always check these to ensure you are not violating any rules. Trustpilot, like many other sites, may have restrictions on scraping, and it's possible they offer an API for accessing reviews in a more structured and legitimate way.

Assuming you have verified that scraping Trustpilot doesn't violate any terms of service or legal agreements, here's a general approach to monitor for new reviews:

Step 1: Identify the URL of the Trustpilot page containing the reviews

First, you need to find the URL for the specific company page on Trustpilot where reviews are posted.

Step 2: Make HTTP requests to retrieve the content

You'll need to use a library to make HTTP requests to the Trustpilot page to fetch the HTML content containing the reviews.

In Python, you can use requests:

import requests

url = ''  # Replace with the actual URL
headers = {
    'User-Agent': 'Your User-Agent'  # Replace with a user-agent string from your browser

response = requests.get(url, headers=headers)
html_content = response.text

Step 3: Parse the HTML content

Use an HTML parser like BeautifulSoup in Python to extract review information:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_content, 'html.parser')
reviews = soup.find_all('article', class_='review')  # Replace with the actual class or tag used by Trustpilot

for review in reviews:
    # Extract review data, e.g., rating, title, content, etc.
    # This will depend on the actual HTML structure of the Trustpilot page

Step 4: Detect new reviews

To detect new reviews, you'll need to keep track of which reviews you've already seen. You could do this by storing the review IDs or dates in a database or a file and checking if the parsed reviews are already recorded.

Step 5: Schedule the scraping

Automate the scraping process by running your script at regular intervals. This could be done using a cron job in Unix-based systems or Task Scheduler in Windows.

Step 6: Respect the website's robots.txt file and terms of service

Before starting your scraping, check robots.txt on the Trustpilot website by visiting This file may contain directives that disallow scraping.

Here's an example using Python's schedule library to run the scraping task periodically:

import schedule
import time

def scrape_trustpilot():
    # Your scraping logic here

# Run the scraping function every hour

while True:

Note on Legal and Ethical Considerations:

Web scraping can be legally complex and carries ethical considerations, especially if it involves personal data. Always ensure you:

  • Comply with the website's terms of service and privacy policy.
  • Do not overload the website's server with too many requests in a short period.
  • Consider the legal implications of storing and using scraped personal data.

Alternative: Trustpilot API

If Trustpilot offers an API, it's recommended to use that instead of scraping. APIs are designed to provide data in a structured format and are usually the preferred and legal way to access data programmatically. Check the Trustpilot developer website for more information on their API offerings and how to use them.

In summary, while it is technically feasible to scrape Trustpilot for new reviews, it's crucial to do so in a way that respects their terms of service, the legal framework concerning data protection, and ethical considerations. If available, using an official API is the safest and most reliable approach.

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping