Imagine tapping into the vast data resources of the internet to gain insights, make informed decisions, and stay ahead of the competition. What is web scraping? Web scraping unlocks this potential by automating the extraction of large amounts of data from websites in a structured manner. Discover the inner workings, applications, tools, and best practices in our comprehensive guide on web scraping.
Short Summary
- Web scraping is a powerful tool to extract data from websites for various purposes such as business intelligence and marketing.
- It involves making HTTP requests, parsing & extracting data, and storing the relevant information locally.
- Responsible web scraping requires adhering to robots.txt files, TOSs, privacy laws and best practices like using user agents & proxies with appropriate crawl rates during off peak hours when necessary.
Understanding Web Scraping
Web scraping is the process of extracting data from websites, enabling businesses and individuals to gather valuable insights from the vast ocean of information available online. At its core, web scraping has two components: a web crawler and a web scraper. The web crawler navigates the complex web of interconnected pages, while the web scraper extracts the desired data from these pages.
The extracted data can be used for various purposes, such as e-commerce business intelligence, investor research, and marketing teams looking to gain a competitive edge with comprehensive insights. To perform web scraping, a myriad of tools and techniques are available, ranging from Python scripts to cloud-based web scraping services.
Key Components: Crawlers and Scrapers
Web crawlers, sometimes referred to as "spiders," are artificial intelligence programs that browse the internet by following links and exploring content. Web scrapers, on the other hand, are specialized tools designed to extract data efficiently and accurately from a website. These two components work together to obtain the relevant data from web pages.
Before initiating the web scraping process, it is essential to specify the data to be collected to prevent the acquisition of excessive information that would later need to be processed. By refining the target data, web crawlers and scrapers can efficiently navigate and extract the necessary information from websites.
The Web Scraping Process
The web scraping process can be broken down into three primary steps: making an HTTP request, parsing, and extracting data, and storing the relevant data locally. The initial stage involves crawling the internet or a specific website to identify URLs that can be processed by the scraper for further examination.
Web scraping bots adhere to three fundamental principles: HTTP request, data parsing and extraction, and data storage and formatting. By following these principles, automated web scraping tools can efficiently collect and analyze data from multiple web pages in a structured format.
HTTP Requests and Responses
HTTP (HyperText Transfer Protocol) is an application layer protocol used to facilitate communication between clients and servers over the internet. In the server-client model, data is transferred across the network from one point to another.
When the client requests data from the server, a GET response is required. The server responds to the HTTP request by providing the data. This allows the scraper to read and access HTML or XML pages.
Parsing and Extracting Data
Once the web page's code is obtained, the next step in the web scraping process is parsing and extracting data. Parsing involves breaking down the web page's code into its constituent parts, while extraction focuses on obtaining the necessary data from the website's code. The extracted data can be loaded into a database or copied into a spreadsheet for further analysis.
A scraper can extract various types of information, such as text, ratings, classes, tags, IDs, or other relevant data points. When using a scraper for specific data extraction, such as book reviews, it is essential to specify information like the book title, author name, and rating.
Storing and Formatting Data
After the data has been extracted, it is crucial to store and format it properly to ensure its usability and comprehensibility. Recommended practices for storing and formatting data include adhering to file formats such as comma-separated values (CSV), Google Sheets, or Excel spreadsheets, documenting the data, using non-proprietary formats, and labeling columns to identify the data.
Taking into account metadata for data sets and carefully considering data storage options are also important aspects of this process.
Applications of Web Scraping
)
Web scraping has a wide range of applications across various industries, providing a valuable source of structured web data that can be harnessed to build and enhance business applications. From e-commerce to market research, web scraping enables businesses to extract large quantities of data from websites in a consistent and organized manner, making it an indispensable tool for data analytics.
Some applications of web scraping include monitoring a competitor's prices, tracking brand sentiment, assessing investment opportunities, and analyzing market trends. News and content monitoring through web scraping also offer an optimal solution for tracking and assembling the most essential stories from your industry.
Price Intelligence and Monitoring
Price intelligence and monitoring are essential use cases for web scraping, particularly in the e-commerce industry. By obtaining product and pricing information from competitor websites, businesses can make data-driven decisions that adapt to market changes and maintain a competitive edge.
Here's an example of automated price monitoring:
import requests
from bs4 import BeautifulSoup
import time
from datetime import datetime
import pandas as pd
def monitor_price(product_url, target_price):
"""Monitor product price and alert when target is reached"""
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
response = requests.get(product_url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
# Extract price (adjust selector based on target website)
price_element = soup.find('span', class_='price')
if price_element:
price_text = price_element.text.strip()
# Remove currency symbols and convert to float
current_price = float(price_text.replace('$', '').replace(',', ''))
print(f"Current price: ${current_price} at {datetime.now()}")
if current_price <= target_price:
print(f"🚨 Price alert! Target reached: ${current_price}")
# Add notification logic here (email, SMS, etc.)
return current_price
return None
# Example usage for competitive price tracking
def track_competitor_prices():
"""Track prices across multiple competitors"""
competitors = {
'Competitor A': 'https://competitor-a.com/product/123',
'Competitor B': 'https://competitor-b.com/item/456',
'Competitor C': 'https://competitor-c.com/products/789'
}
price_data = []
for name, url in competitors.items():
price = monitor_price(url, 100.0)
if price:
price_data.append({
'competitor': name,
'price': price,
'timestamp': datetime.now(),
'url': url
})
# Save to CSV for analysis
df = pd.DataFrame(price_data)
df.to_csv('competitor_prices.csv', index=False)
return df
# Monitor every hour
while True:
track_competitor_prices()
time.sleep(3600) # Wait 1 hourWeb scraping empowers businesses to stay informed and agile in a rapidly evolving market landscape.
Market Research and Analysis
High-quality web-scraped data provides valuable insights for market research and analysis. Data extracted from websites can reveal customer requirements, market influences, and industry trends, enabling businesses to make informed decisions and drive growth.
Here's an example of scraping social media sentiment for market research:
import requests
from bs4 import BeautifulSoup
import pandas as pd
from textblob import TextBlob
import matplotlib.pyplot as plt
def analyze_product_sentiment(product_name, review_urls):
"""Analyze sentiment of product reviews across multiple platforms"""
all_reviews = []
for platform, url in review_urls.items():
reviews = scrape_reviews(url, platform)
for review in reviews:
# Analyze sentiment using TextBlob
sentiment = TextBlob(review['text']).sentiment
review.update({
'platform': platform,
'polarity': sentiment.polarity, # -1 to 1
'subjectivity': sentiment.subjectivity, # 0 to 1
'sentiment_label': 'positive' if sentiment.polarity > 0.1
else 'negative' if sentiment.polarity < -0.1
else 'neutral'
})
all_reviews.append(review)
# Create DataFrame for analysis
df = pd.DataFrame(all_reviews)
# Generate insights
sentiment_summary = df.groupby('platform')['sentiment_label'].value_counts()
avg_sentiment = df.groupby('platform')['polarity'].mean()
return df, sentiment_summary, avg_sentiment
def scrape_reviews(url, platform):
"""Scrape reviews from different platforms"""
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
reviews = []
# Platform-specific selectors
if platform == 'amazon':
review_elements = soup.find_all('div', {'data-hook': 'review'})
for element in review_elements:
text_elem = element.find('span', {'data-hook': 'review-body'})
rating_elem = element.find('i', class_='review-rating')
if text_elem and rating_elem:
reviews.append({
'text': text_elem.get_text().strip(),
'rating': extract_rating(rating_elem.get('class')),
'platform': platform
})
elif platform == 'trustpilot':
review_elements = soup.find_all('div', class_='review-content')
for element in review_elements:
text_elem = element.find('p', class_='review-text')
rating_elem = element.find('div', class_='star-rating')
if text_elem and rating_elem:
reviews.append({
'text': text_elem.get_text().strip(),
'rating': extract_rating(rating_elem.get('class')),
'platform': platform
})
return reviews
# Example usage
product_urls = {
'amazon': 'https://amazon.com/product-reviews/B123456',
'trustpilot': 'https://trustpilot.com/review/company-name'
}
df, sentiment_summary, avg_sentiment = analyze_product_sentiment(
'iPhone 15', product_urls
)
print("Sentiment Analysis Results:")
print(sentiment_summary)
print("\nAverage Sentiment by Platform:")
print(avg_sentiment)In today's data-driven world, web scraping is an indispensable tool for market research and business intelligence.
Lead Generation and Marketing
Lead generation is a top challenge for marketers, but web scraping can alleviate this burden by providing structured lead lists. By collecting contact information from target audiences, such as names, job titles, email addresses, and cellphone numbers, web scraping enables businesses to pinpoint potential customers and target them with relevant marketing initiatives.
Leveraging web scraping for lead generation and marketing efforts can result in increased efficiency and a higher return on investment.
Web Scraping Tools and Techniques
To harness the power of web scraping, various tools and techniques are available to automate the extraction of large amounts of data from websites in a structured manner. Python libraries like BeautifulSoup and Scrapy, as well as cloud-based web scraping services, are commonly used to perform web scraping tasks efficiently and effectively.
These tools and techniques offer advanced features, such as recognizing unique HTML site structures, extracting reformatting and storing data from APIs, managing cookies, and circumventing any Terms of Use that restrict or prohibit content scraping. By employing these tools and techniques, businesses can unlock the true potential of web data and gain a competitive edge in their industries.
Python Libraries: BeautifulSoup and Scrapy
Python is the most popular language for web scraping due to its simplicity and powerful libraries. Here are the two main approaches:
BeautifulSoup - Simple and Beginner-Friendly:
import requests
from bs4 import BeautifulSoup
import time
def scrape_quotes():
"""Scrape quotes from a quotes website"""
url = "http://quotes.toscrape.com/"
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
quotes = []
for quote in soup.find_all('div', class_='quote'):
text = quote.find('span', class_='text').text
author = quote.find('small', class_='author').text
tags = [tag.text for tag in quote.find_all('a', class_='tag')]
quotes.append({
'text': text,
'author': author,
'tags': tags
})
return quotes
# Usage
quotes = scrape_quotes()
for quote in quotes[:3]:
print(f"'{quote['text']}' - {quote['author']}")Scrapy - Enterprise-Grade Framework:
# Create a new Scrapy project
# scrapy startproject quotes_scraper
import scrapy
class QuotesSpider(scrapy.Spider):
name = 'quotes'
start_urls = ['http://quotes.toscrape.com/']
def parse(self, response):
# Extract quotes from current page
for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
# Follow pagination links
next_page = response.css('li.next a::attr(href)').get()
if next_page is not None:
yield response.follow(next_page, self.parse)
# Run with: scrapy crawl quotes -o quotes.jsonKey Differences:
- BeautifulSoup: Best for simple, one-off scraping tasks
- Scrapy: Ideal for large-scale, complex scraping projects with built-in features like:
- Automatic request retries
- Concurrent request handling
- Built-in data export (JSON, CSV, XML)
- Middleware for proxies and user agents
- Robust error handling
Other Popular Python Libraries:
# Selenium for JavaScript-heavy sites
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("https://dynamic-content-site.com")
element = driver.find_element(By.CLASS_NAME, "dynamic-content")
print(element.text)
driver.quit()
# Requests-HTML for modern web scraping
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://example.com')
r.html.render() # Execute JavaScript
data = r.html.find('.content', first=True).textCloud-Based Web Scraping Services
Cloud-based web scraping services offer an alternative to traditional web scraping tools, providing a more flexible and scalable solution for extracting data from websites. These services are hosted on off-site servers provided by third-party vendors, offering flexible pricing options and eliminating the need to install any software on local machines.
Cloud-based web scrapers are particularly suitable for scraping large numbers of URLs, as the crawling and data extraction processes are conducted on off-site servers, reducing the load on local machines. By leveraging cloud-based web scraping services, businesses can collect and parse raw data from the web without the need for complex setup and maintenance.
Legal and Ethical Considerations
While web scraping offers numerous benefits and applications, it also raises legal and ethical considerations that must be addressed before embarking on a web scraping project. The legality of web scraping depends on various factors, such as the purpose of the scraping, the data accessed, the website's Terms of Use, and data sovereignty laws. To ensure compliance with these laws and regulations, it is important to respect the website's robots.txt file, adhere to its Terms of Service (TOS), and be aware of data protection and privacy laws.
By following best practices and being mindful of legal and ethical considerations, web scraping can be conducted in a responsible and compliant manner. However, it is essential to remain vigilant of the potential risks associated with web scraping, such as fraudulent activities, scams, intellectual property theft, and extortion.
Respecting Robots.txt and TOS
Before starting any web scraping project, always check the website's robots.txt file and Terms of Service. Here's how to programmatically check robots.txt:
import urllib.robotparser
from urllib.parse import urljoin
def check_robots_txt(url, user_agent='*'):
"""Check if scraping is allowed according to robots.txt"""
robots_url = urljoin(url, '/robots.txt')
try:
rp = urllib.robotparser.RobotFileParser()
rp.set_url(robots_url)
rp.read()
can_fetch = rp.can_fetch(user_agent, url)
crawl_delay = rp.crawl_delay(user_agent)
return {
'can_fetch': can_fetch,
'crawl_delay': crawl_delay,
'robots_url': robots_url
}
except Exception as e:
print(f"Error checking robots.txt: {e}")
return None
# Example usage
url = "https://example.com/products"
robot_rules = check_robots_txt(url)
if robot_rules and robot_rules['can_fetch']:
print("✅ Scraping allowed")
if robot_rules['crawl_delay']:
print(f"⏱️ Recommended delay: {robot_rules['crawl_delay']} seconds")
else:
print("❌ Scraping not allowed")Common robots.txt directives:
User-agent: *
Disallow: /admin/
Disallow: /private/
Crawl-delay: 1
User-agent: Googlebot
Allow: /
Sitemap: https://example.com/sitemap.xmlAlways respect these guidelines to maintain ethical scraping practices and avoid legal issues.
Data Protection and Privacy Laws
Data protection and privacy laws are legal frameworks designed to safeguard an individual's personal information and privacy. These laws vary from country to country and dictate the guidelines for collecting, processing, storing, and sharing personal data.
When engaging in web scraping activities, it is essential to adhere to these laws and regulations to ensure the responsible and ethical handling of personal information. By being mindful of data protection and privacy laws, web scrapers can avoid potential legal issues and maintain a responsible approach to data extraction.
Best Practices for Web Scraping
To ensure a successful and efficient web scraping project, several best practices should be followed. These include using user agents and proxies to simulate genuine users, setting a crawl rate and off-peak hours to avoid overloading the target website's server, and utilizing JavaScript (JS) rendering to handle dynamic content. By adhering to these best practices, web scrapers can overcome common challenges and enhance the overall efficiency of their data extraction efforts.
In addition to these best practices, it is crucial to remain vigilant of potential security risks associated with web scraping, such as scams, intellectual property theft, and extortion. By employing a responsible and ethical approach to web scraping, businesses can unlock the full potential of web data extraction while minimizing potential risks and legal issues.
User Agents and Proxies
User agents and proxies are essential tools for maintaining anonymity and avoiding detection during web scraping.
User Agent Rotation:
import random
import requests
# List of common user agents
user_agents = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15'
]
def get_random_headers():
"""Generate random headers for requests"""
return {
'User-Agent': random.choice(user_agents),
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate',
'Connection': 'keep-alive'
}
# Usage
response = requests.get('https://example.com', headers=get_random_headers())Proxy Rotation:
import itertools
import requests
# List of proxy servers
proxies_list = [
{'http': 'http://proxy1:8080', 'https': 'https://proxy1:8080'},
{'http': 'http://proxy2:8080', 'https': 'https://proxy2:8080'},
{'http': 'http://proxy3:8080', 'https': 'https://proxy3:8080'},
]
# Create a cycle iterator for proxy rotation
proxy_cycle = itertools.cycle(proxies_list)
def make_request_with_proxy(url):
"""Make request using rotating proxies"""
proxy = next(proxy_cycle)
headers = get_random_headers()
try:
response = requests.get(url, proxies=proxy, headers=headers, timeout=10)
return response
except requests.exceptions.RequestException as e:
print(f"Request failed with proxy {proxy}: {e}")
return None
# Advanced proxy management with requests-sessions
class ProxyRotator:
def __init__(self, proxies):
self.proxies = itertools.cycle(proxies)
self.session = requests.Session()
def get(self, url, **kwargs):
proxy = next(self.proxies)
self.session.proxies.update(proxy)
self.session.headers.update(get_random_headers())
return self.session.get(url, **kwargs)
# Usage
rotator = ProxyRotator(proxies_list)
response = rotator.get('https://example.com')By rotating user agents and proxies, scrapers can maintain anonymity and avoid IP-based blocking mechanisms.
Crawl Rate and Off-Peak Hours
Another key best practice in web scraping is to set an appropriate crawl rate and schedule scraping activities during off-peak hours. By doing so, web scrapers can avoid overloading the target website's server and minimize the risk of detection by anti-scraping technologies.
This approach ensures that the web scraping process is conducted in a responsible and efficient manner, minimizing potential disruptions to the target website's operations.
JavaScript Rendering
Modern websites heavily rely on JavaScript to load content dynamically. Traditional HTTP requests only retrieve the initial HTML, missing content loaded by JavaScript. Here's how to handle JavaScript-rendered content:
Using Selenium WebDriver:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
def scrape_spa_content(url):
"""Scrape Single Page Application content"""
# Configure Chrome options
chrome_options = Options()
chrome_options.add_argument('--headless') # Run in background
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome(options=chrome_options)
try:
driver.get(url)
# Wait for specific element to load
wait = WebDriverWait(driver, 10)
products = wait.until(
EC.presence_of_all_elements_located((By.CLASS_NAME, "product-item"))
)
# Extract data after JavaScript execution
scraped_data = []
for product in products:
data = {
'name': product.find_element(By.CLASS_NAME, 'product-name').text,
'price': product.find_element(By.CLASS_NAME, 'price').text,
'rating': product.get_attribute('data-rating')
}
scraped_data.append(data)
return scraped_data
finally:
driver.quit()
# Usage
products = scrape_spa_content('https://spa-ecommerce-site.com')
print(f"Found {len(products)} products")Using Playwright (Modern Alternative):
from playwright.sync_api import sync_playwright
def scrape_with_playwright(url):
"""Modern browser automation with Playwright"""
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
# Set user agent and viewport
page.set_user_agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36')
page.set_viewport_size({"width": 1920, "height": 1080})
page.goto(url)
# Wait for network to be idle (all requests finished)
page.wait_for_load_state('networkidle')
# Execute custom JavaScript if needed
page.evaluate('window.scrollTo(0, document.body.scrollHeight)')
# Extract data
products = page.query_selector_all('.product-item')
scraped_data = []
for product in products:
name = product.query_selector('.product-name').inner_text()
price = product.query_selector('.price').inner_text()
scraped_data.append({'name': name, 'price': price})
browser.close()
return scraped_dataHandling AJAX Requests:
# Monitor network requests to capture AJAX data
def intercept_api_calls(url):
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
api_responses = []
# Intercept network responses
def handle_response(response):
if 'api/' in response.url and response.status == 200:
api_responses.append({
'url': response.url,
'data': response.json()
})
page.on('response', handle_response)
page.goto(url)
page.wait_for_timeout(5000) # Wait for AJAX calls
browser.close()
return api_responsesWhen to Use JavaScript Rendering:
- Content loaded via AJAX/fetch requests
- Single Page Applications (SPAs)
- Infinite scroll pages
- Interactive elements requiring user actions
- Content behind login walls
Performance Considerations:
- JavaScript rendering is slower than static HTML parsing
- Requires more system resources (memory, CPU)
- Consider headless mode for better performance
- Use browser pools for concurrent scraping
Summary
Web scraping offers incredible potential for businesses and individuals to harness the vast data resources of the internet, providing valuable insights and a competitive edge in various industries. By understanding the intricacies of web scraping, employing powerful tools and techniques, and adhering to best practices and legal considerations, web scraping can be conducted in a responsible and efficient manner. Unleash the power of web scraping and unlock a world of data-driven insights to propel your business forward.
Frequently Asked Questions
What is web scraping used for?
Web scraping is a powerful tool used by businesses to extract important data and information from websites. It can be used to collect a range of data, including contact information, pricing data, user reviews, and product availability.
Companies use web scraping to gain valuable insights into their markets and competitors.
Is web scraping legal?
Overall, web scraping is generally legal as long as it involves data that is publicly available. However, it is important to adhere to international regulations to protect personal data, intellectual property, and confidential information.
What is an example of web scraping?
Web scraping is a powerful technique used to extract and analyze data from websites. An example of this would be gathering product information from an ecommerce website and transferring it into an Excel spreadsheet, making the data easier to use.
Automated tools can make web scraping more efficient, but manual web scraping is still possible.
Do hackers use web scraping?
Yes, hackers do use web scraping as a tool to extract data from websites. It is often used in combination with other methods of acquiring information, allowing them to collect data in bulk from multiple sources in order to carry out their illegal activities.
Modern Web Scraping Solutions
As web scraping becomes more complex in 2025, developers face increasing challenges with anti-bot measures, JavaScript-heavy sites, and infrastructure management. WebScraping.AI provides solutions for these common web scraping challenges:
Key Features:
🛡️ Automatic Proxy Management
- Rotates IP addresses automatically to prevent blocking
- Handles proxy failures and retries seamlessly
- Supports residential and datacenter proxy pools
🌐 Real Browser Rendering
- Uses real Chrome browsers to execute JavaScript
- Handles modern SPA (Single Page Applications)
- Supports complex user interactions and AJAX requests
⚡ Intelligent HTML Parsing
- Request specific data fields instead of full HTML
- Server-side parsing reduces client processing
- Structured JSON responses for easy integration
📊 Usage Analytics & Monitoring
- Track scraping success rates and performance
- Monitor data quality and extraction accuracy
- Real-time alerts for blocked requests
Example Integration:
import requests
# Simple API call to scrape any website
def scrape_with_webscraping_ai(url, selector=None):
"""
Scrape websites using WebScraping.AI API
"""
api_url = "https://api.webscraping.ai/html"
params = {
'url': url,
'api_key': 'YOUR_API_KEY',
'js': 'true', # Enable JavaScript rendering
'timeout': 10000
}
if selector:
params['selector'] = selector # Extract specific elements
response = requests.get(api_url, params=params)
if response.status_code == 200:
return response.text
else:
raise Exception(f"API Error: {response.status_code}")
# Extract product information
product_data = scrape_with_webscraping_ai(
'https://example-shop.com/product/123',
selector='.product-info'
)
print(product_data)This approach allows developers to focus on data analysis and business logic instead of managing infrastructure and dealing with anti-scraping measures.
