Where can I find examples of real-world web scraping using Reqwest?

Reqwest is a modern Rust library for making HTTP requests, which is akin to what requests is in Python. While Reqwest itself is not a web scraping tool, it can be used as part of a web scraping solution in Rust by handling the HTTP requests to fetch web pages. Actual web scraping—parsing and extracting data—would typically be performed by another library, such as scraper which is inspired by Python's BeautifulSoup.

Here's an example of how you might use Reqwest and Scraper together in a real-world Rust web scraping scenario:

use reqwest;
use scraper::{Html, Selector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Use Reqwest to download the HTML content of a web page
    let res = reqwest::get("https://www.example.com").await?;
    let body = res.text().await?;

    // Parse the HTML using the Scraper crate
    let document = Html::parse_document(&body);

    // Create a Selector for the elements you're interested in
    let selector = Selector::parse(".some-class").unwrap();

    // Iterate over elements matching the selector
    for element in document.select(&selector) {
        // Extract text or attribute value from the element
        let text = element.text().collect::<Vec<_>>();
        println!("{:?}", text);
    }

    Ok(())
}

In the above example, Reqwest is used to asynchronously fetch a web page, and Scraper is employed to parse the HTML and extract elements with a class of some-class. Note that this is a simplified example; in real-world scenarios, you would need to handle potential errors more gracefully and also respect the website's robots.txt rules and terms of service.

For JavaScript, web scraping is commonly done using libraries like Axios for HTTP requests and Cheerio or Puppeteer for parsing and interacting with HTML content. Here's a similar example using Axios and Cheerio:

const axios = require('axios');
const cheerio = require('cheerio');

// Use Axios to download the HTML content of a web page
axios.get('https://www.example.com')
  .then(response => {
    const html = response.data;

    // Load the HTML into Cheerio
    const $ = cheerio.load(html);

    // Use a selector to find elements with the class 'some-class'
    $('.some-class').each((index, element) => {
      // Extract text or attribute value from the element
      const text = $(element).text();
      console.log(text);
    });
  })
  .catch(console.error);

In the JavaScript example, the Axios library fetches the HTML from a web page, and Cheerio is used to parse the HTML and select elements with a class of some-class, extracting and logging their text content.

Remember, when scraping websites, always:

1. Check the robots.txt file of the website to see if scraping is permitted.
2. Read through the website's Terms of Service to ensure you're not violating any terms.
3. Be respectful with your requests; do not send too many requests in a short period, which could overload the website's server.
4. Consider caching pages and setting a user-agent string that identifies your bot.