Can Ruby be used to scrape data from APIs instead of HTML?

Yes, Ruby can be used to scrape data from APIs. In fact, interacting with APIs is often simpler and more reliable than scraping HTML because APIs are designed to be consumed by programs and provide structured data, typically in JSON or XML format. Ruby, like many other programming languages, has libraries that make it easy to send HTTP requests and process the data returned by APIs.

Here's an example of how you might use Ruby to scrape data from a JSON API:

First, you'll want to install the httparty gem, which simplifies the process of making HTTP requests:

gem install httparty

Then you can use the following Ruby script to make a request to an API and process the JSON response:

require 'httparty'
require 'json'

# Replace the URL below with the API endpoint you want to scrape
url = 'https://api.example.com/data'

# Make the HTTP GET request
response = HTTParty.get(url)

# Check if the request was successful
if response.code == 200
  # Parse the JSON response
  data = JSON.parse(response.body)

  # Now you can work with the data, for example, print it
  puts data

  # If the API returns an array of items, you can iterate over them
  if data.is_a?(Array)
    data.each do |item|
      puts item
    end
  end
else
  puts "Error: #{response.code}"
end

Keep in mind that you should always check the API's documentation to understand the structure of the response and to see if there are any authentication or rate-limiting considerations.

Authentication:

Many APIs require some form of authentication. Here's an example of how you might include an API key in your request headers:

headers = {
  'Authorization' => 'Bearer YOUR_API_KEY',
  'Accept' => 'application/json'
}

response = HTTParty.get(url, headers: headers)

Rate Limiting:

APIs often have rate limits to prevent abuse. Be sure to check the API documentation for any rate limits and handle them in your code. This might involve adding delays between requests or gracefully handling 429 Too Many Requests responses.

Error Handling:

It's important to handle errors and exceptions that may occur when making requests. You should check for HTTP error codes and potentially rescue exceptions that may be thrown by the HTTParty library.

Ruby is a powerful language for interacting with APIs, and libraries like HTTParty make it straightforward. Remember to respect the terms of service for the API you are using, as scraping can be a violation in some cases.