How do I scrape and parse JSON data with Ruby?

To scrape and parse JSON data with Ruby, you can use a few different libraries and modules. The core approach involves fetching the web content (which contains JSON) and then parsing it to extract the relevant information. Here's a step-by-step guide on how to do it:

Step 1: Install Required Gems

First, you'll need to install the httparty gem if you don't have it already. This gem makes it easy to make HTTP requests. You'll also need the json gem to parse the JSON content, though it is included with Ruby by default, so you may not need to install it separately.

gem install httparty

Step 2: Fetch the Web Content

Use HTTParty to make a GET request to the URL that returns JSON data.

require 'httparty'
require 'json'

url = 'http://example.com/data.json'
response = HTTParty.get(url)

Step 3: Parse the JSON Data

Once you have the response, you can parse the JSON data using Ruby's built-in JSON library.

parsed_data = JSON.parse(response.body)

Step 4: Access the Data

After parsing the JSON, you can access the data as you would with any other Ruby hash or array, depending on the structure of the JSON.

# If JSON is an array of objects
parsed_data.each do |item|
  puts item['key'] # Replace 'key' with the actual key you want to access
end

# If JSON is a single object
puts parsed_data['key'] # Replace 'key' with the actual key you want to access

Full Example

Here is a full example script that puts all these steps together to scrape and parse JSON data from a web service:

require 'httparty'
require 'json'

# The URL that returns JSON data
url = 'http://example.com/data.json'

# Fetch the data
response = HTTParty.get(url)

# Check if the request was successful
if response.code == 200
  # Parse the JSON data
  parsed_data = JSON.parse(response.body)

  # Access and print the data
  # Assuming the JSON is an array of objects
  parsed_data.each do |item|
    puts item['key'] # Replace 'key' with the actual key you want to access
  end
else
  puts "Error: Unable to fetch data (Status code: #{response.code})"
end

Remember to replace 'http://example.com/data.json' with the actual URL you want to scrape, and adjust the keys you access based on the structure of the JSON you're working with.

Error Handling

It's important to handle possible errors, such as network issues or unexpected data formats. The above code checks the response code to ensure the request was successful before attempting to parse the JSON.

Conclusion

With these steps, you can scrape and parse JSON data with Ruby. Make sure to respect the website's robots.txt file and terms of service when scraping data, and consider the legal and ethical implications of your scraping activity.