How do I parse XML with Ruby for web scraping?

Parsing XML with Ruby is quite straightforward, thanks to the built-in libraries like REXML and Nokogiri. Nokogiri is more popular and generally preferred for its ease of use and speed. Below is a step-by-step guide to parsing XML with Nokogiri for web scraping purposes.

Step 1: Install Nokogiri

Before you start, you need to install the Nokogiri gem if you haven't already. You can do this by running the following command in your terminal:

gem install nokogiri

Step 2: Require Nokogiri in Your Ruby Script

In your Ruby script, require the Nokogiri gem at the top of the file:

require 'nokogiri'
require 'open-uri' # If you plan to scrape data from the web.

Step 3: Load XML Content

You can parse XML content from a string, a file, or directly from the web.

From a String:

xml_str = <<-XML
<root>
  <item>
    <title>Item 1</title>
    <link>http://example.com/1</link>
  </item>
  <item>
    <title>Item 2</title>
    <link>http://example.com/2</link>
  </item>
</root>
XML

doc = Nokogiri::XML(xml_str)

From a File:

# Assuming 'example.xml' is your XML file.
doc = Nokogiri::XML(File.open('example.xml'))

From the Web:

url = 'http://example.com/data.xml'
xml_data = open(url)
doc = Nokogiri::XML(xml_data)

Step 4: Parse XML Content

With the XML content loaded into a Nokogiri document, you can now parse it using various methods provided by Nokogiri.

Find Nodes by XPath:

items = doc.xpath('//item')
items.each do |item|
  title = item.at_xpath('title').content
  link = item.at_xpath('link').content
  puts "Title: #{title}, Link: #{link}"
end

Find Nodes by CSS Selectors:

items = doc.css('item')
items.each do |item|
  title = item.at_css('title').content
  link = item.at_css('link').content
  puts "Title: #{title}, Link: #{link}"
end

Handle Namespaces:

If the XML you are parsing uses namespaces, you might need to handle them to select nodes correctly:

doc.remove_namespaces! # Removes namespaces from all nodes.

# Alternatively, define namespaces to use them in your XPath queries.
namespaces = {
  'ns' => 'http://example.com/ns'
}

items = doc.xpath('//ns:item', namespaces)

Step 5: Handle Errors and Encoding

Nokogiri can automatically handle different encodings, but you should be aware of potential errors:

begin
  # Parsing code here.
rescue Nokogiri::XML::SyntaxError => e
  puts "Caught an exception: #{e}"
end

Example Usage:

require 'nokogiri'
require 'open-uri'

url = 'http://example.com/data.xml'
xml_data = open(url)
doc = Nokogiri::XML(xml_data)

items = doc.xpath('//item')
items.each do |item|
  title = item.at_xpath('title').content
  link = item.at_xpath('link').content
  puts "Title: #{title}, Link: #{link}"
end

This script will fetch XML data from the specified URL, parse it, and print out the title and link for each <item> element.

Remember to handle exceptions and edge cases in your production code, especially when dealing with web scraping, to account for network issues, unexpected data formats, and changes in the target website's structure or availability.