Can I use Ruby to scrape and interact with forms on websites?

Yes, you can use Ruby to scrape and interact with forms on websites. To perform web scraping with Ruby, you typically use libraries such as Nokogiri for parsing HTML and Mechanize for interacting with web forms.

Here's how you can use these libraries to scrape and interact with forms:

Step 1: Install the Required Gems

First, you need to install the necessary Ruby gems. You can do this by running the following commands in your terminal:

gem install nokogiri
gem install mechanize

Step 2: Use Nokogiri to Scrape Content

Nokogiri is a powerful HTML, XML, SAX, and Reader parser with the ability to search documents via XPath or CSS3 selectors.

require 'nokogiri'
require 'open-uri'

url = 'http://example.com'
html = open(url)

doc = Nokogiri::HTML(html)
titles = doc.css('h1')

titles.each do |title|
  puts title.content
end

Step 3: Use Mechanize to Interact with Forms

Mechanize is a library used for automating interaction with websites, which includes submitting forms.

Here's an example of how to use Mechanize to submit a form:

require 'mechanize'

agent = Mechanize.new
page = agent.get('http://example.com/login')

# Get the login form
login_form = page.form_with(id: 'login-form')

# Fill in the login details
login_form.field_with(name: 'username').value = 'your_username'
login_form.field_with(name: 'password').value = 'your_password'

# Submit the form
page = agent.submit(login_form)

# Now you are logged in, and you can navigate to other pages or interact with other forms

Example: Scraping and Interacting with a Search Form

Here's a more concrete example that demonstrates scraping a website and then interacting with a search form:

require 'mechanize'

# Initialize Mechanize agent
agent = Mechanize.new

# Fetch the page
page = agent.get('http://example.com')

# Look for the search form
search_form = page.form_with(id: 'search-form')

# Check if the form is found
if search_form
  # Set the search query
  search_form.field_with(name: 'query').value = 'web scraping'

  # Submit the form
  results_page = agent.submit(search_form)

  # Parse the results page
  results_page.search('.search-result').each do |result|
    title = result.at('.result-title').text.strip
    description = result.at('.result-description').text.strip
    puts "Title: #{title}"
    puts "Description: #{description}"
  end
else
  puts "Search form not found"
end

In this example, Mechanize is used to fetch the page, locate the search form, fill in the search query, and submit the form. After submitting the search form, the results page is parsed to extract the titles and descriptions of the search results.

Remember to always respect the website's robots.txt file and terms of service when scraping, and ensure that your activities do not violate any laws or regulations.