Nokogiri is a Ruby library for parsing HTML, XML, SAX, and Reader APIs. It is often used for web scraping purposes because it can easily navigate and search through the DOM of a webpage. However, Nokogiri itself does not have the capability to interact with JavaScript or to render a page like a browser would. It can only parse the static HTML content that it is given.
Headless browsers, on the other hand, are browsers without a graphical user interface that can be controlled programmatically. They can render pages just like a regular browser, execute JavaScript, and interact with dynamic content. Examples of headless browsers include Headless Chrome, Headless Firefox, and tools like Puppeteer (for Chrome) and Playwright (which can control multiple browsers).
To scrape content from a web page that relies heavily on JavaScript, you would first use a headless browser to render the page and execute the JavaScript, and then you could use Nokogiri to parse the HTML content.
Here's how you can integrate Nokogiri with a headless browser, using Headless Chrome controlled by the Selenium WebDriver:
First, you'll need to install the necessary Ruby gems:
gem install selenium-webdriver
gem install nokogiri
Next, you can use the following Ruby script to scrape dynamic content using a headless browser and then parse the resulting HTML with Nokogiri:
require 'selenium-webdriver'
require 'nokogiri'
# Set up headless Chrome
options = Selenium::WebDriver::Chrome::Options.new(args: ['headless'])
driver = Selenium::WebDriver.for(:chrome, options: options)
# Navigate to the page you want to scrape
driver.get('https://example.com')
# Wait for the page to load or for a specific element to appear
# driver.manage.timeouts.implicit_wait = 10 # seconds
# Get the HTML content of the page
html = driver.page_source
# Parse the HTML with Nokogiri
doc = Nokogiri::HTML(html)
# Now you can use Nokogiri methods to search and manipulate the HTML
# For example, to get the text of the first <h1> element:
h1_text = doc.at_css('h1').text.strip
puts h1_text
# Don't forget to quit the driver session
driver.quit
In this example, Selenium WebDriver is used to control Headless Chrome and navigate to the desired webpage. We then grab the rendered HTML source code of the page and parse it with Nokogiri to extract the information we're interested in.
Please note that when using headless browsers, you should always be respectful of the terms of service of the website you're scraping and ensure that your activities are legal. Additionally, heavy usage of headless browsers may put a load on the target website's servers, so it's important to use them responsibly.
