Ruby Web Scraping: The Libraries and Frameworks Worth Using in 2026

The short answer for 2026: Faraday or HTTParty to fetch, Nokogiri to parse, Ferrum when the page needs JavaScript, Mechanize when you need a stateful session. That's the whole stack for most jobs. Everything else in the Ruby scraping ecosystem is either a wrapper around those four or a project that stopped shipping years ago — and a lot of published Ruby scraping advice still recommends the latter.

This guide covers what each gem is actually for, which ones are still maintained (with release dates, not vibes), and the version-specific gotchas that break copy-pasted examples.

Key Takeaways

• Nokogiri::HTML is the HTML4/libxml2 parser. Use Nokogiri.HTML5(...) instead — it's the WHATWG-spec parser, so it builds the same DOM your browser does on modern markup.
• Ferrum talks to Chrome over CDP with no WebDriver in the loop — it's the fastest way to render JS in Ruby, and it's actively developed (0.17.2, March 2026).
• Watir is effectively frozen: last gem release 7.3.0 in August 2023, last commit to the repo May 2024. Don't start new work on it.
• Kimurai came back from the dead. It sat at 1.4.0 from January 2019 until 2.0.0 shipped in December 2025; 2.2.0 landed January 2026. Usable again, but verify before you bet a project on it.
• Delete gem "webdrivers" from your Gemfile. Its own maintainer says to stop requiring it on Selenium 4.11+, which manages drivers itself.
• Ruby 3.2 hit end of life on 2026-04-01 and 3.3 is security-fixes-only. Target 3.4 or 4.0 — selenium-webdriver 4.46 already requires Ruby >= 3.3.

Which Ruby scraping gem should you use?

Decision rule: start with Faraday + Nokogiri. Add Mechanize only when you find yourself hand-rolling a cookie jar. Reach for Ferrum only after you've confirmed the data isn't in the initial HTML — check by disabling JavaScript in DevTools and reloading, or by looking for the underlying JSON endpoint in the Network tab. A browser is roughly two orders of magnitude more expensive per page than an HTTP request.

Is Ruby still a good language for web scraping?

For fetching and parsing, yes — Nokogiri is a mature libxml2/gumbo binding, and Ruby's threading model handles network-bound work well (MRI releases the GVL during blocking I/O, so a thread pool gives you real concurrency on requests).

Where Ruby is genuinely behind Python: there's no equivalent of Scrapy's full crawling framework with the same ecosystem depth, and browser automation has fewer options — no first-party Playwright support, for example. If you're standing up a large distributed crawler from scratch, that's a real consideration. If you're adding scraping to an existing Rails app, the gems below are perfectly adequate and you keep your deployment, job queue, and models.

Fetching pages: Faraday vs HTTParty

HTTParty is the shorter one:

require "httparty"

UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " \
     "(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"

response = HTTParty.get(
  "https://example.com/products",
  headers: { "User-Agent" => UA, "Accept-Language" => "en-US,en;q=0.9" },
  timeout: 15
)

raise "HTTP #{response.code}" unless response.success?
html = response.body

Faraday costs a few more lines and gives you a middleware stack, which is what you want once retries stop being optional:

require "faraday"
require "faraday/retry"   # Faraday 2.x moved retry OUT of core — separate gem

conn = Faraday.new(
  headers: { "User-Agent" => UA },
  request: { timeout: 15, open_timeout: 5 }
) do |f|
  f.request :retry,
            max: 3,
            interval: 0.5,
            backoff_factor: 2,                            # 0.5s, 1s, 2s
            retry_statuses: [429, 500, 502, 503, 504],
            exceptions: [Faraday::TimeoutError, Faraday::ConnectionFailed]
  f.response :raise_error                                 # 4xx/5xx become exceptions
end

html = conn.get("https://example.com/products").body

The Faraday 2 gotcha: f.request :retry raises Faraday::Error unless you add gem "faraday-retry" and require it. Retry middleware lived in Faraday core in 1.x and was extracted in 2.0, which is why so many older snippets fail on a fresh install.

Ruby's stdlib Net::HTTP works too, and URI.open from open-uri shows up in every beginner tutorial. Skip URI.open for scraping: it doesn't let you set a timeout cleanly, and passing user-controlled input to it is a known command-injection vector.

Parsing with Nokogiri: use the HTML5 parser

This is the single most valuable correction to make to old Ruby scraping code:

require "nokogiri"

doc = Nokogiri::HTML(html)    # HTML4 parser (libxml2) — HTML is an alias for HTML4
doc = Nokogiri.HTML5(html)    # WHATWG HTML5 parser (gumbo) — use this

Nokogiri::HTML has been an alias for Nokogiri::HTML4 since 1.12. The HTML4 path uses libxml2's parser, which predates the HTML5 spec and recovers from broken markup differently than a browser does — nested tables, <template>, and unclosed tags are where you notice. Nokogiri.HTML5 implements the WHATWG parsing algorithm, so the tree you query matches the DOM you inspected in DevTools.

Extraction, with the failure modes handled:

doc = Nokogiri.HTML5(html)

products = doc.css("li.product").map do |node|
  {
    name:  node.at_css("h2")&.text&.strip,
    price: node.at_css(".price")&.text.to_s[/[\d,.]+/]&.delete(",")&.to_f,
    url:   URI.join("https://example.com", node.at_css("a")&.attr("href").to_s).to_s,
    sku:   node["data-sku"]
  }
end.reject { |p| p[:name].nil? }

Three things worth copying:

• at_css returns one node or nil; css returns a NodeSet. Calling .text on an empty NodeSet gives you "", not an error — which is how silent data loss happens. Prefer at_css + &. so a layout change gives you nil you can filter on.
• node["href"] reads an attribute and returns a String directly. The .attribute("href").value form you see in older posts blows up with NoMethodError on missing attributes.
• URI.join for relative links. Never string-concatenate a base URL with an href.

CSS selectors cover most cases; XPath earns its keep for axes CSS can't express — "the <td> following the <th> whose text is 'SKU'" is one XPath expression and no CSS selector. Our XPath cheat sheet has the patterns worth memorising.

Sessions, logins, and forms: Mechanize

Mechanize bundles an HTTP client, a cookie jar, and Nokogiri into an object that remembers where it's been:

require "mechanize"

agent = Mechanize.new
agent.user_agent_alias = "Mac Safari"
agent.history_added = proc { sleep 0.5 }   # rate limit every navigation

login = agent.get("https://example.com/login")
form = login.form_with(action: /login/)
form.field_with(name: "email").value    = ENV.fetch("SCRAPER_EMAIL")
form.field_with(name: "password").value = ENV.fetch("SCRAPER_PASSWORD")
dashboard = agent.submit(form)

# Cookies persist automatically across subsequent requests
report = agent.get("https://example.com/reports/monthly")
rows   = report.search("table.data tr")   # Nokogiri under the hood

history_added is the underrated feature: it fires on every page load, so one line gives you a global delay without threading a sleep through your crawl logic. Mechanize is maintained (2.14.0, January 2025; commits through May 2026) and is the right tool whenever authentication or multi-step forms are involved. Our Mechanize guide goes deeper on form handling.

What Mechanize can't do is run JavaScript — it has no JS engine at all. If a form submits over fetch() and updates the DOM, Mechanize sees nothing.

Scraping JavaScript pages: Ferrum

Ferrum drives Chrome over the Chrome DevTools Protocol directly — no ChromeDriver, no WebDriver process, no webdrivers gem. It's the modern default for headless browser work in Ruby:

require "ferrum"
require "nokogiri"

browser = Ferrum::Browser.new(
  headless: true,
  timeout: 20,
  window_size: [1366, 768],
  browser_options: { "disable-blink-features" => "AutomationControlled" }
)

page = browser.create_page
page.headers.set_overrides(user_agent: UA)

# Block what you don't parse — images and trackers are most of the page weight
page.network.blacklist = [
  %r{\.(png|jpe?g|gif|webp|svg|woff2?)(\?|$)},
  /googletagmanager\.com/,
  /doubleclick\.net/
]

begin
  page.go_to("https://example.com/spa-products")
  page.network.wait_for_idle(timeout: 10)   # settle XHR, not an arbitrary sleep

  raise "blocked: #{page.network.status}" unless page.network.status == 200

  doc = Nokogiri.HTML5(page.body)           # hand the rendered DOM to Nokogiri
  puts doc.css("li.product").size
ensure
  browser.quit                              # always kill the Chrome process
end

Two habits that matter more than the gem choice:

• network.wait_for_idle instead of sleep. It waits for in-flight connections to drain, so slow pages still work and fast pages don't cost you three wasted seconds each.
• network.blacklist aborts requests matching your patterns. Dropping images, fonts, and analytics typically cuts page load time and bandwidth substantially — and you were never going to parse them.

If your team already writes Capybara specs, Cuprite is the same engine behind Capybara's DSL (visit, find, all), which makes scraping code look like your test suite. It's a thinner layer over Ferrum, not a different browser.

What about Watir, Selenium, and Kimurai?

This is where most Ruby scraping articles are out of date, so here's the current state with dates:

Watir — avoid for new projects. Version 7.3.0 shipped in August 2023 and the GitHub repo's last commit was May 2024. It's a wrapper over selenium-webdriver (declared as ~> 4.2), and Selenium has released dozens of versions since. Nothing is broken today, but an unmaintained wrapper over a fast-moving dependency is a maintenance bill waiting to arrive.

selenium-webdriver — fine, just not the first choice. 4.46.0 (July 2026) is actively maintained and requires Ruby >= 3.3. Use it when you need a browser Ferrum can't drive (Firefox, Safari) or a Selenium Grid. For Chrome-only scraping, Ferrum has less machinery in the path. And delete the webdrivers gem: its README tells you outright to stop requiring it on Selenium 4.11+, because Selenium Manager now downloads and manages drivers itself.

Kimurai — genuinely revived, verify before committing. The framework was dormant at 1.4.0 from January 2019 through late 2025; 2.0.0 landed in December 2025 and 2.2.0 in January 2026, now requiring Ruby >= 3.2 and building on Capybara ~> 3.40. It gives you Scrapy-like spider classes, built-in throttling, and pipeline hooks. The caveat is that the revival is recent and the repo's last activity was January 2026 — run your own smoke test before making it load-bearing.

Concurrency without a framework

Scraping is I/O-bound, and MRI releases the GVL during blocking I/O, so a plain thread pool works well:

require "faraday"

def fetch_all(urls, concurrency: 5)
  queue   = Queue.new
  results = Queue.new
  urls.each { |u| queue << u }

  workers = Array.new(concurrency) do
    Thread.new do
      conn = Faraday.new(headers: { "User-Agent" => UA },
                         request: { timeout: 15 })
      loop do
        url = begin
          queue.pop(true)       # non-blocking; raises when the queue is empty
        rescue ThreadError
          break
        end

        begin
          results << [url, conn.get(url).body]
        rescue Faraday::Error => e
          warn "#{url}: #{e.class}"
        end
        sleep 0.2 + rand(0.3)   # jittered politeness delay
      end
    end
  end

  workers.each(&:join)
  Array.new(results.size) { results.pop }
end

Each thread gets its own Faraday connection — connection objects aren't guaranteed thread-safe to share. Start at 5 concurrent requests and only go higher against infrastructure you own or a service you're paying for.

When the gems aren't the problem

Past a certain point your scraper stops failing because of parsing and starts failing because the target doesn't want automated traffic. The escalation ladder, in order of effort:

1. A real User-Agent and matching headers. The cheapest fix, and it resolves a surprising share of 403s — see User-Agent rotation.
2. Slow down. Concurrency 2, one second between requests. Rate limits are the most common invisible block.
3. Change IPs. Datacenter proxies first; move to residential only when datacenter IPs get blocked, since they cost far more per request. Our guide to proxy types covers the tradeoff.
4. Render in a real browser with a plausible fingerprint — headless Chrome with default settings is detectable on its own.

Before scaling any of this, read the target's terms and robots.txt. Is web scraping legal covers the actual case law rather than the usual hand-waving; the summary is that it depends heavily on what data you take and how you got it.

Using WebScraping.AI from Ruby

Steps 3 and 4 above are infrastructure work: proxy pools, browser fleets, fingerprint maintenance. If that isn't the product you're building, our API does the fetch and hands your Ruby code the HTML — and there's a first-party gem for it (this site is a Rails app, so the Ruby client is one we use ourselves):

# Gemfile: gem "webscraping_ai", "~> 4.0"
require "webscraping_ai"
require "nokogiri"

client = WebScrapingAI::Client.new(api_key: ENV.fetch("WEBSCRAPING_AI_API_KEY"))

# Rendered HTML through a rotating proxy — then parse as usual
html = client.html(
  "https://example.com/spa-products",
  js: true,
  wait_for: "li.product",       # wait for a selector, not a fixed timeout
  proxy: "residential",
  country: "us"
)
products = Nokogiri.HTML5(html).css("li.product")

# Or skip selectors entirely and describe the fields you want
data = client.fields(
  "https://example.com/product/42",
  fields: {
    title: "Product title",
    price: "Current price in USD, numbers only",
    in_stock: "true or false, whether the item is purchasable"
  }
)
# => {"title" => "...", "price" => "39.99", "in_stock" => "true"}

Errors come back as a typed hierarchy, so retry logic stays readable:

begin
  client.html(url, js: true)
rescue WebScrapingAI::RateLimitError
  sleep 1
  retry
rescue WebScrapingAI::GatewayTimeoutError
  client.html(url, js: true, timeout: 30_000)   # page needed longer
end

Requests are priced in credits: 1 for a plain datacenter fetch, 5 with JavaScript rendering, 10 and 25 for residential without and with JS, 50 for stealth, and +5 when you use the AI extraction endpoints. Failed requests don't cost credits. The free tier is 2,000 credits a month with no credit card; full parameter reference is in the docs, and the AI extraction endpoints are documented there too. Teams running this for price monitoring usually pair it with a Sidekiq job per target.

Frequently asked questions

What is the best Ruby gem for web scraping? Nokogiri, for parsing — it's in every Ruby scraping stack regardless of how you fetch. For fetching, Faraday if you want middleware for retries and logging, HTTParty if you want the shortest possible code. There is no single gem that does the whole job well; Ruby's ecosystem composes small ones.

Which Ruby scraping framework should I use? Kimurai is the only real Scrapy analogue in Ruby and it's maintained again as of December 2025 (2.2.0 in January 2026). For most projects a plain Ruby class plus Faraday, Nokogiri, and a Sidekiq job is less machinery and easier to debug than a framework.

How do I scrape JavaScript-rendered pages in Ruby? Use Ferrum, which drives headless Chrome over CDP with no WebDriver dependency, and pass page.body to Nokogiri once page.network.wait_for_idle returns. Before that, check whether the page fetches its data from a JSON endpoint you can call directly — that's faster and far more stable than driving a browser.

Is Nokogiri::HTML the same as Nokogiri.HTML5? No. Nokogiri::HTML is an alias for Nokogiri::HTML4, which uses libxml2's pre-HTML5 parser. Nokogiri.HTML5 uses the WHATWG-conformant gumbo parser and builds the same tree a browser would. Use HTML5 for anything scraped off the modern web.

Is Watir still maintained? Not actively. The last release, 7.3.0, was August 2023 and the repository's last commit was May 2024. It still works against selenium-webdriver ~> 4.2, but new Ruby browser automation should start with Ferrum or selenium-webdriver directly.

Do I still need the webdrivers gem? No. Selenium 4.11+ ships Selenium Manager, which downloads and manages browser drivers itself; the webdrivers maintainer explicitly recommends dropping the gem if you can run Selenium 4.11 or newer.

Can I scrape from inside a Rails app? Yes — that's one of Ruby's practical advantages. Put the fetch in an ActiveJob/Sidekiq worker rather than a request cycle, persist results with ActiveRecord, and cache raw HTML so a parser bug doesn't mean re-fetching everything.