Mechanize Web Scraping in Ruby and Python

Mechanize is a stateful programmatic browser: it keeps cookies, submits forms, follows redirects and links, and never runs a line of JavaScript. Two separate libraries share the name — the Ruby gem and the older Python package — and in 2026 they are in very different health. The Ruby gem is stable and safe to depend on; the Python package has not had a release since April 2024 and new Python projects should use MechanicalSoup or requests instead. This guide gives working code for both, the version facts behind that recommendation, and the point at which neither is the right tool.

Key Takeaways

• Ruby: mechanize 2.14.0 (January 2025), 41M downloads, repository still active with dependency and security updates. Feature-frozen, not abandoned
• Python: mechanize 0.4.10 (April 2024), no repository commits since May 2025. Working but dormant — prefer MechanicalSoup 1.4.0, which is actively developed
• Mechanize executes no JavaScript at all. Anything rendered client-side is invisible to it, and no configuration changes that
• Its value is session state: log in once and every later request on the same agent carries the cookie. Three lines instead of manual cookie juggling
• In Ruby, every Mechanize::Page is a Nokogiri document, so your existing CSS and XPath knowledge applies directly
• Its HTTP fingerprint is uniform and non-browser-like, so Cloudflare-class protection blocks it whatever User-Agent you set

Is Mechanize still maintained?

Worth answering precisely, because most Mechanize tutorials on the web were written between 2015 and 2020 and none of them tell you this:

The Ruby picture is a mature library in maintenance mode: 2026 activity is dependency bumps and a security policy rather than features, with only a handful of open issues. That is what a finished library looks like, not a dying one — Mechanize solves a problem that stopped changing a decade ago.

The Python picture is different. mechanize predates the Ruby gem, was handed to a new maintainer in 2017, and has now gone over two years without a release. It still works on modern Python, but "still works" is a weaker guarantee than most projects want from a dependency. MechanicalSoup — Requests plus Beautiful Soup wrapped in the same browser metaphor — is where that community moved.

Mechanize in Ruby

gem install mechanize

The Mechanize object is the browser; it persists cookies and history across requests:

require 'mechanize'

agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
agent.follow_meta_refresh = true
agent.read_timeout = 20

page = agent.get('https://quotes.toscrape.com/')

page.search('.quote').each do |quote|
  puts "#{quote.at('.author').text}: #{quote.at('.text').text}"
end

Following pagination

Link navigation is the reason Mechanize exists. Rather than reconstructing URLs, you click the links a user would:

page = agent.get('https://quotes.toscrape.com/')

loop do
  page.search('.quote .text').each { |q| puts q.text }

  next_link = page.link_with(text: 'Next →')
  break unless next_link

  page = next_link.click
  sleep 1                    # be polite; the target has no obligation to serve you
end

link_with also matches on href:, dom_class:, and regular expressions, and links_with returns all matches. Relative URLs are resolved against the current page automatically — a surprisingly large fraction of hand-rolled scraper bugs.

Logging in

page = agent.get('https://example.com/login')
form = page.form_with(action: '/login')
form.username = 'user@example.com'
form.password = 'secret'
dashboard = agent.submit(form)

# The agent now carries the session cookie for every subsequent request:
orders = agent.get('https://example.com/orders')

Form fields become methods named after their name attribute. When a field name isn't a valid Ruby identifier, use form['field-name'] = 'value'. Checkboxes and radios have .check, and form.file_uploads handles uploads.

Every page wraps a Nokogiri document, so page.search and page.at accept both CSS selectors and XPath expressions — there is nothing new to learn if you have used Nokogiri directly. The broader Ruby toolkit, including HTTParty for plain API calls and Ferrum for real-browser work, is covered in our Ruby web scraping libraries roundup.

Mechanize in Python

The Python package uses a Browser object with the same model:

import mechanize

br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent', 'Mozilla/5.0')]

br.open('https://example.com/login')
br.select_form(nr=0)
br['username'] = 'user@example.com'
br['password'] = 'secret'
br.submit()

html = br.open('https://example.com/orders').read()

It bundles no parser, so you pair it with Beautiful Soup or lxml for extraction.

Use MechanicalSoup instead

For anything new, MechanicalSoup gives you the same workflow on maintained foundations — Requests underneath for HTTP, Beautiful Soup for parsing:

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser(user_agent='Mozilla/5.0')
browser.open('https://example.com/login')

browser.select_form('form[action="/login"]')
browser['username'] = 'user@example.com'
browser['password'] = 'secret'
browser.submit_selected()

browser.open('https://example.com/orders')
for row in browser.page.select('tr.order'):        # browser.page is a BeautifulSoup object
    print(row.select_one('.total').get_text(strip=True))

The gain is not ergonomics — it is that browser.session is an ordinary requests.Session, so proxies, retries, timeouts, and TLS configuration all work the way the rest of the Python ecosystem works. If you don't actually need the form-and-session abstraction, a plain requests.Session plus Beautiful Soup covers the same ground with one fewer dependency; our Python web scraping libraries guide compares the options, and Scrapy is the better answer for crawls above a few thousand pages.

When does Mechanize stop working?

Two hard walls, and neither has a workaround inside Mechanize:

JavaScript. Mechanize parses HTML and speaks HTTP. React and Vue apps, infinite scroll, content loaded by XHR after page load, and anything behind a client-side router are simply not in the document it receives. The diagnostic takes ten seconds: disable JavaScript in your browser and reload the target. If the content is still there, Mechanize can scrape it. If not, no amount of Mechanize configuration will help.

Fingerprinting. Mechanize presents a consistent, obviously-not-a-browser TLS handshake and header ordering. Cloudflare, DataDome, and PerimeterX identify it in the first request, before any of your headers are even considered. Setting user_agent_alias changes what appears in a log file, not what a bot-detection vendor measures.

There is one intermediate case worth trying before you escalate: many "JavaScript-rendered" pages fetch their data from a JSON endpoint you can call directly. Open the network tab, find the XHR, and point Mechanize at that URL — it is faster than any browser and the response is already structured.

When that doesn't apply, the options are a real browser (Ferrum or Selenium in Ruby, Playwright or Puppeteer elsewhere — see our headless browser guide for the memory and detection trade-offs), or delegating the fetch.

Keeping the Mechanize workflow, delegating the fetch

Delegation keeps the shape of your code. You lose the form-filling abstraction — the API fetches a URL rather than driving a session — but the parsing side is untouched:

require 'net/http'
require 'nokogiri'

uri = URI('https://api.webscraping.ai/html')
uri.query = URI.encode_www_form(
  api_key: ENV.fetch('WEBSCRAPING_AI_KEY'),
  url: 'https://example.com/js-heavy-page',
  js: true,
  proxy: 'residential',
  wait_for: '.results-loaded'      # hold the response until this selector exists
)

doc = Nokogiri::HTML(Net::HTTP.get(uri))
doc.css('.result').each { |r| puts r.at('.title').text }

wait_for is the piece that replaces the "did the content load yet?" guessing — the response doesn't come back until that selector is in the DOM. If you only need a few values, /ai/fields returns JSON keyed by field descriptions you write in plain English, which survives markup changes that would break a CSS selector.

WebScraping.AI has a first-party Ruby gem and Python package if you'd rather not build the query string by hand. Pricing is 1 credit for a plain fetch, 5 with JavaScript rendering, 10/25 for residential proxies, and failed requests are free — so targets that block you don't show up on the bill. The free tier is 2,000 credits a month with no card. This matters most for the long-running job, the b2b lead generation or job listing aggregation pipeline that has to keep working after a target adds bot protection.

Whichever route you take, know what you're allowed to collect before you build it — is web scraping legal covers the terms-of-service and personal-data questions honestly.

Frequently asked questions

Is the Mechanize gem still maintained in 2026? Yes, in maintenance mode. Version 2.14.0 shipped in January 2025 and the repository was still receiving dependency and security updates in May 2026, with a handful of open issues. There are no new features coming, which for a library this mature is a reasonable state rather than a warning sign.

Should I use Mechanize for Python? Not for new projects. The Python mechanize package last released 0.4.10 in April 2024 and its repository has been quiet since May 2025. MechanicalSoup offers the same browser-and-forms model on top of Requests and Beautiful Soup and is actively developed. Existing Python mechanize code doesn't need an emergency rewrite — just don't start there.

Can Mechanize handle JavaScript? No, neither version, and this is architectural rather than a missing feature. Mechanize fetches HTML over HTTP and parses it; there is no JavaScript engine and no DOM to run one against. Use a headless browser or a rendering API for client-side content.

Mechanize or Nokogiri? They do different jobs and Mechanize uses Nokogiri internally. Nokogiri parses HTML you already have; Mechanize fetches pages, tracks cookies, and submits forms, handing you a Nokogiri document at the end. If you need a session, use Mechanize. If you already have the HTML, use Nokogiri directly.

How do I use a proxy with Mechanize? In Ruby, agent.set_proxy('proxy.example.com', 8080, 'user', 'pass'). In Python, br.set_proxies({"http": "http://proxy.example.com:8080"}). Both route requests through the proxy, but neither changes the TLS fingerprint that anti-bot systems inspect — a proxy fixes IP-reputation and rate-limit blocks, not detection.