Mechanize examples

Note: Several examples show methods chained to the end of do/end blocks. Do...end is the same as curly braces ({...}). For example, do ... end.submit is the same as { ... }.submit.

Google

require 'rubygems'
require 'mechanize'

a = Mechanize.new { |agent|
  agent.user_agent_alias = 'Mac Safari'
}

a.get('http://google.com/') do |page|
  search_result = page.form_with(:name => 'f') do |search|
    search.q = 'Hello world'
  end.submit

  search_result.links.each do |link|
    puts link.text
  end
end

Rubyforge

require 'rubygems'
require 'mechanize'

a = Mechanize.new
a.get('http://rubyforge.org/') do |page|
  # Click the login link
  login_page = a.click(page.link_with(:text => %rLog In/))

  # Submit the login form
  my_page = login_page.form_with(:action => '/account/login.php') do |f|
    f.form_loginname  = ARGV[0]
    f.form_pw         = ARGV[1]
  end.click_button

  my_page.links.each do |link|
    text = link.text.strip
    next unless text.length > 0
    puts text
  end
end

File Upload

Upload a file to flickr.

require 'rubygems'
require 'mechanize'

abort "#{$0} login passwd filename" if (ARGV.size != 3)

a = Mechanize.new { |agent|
  # Flickr refreshes after login
  agent.follow_meta_refresh = true
}

a.get('http://flickr.com/') do |home_page|
  signin_page = a.click(home_page.link_with(:text => %rSign In/))

  my_page = signin_page.form_with(:name => 'login_form') do |form|
    form.login  = ARGV[0]
    form.passwd = ARGV[1]
  end.submit

  # Click the upload link
  upload_page = a.click(my_page.link_with(:text => %rUpload/))

  # We want the basic upload page.
  upload_page = a.click(upload_page.link_with(:text => %rbasic Uploader/))

  # Upload the file
  upload_page.form_with(:method => 'POST') do |upload_form|
    upload_form.file_uploads.first.file_name = ARGV[2]
  end.submit
end

Pluggable Parsers

Lets say you want html pages to automatically be parsed with Rubyful Soup. This example shows you how:

require 'rubygems'
require 'mechanize'
require 'rubyful_soup'

class SoupParser < Mechanize::Page
  attr_reader :soup
  def initialize(uri = nil, response = nil, body = nil, code = nil)
    @soup = BeautifulSoup.new(body)
    super(uri, response, body, code)
  end
end

agent = Mechanize.new
agent.pluggable_parser.html = SoupParser

Now all HTML pages will be parsed with the SoupParser class, and automatically give you access to a method called 'soup' where you can get access to the Beautiful Soup for that page.

Using a proxy

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
agent.set_proxy('localhost', '8000')
page = agent.get(ARGV[0])
puts page.body

The transact method

transact runs the given block and then resets the page history. I.e. after the block has been executed, you're back at the original page; no need count how many times to call the back method at the end of a loop (while accounting for possible exceptions).

This example also demonstrates subclassing Mechanize.

require 'rubygems'
require 'mechanize'

class TestMech < Mechanize
  def process
    get 'http://rubyforge.org/'
    search_form = page.forms.first
    search_form.words = 'WWW'
    submit search_form

    page.links_with(:href => %r{/projects/} ).each do |link|
      next if link.href =~ %r{/projects/support/}

      puts 'Loading %-30s %s' % [link.href, link.text]
      begin
        transact do
          click link
          # Do stuff, maybe click more links.
        end
        # Now we're back at the original page.

      rescue => e
        $stderr.puts "#{e.class}: #{e.message}"
      end
    end
  end
end

TestMech.new.process

Client Certificate Authentication (Mutual Auth)

In most cases a client certificate is created as an additional layer of security for certain websites. The specific case that this was initially tested on was for automating the download of archived images from a banks (Wachovia) lockbox system. Once the certificate is installed into your browser you will have to export it and split the certificate and private key into separate files. Exported files are usually in .p12 format (IE 7 & Firefox 2.0) which stands for PKCS #12. You can convert them from p12 to pem format by using the following commands:

openssl.exe pkcs12 -in input_file.p12 -clcerts -out example.key -nocerts -nodes openssl.exe pkcs12 -in input_file.p12 -clcerts -out example.cer -nokeys

require 'rubygems'
require 'mechanize'

# create Mechanize instance
agent = Mechanize.new

# set the path of the certificate file
agent.cert = 'example.cer'

# set the path of the private key file
agent.key = 'example.key'

# get the login form & fill it out with the username/password
login_form = agent.get("http://example.com/login_page").form('Login')
login_form.Userid = 'TestUser'
login_form.Password = 'TestPassword'

# submit login form
agent.submit(login_form, login_form.buttons.first)