Posts tagged "tests"

TL;DR: test quality is not just about verifying correctly whether your code works, it’s also about making your test easy to read and understand. You can do that by structuring your test using the four-phases xUnit standard.

The 4 phases of a test: setup, exercise, verify and teardown

People don’t write tests to be read, they write them to be executed

One of the main reasons to write tests is to have an automated way to check if your code is doing what you expect it to do. That means, trying to verify its correctness. Your test suite acts as a safety net that guarantees your software will continue to work as expected while you refactor, build new features or fix bugs. That’s amazing! But, throughout the years, software developers discovered that tests can be even more than a safety net.

Write tests as examples of how to use your code

A test is an example of how to use your code, not just a way to verify its correctness. Seeing tests as examples of how to use your code changes a little bit the priorities you have when writing them. If the test should serve as an example, then it should be easy to read and understand. Therefore, you should also focus on test readability, not just test “executability” (I know, weird word). One way to improve readability of a piece of text (or code) is to write it in a way the readers are used to, a structure that they expect, some standard way… Let’s think about that.

Back at your school days, you learned that when writing an essay, you’re supposed to structure it in: introduction, body and conclusion. Why? Because that structure helps you to better express your ideas. That means, it helps the reader to understand your message. Is there any equivalent of that for automated tests writing? In fact, there is. It’s called the xUnit structure.

Structure your tests using the xUnit standard

First, let’s see a test that can have its readability improved:

describe Stack do
  describe "#push" do
    it "puts an element at the top of the stack" do
      stack = Stack.new
      stack.push(1)
      stack.push(2)
      expect(stack.top).to eq(2)
    end
  end
end

One can understand the test above, but still, it’s not easy to quickly scan the test and see that it has logical parts. Those parts would be the the xUnit phases.

The standard xUnit test structure is composed of 4 phases: setup, exercise, verify and teardown.

  • Setup: this where you put the object under test in the necessary state for the behavior you want to check;
  • Exercise: when you send a message to your object;
  • Verify: here, you should check if the object under test behaved the way you expected;
  • Teardown: basically where you clean up stuff in order to get your system back to the initial state.

Now, let’s re-organize the test above making explicit that there are different logical parts:

describe Stack do
  describe "#push" do
    it "puts an element at the top of the stack" do
      # setup
      stack = Stack.new
 
      # exercise
      stack.push(1)
      stack.push(2)
 
      # verify
      expect(stack.top).to eq(2)
    end
  end
end

It’s easier to scan, isn’t it?

About the comments, no, we don’t need them. I added them in order to make the example clear. Let’s remove them and keep this structure:

describe Stack do
  describe "#push" do
    it "puts an element at the top of the stack" do
      stack = Stack.new
 
      stack.push(1)
      stack.push(2)
 
      expect(stack.top).to eq(2)
    end
  end
end

One can say that we just added two line breaks, that’s true. But that’s just the how, not the what. The what is: improving test readability. The how is: structuring the code based on the xUnit four-phase standard, by adding two line breaks. Got it?

Using a standard structure to ease the communication of an idea is not something new. As an example, Rails does that when it generates a standard directory structure. When entering on a new Rails project and scanning it, you know your way and where stuff are because you already expect a defined structure and you are used to it. It’s not something completely new, you’re used to that structure. I could also say that even Ruby uses that concept when it talks about the “principle of least surprise”, but maybe I would be going too far. So, let’s get that wrapped up.

Why care about test readability?

So, why should I care about all of that stuff? I mean, isn’t just having my test suite on green enough? No.

Test readability will be really important in a lot of situations. Like when a test gets red, someone needs to fix it. In order to do that, one needs to understand what the test is about. If the test is well structured and easy to read, they can fix it faster.

Also, if you think about your tests as examples of how to use your code, someone that is trying to use a class that you wrote, can see how it’s done in the tests. The test readability will be equally important here too.

So, what about you, how do you improve your test’s quality? How do you improve your test’s readability?

We, Rails developers, have always worried about improving the performance of our test suites. Today I would like to share three quick tips we employ in our projects that can drastically speed up your test suite.

1. Reduce Devise.stretches

Add the following to your spec/test helper:

Devise.stretches = 1

Explanation: Devise uses bcrypt-ruby by default to encrypt your password. Bcrypt is one of the best choices for such job because, different from other hash libraries like MD5, SHA1, SHA2, it was designed to be slow. So if someone steals your database it will take a long time for them to crack each password in it.

That said, it is expected that Devise will also be slow during tests as many tests are generating and comparing passwords. For this reason, a very easy way to improve your test suite performance is to reduce the value in Devise.stretches, which represents the cost taken while generating a password with bcrypt. This will make your passwords less secure, but that is ok as long as it applies only to the test environment.

Latest Devise versions already set stretches to one on test environments in your initializer, but if you have an older application, this will yield a nice improvement!

2. Increase your log level

Add the following to your spec/test helper:

Rails.logger.level = 4

Explanation: Rails by default logs everything that is happening in your test environment to “log/test.log”. By increasing the logger level, you will be able to reduce the IO during your tests. The only downside of this approach is that, if a test is failing, you won’t have anything logged. In such cases, just comment the configuration option above and run your tests again.

3. Use shared connection with transactional fixtures

If you are using Capybara for javascript tests and Active Record, add the lines below to your spec/test helper and be sure you are running with transactional fixtures equals to true:

class ActiveRecord::Base
  mattr_accessor :shared_connection
  @@shared_connection = nil
 
  def self.connection
    @@shared_connection || retrieve_connection
  end
end
 
# Forces all threads to share the same connection. This works on
# Capybara because it starts the web server in a thread.
ActiveRecord::Base.shared_connection = ActiveRecord::Base.connection

Explanation: A long time ago, when Rails was still in 1.x branch, a new configuration option called use_transactional_fixtures was added to Rails. This feature is very simple: before each test Active Record will issue a begin transaction statement and issue a rollback after the test is executed. This is awesome because Active Record will ensure that no data will be left in our database by simply using transactions, which is really, really fast.

However, this approach may not work in all cases. Active Record connection pool works by creating a new connection to the database for each thread. And, by default, database connections do not share transactions state. This means that, if you create data inside a transaction in a thread (which has its own database connection), another thread cannot see the data created at all! This is usually not an issue, unless if you are using Capybara with Javascript tests.

When using Capybara with javascript tests, Capybara starts your Rails application inside a thread so the underlying browser (Selenium, Webkit, Celerity, etc) can access it. Since the test suite and the server are running in different threads, if our test suite is running inside a transaction, all the data created inside the test suite will no longer be available in the server. Alternatively, since the server is outside the transaction, data created by the server won’t be cleaned up. For this reason, many people turn off use_transactional_fixtures and use Database Cleaner to clean up their database after each test. However, this affects your test suite performance badly.

The patch above, however, provides a very simple solution to both problems. It forces Active Record to share the same connection between all threads. This is not a problem in your test suite because when the test thread is running, there is no request in the server thread. When the server thread is running, the test thread is waiting for a response from the server. So it is unlikely that both will use the connection at the same time. Therefore, with the patch above, you no longer need to use Database Cleaner (unless you are using another database like Mongo) and, more importantly, you must turn use_transactional_fixtures back to true, which will create a transaction wrapping both your test and server data, providing a great boost in your test suite performance.

Finally, if any part of your code is using threads to access the database and you need to test it, you can just set ActiveRecord::Base.shared_connection = nil during that specific test and everything should work great!

Conclusion

That’s it! I hope you have enjoyed those tips and, if they helped you boost your test suite performance, please let us know in the comments the time your test suite took to run before and after those changes! Also, please share any tips you may have as well!

A while ago we were working on an application that had an entire version specially created for mobiles, such as the iPhone. This specific application was entirely tested with Capybara, Steak and Selenium Webdriver. Although the test suite wasn’t the fastest one in the world, the web application was very well tested, and to guarantee that we would also be testing the mobile version, we would have to simulate an iPhone user agent accessing the application.

But wait, you might be thinking that we are not able to change browser headers while dealing with Selenium. Capybara has a nice API to define new drivers and Selenium allows us to define different profiles with custom configurations for each driver. Lets see how we can put all this together to handle that:

Capybara.register_driver :iphone do |app|
  require 'selenium/webdriver'
  profile = Selenium::WebDriver::Firefox::Profile.new
  profile['general.useragent.override'] = "iPhone"
 
  Capybara::Driver::Selenium.new(app, :profile => profile)
end

Yup, it’s that simple =). We are creating a new driver for Capybara called :iphone, that will use Selenium with Firefox, but with a different profile, overriding the user agent string. This way you can pretend to your application that you are accessing through a “real” iPhone, by giving the “iPhone” string as user agent. You could also configure an :android driver, for instance, by simply changing the user agent string.

So now, how do we make use of that new driver in our specs? Here comes a simple example:

scenario 'access phone information using a modal box', :driver => :iphone do
  visit root_path
 
  page.should have_no_css "#fancybox-wrap"
  page.should have_no_content "0800 123456"
 
  within("header") { click_link "Telefones úteis" }
 
  within("#fancybox-wrap") do
    page.should have_content "0800 123456"
  end
end

We are just passing the :driver => :iphone option to our scenario. Remember that the latest Capybara versions use RSpec metadata options and will apply the :driver option automatically, changing the current driver to our registered :iphone in this case. For more info please refer to Capybara’s README.

You are now able to configure different user agents based on your application requirements, and test it in a full stack way. How about you, do you have any quick hint on how to test different user agents using another driver? Let us know in the comments :)

Updates (04/04/2014)

We were told that if you’re using Selenium Webdriver version 2.41.0, the code above will raise an exception. In order to fix that problem, you just need to replace Capybara::Driver::Selenium.new by Capybara::Selenium::Driver.new. Thanks Michael Joseph for suggesting that update.

Here at PlataformaTec we like to use Capybara for acceptance tests. Recently we have discovered the custom selectors feature in Capybara and we would like to share with you how that feature helped us to improve our tests.

Sometimes we need to implement features that involves showing some ordered items to the user, like a ranking feature. The HTML for a feature like that could be:

<ol id="overall-ranking">
  <% @top_users.each do |user| %>
    <li><%= user.name %></li>
  <% end %>
</ol>

The acceptance tests for this ranking could be written as follows:

scenario "The user can see an overall ranking" do
  Factory(:user, :name => "Hugo",  :score => 5000)
  Factory(:user, :name => "Ozaki", :score => 3000)
  Factory(:user, :name => "João",  :score => 4000)
 
  visit overall_ranking_path
 
  within("#overall-ranking") do
    find(:xpath, './/li[1]').text.should match("Hugo")
    find(:xpath, './/li[2]').text.should match("João")
    find(:xpath, './/li[3]').text.should match("Ozaki")
  end
end

Generally, I don’t like to see those XPath selectors inside my acceptance tests. And sometimes it can get really ugly! So, in order to improve our tests, we can create a custom selector with Capybara as follows:

# spec/spec_helper.rb
 
RSpec.configure do |config|
 
  Capybara.add_selector(:li) do
    xpath { |num| ".//li[#{num}]" }
  end
 
end

After that, we can refactor our test as shown below:

scenario "The user can see an overall ranking" do
  Factory(:user, :name => "Hugo",  :score => 5000)
  Factory(:user, :name => "Ozaki", :score => 3000)
  Factory(:user, :name => "João",  :score => 4000)
 
  visit overall_ranking_path
 
  within("#overall-ranking") do
    find(:li, 1).text.should match("Hugo")
    find(:li, 2).text.should match("João")
    find(:li, 3).text.should match("Ozaki")
  end
end

If you wanna know more about Capybara’s custom selectors, check its README.

And you? Any tips about using Capybara or improving your acceptance/integration tests?

One of the beauties in the Open Source world is the possibility of reading other people source code and learn new things. However, lately I found out that not only the library code, but the test suite of several open source projects are full lessons for us.

In this post, I want to tell you which are the three test suites that I admire the most and why.

Integration award: Railties

Rails 3 has several improvements and not all of them may be visible to the application developer. One of the hidden unicorns is Railties test suite. As Yehuda stated in a blog post during the refactoring of version 2.3 to 3.0:

“Although the Rails initializer tests covered a fair amount of area, successfully getting the tests to pass did not guarantee that Rails booted.”

This happened because, in order to have fast tests, Rails 2.3 suite stubbed and mocked a significant part of the booting process. The new test suite is able to create a new application using the application generator, change configuration options, add plugins and engines, boot it and even make HTTP requests using Rack::Test.

For instance, take a look at this test which ensures that app/metals inside plugins are successfully added to the application middleware stack:

def test_plugin_metals_added_to_middleware_stack
  @plugin.write 'app/metal/foo_metal.rb', <<-RUBY
    class FooMetal
      def self.call(env)
        [200, { "Content-Type" => "text/html"}, ["FooMetal"]]
      end
    end
  RUBY
 
  boot_rails
  require 'rack/test'
  extend Rack::Test::Methods
 
  get "/not/slash"
  assert_equal 200, last_response.status
  assert_equal "FooMetal", last_response.body
end

The most important lesson here is: whenever mocking or stubbing in our tests, we still need to add tests without the mocks and stubs to ensure all API contracts are respected.

Readability award: Capybara

Capybara is a tool to aid writing acceptance tests for web applications. Capybara can use several drivers to interact with a web application, as Selenium, Celerity or even Rack::Test. Each driver needs a different setup and has different features. For instance, both Selenium and Celerity can handle javascript, but not Rack::Test.

As you may imagine, all these different drivers can make a test suite become a real spaghetti. However, Jonas Nicklas was able to transform a potential problem into a very elegant and readable test suite with Rspec help. Here is, for instance, the tests for selenium:

describe Capybara::Driver::Selenium do
  before do
    @driver = Capybara::Driver::Selenium.new(TestApp)
  end
 
  it_should_behave_like "driver"
  it_should_behave_like "driver with javascript support"
end

Each behavior group above (“driver” and “driver with javascript support”) is inside Capybara library allowing everyone to develop its own extensions using a shared suite. For instance, if a driver has javascript support, it means the following tests should pass:

shared_examples_for "driver with javascript support" do
  before { @driver.visit('/with_js') }
 
  describe '#find' do
    it "should find dynamically changed nodes" do
      @driver.find('//p').first.text.should == 'I changed it'
    end
  end
 
  describe '#drag_to' do
    it "should drag and drop an object" do
      draggable = @driver.find('//div[@id="drag"]').first
      droppable = @driver.find('//div[@id="drop"]').first
      draggable.drag_to(droppable)
      @driver.find('//div[contains(., "Dropped!")]').should_not be_nil
    end
  end
 
  describe "#evaluate_script" do
    it "should return the value of the executed script" do
      @driver.evaluate_script('1+1').should == 2
    end
  end
end

Capybara test suite is one of the best examples of using tests as documentation. By skimming the test suite you can easily know which features are supported by each driver! Sweet, isn’t it?

Friendliness award: I18n

When you are a big Open Source project, your test suite needs to be easy to run in order to new developers can create patches without hassle. The I18n library for Ruby definitely meets the big Open Source project requirement since it’s widely used and provides several extensions.

However, some of these extensions depends on ActiveRecord, some in ruby2ruby, others in ruby-cldr… and soon it will even support a few Key-Value stores, as Tokyo and Redis. Due to all these dependencies, you would probably imagine that running I18n test suite would require several trials and a lot of configuration before it finally works, right?

WRONG! If you don’t have ActiveRecord, I18n will say: “hey, you don’t have ActiveRecord” but still run the part of test suite that does not depend on it. So if a developer wants to fix or add something trivial, he doesn’t need to worry with installing all sorts of dependencies.

Besides, as mentioned a couple months ago, the I18n library allows you to create several combinations of backends. In other words, the I18n test suite needs to ensure that all these different combinations work as expected.

This problem is quite similar to the one in Capybara which needs to test different drivers. However, I18n uses Test::Unit thus it cannot use shared examples groups as in Rspec. So how were I18n developers able to solve this issue? Using Ruby modules!

Here are the tests for the upcoming KeyValue backend:

require 'test_helper'
require 'api'
 
class I18nKeyValueApiTest < Test::Unit::TestCase
  include Tests::Api::Basics
  include Tests::Api::Defaults
  include Tests::Api::Interpolation
  include Tests::Api::Link
  include Tests::Api::Lookup
  include Tests::Api::Pluralization
  # include Tests::Api::Procs
  include Tests::Api::Localization::Date
  include Tests::Api::Localization::DateTime
  include Tests::Api::Localization::Time
  # include Tests::Api::Localization::Procs
 
  STORE = Rufus::Tokyo::Cabinet.new('*')
 
  def setup
    I18n.backend = I18n::Backend::KeyValue.new(STORE)
    super
  end
 
  test "make sure we use the KeyValue backend" do
    assert_equal I18n::Backend::KeyValue, I18n.backend.class
  end
end

Each included module above adds a series of tests to the backend. Since key-value backends cannot store procs, we don’t include any test related to procs.

Wrapping up

These three are my favorite test suites and also part of my favorite open source projects!

We’ve adopted Capybara as the official testing tool at PlataformaTec for some time already and I18n is one of the subjects of my upcoming book about Rails 3. In one specific chapter, we will build a tool that stores I18n translations into TokyoCabinet, which allows us to create and update translations through a web interface, similarly to ActiveRecord. The only difference is that TokyoCabinet is waaaay faster.

Finally, the fact you can mimic several of Rspec features using simple Ruby (like Capybara using shared example groups and I18n simply using modules) will be part of my talk in Euruko 2010 entitled DSL or NoDSL: The power is in the middle. The talk will show cases where DSLs mimics much of the behavior provided by Ruby and discuss what we are winning and/or losing in such cases.

Keep following us and, until the next blog post is out, we would love to hear in the comments which are your favorite test suites!