Erik DeBill has put two interesting benchmarks on his blog. The first one compares the performance of different Ruby implementations in Rails development mode while the second compares their performance in Rails boot time. If you haven’t read them yet, please do it now.

Benchmarking code is an important practice, but it can be misleading if you fail to understand the root causes that lead to the different results.

Performance in development mode

In the first blog post, it is guessed that the root case for having slow requests in development is because Rails eager loads all models and controllers for each request:

Now, what I’d really like is a way to avoid recompiling everything every time. If I could have Rails recompile just the model or controller I’m working on and skip all the others, that’d be grand. I’ve taken a couple stabs at it, but I haven’t succeeded yet.

This is wrong! Rails, in development, only loads the model and the controller you are using in that specific request. This is very easy to verify if you create a new application, scaffold two resources and add a puts self.name in their class definition. If you access one controller, it will only load the model explicitly referenced in that controller. Even the model associations try to be lazy in that aspect, always loading the minimum it can.

So you may ask, why Rails is getting so slow after adding more scaffolds?

It happens because Rails 3.0 includes all helpers by default in your ApplicationController. So at the beginning of each request, Rails needs to load all helpers. Loading a helper in development mode is slow because ActiveSupport::Dependencies needs to track which dependencies were added when a file is loaded. This tracking basically happens by checking which constants were added invoking Object.constants before and after the file was loaded. Tracking these constants take more than 50% of the time in the request, mainly because invoking Object.constants is slow.

In other words, the main reason for an implementation to perform better in the benchmarks showed in the blog post is if it can calculate Object.constants faster. Those results do not mean at all that an implementation is more suitable than other for Rails development. In order to have real results, we would need a real application that is not made of 1000 scaffold (or, in this case, 1000 helpers).

In any case, if the root cause is in loading all helpers, how can we make it better? There are a few things:

1) Obviously, the problem can be fixed by having less helper files. Since Rails scaffold automatically generates helper files, it is common that applications have a bunch of empty helpers. Get rid of them. If you prefer you can even turn off the automatic generation of helpers in scaffold by adding the following to your application configuration:

config.generators.helper = false

2) If you simply don’t want to include all helpers, there is a method called clear_helpers that you could invoke in your ApplicationController. This method won’t fix the problem because it is invoked too late, after all the helpers were already loaded. So you get the feature, but not the performance improvement.

3) Rails master (upcoming Rails 3.1) has a configuration option that allows you to effectively turn these helpers off getting both the feature and the performance improvement:

config.action_controller.include_all_helpers = false

Boot performance

The second blog post shows how Rails boot time performs in different implementations. Since it was not made explicit in which environment those benchmarks were executed, I will assume it happened on development.

At the end of the second blog post, it tries to associate the performance of booting Rails in development with the amount of code inside the app. However, when you boot an application in development, no model, controller or helper is loaded at all unless you explicitly access them in an initializer or in your routes file. Once again, you can check that by adding some puts to your classes.

So, you may ask one more time, what makes booting up so slow?

Rails 3 has a new router that can match paths very fast, but in order to do so, it needs to compile each route into a regular expression and that takes some time (although it could probably be made faster). And it is exactly the routes compilation that is slow on boot time. We can easily reproduce it by adding the following to our router:

Foo::Application.routes.draw do
  1000.times do |index|
    resources :"posts#{index}"
  end
end

This took 55 seconds on my machine using REE which is quite close to the value that he showed on his benchmark.

Again, benchmarking code is important, but more important is to correctly interpret the results. In his example, it is likely that most of Rails booting time is spent on compiling the routes and the benchmark just shows how good different Ruby implementations are in handling all these regular expressions.

Wrapping up

Much more interesting benchmarks for Rails boot time would actually be performed in production environment, which actually has to load all the code inside the app folder and compile the routes file. Regardless, developers starting new applications should always be skeptical about choosing a Ruby implementation based on other application’s benchmarks.

When starting out a new application, any Ruby implementation should suit you just fine unless you have a stronger constraint (like Java integration). Once your application starts to grow and you want to evaluate how well it performs in different implementations, you should do your own benchmarks and see how it goes. In any case, don’t jump into conclusions. If you need to investigate deeper, each implementation has its own sets of benchmarking and profiling tools that may help you really understand what is actually slow and how to improve it.

I also want to thank ruby-prof authors and maintainers, for such an amazing tool, and Yehuda Katz, who helped me profile a demo Rails application in order to write this detailed response.

And you? Have you done benchmarks in your applications and found any interesting data you would like to share?