CI JOE, can you please watch my windows?

Suppose you park your car in a neighborhood with a lot of occurrences of vandalism and crimes. If you return two weeks later, there is a good chance that you will find the car in the same state that it were when you left it there.  Now imagine if you parked the same car but with a broken window. There is a good chance that you will see an engineless car, without tires or many other parts. Of course, we did not conduct this social experiment, but we know about a theory called Broken windows theory that says if we have something intact, it has a tendency to remain intact. But if we have something broken, there is a tendency that it will be even more damaged after some time.

But how does it apply to software development? Suppose you have some developers working on a project with about a thousand tests covering all the code and all of them are passing. If a developer implements a feature and it breaks some tests, it seems reasonable that he will make an effort to make the necessary modifications so all tests pass, or maybe he’ll remove them if they don’t make sense anymore. Also, it is expected that he will write new tests to cover the new feature.

Now suppose a different scenario: the project has a thousand tests and a hundred of them are not passing. Even worse, the code coverage is about 70%. As the developer writes the new feature, there is a tendency that he may not create tests and not fix the broken tests that this implementation caused. There is an analogy with the broken windows and failing tests: if you allow a failing test to stay there, there is a good chance that your team may not work on correcting tests and the number of failing tests may increase during time, leading to a poor quality software.

OK, you have heard about the problem, but how can we solve that? A first idea is to hire a monkey that keeps an eye in the version control, and everytime there is an update there, it presses a key that runs all the tests. The monkey is actually very smart and it will scream if it sees red F’s in the screen. And every time the monkey screams, you know that somebody commited a broken test and you also know that every commit will make the monkey scream. So it’s time to fix the broken test. But you may have problems with Greenpeace doing that. So, it is a good idea to have something that does not depend on living animals. So maybe a robot? Yes, this is what we call Continuous Integration tools.

Continuous integration is what we call the process where developers integrate their codes continuously. Continuous Integration tool is an application that executes the build every time it is updated in the version control. This build can watch for compilation/interpretations errors, not passing tests, performance issues or anything you may think reasonable to watch.

There are lots of continuos integration solutions suitable for Ruby (and consequently Ruby on Rails). Some java-based (Hudson) or some in Ruby (Cruise Control rb or Signal). We tried using Signal at first, but we have some problems related to the gem environment (we could not make it change the environment to the project gemset instead of the Signal gemset environment). So we tried the minimalistic CI JOE.

CI JOE is a CI server created by the Githubber Chris Wanstrath. It is sinatra based and does not need even a database to run: all of its configurations are recorded in the .git/config file and it uses git hooks to run the desired actions. There are only two requirements: your project must use git for version control, and the build should return a zero shell status for success and anything different than zero for not successful builds. You can use it with any programming language or framework. I strongly encourage you to configure to pull the build trigger every time you do a git push (you can do that simply doing a POST to a URL).

The most awesome thing about CI JOE is that it is dead simple to configure it to do almost anything. You can configure Campfire notifications, e-mail deliveries or an alarming buzzing when it fails. The only requirement is that a shell command exists (if it requires a sequence a of shell commands, you can join them using ‘;’). A great idea is to integrate electric devices like leds, buzzers or even a semaphore.

The project comprehensive README has a lot of examples, including how to protect it with HTTP basic authentication or how to configure a queue of build requests. For our projects, we usually customize it by setting a git after-reset hook to execute “bundle install” (it always execute this git hook before the build starts) and then we modify the html template to include the project logo. Next, CI is ready to rock and roll!

And you? What do you think about CI JOE? Are you customizing it in any way? Are you using another CI solution for your Ruby on Rails projects?

7 responses to “CI JOE, can you please watch my windows?”

  1. Francisco J. says:

    Wow, great analogy. I really understand much better now the need for a CI solution. However I always ask myself the same question. Is it really needed in a very small team? lets say… 2 people?

    In my little experience when you TDD or BDD in a small team, the other developers they use to work as the “CI monkey”, telling you when you’ve broken something. I’m not saying that it’s not needed… but what I mean is, does it really worth the effort to set up a CI server when you do something small with a really small team? I’m really curious about what others think about it 🙂

  2. I love cijoe for it’s simplicity. The fact that it doesn’t use a database makes it really easy to install.

    We actually run all of our PHP tests using cijoe via a github hook. cijoe then posts the results to our Campfire room. Nothing fancy, but it gives our team immediate feedback on our test suite and has caught quite a few issues before they were pushed to production.

  3. drnic says:

    For all your projects do you run the tests in the same environment as the CI master or us VMs or slaves to have isolated, production-similar environments for running tests?

    Bonus question – how do you protect against destroying your production databases? 🙂

  4. drnic says:

    Hudson is trivial to setup. Wonderful CI platform.

  5. Marcin Kulik says:

    It’s always worth setting up. Even for such small teams. From my experience I know that people sometimes forget running tests and CI server catches all these situations. And if setting up CI Joe involves almost no effort you have no excuse to not try it 😉

  6. Well, I think a good protection against destroying production databases is to not have them configured in your version control. Only in a folder in the production environment and capistrano does a symbolic link on each deploy. I believe that we do not need to do that in a production-similar environment. We usually have an approval environment similar to production. Of course it is not fully tested as a CI environment, but we usually approve each feature with our clients. And we only do the deploy when tests pass, so we think that there is no need for an isolated VM for each CI server instance. And of course, since Bundler exists, configuring an Rails environment is dead simple, so we have to be less concerned about environment problems :-).

  7. We’ve been using TeamCity for a while. It comes with custom formatters for RSpec and Cucumber which allows it to display the number of test passed / failed / pending while running the tests. You get alerted as soon as one test fails. You can also see each test duration and all kind of metrics which helps… sometimes. 🙂

    However, TeamCity does not provide autobranching of git repository. Since we create a topic branch for every feature / bug / refactoring we do, we want each branch to be green before we code review them.

    The only CI server with auto-branching we found was Integrity. It’s pretty much like CI Joe. It’s very simple, it’s also quite buggy but we really missed all the metrics (duration / code coverage / fog results…)

    We decided to make a small sinatra app which would copy projects on a CI server for every branch we push too. TeamCity does not provide any REST API so we moved to Hudson. It’s ugly, quite complex, but very easy to setup, damn easy to setup agents (Hudson would connect via ssh to an agent, bootstrap it and run builds) and we get all the metrics back!

    Those willing to autobranch on Hudson using Github Post Commit Hook can use this little sinatra app: