So, you already did the right choice of using Delayed Job for your background processing, great! But, how are you going to be certain that your background processing will still be happening while you are sleeping? And if your Delayed Job process goes down, are you going to wake up in the dawn e restart it manually? I wouldn’t do that, I really appreciate my sleep. So, what’s the solution?

As rubyists and railers, we already know there are solutions, like God, that do this job for us. However, there are another solutions, like Bluepill. Bluepill is a process monitoring tool like God, but, unlike God, it doesn’t have memory leak, according to its authors.

Well, as I don’t want to wake up in the dawn to restart my Delayed Job process, and neither I want to restart my God process because of memory leaking, I decided to use Bluepill. But, how I use Bluepill to monitor my Delayed Job process?

In order to configure Bluepill to monitor Delayed Job, and use Capistrano to automate some tasks, we have basically 4 steps:

  1. Install and configure Delayed Job
  2. Install and configure Bluepill
  3. Write a Capistrano Recipe for Bluepill
  4. Set some stuff in /etc/sudoers

Let’s take a look at each step.

Installing and configuring Delayed Job

Installing and configuring Delayed Job is super simple, just read the project’s README, which is very clear. I also recommend watching the RailsCast episode about Delayed Job, there is some good information there, like using the Delayed Job CollectiveIdea’s fork, instead of the original repo’s version.

Installing and configuring Bluepill

There’s no secret in installing Bluepill too, you just need to read the project’s README and follow its steps.

In the confiuration part, you can see all the options in the README too. In my case, the configuration file is at RAILS_ROOT/config/production.pill.

Bluepill.application("my_app") do |app|
  app.process("delayed_job") do |process|
    process.working_dir = "/home/deploy/my_app/current"
 
    process.start_grace_time    = 10.seconds
    process.stop_grace_time     = 10.seconds
    process.restart_grace_time  = 10.seconds
 
    process.start_command = "ruby script/delayed_job -e production start"
    process.stop_command  = "ruby script/delayed_job -e production stop"
 
    process.pid_file = "/home/deploy/my_app/shared/pids/delayed_job.pid"
    process.uid = "deploy"
    process.gid = "deploy"
  end
end

However, I had a little problem between the interaction of Bluepill and Delayed Job. Delayed Job is not interpreting very well the -e production flag. You can see more details about that in an issue I opened.

The first solution I thought was to use RAILS_ENV=production ruby script/delayed_job start, however, for some reason that I don’t know exactly, it didn’t work.

So, the solution I came up with was to modify the file in RAILS_ROOT/script/delayed_job to the following one:

1
2
3
4
5
6
7
8
9
#!/usr/bin/env ruby
 
# TODO improve the line of code below
# The line below is just a hack while we wait the delayed job guys answer the following issue
# http://github.com/collectiveidea/delayed_job/issues#issue/38
ENV['RAILS_ENV'] ||= "production"
require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
require 'delayed/command'
Delayed::Command.new(ARGV).daemonize

As the -e production was not being interpreted properly by Delayed Job, I’m setting manually the RAILS_ENV to production (line 6), therefore I’m supposing that you are using Bluepill to monitor processes that are in a production environment.

Capistrano recipe for Bluepill

In order to automate the start and stop Bluepill’s tasks, I wrote the following capistrano’s recipe:

# deploy credentials
set :user, "deploy"
set :deploy_to, "/home/deploy/#{application}"
set :use_sudo, false
set :rails_env, "production"
 
# Bluepill related tasks
after "deploy:update", "bluepill:quit", "bluepill:start"
namespace :bluepill do
  desc "Stop processes that bluepill is monitoring and quit bluepill"
  task :quit, :roles => [:app] do
    sudo "bluepill stop"
    sudo "bluepill quit"
  end
 
  desc "Load bluepill configuration and start it"
  task :start, :roles => [:app] do
    sudo "bluepill load /home/deploy/my_app/current/config/production.pill"
  end
 
  desc "Prints bluepills monitored processes statuses"
  task :status, :roles => [:app] do
    sudo "bluepill status"
  end
end

Note that instead of using the run capistrano’s method, I’m using the sudo method. I’m doing this because the bluepill command must be run as root. And that, takes us to the next topic.

/etc/sudoers

Since I need to run the bluepill command as root, should I change from set :user, "deploy" to set :user, "root"? I think it’s not a good idea, we don’t like to give root access to anything, even to deployment. So, what should I do? It’s simple, you just need to edit your sudoers file.

In order to do this, you need to use the visudo command to open and edit the /etc/sudoers file. Once with the file opened, just add the following line to the end of the file:

deploy ALL=(ALL) NOPASSWD: /usr/local/bin/bluepill

Now, you’re done, the deploy user already can do sudo bluepill without giving any password. Problem solved without opening security holes.

After these 4 steps, you’re ready to sleep at night without worrying about your Delayed Job process. And, if you want to know the status of the monitored processes by Buepill, you just need to run the following capistrano task in your local machine:

cap bluepill:status

And you, what are your solutions to sleep in peace?

Update: if are having trouble with restarting Bluepill with Capistrano, take a look at this.

Tags: , , ,

This entry was posted on Wednesday, February 10th, 2010 at 3:47 pm and is filed under English. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

  • Frederic

    “The first solution I thought was to use RAILS_ENV=production ruby script/delayed_job start, however, for some reason that I don’t know exactly, it didn’t work.”

    I’m pretty sure (my quick dig of bluepill/daemonize.rb/ruby source code show me so) that the string you give to the start_command/stop_command is ultimately ran using an sh command (on Unix & alike, cmd on windows), however the construct “VAR=val command” is specific to the bash shell.

    My guess is that written “RAILS_ENV=production; ruby script/delayed_job start” (with the semicolon) your start_command would have worked (not sure for Windows, but who cares ;-)

  • Frederic

    “The first solution I thought was to use RAILS_ENV=production ruby script/delayed_job start, however, for some reason that I don’t know exactly, it didn’t work.”

    I’m pretty sure (my quick dig of bluepill/daemonize.rb/ruby source code show me so) that the string you give to the start_command/stop_command is ultimately ran using an sh command (on Unix & alike, cmd on windows), however the construct “VAR=val command” is specific to the bash shell.

    My guess is that written “RAILS_ENV=production; ruby script/delayed_job start” (with the semicolon) your start_command would have worked (not sure for Windows, but who cares ;-)

  • http://scharfie.com Chris Scharf

    Maybe this would work?

    /usr/bin/env RAILS_ENV=production ruby script/delayed_job start

  • http://scharfie.com Chris Scharf

    Maybe this would work?

    /usr/bin/env RAILS_ENV=production ruby script/delayed_job start

  • Hugo Baraúna

    @frederic, @chris

    It seem that the solution to that problem is the one that Chris said. You can view more about that in this issue.

  • Hugo Baraúna

    @frederic, @chris

    It seem that the solution to that problem is the one that Chris said. You can view more about that in this issue.

  • Frederic

    My solution was missing an “export RAILS_ENV;” to work properly (but the rest of the comment is still correct: “sh -c” (hidden in a Kernel.exec(“cmd-string”) call) does not accept bash specific syntax.)

    Chris’ solution is neater and shorter, but it can even be made shorter by not prefixing env with its path ;-).

    Bash syntax “VAR=val cmd” is just a shortcut for “env VAR=val cmd” when you do not need to use the -i option of the env command to have it “to completly ignore the environment it inherits” — env(1).

  • Frederic

    My solution was missing an “export RAILS_ENV;” to work properly (but the rest of the comment is still correct: “sh -c” (hidden in a Kernel.exec(“cmd-string”) call) does not accept bash specific syntax.)

    Chris’ solution is neater and shorter, but it can even be made shorter by not prefixing env with its path ;-).

    Bash syntax “VAR=val cmd” is just a shortcut for “env VAR=val cmd” when you do not need to use the -i option of the env command to have it “to completly ignore the environment it inherits” — env(1).

  • Pingback: Caffeine Driven Development » Blog Archive » L33t Links #79

  • http://jonkinney.com Jon Kinney

    Why do we need to stop and re-start bluepill after a deploy? Aside from it not working properly for me on OpenSUSE 11, I would think that an init script that starts when the server starts would be the way to go, no?

  • http://jonkinney.com Jon Kinney

    Why do we need to stop and re-start bluepill after a deploy? Aside from it not working properly for me on OpenSUSE 11, I would think that an init script that starts when the server starts would be the way to go, no?

  • Pingback: Weekend Links « a work on process