Everyone already heard about scalability at least once. Everyone already heard about memcached as well. What not everyone might heard is the dog-pile effect and how to avoid it. But before we start, let’s take a look on how to use Rails with memcached.

Rails + Memcached =

First, if you never used memcached with rails or never read/heard a lot about scalability, I recommend checking out Scaling Rails episodes done by Gregg Pollack, in special the episode about memcached.

Assuming that you have your memcached installed and want to use it on your application, you just need to add the following to your configuration files (for example production.rb):

config.cache_store = :mem_cache_store

By default, Rails will search for a memcached process running on localhost:11211.

But wait, why would I want to use memcached? Well, imagine that your application has a page where a slow query is executed against the database to generate a ranking of blog posts based on the author’s influence and this query takes on average 5 seconds. In this case, everytime an user access this page, the query will be executed and your application will end up having a very high response time.

Since you don’t want the user to wait 5 seconds everytime he wants to see the ranking, what do you do? You store the query results inside memcached. Once your query result is cached, your app users do not have to wait for those damn 5 seconds anymore!

What is the dog-pile effect?

Nice, we start to cache our query results, our application is responsive and we can finally sleep at night, right?

That depends. Let’s suppose we are expiring the cache based on a time interval, for example 5 minutes. Let’s see how it will work in two scenarios:

1 user accessing the page after the cache was expired:

In this first case, when the user access the page after the cache was expired, the query will be executed again. After 5 seconds the user will be able to see the ranking, your server worked a little and your application is still working.

N users accessing the page after the cache was expired:

Imagine that in a certain hour, this page on your application receives 4 requests per second on average. In this case, between the first request and the query results being returned, 5 seconds will pass and something around 20 requests will hit your server. The problem is, all those 20 requests will miss the cache and your application will try to execute the query in all of them, consuming a lot of CPU and memory resources. This is the dog-pile effect.

Depending on how many requests hit your server and the amount of resources needed to process the query, the dog-pile effect can bring your application down. Holy cow!

Luckily, there are a few solutions to handle this effect. Let’s take a look at one of them.

Dog pile effect working on your application!

How to avoid the dog-pile effect?

The dog-pile effect is triggered because we allowed more than one request to execute the expensive query. So, what if we isolate this operation to just the first request and let the next requests use the old cache until the new one is available? Looks like a good idea, so let’s code it!

Since Rails 2.1, we have an API to access the cache, which is defined by an abstract class called ActiveSupport::Cache::Store. You can read more about it in this post or in this excellent railscast episode.

The code below simply implements a new and smarter memcached store on top of the already existing MemCacheStore:

module ActiveSupport
  module Cache
    class SmartMemCacheStore  expires_at && !exist?("lock_#{key}")
          orig_write("lock_#{key}", true, :expires_in => lock_expires_in)
          return nil
        else
          data
        end
      end

      def write(key, value, options = nil)
        expires_delta = options.delete[:expires_delta] if !options.nil?
        expires_delta ||= 300

        expires_at = Time.now + expires_delta
        package = [value, expires_at]
        orig_write(key, package, options)
        delete("lock_#{key}")
      end
    end
  end
end

The code above is mainly doing:

1. Suppose that your query is already cached;
2. In the first five minutes, all requests will hit the cache;
3. In the next minutes, the first request will notice that the cache is stale (line 17) and will create a lock so only it will calculate the new cache;
4. In the next 5 seconds, the new query is calculated and all requests, instead of missing the cache, will access the old cache and return it to the client (lines 17 and 21)k;
5. When the query result is returned, it will overwrite the old cache with the new value and remove the lock (lines 31 and 32);
6. From now on, all new requests in the next five minutes will access the fresh cache and return it (lines 17 and 21).

Fallbacks and a few things to keep in mind

First, is not recommend to set the :expires_in value in your cache:

Rails.cache.write('my_key', 'my_value', :expires_in => 300)

With the solution proposed above, you just need to set :expires_delta. This is due to the fact that our application will now be responsible to expire the cache and not memcached.

Rails.cache.write('my_key', 'my_value', :expires_delta => 300)

However, there are a few cases where memcached can eventually expire the cache. When you initialize memcached, it allocates by default 64MB in memory. If eventually those 64MB are filled, what will memcached do when you try to save a new object? It uses the LRU algorithm and deletes the less accessed object in memory.

In such cases, where memcached removes a cache on its own, the dog pile effect can appear again. Suppose that the ranking is not accessed for quite some time and the cached ranking is discarded due to LRU. If suddenly a lot of people access the page in the five initial seconds where the query is being calculated, requests will accumulate and once again the dog-pile effect can bring your application down.

It’s important to have this scenario in mind when you are sizing your memcached, mainly on how many memory will be allocated.

Now I can handle the dog-pile effect and sleep again!

Summarizing, when your are using a cache strategy, you will probably need to expire your cache. In this process, the dog-pile effect can appear and haunt you down. Now you have one (more) tool to solve it.

You just need to add the SmartMemCacheStore code above to your application (for example in lib/), set your production.rb (or any appropriated environment) to use the :smart_mem_cache_store. If you use Rails default API to access the cache (Rails.cache.read, Rails.cache.write) and designed well your memcached structure, you will be protected from the dog-pile effect.

A real dog-pile! =p