Mocks and explicit contracts

A couple days ago I expressed my thoughts regarding mocks on Twitter:

The blame is not on mocks though, they are actually a useful technique for testing. However our test tools often maks it very easy to abuse mocks and the goal of this post is to provide better guidelines on using them.

What are mocks?

The wikipedia definition is excellent: mocks are simulated entities that mimic the behavior of real entities in controlled ways. I will emphasize this later on but I always consider “mock” to be a noun, never a verb.

Case study: external APIs

Let’s see a common practical example: external APIs.

Imagine you want to consume the Twitter API in your web application and you are using something like Phoenix or Rails. At some point, a web request will come-in, which will be dispatched to a controller which will invoke the external API. Let’s imagining this is happening directly from the controller:

defmodule MyApp.MyController do
  def show(conn, %{"username" => username}) do
    # ...
    # ...

The code may work as expected but, when it comes to make the tests pass, a common practice is to just go ahead and mock (warning! mock as a verb!) the underlying HTTPClient used by MyApp.TwitterClient:

mock(HTTPClient, :get, to_return: %{..., "username" => "josevalim", ...})

You proceed to use the same technique in a couple other places and your unit and integration test suites pass. Time to move on?

Not so fast. The whole problem with mocking the HTTPClient is that you just coupled your application to that particular HTTPClient. For example, if we decide to use a new and faster HTTP client, a good part of your integration test suite will now fail because it all depends on mocking HTTPClient itself, even when the application behaviour is the same. In other words, the mechanics changed, the behaviour is the same, but your tests fail anyway. That’s a bad sign.

Furthermore, because mocks like the one above changes modules globally, they are particularly aggravating in Elixir as changing global values means you can no longer run that part of your test suite concurrently.

The solution

Instead of mocking the whole HTTPClient, we could replace the Twitter client (MyApp.TwitterClient) by something else during tests. Let’s explore how the solution would look like in Elixir.

In Elixir, all applications ship with configuration files and a mechanism to read them. Let’s use this mechanism to be able to configure the Twitter client for different environments. The controller code should now look like this:

defmodule MyApp.MyController do
  @twitter_api Application.get_env(:my_app, :twitter_api)

  def show(conn, %{"username" => username}) do
    # ...
    # ...

And now we can configure it per environment as:

# In config/dev.exs
config :my_app, :twitter_api, MyApp.Twitter.Sandbox

# In config/test.exs
config :my_app, :twitter_api, MyApp.Twitter.InMemory

# In config/prod.exs
config :my_app, :twitter_api, MyApp.Twitter.HTTPClient

This way we can choose the best strategy to retrieve data from Twitter per environment. The sandbox one is useful if Twitter provides some sort of sandbox for development. The HTTPClient is our previous implementation while the in memory avoids HTTP requests altogether, by simply loading and keeping data in memory. Its implementation could be defined in your test files and even look like:

defmodule MyApp.Twitter.InMemory do
  def get_username("josevalim") do
      username: "josevalim"

which is as clean and simple as you can get. At the end of the day, MyApp.Twitter.InMemory is a mock (mock as a noun, yay!), except you didn’t need any fancy library to define one! The dependency on HTTPClient is gone as well.

The need for explicit contracts

Because a mock is meant to replace a real entity, such replacement can only be effective if we have explicitly defined how the real entity should behave. Failing this, you will find yourself in the situation where the mock entity grows more and more complex with time, increasing the coupling between the components being tested, but you won’t ever notice it because the contract was never explicit.

Furthermore, we have already defined three implementations of the Twitter API, so we better make it all explicit. In Elixir we do so by defining a behaviour with callback functions:

defmodule MyApp.Twitter do
  @doc "..."
  @callback get_username(username :: String.t) :: %MyApp.Twitter.User{}
  @doc "..."
  @callback followers_for(username :: String.t) :: [%MyApp.Twitter.User{}]

Now add @behaviour MyApp.Twitter on top of every module that implements the behaviour and Elixir will help you provide the expected API.

It is interesting to note we rely on such behaviours all the time in Elixir: when you are using Plug, when talking to a repository in Ecto, when testing Phoenix channels, etc.

Testing the boundaries

Previously, because we didn’t have a explicit contract, our application boundaries looked like this:

[MyApp] -> [HTTP Client] -> [Twitter API]

That’s why changing the HTTPClient could break your integration tests. Now our app depends on a contract and only one implementation of such contract/behaviour rely on HTTP:

[MyApp] -> [MyApp.Twitter (contract)]
[MyApp.Twitter.HTTP (contract impl)] -> [HTTPClient] -> [Twitter API]

Our application tests are now isolated from both the HTTPClient and the Twitter API. However, how can we test MyApp.Twitter.HTTP (the second line)?

The challenge in testing a large system is exactly in finding the proper boundaries. If you define and isolate too many of them without integration tests, your tests can become brittle and you will end-up finding errors only in production. On the other hand, defining very few boundaries can lead to slow suites and hard to debug tests. Often times there is no correct answer and boundaries will vary depending on the team confidence and other external factors.

Personally, I would test MyApp.Twitter.HTTP by directly reaching the Twitter API. We would run those tests only when needed in development and configure them as necessary in our build system. The @tag system in ExUnit, Elixir’s test library, provides conveniences to help us with that:

defmodule MyApp.Twitter.HTTPTest do
  use ExUnit.Case, async: true

  # All tests will ping the twitter API
  @moduletag :twitter_api

  # Write your tests here

In your test helper, you want to exclude the Twitter API test by default:

ExUnit.configure exclude: [:twitter_api]

But you can still run the whole suite with the tests tagged :twitter_api if desired:

mix test --include twitter_api

Or run only the tagged tests:

mix test --only twitter_api

Although I prefer this approach, external conditions like rate limiting may make such solution inpractical. In such cases, we may actually need a mock HTTPClient, as long as we don’t break the properties we have previously defined:

  1. If you change your HTTP client, your application suite won’t break but only the tests for MyApp.Twitter.HTTP

  2. You won’t mock (warning! mock as a verb) your HTTP client. Instead, you will pass it as a dependency via configuration, similar to how we did for the Twitter API

  3. You still need a way to test, that is not in production, that your client works as expected

Alternatively, you may avoid mocking the HTTP client by running a dummy webserver that emulates the Twitter API. bypass is one of many projects that can help with that. Those are all options you should discuss with your team.

Other notes

I would like to finish this article by bringing up some common concerns and comments whenever the mock discussion comes up.

Making the code “testable”

Quoting from elixir-talk mailing list:

So the proposed solution is to change production code to be “testable” and making production code to call Application configuration for every function call? This doesn’t seem like a good option as it’s including a unnecessary overhead to make something “testable”.

I’d argue it is not about making the code “testable”, it is about improving the design of your code.

A test is a consumer of your API like any other code you write. One of the ideas behind TDD is that tests are code and no different from code. If you are saying “I don’t want to make my code testable”, you are saying “I don’t want to decouple some modules” or “I don’t want to think about the contract behind these components”.

Just to clarify, there is nothing wrong with “not wanting to decouple some modules”. For example, you don’t want to decouple every time you call the URI module. But if we are talking about something as complex as an external API or such, defining an explicit contract and making the contract implementation configurable is going to make your code wonders and make it easier to manage its complexity.

Finally, the overhead is also minimum. Application configuration in Elixir is stored in ETS tables which means they are directly read from memory.

Mocks as locals

Although we have used the application configuration for solving the external API issue, sometimes it is easier to just pass the dependency as argument. Imagine this example in Elixir where some function may perform heavy work which you want to isolate in tests:

defmodule MyModule do
  def my_function do
    # ...
    SomeDependency.heavy_work(arg1, arg2)
    # ...

You could remove the dependency by passing it as an argument, which can be done in multiple ways. If your dependency surface is tiny, an anonymous function will suffice:

defmodule MyModule do
  def my_function(heavy_work \\ &SomeDependency.heavy_work/2) do
    # ...
    heavy_work.(arg1, arg2)
    # ...

And in your test:

test "my function performs heavy work" do
  heavy_work = fn _, _ ->
    # Simulate heavy work by sending self() a message
    send self(), :heavy_work


  assert_received :heavy_work

Or define the contract, as explained in the previous section of this post, and pass a module in:

defmodule MyModule do
  def my_function(dependency \\ SomeDependency) do
    # ...
    dependency.heavy_work(arg1, arg2)
    # ...

Now in your test:

test "my function performs heavy work" do
  # Simulate heavy work by sending self() a message
  defmodule TestDependency do
    def heavy_work(_arg1, _arg2) do
      send self(), :heavy_work


  assert_received :heavy_work

Finally, you could also make the dependency a data structure and define the contract with a protocol.

In fact, passing the dependency as argument is much simpler and therefore should be preferred over relying on configuration files and Application.get_env/3. When not possible, the configuration system is a good fallback.

Mocks as nouns

Another way to think about mocks is to treat them as nouns. You shouldn’t mock an API (verb), instead you create a mock (noun) that implements a given API.

Most of the bad uses of mocks come when they are used as verbs. That’s because, when you use mock as a verb, you are changing something that already exists, and often those changes are global. For example, when we say we will mock the SomeDependency module:

mock(SomeDependency, :heavy_work, to_return: true)

When you use mock as a noun, you need to create something new, and by definition it cannot be the SomeDependency module because it already exists. So “mock” is not an action, it is something you pass around. I’ve found the noun-verb guideline to be very helpful when spotting bad use of mocks. Your mileage may vary.

Mock libraries

With all that said, should you discard your mock library?

It depends. If your library incentivates you to break the guidelines above, like using mocks to replace global entities (or using mock as a verb), changing static methods in OO or replacing modules in FP languages, you should definitely consider how the library is being used in your codebase and potentially discard it.

However there are mock libraries that does not promote any of the “anti-patterns” above and are mostly convenience to define “mock objects” or “mock modules” that you would pass to the testing system as argument and collect information how many times the mock was invoked, with which arguments and so on.

Summing up

Part of testing your system is to find the proper contracts and proper boundaries between the components. If you follow closely a guideline that mocks will be used only if you define a explicit contract, it will:

  1. protect you from overmocking as it will push you to define contracts only for the parts of your system that matters (as mentioned, you very likely don’t want to hide interactions with the URI or Enum modules behind a contract)

  2. make it easier to manage the complexity between the components themselves. Every time you need a new function from your dependency, you need to add it to the contract (a new @callback in our Elixir code). If the list of @callbacks are getting bigger and bigger, it will be noticiable as the knowledge is in one place and you will be able to act on it

  3. make it easier to test your system because it will push you to isolate the interaction between complex components

Defining contracts allows us to see the complexity in our dependencies. Your application will always have complexity, so always make it as explicit as you can.

Subscribe to our blog
Share on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInEmail this to someone
  • pcreux

    Hey José,

    Thanks for sharing this. In my experience, injecting dependencies make testing way easier an cleaner. As long as your dependency has a simple API, it’s really easy to swap it for a mock in test environment. I rely on this technique heavily for service objects in the Rails world. For instance:

    class GetReport
    include Virtus.model

    attribute :api_key, String
    attribute :newsletter_uuid, String

    attribute :mandrill_client, Object, default: proc do
    Rails.env.test? ? :

  • Dexter Miguel

    Great article. This is useful in all languages, not just Elixir. Thanks.

  • +1. In the example above you are creating a mock, you are not mocking an existing entity (noun, not a verb). However, I wouldn’t use “Rails.env.test?” in my code. I’d use the Rails configuration and read “Rails.config.mandrill_client” in the attribute. The configuration would be set to the mock in the config/test.rb (or in the test helper).

    You are absolutely right when you said “as long as your dependency has a simple API”. The whole idea behind explicit contracts is to help you guarantee the API continues simple in the long term, as it will be explicitly defined in a single place. It is a pity Ruby does not provide anything for this out of the box but there are certainly packages that could help with that.

  • pcreux

    +1 to not use `Rails.env.test?` but use the Rails config instead. :)

  • rcillo

    I would like to know if do you usually add `@spec` to function definitions so to make contracts stricter. I feel like it would produce even safer mock tests.

  • samueltonini

    Great article! Thanks for writing it. :-)

  • You could but they are usually the same as the `@callback` so there is no need. Unless they are meant to accept only a subset of the callback. So you could use `@spec` to restrict it further.

  • X4lldux

    And even when someone is still worried about the overhead of calling to a function which is calling to Application which is calling to ETS, you can always use macros like `defmacro twitter_api, do: unquote( Application.get_env :my_app, :twitter_api )` .

    @josevalim:disqus can you name some of those “good” not_anti_pattern_mock libraries?

  • I personally don’t use any. With Elixir it is easy enough to create a module whenever you want to track stuff.

  • From what I have observed, the current discussion around “mocks” seems to be isolating ourselves from inconvenient dependencies such as external API’s. Does anyone have any opinions regarding using mocks as a discovery tool for TDD (aka London-style, GOOS)? Even better any links? It seems that José is touching on some issues addressed in the paper “Mock Roles, not Objects” such as only mock what you own (i.e don’t mock third-party libraries). Cheers!

  • Pedro Assumpcao

    Great article! Thanks.

  • Brenton Annan

    Hey José,

    Great article, just what I’ve been looking for – I’d felt a bit strange about using dependency inject for testing, but your thoughts on tests being an equal client of code make a lot of sense.

    Is it possible to use this sort of methodology to build mocks when `use`ing a module, e.g. when testing that callbacks are working correctly in `HTTPoison.Base` ( ), as I’d like to do in this gist: ?

  • TheFirstAvenger

    If I wanted to use a different auth plug for test, and I defined the module in my config, how would I go about using that plug in a phoenix router? My first attempt was this, which failed:
    import Application.get_env(:server_commander, :auth_plug)

  • That’s because import works at expansion/compile time and the Application.get_env is performed at runtime. You won’t be able to use import. You will need to use the `module.function(args)` syntax.

  • TheFirstAvenger

    How exactly would the pipeline definition look in the Phoenix Router. For example, I have the following two pipelines defined which functioned correctly before adding in this test mock config:

    pipeline :secure do
    plug :fetch_access_token
    plug :authenticate_user
    plug :fetch_user
    plug :validate_permission, [level: :user, action: :edit]
    plug :validate_permission, [level: :user, action: :view]

    pipeline :admin_edit do
    plug :validate_permission, [level: :admin, action: :edit]

  • Andrew Bruce

    Hi José,

    You mention using data structures and defining protocols. On day one of learning Elixir this was what I reached for, but ran into a problem loading the fake implementation in test.

    I created my project with ‘mix new’. It has a protocol named RemoteCaller, and I’m passing in a %FakeRemoteCaller{} to a function. That fake looks like this:

    defmodule FakeRpcCaller do
    defstruct []

    defimpl RemoteCaller, for: FakeRpcCaller do
    def call(node, mod, fun, args) do
    [0, [:something], nil]

    The test is doing Code.load_file("test/fake_remote_caller.ex"). With fake_remote_caller.ex (or .exs) in the test/ dir, I get:

    ** (Protocol.UndefinedError) protocol RemoteCaller not implemented for %FakeRpcCaller{}

    However, when it’s in the lib/ dir, all works fine. What’s going on here? Thanks!

  • Excellent question! I am assuming you are using Elixir v1.2 as since this version Elixir consolidate protocols by default during compilation, which means that any protocol defined during tests won’t be part of the consolidation. You have two options here.

    1. The first is to include “test/support” in your compilation paths for the test environment. Phoenix for examples does it for new applications:

    2. The second option is to disable protocol consolidation for tests. Just set “consolidate_protocols: Mix.env != :test” in def project in your mix.exs.

  • Andrew Bruce

    Great, I chose (1), as it’s a bit more explicit. This also removed the need for the Code.load_file. Thanks!