There is no arguing about how important tests are for our application.

But from time to time, when we are dealing with it, some questions came up on a daily basis.

A very common day-do-day case is our application relying on APIs and external libs, but one of the things we don’t want our test suite to do is to access the external world.

This can lead us to many undesired scenarios, for example: if we have a connection problem on an API test that hits the internet, the test will fail, causing our test suite to be intermittent.

How do we mock or stub a request?

Some libs can help us, I will talk about two of them.

Bypass

The first one I wanna talk about is Bypass.
It is a lib that simulates an External Server, it works to stub the layer closer to the request. The idea is that you remove the external layer and start to call Bypass, so now, instead of talking to the internet, you are talking to a local server that you have control of. This way you can keep your tests protected.

Let’s think about an example, we will build an application that needs to fetch data from an API, and we will create a LastfmClient module that will be responsible for doing that.

defmodule MyApp.Lastfm.Client do
  @moduledoc"""
  Search tracks

  https://www.last.fm/api/show/track.search
  """

  @api_url "https://ws.audioscrobbler.com/2.0/"

  def search(term, url \\ @api_url) do
    response = Mojito.request(method: :get, url: search_url(url, term))

    case response do
      {:ok, %{status_code: 200, body: body}} ->
        {:ok, response(body)}
      {:ok, %{status_code: 404}} ->
        {:not_found, "Not found"}
      {_, response} ->
        {:error, response}
    end
  end
end
Code language: PHP (php)

What this client does is fetch the tracks from our API and according to the type of the response return something. Not that different from most of the clients we wrote on a daily basis. Let’s test it now.

describe "search/2" do
  test "searches tracks by the term" do
    response = Client.search("The Kooks")

    assert {:ok, [
      %{
        "artist" => "The Kooks",
        "name" => "Seaside",
        "url" => "https://www.last.fm/music/The+Kooks/_/Seaside"
      }
    ]} = response
  end
end
Code language: PHP (php)

What’s the problem with this test?

It seems pretty straight forward we exercise the SUT and verify the expected outcome, but this is a fragile test because it is accessing the external world.

Every time we call the Client.search/2 we are hitting the internet. A lot of problems can happen here: if the internet is down the test will fail, if the internet is slow your suit test will be slow, you won’t have the feedback as fast as you need and will be less inclined to run the test suit, or your suit test will become intermittent and you won’t trust your tests anymore, meaning that when a real failure happens you won’t care.

How can we fix it?

And that’s when Bypass comes to our aid.

First, you will need to setup Bypass in your tests.

setup do
  bypass = Bypass.open()

  {:ok, bypass: bypass}
end
Code language: JavaScript (javascript)

And in your test, you will set up which scenarios you want to test. Is it a success call? A not found? How should your code behave in each scenario? Tell Bypass that.

describe "search/2" do
  test "searches tracks by the term", %{bypass: bypass} do
    Bypass.expect bypass, fn conn ->
      Plug.Conn.resp(conn, 200, payload())
    end

    response = Client.search("The Kooks", "http://localhost:/#{bypass.port}/")

    assert {:ok, [
      %{
        "artist" => "The Kooks",
        "name" => "Seaside",
        "url" => "https://www.last.fm/music/The+Kooks/_/Seaside"
      }
    ]} = response
  end
end

defp payload do
  ~s(
    {
      "results": {
        "@attr": {},
        "opensearch:Query": {
          "#text": "",
          "role": "request",
          "startPage": "1"
        },
        "opensearch:itemsPerPage": "20",
        "opensearch:startIndex": "0",
        "opensearch:totalResults": "51473",
        "trackmatches": {
          "track": [
            {
              "artist": "The Kooks",
              "image": [
                {
                  "#text": "https://lastfm.freetls.fastly.net/i/u/34s/2a96cbd8b46e442fc41c2b86b821562f.png",
                  "size": "small"
                },
                {
                  "#text": "https://lastfm.freetls.fastly.net/i/u/64s/2a96cbd8b46e442fc41c2b86b821562f.png",
                  "size": "medium"
                },
                {
                  "#text": "https://lastfm.freetls.fastly.net/i/u/174s/2a96cbd8b46e442fc41c2b86b821562f.png",
                  "size": "large"
                },
                {
                  "#text": "https://lastfm.freetls.fastly.net/i/u/300x300/2a96cbd8b46e442fc41c2b86b821562f.png",
                  "size": "extralarge"
                }
              ],
              "listeners": "851783",
              "mbid": "c9b89088-01cd-4d98-a1f4-3e4a00519320",
              "name": "Seaside",
              "streamable": "FIXME",
              "url": "https://www.last.fm/music/The+Kooks/_/Seaside"
            }
          ]
        }
      }
    }
  )
end
Code language: PHP (php)

When to use Bypass?

When you have code that needs to make an HTTP request. You need to know how your application will behave. For instance, if the API is down, will your application stop working?

But then comes some questions, if I have a module that depends on this Client implementation, will I need to repeat this Bypass code every time in my tests?
Why does another module need to know these implementation details if it is not dealing with the request?

Mox

Mox can help us with that. It forces you to implement explicit contracts in your application so we know what to expect.

Going back to our example, let’s implement a module called Playlist that will be responsible for fetching a list of songs by artist and give it a name.

defmodule MyApp.Playlist do
  alias MyApp.Lastfm.Client

  def artist(name) do
    {:ok, songs} = Client.search(name)

    %{
      name: "This is #{name}",
      songs: songs
    }
  end
end
Code language: JavaScript (javascript)

The simplest test we can write to this code would be something like:

describe "artist/1" do
  test "returns the songs by artist" do
    result = Playlist.artist("The Kooks")

    assert result["name"] == "This is The Kooks"
    assert Enum.any?(result["songs"], fn song ->
      song["artist"] == "The Kooks"
    end)
  end
end
Code language: JavaScript (javascript)

Since the Playlist depends on the Client, to have an accurate test we would need to stub the request with the payload response from the Lastfm API so we can make sure the Playlist behaves accordingly.

You don’t need to stub all the Client requests in the Playlist tests, you need to know what it returns and handle the responses, you need to have a explicit contract.

Let’s see how we can implement those contracts.

Behaviours

Elixir uses behaviours as a way to define a set of functions that have to be implemented by a module. You can compare them to interfaces in OOP.

Let’s create a file at lib/my_app/music.ex that says what our Client expects as an argument and what it returns:

defmodule MyApp.Music do
  @callback search(String.t()) :: map()
end
Code language: JavaScript (javascript)

In our config/config.exs file, let’s include two lines. The first one says which client we are using and the second one is the Lastfm API that we will remove from the default argument, just to keep the callback simple.

config :my_app, :music, MyApp.Lastfm.Client
config :my_app, :lastfm_api, "https://ws.audioscrobbler.com/2.0/"
Code language: JavaScript (javascript)

In our config/test.exs file, let’s include our mock module.

config :my_app, :music, MyApp.MusicMock
Code language: CSS (css)

In the test/test_helper file, let’s tell Mox which is the mock module that is responsible for mocking the calls to our behaviour.

Mox.defmock(MyApp.MusicMock, for: MyApp.Music)
Code language: CSS (css)

Let’s go back to our Playlist module, and let’s change the way we call the Client module, to use the config we just created.

defmodule MyApp.Playlist do
  @music Application.get_env(:my_app, :music)

  def artist(name) do
    {:ok, songs} = @music.search(name)

    %{
      "name" => "This is #{name}",
      "songs" => songs
    }
  end
end
Code language: PHP (php)

In our Client module let’s adopt the behaviour we created, and let’s change the API url to fetch from the config we created.

defmodule MyApp.Lastfm.Client do
  @moduledoc"""
  Search tracks

  https://www.last.fm/api/show/track.search
  """

  @behaviour MyApp.Music

  def search(term) do
    url = lastfm_api_url()
    response = Mojito.request(method: :get, url: search_url(url, term))

    case response do
      {:ok, %{status_code: 200, body: body}} ->
        {:ok, response(body)}
      {:ok, %{status_code: 404}} ->
        {:not_found, "Not found"}
      {_, response} ->
        {:error, response}
    end
  end

  defp lastfm_api_url do
    Application.get_env(:my_app, :lastfm_api)
  end
end
Code language: PHP (php)

We will need to change the test/my_app/lastfm/client_test.exs to change the env config for the API url on the setup of the test, but I’ll leave it to you to do that.

Finally, in our PlaylistTest we will need to import Mox.

import Mox

# Make sure mocks are verified when the test exits
setup :verify_on_exit!
Code language: PHP (php)

And in our test, we need to tell our MusicMock what is expected to return.

describe "artist/1" do
  test "returns the songs by artist" do
    MusicMock
    |> expect(:search, fn _name ->
      {
        :ok,
        [
          %{
            "artist" => "The Kooks",
            "name" => "Seaside",
            "url" => "https://www.last.fm/music/The+Kooks/_/Seaside"
          }
        ]
      }
    end)

    result = Playlist.artist("The Kooks")

    assert result["name"] == "This is The Kooks"
    assert Enum.any?(result["songs"], fn song ->
      song["artist"] == "The Kooks"
    end)
  end
end
Code language: PHP (php)

What’s the difference?

Looking at the code it seems that we still need to pass the list of music in the tests. But there is a difference, the music’s list is a dependency of the Playlist module, but we don’t know its internals, we don’t know from where we are fetching it, how the request response works, and all these details. The only thing we need to know at the Playlist module is that it depends on a list of songs.

One last thing before we go

We went through all that trouble to make sure the tests are protected from the outside world, but you know, Elixir has this amazing Doctest feature, and one can argue that this replaces the application tests.

That’s not the case, Doctests are not tests and you shouldn’t rely on it to make sure your application behaves the way you expect it to. Make sure you don’t use any code that hits the external world when you are writing your documentation, there is no way to mock calls and this can cause all the problems we already discussed.

That’s all folks

The code from this example can be found here, I hope it helps!

The post Elixir: What about tests? first appeared on Plataformatec Blog.

To illustrate this problem, consider a simple application of storing and listing contacts. You have an actor user that can sign up, sign in, then can CRUD their contacts with first name, last name and phone number. We are in a context of multiple developers working together, programming and delivering features every day. For the examples below I’ll use Ruby, RSpec, Capybara and Ruby on Rails.

Suppose we have a UI testing using RSpec features test like this:

#spec/features/contacts_spec.rb
feature 'User can create new contact' do
  scenario 'when user provides valid contact information' do
    login_with email: 'default@mail.com', password: 'default'

    click_link 'Contacts'

    expect(page).to have_content('Listing Contacts')
    expect(page).to have_content("There're no contacts.")

    click_link 'New Contact'

    fill_in 'First name', with: 'Jon'
    fill_in 'Last name', with: 'Doe'
    fill_in 'Phone number', with: '1199999999'

    click_on 'Create Contact'

    expect(page).to have_content('Listing Contacts')
    expect(page).to have_content('Jon Doe - (11) 9999-9999')
  end
end

Simple and straight forward. The good part of this test is it checks the entire application stack: authentication, rendering, storing stuff on the database and the logic of creating contacts. Some developer may be confident with this test and start to question: Should I add more unit tests? For phone number formatting? For displaying complete names? For user authentication? For contact validation and creation? If one of these features fail, this feature test will fail too. Why should I care to add more tests? Isn’t it redundancy? Waste of time? The short answer is you should add more unit and integration tests because of feedback. Now, let’s expand the topic.

Speed matters

We know the UI test brings confidence since it tests the entire stack of your application. But overusing this type of tests will severely impact your test suite speed.

In our contacts creation test example, we have some edge cases that were not covered. What happens if we use a different size of numbers in the telephone number field? How will it be displayed? We can add a new UI test to cover this case, but a unit test suits better and faster. If the number formatting is a view helper we can check it with a helper test, for example:

describe '#format_phone_number' do
  it 'formats cell phones numbers of 10 digits in "(XX) XXXXX-XXXX" format' do
    formatted_number = helper.format_phone_number('11999998888')
    expect(formatted_number).to eq '(11) 99999-8888'
  end

  it 'does not format others digits different of 10' do
    formatted_number = helper.format_phone_number('11888')
    expect(formatted_number).to eq '11888'
  end
end

This test is way easier to write, read, cheaper and faster than adding a new UI to test this specific case. Keeping this practice, you will have fewer UI tests and more integration and unit tests. Such distribution of the different kind of tests resembles the format of a pyramid:

Few UI tests on top and a solid base of unit tests. If you want more information about the test pyramid, I recommend you the following links:

• The Forgotten Layer of the Test Automation Pyramid by Mike Cohn
• The Test Pyramid by Kenneth Truyers
• TestPyramid by Martin Fowler
• We have a talk and a video explaining the test pyramid in portuguese.

Helpful messages

In the daily routine of a development team, the code is constantly evolving. The test server is running and warning when the test suite is passing or broken. A passing test suite is a requirement to deploy a new change in production in most of the clients that we have been working on. Then when the test suite fails, it is important for developers to act and fix the broken tests fast. Without helpful messages, it is critically hard.

Let’s use again our contacts UI test, let’s use a scenario where we just have that test. The application has received new changes and the test server warns the team that we have an error, and the output is:

  1) User can create new contact when user provides valid contact information
     Failure/Error: expect(page).to have_content("Jon Doe - (11) 9999-9999")
       expected to find text "Jon Doe - (11) 9999-9999" in "default@mail.com Log out Contacts Contact was successfully created. Contact was successfully created. Listing Contacts Jon Doe - 1199999

We know the test failed in the last step, the step of checking the recent contact content in the listing page. In this sample application, we have a small text to search what’s going on. In a real world application, this content can be a lot bigger and unproductive to read. From here the developer has a lot of possibilities to search what the root cause of the problem. For example:

• Is it in the creation?
• Is it in the contacts listing?
• Is it in the telephone format method?
• Is it in the test?
• Had been changes to asynchronous request and expectation didn’t have time to assert the new content?

The list can go on depending on your creativity. Now let’s build a scenario where we also have the format_phone_number helper unit test. The tests are failing, but now have this output:

1) User can create new contact when user provides valid contact information
     Failure/Error: expect(page).to have_content("Jon Doe - (11) 9999-9999")
       expected to find text "Jon Doe - (11) 9999-9999" in "default@mail.com Log out Contacts Contact was successfully created. Contact was successfully created. Listing Contacts Jon Doe - 1199999999 Show Edit Destroy New Contact"

2) ContactsHelper#format_phone_number formats cell phones numbers of 10 digits in "(XX) XXXXX-XXXX" format
   Failure/Error: expect(helper.format_phone_number('11999998888')).to eq '(11) 99999-8888'

     expected: "(11) 99999-8888"
          got: "11999998888"

     (compared using ==)

Now we have two failing tests. The second error message is way easier to guess why it is broken. Some change on the format_phone_number helper has broken the cell phone formatting. Knowing it, looking again at the first error message you may connect the dots and find out the errors are connected.

If two tests are failing for the same reason, this is not the same thing of worthless redundancy. Focused behavior tests bring you a lot more information for debugging, helping you solve problems fast. This is another benefit of the test pyramid.

Conclusion

Some people tend to be relaxed about worry too much about creating focused tests. I think the main reason is they are very used to code base context. If something fails they can navigate, debug and figure out fast what happened. But unfortunately, it not always true for everyone, specially for the newcomers.

Another problem is when the team becomes careless about covering exceptional cases with unit and integrations tests. The test suite is always green but problems are increasing in production. Your test suites become less reliable and leaving the entire team insecure about deploying new changes.

To keep your test suite fast and worthy of trust I recommend you use the test pyramid technique. In Rails applications will be something like:

• Fewer UI tests with Capybara. Use them to test the user main journey and some exceptional cases.
• Test beyond the expected behavior. Exercise the exceptional behaviors with integration and unit tests. Write more models, controllers, requests, views, helpers, or mailers tests.
• Consider adding JavaScript unit tests in your test suite.

These practices help us a lot to maintain a healthy code base and allow us to adapt and be more agile to change. What do you think of this practices? Which techniques and practices are you doing to keep your test suite fast and with helpful messages? Let me know with a comment below.

The post Using the test pyramid to have better feedback from your test suite first appeared on Plataformatec Blog.

A Tale of Two Adapters

Back in November, I worked on integrating a payment gateway from scratch into one of our client projects, through a gem that abstracts the HTTP interface of this external service. On this payment flow we had to first authorize the payment data with the gateway, which would return the transaction data for us to capture the payment in the background and go on with the business logic that depended on a successful payment flow.

If you ever worked on something similar, you probably remember a few rough edges that we need to deal in cases like this: how to test the integration with the right credit card numbers for each possible outcome? Do we have a sandbox available for development and test environments? How can we control the performance and stability costs that this external dependency might bring to our application, and the coupling between the code that supports this core feature and this gem?

Our attempt to handle the coupling and maintenance cost of this dependency was to push all the integration code behind an abstraction layer responsible for dealing with this payment flow logic under a Payments namespace.

require 'gateway-xyz-gem'

module Payments
  class GatewayXYZ
    def authorize(order, credit_card)
      # Uses the `Order` details and the user `CreditCard` data to authorize
      # a new transaction on the XYZ Payment Gateway through the
      # `gateway-xyz-gem` classes.
    end

    def capture(payment_id)
      # Capture the payment information for a transaction that was previously
      # authorized.
    end
  end
end

Somewhere down our orders#create action (but not directly in the controller method itself) we call GatewayXYZ#authorize with the order record and a credit_card value object and our integration with the external service is done.

We might have a nice set of well-defined methods on the GatewayXYZ class but our job on these abstractions is far from done. We might unit test it with something like WebMock or VCR to handle the external service dependency, but every other piece of our system that interacts with this abstraction will also depend on the external API to work properly – the OrdersController, the background job that captures the payment and the Order model itself that might trigger the initial authorize call. Should we just sprinkle the existing stubs all over our test suite and call it a day?

We added a gateway implementation that mimics the expected behavior of the GatewayXYZ (with the same method signatures as the real gateway) and doesn’t depend on external resources. It also has a predefined behavior for specific inputs so we can test different code paths of their collaborators based on the test input.

module Payments
  class Memory
    def authorize(order, credit_card)
      if BAD_CREDIT_CARD_NUMBERS.include?(credit_card.number)
        bad_response
      else
        ok_response
      end
    end
  end
end

Dealing with environment specific setups

Now we need to make our Payments::Memory the go-to implementation for our test cases that depend on our payment abstractions. There are a few different ways we can do this on a Rails app.

Rails.application.config

We can expose a configuration setting in app that says which implementation it should use, similar to how Action Mailer picks the delivery method for your emails or how Active Job might have different queue adapters for your background jobs.

# config/application.rb
Rails.application.config.x.payment_gateway = Payments::GatewayXYZ

# config/environments/test.rb
Rails.application.config.x.payment_gateway = Payments::Memory

# app/models/order.rb
class Order
  def authorize(credit_card)
    gateway = build_gateway
    gateway.authorize(self, credit_card)
  end

  private

  def build_gateway
    klass = Rails.application.config.x.payment_gateway
    klass.new
  end
end

Module.mattr_accessor macro

You can set a class level macro on the classes that depend on a configurable value and change as you want in your code. This approach can be useful if you want to keep the configuration closer to the implementation that relies on it, instead of jumping between app code and configuration code if you want to debug something or be able to change it during runtime.

# app/models/order.rb
class Order
  cattr_accessor :payment_gateway do
    Payments::GatewayXYZ
  end

  def authorize(credit_card)
    gateway = payment_gateway.new
    gateway.authorize(self, credit_card)
  end
end

# test/test_helper.rb
Order.payment_gateway = Payments::Memory

Factory method

This approach is useful when you want to hide away how to create an instance of a gateway implementation, so other classes that depend on it can have a way to just ask for a gateway object without worrying on how to create it.

# app/models/payments.rb
module Payments
  matt_accessor :gateway do
    Payments::GatewayXYZ
  end

  def build_gateway
    gateway.new
  end

  module_function :build_gateway
end

# test/test_helper.rb
Payments.gateway = Payments::Memory

I don’t believe that there is a Single Way to Do It this kind of dependency injection, so you should feel free to pick a strategy that suits the interfaces you are building and the coding style of your team – I’m personally a fan of the factory method and the cattr_accessor approaches as they feel more detached from the configuration and closer to the application code, although the configuration way feels more aligned with global APIs from 3rd party gems.

Skipping Hash driven development

Our GatewayXYZ and Memory implementations have the same methods with the same arguments but there is a second piece of making a uniform API that we need to think about: what those methods should return?

Our authorize needs to return more than a truthy/falsy value, as we need to gather more information about the payment transaction on our end, like the payment_id from the transaction, or a reason of why it might have failed (was the credit card denied? There is invalid data in the request), details for logging or instrumentation, etc. And if we think about implementing this API for multiple services (let’s say we need a Payments::PayPal now, for instance), those services will return this data in different formats that we need to normalize so these differences don’t leak to the rest of the system.

One might say that a Hash with all this junk would do it, but going that path opens too many doors for inconsistency and bugs as the hash is a weak abstraction that can be mutated anywhere and won’t enforce any specific format or requirements on the return values.

For that, we can implement a Payments::Result struct/value object to represent the outcome of our authorize action, and return it from each gateway implementation in our system, enforcing the interface we want to have.

module Payments
  class Result < Struct.new(:payment_id, :errors)
    def ok?
      errors.blank?
    end

    def failed?
      !ok?
    end
  end
end

Our Result class has the minimal information that our client code needs, and each gateway is responsible for constructing a Result from its own data. The Memory gateway can do something as straightforward as this:

module Payments
  class Memory
    def authorize(order, credit_card)
      Result.new(
        payment_id: SecureRandom.hex,
        errors: SAMPLE_ERRORS[credit_card.number])
    end
  end
end

This approach is useful not just for enforcing the interface we want, but also to improve other areas of our code that could use more specific abstractions than a bare Hash instance.

Going forward with contracts and macros

This homemade approach for better contracts between our app and this external service can go a long way, but if you want, you can build strict checks on top of your APIs to ensure that your objects are collaborating as you expect. We haven’t tried yet, but the contracts gem looks very interesting if you want that kind of type constraints that are lacking on Ruby.

You can even write your own checks by wrapping methods into type checking proxies, as refile does with its Refile::BackendMacros module. When extended by a backend implementation, it provides macros to validate the input for methods like #upload(uploadable) or #delete(id), so custom implementations don’t need to worry about validating these arguments on their own.

The post Experimenting with explicit contracts with Ruby first appeared on Plataformatec Blog.

Install Erlang and Elixir

We use Jenkins to run ours builds and Ansible to setup the agents. The following code are snippets from the tasks used for machine provisioning. Our agents are using Ubuntu, but you can check the Elixir Install section to follow the steps for another OS.

Step 1: Add the Erlang Solutions repository:

First, we need to add the Erlang Solution repo and its GPG key before updating the repo. With this step we can find the Erlang and Elixir packages.

- name: Add Erlang Solutions repository
  apt_repository: >
    repo='deb https://packages.erlang-solutions.com/ubuntu/ {{ansible_distribution_release}} contrib'
    state=present
    update_cache=yes

- name: Add the GPG key for Erlang Solutions
  apt_key:
    url: "https://packages.erlang-solutions.com/{{ ansible_distribution | lower }}/erlang_solutions.asc"
    state: present

Step 2: Install Erlang and Elixir packages.

We are using the latest version of Elixir but if you need to manage it you can take a look at a manager like kiex or asdf.

- name: Install Erlang
  apt: pkg=esl-erlang state=latest

- name: Install Elixir
  apt: pkg=elixir state=latest

Step 3: Install Hex and Rebar.

If you are new to Elixir/Erlang, you probably don’t know about these tools. Hex is the package manager for the Erlang ecosystem and you can learn more at https://hex.pm/.

Rebar is an Erlang build tool, and it’s necessary to build the dependency poolboy. The task mix local.rebar will install both rebar and rebar3. You can check the Mix.Tasks.Local.Rebar documentation to install one of them if you need.

- name: Install Elixir Hex
  command: mix local.hex --force

- name: Install Rebar
  command: mix local.rebar

Running your tests

We use Janky, along with Jenkins, which can execute a custom build configuration into script/cibuild that is stored at the root of the repository. Our Phoenix projects use the following script:

#!/bin/bash -ex
env

source /etc/profile

export POSTGRESQL_USER="test"
export POSTGRESQL_PASSWORD="test"

mix ecto.reset
mix deps.get
mix test

Exporting ENV variables

Our project is using Postgres and it isn’t a good practice to store the credentials in the version control. So, we export the credential to environment variables and get it in config/test.exs:

config :my_app, MyApp.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: System.get_env("POSTGRESQL_USER") || "postgres",
  password: System.get_env("POSTGRESQL_PASSWORD") || "postgres",
  database: "my_app_test",
  hostname: "localhost",
  pool: Ecto.Adapters.SQL.Sandbox,
  port: System.get_env("POSTGRESQL_PORT") || 5432

You can read more about this practice at Twelve-Factor App.

Reset your database

Usually, each builder is responsible for provision in a new machine. Instead of this, we setup a machine that will run several builds from different projects while there is activity. Therefore, we need to reset the database to avoid side-effects with different schemas.

The task mix test will run the tasks ecto.create and ecto.migrate (you can check it in test/test_helper.exs). We could use mix ecto.destroy instead of mix ecto.reset, but we prefer to use the latter to be more explicit.

Install dependencies and run the tests

Now, mix deps.get and mix test will install the project dependencies and run your tests.

Which tips do you have to install or use in CI to run Phoenix tests?

The post How to setup CI to run Phoenix projects first appeared on Plataformatec Blog.

In this blog post, we’ll write a few acceptance tests for an expenses report listing page, where we’ll interact with the report and some form elements.

To make possible to interact with the elements on the page, we’ll need a web browser driver. Hound accepts chrome_driver, firefox, phantomjs and selenium. My choice is phantomjs because I want to use a headless browser (I don’t want the driver to open a browser during the execution of the test suite).

Setup

First we’ll need to add Hound into our dependencies, so add the following line in your mix.exs:

{:hound, "~> 0.8"}

Make sure it’ll start during the test suite runtime. To do this we’ll need to add Application.ensure_all_started(:hound) before ExUnit.start in our test helper:

Application.ensure_all_started(:hound)
ExUnit.start

We’ll be using phantomjs as our web driver. Make sure it’s properly installed and that you can start it with phantomjs --wd. To configure it, add this to the config.exs file:

config :hound, driver: "phantomjs"

Take a look at this doc from Hound resources to check if you’d like different configs.

We’ll also need to set the server config in our config/test.exsto true.

config :my_app, MyApp.Endpoint,
  http: [port: 4001]
  server: true

That should do it! Before writing our first test, let’s define an IntegrationCase module, similar to the ModelCase and ConnCase provided by Phoenix, which will include all functionality we need to write our integration tests. Create the test/support/integration_case.ex file and add the following content:

defmodule MyApp.IntegrationCase do
  use ExUnit.CaseTemplate

  using do
    quote do
      use Hound.Helpers

      import Ecto.Model
      import Ecto.Query, only: [from: 2]
      import MyApp.Router.Helpers

      alias MyApp.Repo

      # The default endpoint for testing
      @endpoint MyApp.Endpoint

      hound_session
    end
  end

  setup tags do
    unless tags[:async] do
      Ecto.Adapters.SQL.restart_test_transaction(MyApp.Repo, [])
    end

    :ok
  end
end

There are a few lines that are worth commenting:

• use Hound.Helpers will include the helpers necessary for us to interact with our interface (take a look at the docs and explore them a little, they’ll be very helpful);
• hound_session will make sure that Hound will start and closes its session during the test execution;
• import MyApp.Router.Helpers will include helpers so we can manipulate routes and URLs.

Exercise

Let’s test!

We’re testing a simple list of expenses from a city (that example was extracted from an app we have been working on, but its scope was reduced so we can follow the steps more easily).

Take a look at its template code:

<div>
  <%= form_for @conn, city_expense_path(@conn, :index, @city.id), [as: :q, method: :get], fn f -> %>
    <div>
      <label for="q_status">Status</label>
      <%=
        select f, :status, [{"Paid", "paid"}, {"Cancelled", "cancelled"}], prompt: "All" %>
    </div>

    <div>
      <label for="q_supplier">Supplier</label>
      <%= text_input f, :supplier %>
    </div>

    <%= submit "Submit" %>
  <% end %>
</div>

<table>
  <thead>
    <th>ID</th>
    <th>Status</th>
    <th>Supplier</th>
    <th>Value</th>
    <th>Date</th>
  </thead>
  <tbody>
    <%= for expense <- @expenses do %>
      <tr>
        <td><%= expense.id %></td>
        <td><%= expense.status %></td>
        <td><%= expense.supplier.name %></td>
        <td><%= expense.value %></td>
        <td><%= expense.date %></td>
      </tr>
    <% end %>
  </tbody>
</table>

We’ll need a new test file, let’s put it at test/integration/expenses_list_test.exs. To use Hound’s facilities, we’ll need to use the IntegrationCase module that we have previously created.

defmodule MyApp.ExpenseListTest do
  use MyApp.IntegrationCase

end

We’ll be using three private functions to help us making assertions and interacting with elements in our test file. The first one, expense_on_the_list, will check if a given expense, that is represented by the Ecto model called MyApp.Expense, is in there. The second function is just a helper for getting the expenses list and the third will help us interact with a select input within a form.

defmodule MyApp.ExpenseListTest do
  use MyApp.IntegrationCase

  # ...

  defp expense_on_the_list(expense, list) do
    list
    |> visible_text
    |> String.contains?(expense.id)
  end

  defp expense_list_items do
    find_element(:tag, "tbody")
    |> find_all_within_element(:tag, "tr")
  end

  defp select_status(form, status) do
    form
    |> find_within_element(:id, "q_status")
    |> input_into_field(status)
  end
end

To interact with an element, you’ll always need to find the element on the page and for this, you need to know Hound’s page helpers. I’ve noticed that we ended up using find_element and find_all_within_element most of the time to find the elements on the page or in a context (i.e inside a previously found element).

Since this test is about the City resource, we’ve created just this city and navigated to it directly on the setup, since this would be a requirement for all the tests in this file, and shared it with all the tests through the setup context.

setup do
  city = insert_city!(%{name: "Winterfell"})

  navigate_to("/cities/#{city.id}/expenses")

  {:ok, city: city}
end

Navigation is another important Hound module. It will help us go through our app easily and get info about the page, like the current_path() function that returns the path we’re navigating on that moment.

Now that we’re on the page, we’ll be interacting with the form, by finding elements that are within it and filling or selecting values on them.

test "filter by supplier", %{city: city} do
  supplier = insert_supplier!(%{name: "Ned Stark"})
  supplier_b = insert_supplier!(%{name: "Bell Tower Management"})
  expense = insert_expense!(%{supplier: supplier, city: city, status: "paid"})
  insert_expense!(%{supplier: supplier_b, city: city, status: "paid"})

  search_form = find_element(:tag, "form")
  search_form
  |> find_within_element(:id, "q_supplier")
  |> fill_field("Ned")

  submit_element(search_form)


  items = expense_list_items
  assert length(items) == 1
  assert expense_on_the_list(expense, items)
end

The module responsible for these tasks is Element. It has very useful functions, like fill_field we used above. All of its functions require an element.

In the previous example, the interactions with the form ended with submit_element, but if we need any other action on it after this, we would need to re-assign it (otherwise, we’ll get a ** (RuntimeError) Element does not exist in cache error), like in the following example:

test "filter by statuses", %{city: city} do
  supplier = insert_supplier!(%{name: "Jon Snow"})

  cancelled_expense = insert_expense!(%{supplier: supplier, city: city, status: "cancelled"})
  paid_expense = insert_expense!(%{supplier: supplier, city: city, status: "paid"})

  search_form = find_element(:tag, "form")
  select_status(search_form, "Cancelled")

  submit_element(search_form)

  items = expense_list_items
  assert length(items) == 1
  assert expense_on_the_list(cancelled_expense, items)

  search_form = find_element(:tag, "form")
  select_status(search_form, "Paid")
  submit_element(search_form)

  items = expense_list_items
  assert length(items) == 1
  assert expense_on_the_list(paid_expense, items)
end

Verify

Runtime

One of the things I’ve paid a lot of attention during this experience was the test suite runtime. As expected, it can get slow with acceptance tests. The original app is still really tiny and before adding the acceptance tests, the runtime was:

Finished in 0.6 seconds (0.5s on load, 0.1s on tests)
23 tests, 0 failures, 2 skipped

After including two tests (but with more interactions than the ones presented), it was noticeable the test suite became slower. It tripled the runtime.

Finished in 1.8 seconds (0.5s on load, 1.2s on tests)
25 tests, 0 failures, 2 skipped

This effect is actually expected. We know that acceptance tests are expensive and that they should be a small portion of your test pyramid

There are a few things that can make acceptance tests faster:

• Upcoming Ecto v2.0 will allow us to run tests that use database persistence asyncronously;

• In my experience Phantom.js considerably faster than the other options, I recommend you to go with it.

You will definitely need to write some acceptance tests, but give preference to controller tests in most scenarios and use acceptance tests for important flows of your app (take a look at the user journey concept, that can give you some good insights).

Web driver server execution

Currently, Hound doesn’t execute the browser automatically during tests. You’ll need to start it; otherwise, your tests will fail. There may be some workarounds to achieve it, if you’re on OS X, you can run Phantomjs as a service.

Teardown

I really enjoyed playing with Hound, and I found very simple to work with it. Also, I see it as a potential project if you’re considering contributing to an Open Source Project.

I hope this post was useful and gave you some ideas of how to write acceptance tests with Elixir and Phoenix. If you have any questions or suggestions, I’m all ears (or eyes).

Which tool have you recently found to be useful when writing tests in Elixir?

The post Writing Acceptance tests in Phoenix first appeared on Plataformatec Blog.

A couple days ago I expressed my thoughts regarding mocks on Twitter:

Mocks/stubs do not remove the need to define an explicit interface between your components (modules, classes, whatever). [1/4]

— José Valim (@josevalim) September 9, 2015

The blame is not on mocks though, they are actually a useful technique for testing. However our test tools often makes it very easy to abuse mocks and the goal of this post is to provide better guidelines on using them.

What are mocks?

The wikipedia definition is excellent: mocks are simulated entities that mimic the behavior of real entities in controlled ways. I will emphasize this later on but I always consider “mock” to be a noun, never a verb.

Case study: external APIs

Let’s see a common practical example: external APIs.

Imagine you want to consume the Twitter API in your web application and you are using something like Phoenix or Rails. At some point, a web request will come-in, which will be dispatched to a controller which will invoke the external API. Let’s imagining this is happening directly from the controller:

defmodule MyApp.MyController do
  def show(conn, %{"username" => username}) do
    # ...
    MyApp.TwitterClient.get_username(username)
    # ...
  end
end

The code may work as expected but, when it comes to make the tests pass, a common practice is to just go ahead and mock (warning! mock as a verb!) the underlying HTTPClient used by MyApp.TwitterClient:

mock(HTTPClient, :get, to_return: %{..., "username" => "josevalim", ...})

You proceed to use the same technique in a couple other places and your unit and integration test suites pass. Time to move on?

Not so fast. The whole problem with mocking the HTTPClient is that you just coupled your application to that particular HTTPClient. For example, if you decide to use a new and faster HTTP client, a good part of your integration test suite will now fail because it all depends on mocking HTTPClient itself, even when the application behaviour is the same. In other words, the mechanics changed, the behaviour is the same, but your tests fail anyway. That’s a bad sign.

Furthermore, because mocks like the one above change modules globally, they are particularly aggravating in Elixir as changing global values means you can no longer run that part of your test suite concurrently.

The solution

Instead of mocking the whole HTTPClient, we could replace the Twitter client (MyApp.TwitterClient) with something else during tests. Let’s explore how the solution would look like in Elixir.

In Elixir, all applications ship with configuration files and a mechanism to read them. Let’s use this mechanism to be able to configure the Twitter client for different environments. The controller code should now look like this:

defmodule MyApp.MyController do
  def show(conn, %{"username" => username}) do
    # ...
    twitter_api().get_username(username)
    # ...
  end

  defp twitter_api do
    Application.get_env(:my_app, :twitter_api)
  end
end

And now we can configure it per environment as:

# In config/dev.exs
config :my_app, :twitter_api, MyApp.Twitter.Sandbox

# In config/test.exs
config :my_app, :twitter_api, MyApp.Twitter.InMemory

# In config/prod.exs
config :my_app, :twitter_api, MyApp.Twitter.HTTPClient

This way we can choose the best strategy to retrieve data from Twitter per environment. The sandbox one is useful if Twitter provides some sort of sandbox for development. The HTTPClient is our previous implementation while the in memory avoids HTTP requests altogether, by simply loading and keeping data in memory. Its implementation could be defined in your test files and even look like:

defmodule MyApp.Twitter.InMemory do
  def get_username("josevalim") do
    %MyApp.Twitter.User{
      username: "josevalim"
    }
  end
end

which is as clean and simple as you can get. At the end of the day, MyApp.Twitter.InMemory is a mock (mock as a noun, yay!), except you didn’t need any fancy library to define one! The dependency on HTTPClient is gone as well.

The need for explicit contracts

Because a mock is meant to replace a real entity, such replacement can only be effective if we explicitly define how the real entity should behave. Failing this, you will find yourself in the situation where the mock entity grows more and more complex with time, increasing the coupling between the components being tested, but you likely won’t ever notice it because the contract was never explicit.

Furthermore, we have already defined three implementations of the Twitter API, so we better make it all explicit. In Elixir we do so by defining a behaviour with callback functions:

defmodule MyApp.Twitter do
  @doc "..."
  @callback get_username(username :: String.t) :: %MyApp.Twitter.User{}
  @doc "..."
  @callback followers_for(username :: String.t) :: [%MyApp.Twitter.User{}]
end

Now add @behaviour MyApp.Twitter on top of every module that implements the behaviour and Elixir will help you provide the expected API.

It is interesting to note we rely on such behaviours all the time in Elixir: when you are using Plug, when talking to a repository in Ecto, when testing Phoenix channels, etc.

Testing the boundaries

Previously, because we didn’t have a explicit contract, our application boundaries looked like this:

[MyApp] -> [HTTP Client] -> [Twitter API]

That’s why changing the HTTPClient could break your integration tests. Now our app depends on a contract and only one implementation of such contract rely on HTTP:

[MyApp] -> [MyApp.Twitter (contract)]
[MyApp.Twitter.HTTP (contract impl)] -> [HTTPClient] -> [Twitter API]

Our application tests are now isolated from both the HTTPClient and the Twitter API. However, how can we make sure the system actually works as expected?

Of the challenges in testing large systems is exactly in finding the proper boundaries. Define too many boundaries and you have too many moving parts. Furthermore, by writing tests that rely exclusively on mocks, your tests become less reliable.

My general guideline is: for each test using a mock, you must have an integration test covering the usage of that mock. Without the integration test, there is no guarantee the system actually works when all pieces are put together. For example, some projects would use mocks to avoid interacting with the database during tests but in doing so, they would make their suites more fragile. These is one of the scenarios where a project could have 100% test coverage but still reveal obvious failures when put in production.

By requiring the presence of integration tests, you can guarantee the different components work as expected when put together. Besides, the requirement of writing an integration test in itself is enough to make some teams evaluate if they should be using a mock in the first place, which is always a good question to ask ourselves!

Therefore, in order to fully test our Twitter usage, we need at least two types of tests. Unit tests for MyApp.Twitter.HTTP and an integration test where MyApp.Twitter.HTTP  is used as an adapter.

Since depending on external APIs can be unreliably, we need to run those tests only when needed in development and configure them as necessary in our build system. The @tag system in ExUnit, Elixir’s test library, provides conveniences to help us with that. For the unit tests, one could do:

defmodule MyApp.Twitter.HTTPTest do
  use ExUnit.Case, async: true

  # All tests will ping the twitter API
  @moduletag :twitter_api

  # Write your tests here
end

In your test helper, you want to exclude the Twitter API test by default:

ExUnit.configure exclude: [:twitter_api]

But you can still run the whole suite with the tests tagged :twitter_api if desired:

mix test --include twitter_api

Or run only the tagged tests:

mix test --only twitter_api

Although I prefer this approach, external conditions like rate limiting may make such solution impractical. In such cases, we may actually need a fake HTTPClient. This is fine as long as we follow the guidelines below:

1. If you change your HTTP client, your application suite won’t break but only the tests for MyApp.Twitter.HTTP
2. You won’t mock (warning! mock as a verb) your HTTP client. Instead, you will pass it as a dependency via configuration, similar to how we did for the Twitter API

Alternatively, you may avoid mocking the HTTP client by running a dummy webserver that emulates the Twitter API. bypass is one of many projects that can help with that. Those are all options you should discuss with your team.

Other notes

I would like to finish this article by bringing up some common concerns and comments whenever the mock discussion comes up.

Making the code “testable”

Quoting from elixir-talk mailing list:

So the proposed solution is to change production code to be “testable” and making production code to call Application configuration for every function call? This doesn’t seem like a good option as it’s including a unnecessary overhead to make something “testable”.

I’d argue it is not about making the code “testable”, it is about improving the design of your code.

A test is a consumer of your API like any other code you write. One of the ideas behind TDD is that tests are code and no different from code. If you are saying “I don’t want to make my code testable”, you are saying “I don’t want to decouple some modules” or “I don’t want to think about the contract behind these components”.

Just to clarify, there is nothing wrong with “not wanting to decouple some modules”. For example, we invoke modules such as URI and Enum from Elixir’s standard library all the time and we don’t want to hide those behind contracts. But if we are talking about something as complex as an external API, defining an explicit contract and making the contract implementation configurable is going to do your code wonders and make it easier to manage its complexity.

Finally, the overhead is also minimum. Application configuration in Elixir is stored in ETS tables which means they are directly read from memory.

Mocks as locals

Although we have used the application configuration for solving the external API issue, sometimes it is easier to just pass the dependency as argument. Imagine this example in Elixir where some function may perform heavy work which you want to isolate in tests:

defmodule MyModule do
  def my_function do
    # ...
    SomeDependency.heavy_work(arg1, arg2)
    # ...
  end
end

You could remove the dependency by passing it as an argument, which can be done in multiple ways. If your dependency surface is tiny, an anonymous function will suffice:

defmodule MyModule do
  def my_function(heavy_work \\ &SomeDependency.heavy_work/2) do
    # ...
    heavy_work.(arg1, arg2)
    # ...
  end
end

And in your test:

test "my function performs heavy work" do
  heavy_work = fn _, _ ->
    # Simulate heavy work by sending self() a message
    send self(), :heavy_work
  end

  MyModule.my_function(heavy_work)

  assert_received :heavy_work
end

Or define the contract, as explained in the previous section of this post, and pass a module in:

defmodule MyModule do
  def my_function(dependency \\ SomeDependency) do
    # ...
    dependency.heavy_work(arg1, arg2)
    # ...
  end
end

Now in your test:

test "my function performs heavy work" do
  # Simulate heavy work by sending self() a message
  defmodule TestDependency do
    def heavy_work(_arg1, _arg2) do
      send self(), :heavy_work
    end
  end

  MyModule.my_function(TestDependency)

  assert_received :heavy_work
end

Finally, you could also make the dependency a data structure and define the contract with a protocol.

In fact, passing the dependency as argument is much simpler and should be preferred over relying on configuration files and Application.get_env/3. When not possible, the configuration system is a good fallback.

Mocks as nouns

Another way to think about mocks is to treat them as nouns. You shouldn’t mock an API (verb), instead you create a mock (noun) that implements a given API.

Most of the bad uses of mocks come when they are used as verbs. That’s because, when you use mock as a verb, you are changing something that already exists, and often those changes are global. For example, when we say we will mock the SomeDependency module:

mock(SomeDependency, :heavy_work, to_return: true)

When you use mock as a noun, you need to create something new, and by definition it cannot be the SomeDependency module because it already exists. So “mock” is not an action (verb), it is something you pass around (noun). I’ve found the noun-verb guideline to be very helpful when spotting bad use of mocks. Your mileage may vary.

Mock libraries

With all that said, should you discard your mock library?

It depends. If your mock library uses mocks to replace global entities, to change static methods in OO or to replace modules in functional languages, you should definitely consider how the library is being used in your codebase and potentially discard it.

However there are mock libraries that does not promote any of the “anti-patterns” above and are mostly conveniences to define “mock objects” or “mock modules” that you would pass to the system under the test. Those libraries adhere to our “mocks as nouns” rule and can provide handy features to developers.

Summing up

Part of testing your system is to find the proper contracts and boundaries between components. If you follow closely a guideline that mocks will be used only if you define a explicit contract, it will:

1. protect you from overmocking as it will push you to define contracts for the parts of your system that matters 
2. help you manage the complexity between different components. Every time you need a new function from your dependency, you are required to add it to the contract (a new @callback in our Elixir code). If the list of @callbacks are getting bigger and bigger, it will be noticeable as the knowledge is in one place and you will be able to act on it
3. make it easier to test your system because it will push you to isolate the interaction between complex components

Defining contracts allows us to see the complexity in our dependencies. Your application will always have complexity, so always make it as explicit as you can.

The post Mocks and explicit contracts first appeared on Plataformatec Blog.