Writing Acceptance tests in Phoenix

Acceptance testing seems to be in its first steps in the Elixir ecosystem, but there are already some cool libs that can help us out to do it. I’m going to show you how we did it with Hound.

In this blog post, we’ll write a few acceptance tests for an expenses report listing page, where we’ll interact with the report and some form elements.

To make possible to interact with the elements on the page, we’ll need a web browser driver. Hound accepts chrome_driver, firefox, phantomjs and selenium. My choice is phantomjs because I want to use a headless browser (I don’t want the driver to open a browser during the execution of the test suite).

Setup

First we’ll need to add Hound into our dependencies, so add the following line in your mix.exs:

{:hound, "~> 0.8"}

Make sure it’ll start during the test suite runtime. To do this we’ll need to add Application.ensure_all_started(:hound) before ExUnit.start in our test helper:

Application.ensure_all_started(:hound)
ExUnit.start

We’ll be using phantomjs as our web driver. Make sure it’s properly installed and that you can start it with phantomjs --wd. To configure it, add this to the config.exs file:

config :hound, driver: "phantomjs"

Take a look at this doc from Hound resources to check if you’d like different configs.

We’ll also need to set the server config in our config/test.exsto true.

config :my_app, MyApp.Endpoint,
  http: [port: 4001]
  server: true

That should do it! Before writing our first test, let’s define an IntegrationCase module, similar to the ModelCase and ConnCase provided by Phoenix, which will include all functionality we need to write our integration tests. Create the test/support/integration_case.ex file and add the following content:

defmodule MyApp.IntegrationCase do
  use ExUnit.CaseTemplate

  using do
    quote do
      use Hound.Helpers

      import Ecto.Model
      import Ecto.Query, only: [from: 2]
      import MyApp.Router.Helpers

      alias MyApp.Repo

      # The default endpoint for testing
      @endpoint MyApp.Endpoint

      hound_session
    end
  end

  setup tags do
    unless tags[:async] do
      Ecto.Adapters.SQL.restart_test_transaction(MyApp.Repo, [])
    end

    :ok
  end
end

There are a few lines that are worth commenting:

Exercise

Let’s test!

We’re testing a simple list of expenses from a city (that example was extracted from an app we have been working on, but its scope was reduced so we can follow the steps more easily).

Take a look at its template code:

<div>
  <%= form_for @conn, city_expense_path(@conn, :index, @city.id), [as: :q, method: :get], fn f -> %>
    <div>
      <label for="q_status">Status</label>
      <%=
        select f, :status, [{"Paid", "paid"}, {"Cancelled", "cancelled"}], prompt: "All" %>
    </div>

    <div>
      <label for="q_supplier">Supplier</label>
      <%= text_input f, :supplier %>
    </div>

    <%= submit "Submit" %>
  <% end %>
</div>

<table>
  <thead>
    <th>ID</th>
    <th>Status</th>
    <th>Supplier</th>
    <th>Value</th>
    <th>Date</th>
  </thead>
  <tbody>
    <%= for expense <- @expenses do %>
      <tr>
        <td><%= expense.id %></td>
        <td><%= expense.status %></td>
        <td><%= expense.supplier.name %></td>
        <td><%= expense.value %></td>
        <td><%= expense.date %></td>
      </tr>
    <% end %>
  </tbody>
</table>

We’ll need a new test file, let’s put it at test/integration/expenses_list_test.exs. To use Hound’s facilities, we’ll need to use the IntegrationCase module that we have previously created.

defmodule MyApp.ExpenseListTest do
  use MyApp.IntegrationCase

end

We’ll be using three private functions to help us making assertions and interacting with elements in our test file. The first one, expense_on_the_list, will check if a given expense, that is represented by the Ecto model called MyApp.Expense, is in there. The second function is just a helper for getting the expenses list and the third will help us interact with a select input within a form.

defmodule MyApp.ExpenseListTest do
  use MyApp.IntegrationCase

  # ...

  defp expense_on_the_list(expense, list) do
    list
    |> visible_text
    |> String.contains?(expense.id)
  end

  defp expense_list_items do
    find_element(:tag, "tbody")
    |> find_all_within_element(:tag, "tr")
  end

  defp select_status(form, status) do
    form
    |> find_within_element(:id, "q_status")
    |> input_into_field(status)
  end
end

To interact with an element, you’ll always need to find the element on the page and for this, you need to know Hound’s page helpers. I’ve noticed that we ended up using find_element and find_all_within_element most of the time to find the elements on the page or in a context (i.e inside a previously found element).

Since this test is about the City resource, we’ve created just this city and navigated to it directly on the setup, since this would be a requirement for all the tests in this file, and shared it with all the tests through the setup context.

setup do
  city = insert_city!(%{name: "Winterfell"})

  navigate_to("/cities/#{city.id}/expenses")

  {:ok, city: city}
end

Navigation is another important Hound module. It will help us go through our app easily and get info about the page, like the current_path() function that returns the path we’re navigating on that moment.

Now that we’re on the page, we’ll be interacting with the form, by finding elements that are within it and filling or selecting values on them.

test "filter by supplier", %{city: city} do
  supplier = insert_supplier!(%{name: "Ned Stark"})
  supplier_b = insert_supplier!(%{name: "Bell Tower Management"})
  expense = insert_expense!(%{supplier: supplier, city: city, status: "paid"})
  insert_expense!(%{supplier: supplier_b, city: city, status: "paid"})

  search_form = find_element(:tag, "form")
  search_form
  |> find_within_element(:id, "q_supplier")
  |> fill_field("Ned")

  submit_element(search_form)


  items = expense_list_items
  assert length(items) == 1
  assert expense_on_the_list(expense, items)
end

The module responsible for these tasks is Element. It has very useful functions, like fill_field we used above. All of its functions require an element.

In the previous example, the interactions with the form ended with submit_element, but if we need any other action on it after this, we would need to re-assign it (otherwise, we’ll get a ** (RuntimeError) Element does not exist in cache error), like in the following example:

test "filter by statuses", %{city: city} do
  supplier = insert_supplier!(%{name: "Jon Snow"})

  cancelled_expense = insert_expense!(%{supplier: supplier, city: city, status: "cancelled"})
  paid_expense = insert_expense!(%{supplier: supplier, city: city, status: "paid"})

  search_form = find_element(:tag, "form")
  select_status(search_form, "Cancelled")

  submit_element(search_form)

  items = expense_list_items
  assert length(items) == 1
  assert expense_on_the_list(cancelled_expense, items)

  search_form = find_element(:tag, "form")
  select_status(search_form, "Paid")
  submit_element(search_form)

  items = expense_list_items
  assert length(items) == 1
  assert expense_on_the_list(paid_expense, items)
end

Verify

Runtime

One of the things I’ve paid a lot of attention during this experience was the test suite runtime. As expected, it can get slow with acceptance tests. The original app is still really tiny and before adding the acceptance tests, the runtime was:

Finished in 0.6 seconds (0.5s on load, 0.1s on tests)
23 tests, 0 failures, 2 skipped

After including two tests (but with more interactions than the ones presented), it was noticeable the test suite became slower. It tripled the runtime.

Finished in 1.8 seconds (0.5s on load, 1.2s on tests)
25 tests, 0 failures, 2 skipped

This effect is actually expected. We know that acceptance tests are expensive and that they should be a small portion of your test pyramid

There are a few things that can make acceptance tests faster:

You will definitely need to write some acceptance tests, but give preference to controller tests in most scenarios and use acceptance tests for important flows of your app (take a look at the user journey concept, that can give you some good insights).

Web driver server execution

Currently, Hound doesn’t execute the browser automatically during tests. You’ll need to start it; otherwise, your tests will fail. There may be some workarounds to achieve it, if you’re on OS X, you can run Phantomjs as a service.

Teardown

I really enjoyed playing with Hound, and I found very simple to work with it. Also, I see it as a potential project if you’re considering contributing to an Open Source Project.

I hope this post was useful and gave you some ideas of how to write acceptance tests with Elixir and Phoenix. If you have any questions or suggestions, I’m all ears (or eyes).

Which tool have you recently found to be useful when writing tests in Elixir?


Subscribe to Elixir Radar

15 responses to “Writing Acceptance tests in Phoenix”

  1. Keith Salisbury says:

    Hey, I think this is a great post so thanks for sharing. Something that always bugs me when people do acceptance testing is the tight coupling of implementation to specification by explicit reference to dom elements such as `tbody` and `tr`. What happens if the UI changes to using divs instead? IMHO using dom data attributes or classes reduces the fragility somewhat, enabling the front end implementation to be changed without forcing the spec to change. Just MHO though, opinions may vary.

  2. Igor Florian says:

    Hey Keith, thanks for sharing your opinion! <3

    The code presented in the post is just an example for didact purpose. In a real-case scenario, we'd probably have a more complex DOM with style, so I’d use the same approach you mentioned.

    Although, I don't really have a problem referencing elements on the tests when the template is simple like that. Even if it breaks my suite when the change happens, I actually like my test suite reacting to change.

  3. George Opritescu says:

    Hi Igor, when I tried running the code with server: “phantomjs”, I kept receiving this error:

    ** (exit) exited in: GenServer.call(Hound.SessionServer, {:change_session, #PID, :default, %{}}, 60000)
    ** (EXIT) an exception was raised:
    ** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}
    (hound) lib/hound/request_utils.ex:45: Hound.RequestUtils.send_req/4
    (hound) lib/hound/session_server.ex:67: Hound.SessionServer.handle_call/3
    (stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:661: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

    Upon adding some IO.inspect statements in hound, I discovered that it needed a driver: “phantomjs” instead of the server option. After this change, it was running succesfully on my machine. Thank you for the article!

  4. josevalim says:

    To be clear, you need both the :server option and to start the driver. 🙂

  5. I think George meant that the config.exs keyword is “driver:” rather than “server:”, else Hound will assume selenium is running instead of phantomjs: https://github.com/HashNuke/hound/blob/8ed2c62c1da001eb0193cd9d3602c23ccddc6651/lib/hound/connection_server.ex#L5-L14

  6. josevalim says:

    You are both correct, of course! Thank you, I have fixed the article.

  7. Igor Florian says:

    Thanks for the feedback guys 🙂

  8. gorodetsky says:

    Great post!! I think the correct extenation for `test/support/integration_case` should be `ex` and not `exs`

  9. josevalim says:

    You are correct, it has been fixed!

  10. SamuelMV says:

    Interesting, but I’d prefer a more declarative way to define the acceptance tests. Maybe I’m too used to Cucumber for ATDD and BDD … but I’m thinking in write my tests in Cucumber with steps implementation in ruby or javascript (nodejs).

  11. Igor Florian says:

    That’s cool!

    I have a good feeling that these tools will evolve nicely. A thing that makes me believe that is the possibility of running async test using database (with Ecto). I believe that this will have a good impact on our test performance.

  12. Sharath Kumar says:

    hi Igor, I followed the steps as specified in this article. but i am getting this error.


    1) test GET / (HelloPhoenix.SampleTest)
    test/sample_test.exs:10
    ** (exit) exited in: GenServer.call(Hound.SessionServer, {:change_session, #PID, :default, %{}}, 60000)
    ** (EXIT) an exception was raised:
    ** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}
    (hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4
    (hound) lib/hound/session_server.ex:67: Hound.SessionServer.handle_call/3
    (stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:661: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

    stacktrace:
    (elixir) lib/gen_server.ex:564: GenServer.call/3
    test/sample_test.exs:8: HelloPhoenix.SampleTest.__ex_unit_setup_1/1
    test/sample_test.exs:1: HelloPhoenix.SampleTest.__ex_unit__/2

    12:48:24.548 [error] GenServer Hound.SessionServer terminating

    ** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}
    (hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4
    (hound) lib/hound/session_server.ex:67: Hound.SessionServer.handle_call/3
    (stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:661: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
    .
    Finished in 1.2 seconds (1.0s on load, 0.1s on tests)

    5 tests, 1 failure
    Randomized with seed 358869

    I am running selenium server using a docker container. I get the message as selenium up and running in that terminal.
    Then what might be the problem?? Please help. I cannot understand the error message

  13. Igor Florian says:

    Hey Sharath o

    It looks like your driver isn’t started. Maybe it’s because you’re starting it in docker and you have a different port for it.

    Try to pass the port you’re using in your config file:

    config :hound, driver: “selenium”, port: 5555

  14. Igor Florian says:

    Actually this could not be your solution, it looks like a session change problem.

    @sharathkumar3011:disqus, could you share your test file in a gist?

  15. Sharath Kumar says:

    @disqus_TKggmR6uN0:disqus Thanks for your reply. i have shared my test file and config file with this link. https://gist.github.com/sharathkumar3011/70d393fc35063ce06574 . Even if i start the selenium webdriver with ‘ java -jar selenium-server-standalone-2.51.0.jar ‘ command, I get same error.