Why are they useful?

Inner (or nested) layouts are useful to reuse template code when you have subsections on your website. Usually, in Phoenix applications, we have /templates/layout/app.html.eex that shares template code between all pages. For example:

In the example above, we want to reuse the application logo, title, and navigation menu across all pages. We can solve that by using the Phoenix default layout and it works great. However, sometimes you need to create a new website section inside the parent layout. For example, imagine we want to add a help section to our Phoenix website. Look at the image below:

We still want to reuse the layout header across all pages, but we also want to reuse the left navigation and main content layout in all pages inside the help section. Let’s see how we can do that using Phoenix.

The Inner Layout Solution

I tried some solutions by looking at some examples online. After discussing with the Plataformatec team, we came up with an approach that’s very simple and extensible, thanks to Phoenix explicit layout rendering. The solution works like this:

• You create the nested layout template.
• In the nested layout, you set the parent template by calling a function.
• The parent layout is aware that is possible to have an inner layout and will render it.
• You can invoke your nested layout by using the Phoenix `put_layout/2` function.

The main goal here is to make the nested layout work like any Phoenix layout. This will make it familiar to any Phoenix developer.

 Preparing The Parent Template

First, let’s add a function that allows a template to render the parent layout:

# views/layout_view.ex
defmodule YourAppWeb.LayoutView do
  use YourAppWeb, :view

  def render_layout(layout, assigns, do: content) do
    render(layout, Map.put(assigns, :inner_layout, content))
  end
end

The render_layout/3 will render the given layout by assigning the contents of the inner layout given in the do argument. Now, in the parent layout, we need to render the inner layout contents or the controller view contents. Look at how you can do it:

<%# templates/layout/app.html.eex %>

<%# ... %>

  <%= Map.get(assigns, :inner_layout) || render @view_module, @view_template, assigns %>

<%# ... %>

With the code above, our parent layout is aware that there may be an inner layout and it should render its contents when available. That’s it! Now let’s see how we can use it.

Using the Nested Layout

You can create a nested layout template in the layouts folder and use it like this:

<%# templates/layout/nested_layout.html.eex %>

<%= render_layout "app.html", assigns do %>
  <%# Your HTML markup here %>
  <%= render @view_module, @view_template, assigns %>
  <%# More HTML markup here %>
<% end %>

We render the parent and put the inner content in the do/end blocks in our nested layout template. Any content outside of the do/end blocks will not be rendered. Don’t forget to call render @view_module, @view_template, assigns, or the contents of your controller’s action template will not be rendered.

Now you can use our nested layout by invoking the plug put_layout/2 in your controller or router. For example:

defmodule YourAppWeb.NestedContentController do
  use YourAppWeb, :controller

  plug :put_layout, :nested_layout

  def index(conn, params) do
    # stuff
  end
end

You can organize your nested layout files in a different way. For example, imagine you have a help section in your website and you want to keep the nested layout file in the help folder. In your HelpView you’ll need to import the render_layout/3 function, like this:

# views/help_view.ex

defmodule YourAppWeb.HelpView do
  use YourAppWeb, :view
  import YourAppWeb.LayoutView, only: [render_layout: 3]
end

After that, you can put your nested layout template in templates/help/layout.html.eex. In the controller or router, you can invoke the nested layout like this:

plug :put_layout, {YourAppWeb.HelpView, "layout.html"}

The code above will render your layout.html.eex template in views/help directory using YourAppWeb.HelpView module.

Wrapping Up

Inner layouts come in handy to reuse template code of subsections of your web application. You learned how simple it is to build that in Phoenix. Just be aware of not creating inner layouts of inner layouts. If you do that, your codebase will start to be very hard to maintain. You can see and try by yourself a sample Phoenix app running the inner layout example.

Have you implemented inner layout in a different way? Do you have any feature that you would like to see how we can build it in Phoenix? Let us know in your comments.

The post Nested Layouts With Phoenix first appeared on Plataformatec Blog.

<%= input f, :name %>
<%= input f, :address %>
<%= input f, :date_of_birth %>
<%= input f, :number_of_children %>
<%= input f, :notifications_enabled %>

Each generated input will have the proper markup and classes (we will use Bootstrap in this example), include the proper HTML attributes, such as required for required fields and validations, and show any input error.

The goal is to build this foundation in our own applications in very few lines of code, without 3rd party dependencies, allowing us to customize and extend it as desired as our application changes.

Setting up

Before building our input helper, let’s first generate a new resource which we will use as a template for experimentation (if you don’t have a Phoenix application handy, run mix phoenix.new your_app before the command below):

mix phoenix.gen.html User users name address date_of_birth:datetime number_of_children:integer notifications_enabled:boolean

Follow the instructions after the command above runs and then open the form template at “web/templates/user/form.html.eex”. We should see a list of inputs such as:

<div class="form-group">
  <%= label f, :address, class: "control-label" %>
  <%= text_input f, :address, class: "form-control" %>
  <%= error_tag f, :address %>
</div>

The goal is to replace each group above by a single <%= input f, field %> line.

Adding changeset validations

Still in the “form.html.eex” template, we can see that a Phoenix form operates on Ecto changesets:

<%= form_for @changeset, @action, fn f -> %>

Therefore, if we want to automatically show validations in our forms, the first step is to declare those validations in our changeset. Open up “web/models/user.ex” and let’s add a couple new validations at the end of the changeset function:

|> validate_length(:address, min: 3)
|> validate_number(:number_of_children, greater_than_or_equal_to: 0)

Also, before we do any changes to our form, let’s start the server with mix phoenix.server and access http://localhost:4000/users/new to see the default form at work.

Writing the input function

Now that we have set up the codebase, we are ready to implement the input function.

The YourApp.InputHelpers module

Our input function will be defined in a module named YourApp.InputHelpers (where YourApp is the name of your application) which we will place in a new file at “web/views/input_helpers.ex”. Let’s define it:

defmodule YourApp.InputHelpers do
  use Phoenix.HTML

  def input(form, field) do
    "Not yet implemented"
  end
end

Note we used Phoenix.HTML at the top of the module to import the functions from the Phoenix.HTML project. We will rely on those functions to build the markup later on.

If we want our input function to be automatically available in all views, we need to explicitly add it to the list of imports in the “def view” section of our “web/web.ex” file:

import YourApp.Router.Helpers
import YourApp.ErrorHelpers
import YourApp.InputHelpers # Let's add this one
import YourApp.Gettext

With the module defined and properly imported, let’s change our “form.html.eex” function to use the new input functions. Instead of 5 “form-group” divs:

<div class="form-group">
  <%= label f, :address, class: "control-label" %>
  <%= text_input f, :address, class: "form-control" %>
  <%= error_tag f, :address %>
</div>

We should have 5 input calls:

<%= input f, :name %>
<%= input f, :address %>
<%= input f, :date_of_birth %>
<%= input f, :number_of_children %>
<%= input f, :notifications_enabled %>

Phoenix live-reload should automatically reload the page and we should see “Not yet implemented” appear 5 times.

Showing the input

The first functionality we will implement is to render the proper inputs as before. To do so, we will use the Phoenix.HTML.Form.input_type function, that receives a form and a field name and returns which input type we should use. For example, for :name, it will return :text_input. For :date_of_birth, it will yield :datetime_select. We can use the returned atom to dispatch to Phoenix.HTML.Form and build our input:

def input(form, field) do
  type = Phoenix.HTML.Form.input_type(form, field)
  apply(Phoenix.HTML.Form, type, [form, field])
end

Save the file and watch the inputs appear on the page!

Wrappers, labels and errors

Now let’s take the next step and show the label and error messages, all wrapped in a div:

def input(form, field) do
  type = Phoenix.HTML.Form.input_type(form, field)

  content_tag :div do
    label = label(form, field, humanize(field))
    input = apply(Phoenix.HTML.Form, type, [form, field])
    error = YourApp.ErrorHelpers.error_tag(form, field) || ""
    [label, input, error]
  end
end

We used content_tag to build the wrapping div and the existing YourApp.ErrorHelpers.error_tag function that Phoenix generates for every new application that builds an error tag with proper markup.

Adding Bootstrap classes

Finally, let’s add some HTML classes to mirror the generated Bootstrap markup:

def input(form, field) do
  type = Phoenix.HTML.Form.input_type(form, field)

  wrapper_opts = [class: "form-group"]
  label_opts = [class: "control-label"]
  input_opts = [class: "form-control"]

  content_tag :div, wrapper_opts do
    label = label(form, field, humanize(field), label_opts)
    input = apply(Phoenix.HTML.Form, type, [form, field, input_opts])
    error = YourApp.ErrorHelpers.error_tag(form, field)
    [label, input, error || ""]
  end
end

And that’s it! We are now generating the same markup that Phoenix originally generated. All in 14 lines of code. But we are not done yet, let’s take things to the next level by further customizing our input function.

Customizing inputs

Now that we have achieved parity with the markup code that Phoenix generates, we can further extend it and customize it according to our application needs.

Colorized wrapper

One useful UX improvement is to, if a form has errors, automatically wrap each field in a success or error state accordingly. Let’s rewrite the wrapper_opts to the following:

wrapper_opts = [class: "form-group #{state_class(form, field)}"]

And define the private state_class function as follows:

defp state_class(form, field) do
  cond do
    # The form was not yet submitted
    !form.source.action -> ""
    form.errors[field] -> "has-error"
    true -> "has-success"
  end
end

Now submit the form with errors and you should see every label and input wrapped in green (in case of success) or red (in case of input error).

Input validations

We can use the Phoenix.HTML.Form.input_validations function to retrieve the validations in our changesets as input attributes and then merge it into our input_opts. Add the following two lines after the input_opts variable is defined (and before the content_tag call):

validations = Phoenix.HTML.Form.input_validations(form, field)
input_opts = Keyword.merge(validations, input_opts)

After the changes above, if you attempt to submit the form without filling the “Address” field, which we imposed a length of 3 characters, the browser won’t allow the form to be submitted. Not everyone is a fan of browser validations and, in this case, you have direct control if you want to include them or not.

At this point it is worth mentioning both Phoenix.HTML.Form.input_type and Phoenix.HTML.Form.input_validations are defined as part of the Phoenix.HTML.FormData protocol. This means if you decide to use something else besides Ecto changesets to cast and validate incoming data, all of the functionality we have built so far will still work. For those interested in learning more, I recommend checking out the Phoenix.Ecto project and learn how the integration between Ecto and Phoenix is done by simply implementing protocols exposed by Phoenix.

Per input options

The last change we will add to our input function is the ability to pass options per input. For example, for a given input, we may not want to use the type inflected by input_type. We can add options to handle those cases:

def input(form, field, opts \\ []) do
  type = opts[:using] || Phoenix.HTML.Form.input_type(form, field)

This means we can now control which function to use from Phoenix.HTML.Form to build our input:

<%= input f, :new_password, using: :password_input %>

We also don’t need to be restricted to the inputs supported by Phoenix.HTML.Form. For example, if you want to replace the :datetime_select input that ships with Phoenix by a custom datepicker, you can wrap the input creation into an function and pattern match on the inputs you want to customize.

Let’s see how our input functions look like with all the features so far, including support for custom inputs (input validations have been left out):

defmodule YourApp.InputHelpers do
  use Phoenix.HTML

  def input(form, field, opts \\ []) do
    type = opts[:using] || Phoenix.HTML.Form.input_type(form, field)

    wrapper_opts = [class: "form-group #{state_class(form, field)}"]
    label_opts = [class: "control-label"]
    input_opts = [class: "form-control"]

    content_tag :div, wrapper_opts do
      label = label(form, field, humanize(field), label_opts)
      input = input(type, form, field, input_opts)
      error = YourApp.ErrorHelpers.error_tag(form, field)
      [label, input, error || ""]
    end
  end

  defp state_class(form, field) do
    cond do
      # The form was not yet submitted
      !form.source.action -> ""
      form.errors[field] -> "has-error"
      true -> "has-success"
    end
  end

  # Implement clauses below for custom inputs.
  # defp input(:datepicker, form, field, input_opts) do
  #   raise "not yet implemented"
  # end

  defp input(type, form, field, input_opts) do
    apply(Phoenix.HTML.Form, type, [form, field, input_opts])
  end
end

And then, once you implement your own :datepicker, just add to your template:

<%= input f, :date_of_birth, using: :datepicker %>

Since your application owns the code, you will always have control over the inputs types and how they are customized. Luckily Phoenix ships with enough functionality to give us a head start, without compromising our ability to refine our presentation layer later on.

Summing up

This article showed how we can leverage the conveniences exposed in Phoenix.HTML to dynamically build forms using the information we have already specified in our schemas. Although the example above used the User schema, which directly maps to a database table, Ecto 2.0 allows us to use schemas to map to any data source, so the input function can be used for validating search forms, login pages, and so on without changes.

While we have developed projects such as Simple Form to tackle those problems in our Rails projects, with Phoenix we can get really far using the minimal abstractions that ship as part of the framework, allowing us to get most of the functionality while having full control over the generated markup.

The post Dynamic forms with Phoenix first appeared on Plataformatec Blog.

In my journey as a curious Elixir developer, I’ve come across this seemingly simple question a few times: which applications from third-party libraries do I need to declare in my mix.exs file?

Before we get down to the nitty-gritty of application dependencies, let’s first recap some basics of the initialization process in Elixir OTP applications.

In its essence, an OTP application is a reusable software component, consisting of multiple modules of its own and it can also depend on third-party code. These dependencies can be either library applications — a collection of standalone modules and functions, with no processes involved — or active applications, with their own life cycles and supervision trees. This distinction is quite subtle, it took me a while to fully grasp its implications.

Applications are defined with an application resource file, like my_app.app, which is a metadata file comprised of a single Erlang term. It includes all the information needed to start our application and is managed by Mix. If you want to dig deeper into that topic you can take a look at its documentation. Once we create a new application with mix new my_app a brand new mix.exs file is generated. This is the file that customizes how my_app.app will be assembled by Mix and it is the file we’ll be dealing with in the Elixir world. Here is what it looks like:

defmodule MyApp.Mixfile do
  use Mix.Project

  def project do
    [app: :my_app,
     version: "0.1.0",
     elixir: "~> 1.3",
     build_embedded: Mix.env == :prod,
     start_permanent: Mix.env == :prod,
     deps: deps()]
  end

  # Configuration for the OTP application
  #
  # Type "mix help compile.app" for more information
  def application do
    [applications: [:logger]]
  end

  # Dependencies can be Hex packages:
  #
  #   {:mydep, "~> 0.3.0"}
  #
  # Or git/path repositories:
  #
  #   {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"}
  #
  # Type "mix help deps" for more examples and options
  defp deps do
    []
  end
end

As we can see, there are some important pieces of information in it, such as the app name, version, Elixir version requirements and so on. In addition, the application function lets us explain what is required to boot our application: which other applications need to be started before ours, locally registered processes, which module represents the starting point of our application and also some default values for the application environment. By running mix help compile.app we can get more information about that function. The docs available for the Application behavior — which abstracts the initialization process per se — are pretty extensive and helpful as well.

So far, so good. Now things start to get interesting. Let’s take a look at the mix.exs file of a new Phoenix project. First we generate an application skeleton with mix phoenix.new hello, then we add {:exrm, "~> 1.0"} to our deps, including the Exrm tool for generating our releases.

defmodule Hello.Mixfile do
  use Mix.Project

  def project do
    [app: :hello,
     version: "0.0.1",
     elixir: "~> 1.3",
     # ...
     deps: deps()]
  end

  # Configuration for the OTP application.
  #
  # Type `mix help compile.app` for more information.
  def application do
    [mod: {Hello, []},
     applications: [:phoenix, :phoenix_pubsub, :phoenix_html, :cowboy, :logger, :gettext,
                    :phoenix_ecto, :postgrex]]
  end

  # ...

  defp deps do
    [{:phoenix, "~> 1.2.0"},
     {:phoenix_pubsub, "~> 1.0"},
     {:phoenix_ecto, "~> 3.0"},
     {:postgrex, ">= 0.0.0"},
     {:phoenix_html, "~> 2.6"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:gettext, "~> 0.11"},
     {:cowboy, "~> 1.0"},
     {:exrm, "~> 1.0"}]
  end

  # ...
end

Along with the :logger application we’ve seen before in our vanilla Elixir app, now we have a few other applications we depend upon: some from Phoenix itself, an HTTP server, some i18n utilities and a database driver too. If we pay close attention to the deps and the applications lists, we can see it is almost a 1 to 1 ratio. Every application relates to its dep counterpart, like cowboy, phoenix_pubsub, postgrex and others. The exceptions are :logger, which is pre-built as part of Elixir, exrm and phoenix_live_reload.

That mismatch confused me. I wondered if I wasn’t mixing them up (pun intended).

Library applications must be included too

Why should library-only applications like phoenix_html and gettext be included in my application function then? It’s not that they need to be booted up and start spawning processes or something. It turns out that our application function has more to do with Releases and runtime than with supervision trees. Yes, listing active applications ensures they are started before our application, but including library applications will also make sure they will be included in our installable release packages.

To support that claim, we can see that even exrm warns us when we forget to do so. Let’s try removing phoenix_html and cowboy from our applications:

defmodule Hello.Mixfile do
  # ...
  def application do
    [mod: {Hello, []},
     applications: [:phoenix, :phoenix_pubsub, :logger, :gettext, :phoenix_ecto, :postgrex]]
  end
end

Then make a new release:

mix deps.get
Running dependency resolution
* Getting phoenix (Hex package)
  Checking package (https://repo.hex.pm/tarballs/phoenix-1.2.0.tar)
  Using locally cached package
...

MIX_ENV=prod mix release
Building release with MIX_ENV=prod.

You have dependencies (direct/transitive) which are not in :applications!
The following apps should be added to :applications in mix.exs:

        phoenix_html => phoenix_html is missing from hello
        cowboy       => cowboy is missing from hello

Continue anyway? Your release may not work as expected if these dependencies are required! [Yn]:

Both dependencies would be missing from the final package :bomb:. We certainly don’t want that.

Development, test, docs and optional dependencies stay out

You might still ask, shouldn’t cowboy and phoenix_html be out of my control and get required automatically by phoenix? That way I’d just need to worry about phoenix as a single, rich dependency.

Actually, no. From the Phoenix framework standpoint, none of these applications are actually required. Cowboy isn’t a strict runtime dependency; we can freely run our Phoenix app on another compatible web server of our choice. As for Phoenix HTML and even Gettext, they are not strictly required either. We can totally build our Phoenix app without any HTML output or localization at all, consider an API-only app for example. It’s all up to us.

As we can see in Phoenix’s own mix.exs file, cowboy is declared as optional: true and phoenix_html as only: :test, gettext likewise is only: :test. The reason these dependencies are declared is that if we eventually need them in our app, they must be at least compatible with the Phoenix version currently in use.

That also explains why exrm and phoenix_live_reload can be kept out of the applications list. We don’t want phoenix_live_reload running in production, since there will be no live code reloading. As for exrm, it is a tool used exclusively to package our application, our business code has no need to know anything about it.

Cool. How come phoenix_live_reload works in development though? Well, when we start our application — with mix phoenix.start, for example — the code is available in the load path, but the phoenix_live_reload application is not started yet; it only starts when we connect to its channel. We can try it out by starting up our little Phoenix app and checking which applications get loaded.

iex -S mix phoenix.start
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

[info] Running Hello.Endpoint with Cowboy using http://localhost:4000/
Interactive Elixir (1.3.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Application.loaded_applications
[
  {:plug, 'A specification and conveniences for composable modules between web applications', '1.1.6'},
  {:hex, 'hex', '0.12.1'},
  ...
]

A list of all the applications currently loaded is returned when we call Application.loaded_applications. Now, after we make our first request by opening http://localhost:4000/ then calling Application.loaded_applications once again, we can see two new applications on that list: fs and phoenix_live_reload.

iex(2)> Application.loaded_applications
[
  {:plug, 'A specification and conveniences for composable modules between web applications', '1.1.6'},
  {:hex, 'hex', '0.12.1'},
  ...
  {:fs, 'VXZ FS Listener', '0.9.1'},
  ...
  {:phoenix_live_reload, 'Provides live-reload functionality for Phoenix',  '1.0.5'}
  ...
]

Declare your applications, even when you’re not using releases

I am deploying to Heroku and my application requires no release packaging whatsoever, should I still worry about my list of applications? Yes, definitely. Suppose we need to upgrade some dependencies, one of them used to be just a library application but now has become an active application, e.g. a cache library which now has to keep a reaper process to clean up stale data. Once we upgrade the dep for that undeclared application, a runtime bug is potentially introduced, given we have no guarantee our new dependencies’ applications will get started properly.

As a library author, you should take extra care

The same goes for libraries, regardless of open sourcing them or not. Remember that runtime application dependencies are transitive. Let’s say an application A depends on B, which in turn depends on C, the application responsible for ensuring C is started is B, unless C is an optional dependency. In that case, the author of B should document the pluggable nature of that optional dependency appropriately, or even provide code generators if applicable — as Phoenix does with cowboy.

Elixir tooling is our friend

One of the most noticeable benefits of Elixir is all the tools-support built around it. Not only in runtime, standing on the shoulders of Erlang, but primarily in development and build time. In case you overlooked the announcement of Elixir v1.3 recently released, you should definitely check it out. Among the improvements in dependency tracking are two new built-in mix tasks: app.tree and deps.tree. They are priceless timesavers in this routine task of keeping up with our dependencies.

Summing up

So, when we are evaluating these dependency issues individually, it all boils down to a few simple criteria.

For the deps function in our Mixfile:

• Always add it to deps if it is a direct dependency.
• Don’t forget the only: [...] option if not every environment needs it.
• Optional dependencies must be set as optional: true. It is always good to document and explain how they are supposed to be included.

For the applications key in our application function:

• Add it to applications if it is required at runtime in production.

All said and done, where can we go from here? Well, I’d say start at home, review your mix.exs. Is there anything important missing? Any unused dependency? Then you can start taking a closer look at your external dependencies’ Mixfiles. Are all runtime dependencies properly required? What about their environments? Once you’re there, take your time to contribute, give back. These tiny bits might seem worthless at first, but rest assured, as some clever man used to say, “success is in the details”.

The post Understanding deps and applications in your Mixfile first appeared on Plataformatec Blog.

After practicing deploy and tracing through nodes with Exrm, we got more comfortable knowing that there is a tool we can count on for managing production releases. Our next biggest concern was how could we make the deploy process more manageable. We couldn’t stop thinking about Capistrano, which we normally use for our Rails projects, then we found Edeliver. From Edeliver’s README description:

edeliver is based on deliver and provides a bash script to build and deploy Elixir and Erlang applications and perform hot-code upgrades.

Trying the whole deploy process manually was a bit harsh with some repetitive tasks. Using Edeliver for our first script/deploy was awkwardly easy! In the end, the whole manual process was simplified to:

    #!/bin/bash -ex

    BRANCH=${1:-master};

    mix edeliver build release --branch=BRANCH --verbose
    mix edeliver deploy release to production --verbose
    mix edeliver start production --verbose
    mix edeliver migrate production up --verbose

You’re probably going to need to customize this script, adapting it for your needs. In this case, we’re using this script only for production deploys, but you can customize it for staging servers pretty easily. We’ll explain how environments work further along.

How it works

As we saw before in the README quote, Edeliver makes pretty much everything with bash scripts. The Mix tasks we saw above will be executed with Elixir, but they’ll result in bash script instructions. Part of the instructions are executed in the scripts locally, which will build new instructions that will run remotely via RPC (Remote procedure call).

Let’s go deeper in some aspects of the lib.

Environments

Edeliver is a cool option for launching and distributing releases in multiple environments. It has a concept of three environments: build, staging and production. Among these, only the build environment should get a bit more of attention.

For a release to work in a server, it must have been built in a machine with the same architecture where the release will run. That’s because Edeliver uses Exrm for building its releases. Exrm will internally use its local NIFs (C functions used by Erlang) which may vary in a different architecture, thus causing, for example, an OSX release not working on Linux. You can read more about it in this Phoenix issue where people are discussing cross-compiling issues and there are some other issues in Exrm as well.

In order to use the build environment in our own development machine, it needs to use the same architecture of our staging and production servers, otherwise it won’t work.

To configure our environments, we’ll need to create a .deliver directory in our project and add a config file. Let’s see the suggested configs from Edeliver’s README for this file:

#!/usr/bin/env bash

APP="your-erlang-app" # name of your release

BUILD_HOST="build-system.acme.org" # host where to build the release
BUILD_USER="build" # local user at build host
BUILD_AT="/tmp/erlang/my-app/builds" # build directory on build host

STAGING_HOSTS="test1.acme.org test2.acme.org" # staging / test hosts separated by space
STAGING_USER="test" # local user at staging hosts
TEST_AT="/test/my-erlang-app" # deploy directory on staging hosts. default is DELIVER_TO

PRODUCTION_HOSTS="deploy1.acme.org deploy2.acme.org" # deploy / production hosts separated by space
PRODUCTION_USER="production" # local user at deploy hosts
DELIVER_TO="/opt/my-erlang-app" # deploy directory on production hosts

It’s pretty easy to configure our environments, we only need to make sure we have ssh permission for these servers specified. A cool thing about this whole configuration, as mentioned before, is that it’s possible to distribute the releases through several servers.

How can I include extra tasks to my deploy process?

What Edeliver does is generic for Elixir and Erlang applications. When we’re using Phoenix, for example, we need to run some tasks before generating the release. The most important tasks are brunch build --production and mix phoenix.digest so we can have our assets working on our release.

To make these work, we’ll need to define a hook in our .deliver/config file:

pre_erlang_clean_compile() {
  status "Preparing assets with: brunch build and phoenix.digest"
  __sync_remote "
    # runs the commands on the build host
    [ -f ~/.profile ] && source ~/.profile # load profile (optional)

    # fail if any command fails (recommended)
    set -e

    # enter the build directory on the build host (required)
    cd '$BUILD_AT'

    mkdir -p priv/static # required by the phoenix.digest task

    # installing npm dependencies
    npm install

    # building brunch
    brunch build --production

    # run your custom task
    APP='$APP' MIX_ENV='$TARGET_MIX_ENV' $MIX_CMD phoenix.digest $SILENCE
  "
} 

This was extracted from an Edeliver doc sample, which explains all the possibilities of hooks.

What about my environment variables?

We shared a tip on dealing with environment variables with Exrm in order to avoid exporting them in our build environment and it’s still up! Although, there’s an important detail we’ll need to pay attention.

In order to make the environments replaceable we needed to set RELX_REPLACE_OS_VARS=true before our start command. But that’s not possible with Edeliver because the start task runs locally.

mix edeliver start production

Then a possible solution is to export the RELX_REPLACE_OS_VARS in your production environment.

Considerations

Edeliver seems like a cool option for dealing with our releases and deploy process, I found it really easy to use. I didn’t enter in implementation details in this post, so make sure to read its README and docs, they’re very useful and well-explained.

This was a solution we found to ease our deploy process. How have you been managing your process? Did this post help you?

The post Deploying Elixir applications with Edeliver first appeared on Plataformatec Blog.

The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

In an Elixir project, the config goes in Mix.Config files. Some examples are: config.exs and environment config files (dev.exs, test.exs and prod.exs). These files are generally used by frameworks and libraries, but they have already proven useful for using mocks in our tests.

Let’s take an Ecto config as example:

# config/dev.exs
config :myapp, MyApp.Repo,
adapter: Ecto.Adapters.Postgres,
username: "postgres",
password: "postgres",
database: "myapp_dev",
hostname: "localhost",
pool_size: 10

A well-known approach is using Environment variables to hide and scope these values through different environments. To use it, we just need to have a configured variable and get it in our application. In Elixir we do this easily with System.get_env("ENV_VAR").

We could configure our last example with this approach:

# config/dev.exs
config :myapp, MyApp.Repo,
adapter: Ecto.Adapters.Postgres,
username: System.get_env("DB_USER"),
password: System.get_env("DB_PASSWORD"),
database: System.get_env("DB_NAME"),
hostname: System.get_env("DB_HOST"),
pool_size: 10

This way you won’t expose your database configs and will actually make things more dynamic. In development this is useful because the developers won’t need to make changes on this file, they’ll just need to export these vars.

So far this isn’t much different from what we do in other languages. However, things start to happen differently when we try to generate an Exrm release to deploy our app in production.

ENV vars need to be present during compile time

We all already know that Elixir is a compiled language. And in order to deploy or generate a release we need to compile our application. So everything is compiled, even our config files! Then, there’s an interesting behavior while compiling our config files.

Our System.get_env() calls will be evaluated during the compilation, so the binaries will be generated with the current value of the ENV var. Because of this, we need all of our environment variables to be exported during compiling. When we don’t have them, their value will be nil and we won’t be able to connect to our database, for example. This way, to build a release we’d need all our environment variables where we’re building it (our own machine or a build server).

If we’re working with Phoenix, there is an exception. Phoenix has a special way of configuring an HTTP port with ENV vars that evaluates it during runtime.

config :myapp, MyApp.Endpoint,
http: [port: {:system, "PORT"}],
# ...

It works great and data won’t be fixed in the release, but it’s specific for this Phoenix config. But don’t be sad! There are already some mature discussions around this in the Exrm repo, take a look, you may be able to help!

There’s a way when using Exrm release

I was chatting around Elixir Slack channel when our friend Ranelli mentioned that there was a simple technique that we could use to solve this when we build an Exrm release. Instead of using System.get_env in our configs, we must use "${ENV_VAR}". Then, we just need to run our release with RELX_REPLACE_OS_VARS=true.

RELX_REPLACE_OS_VARS=true rel/myapp/bin/myapp start

This will make our release to use the values represented by these special strings. I’ll explain.

An Exrm release has two important files: sys.config and vm.args. These files are responsible by the data used in production (usually what’s in config.exs and prod.exs) and specific configs that we can make of the Erlang VM respectively.

sys.config

[{sasl,[{errlog_type,error}]},
{logger,
[{console,
[{format,<<"$time $metadata[$level] $message\n">>},
{metadata,[request_id]}]},
{level,info}]},
{myapp,
[{'Elixir.MyApp.Endpoint',
[{root,<<"/Users/igorffs/src/myapp">>},
{render_errors,[{accepts,[<<"html">>,<<"json">>]}]},
{pubsub,
[{name,'Elixir.MyApp.PubSub'},
{adapter,'Elixir.Phoenix.PubSub.PG2'}]},
{http,[{port,<<"${PORT}">>}]},
{url,[{host,<<"localhost">>}]},
{cache_static_manifest,<<"priv/static/manifest.json">>},
{server,true},
{secret_key_base,
<<"${SECRET_KEYBASE}">>}]},
{'Elixir.MyApp.Repo',
[{adapter,'Elixir.Ecto.Adapters.Postgres'},
{username,<<"${DB_USER}">>},
{password,<<"${DB_PASSWORD}">>},
{database,<<"${DB_NAME}">>},
{hostname,<<"localhost">>},
{pool_size,10},
{port,<<"15432">>}]}]},
{phoenix,[{generators,[{migration,true},{binary_id,false}]}]}].

vm.args

## Name of the node
-sname myapp

## Cookie for distributed erlang
-setcookie myapp

## Heartbeat management; auto-restarts VM if it dies or becomes unresponsive
## (Disabled by default..use with caution!)
##-heart

## Enable kernel poll and a few async threads
##+K true
##+A 5

## Increase number of concurrent ports/sockets
##-env ERL_MAX_PORTS 4096

## Tweak GC to run more often
##-env ERL_FULLSWEEP_AFTER 10

Exrm is using a lib called relx under the hood to build its releases. When we exported RELX_REPLACE_OS_VARS=true relx will make a replace of the strings by their correspondent ENV var values in the config files.

{'Elixir.MyApp.Repo',
[{adapter,'Elixir.Ecto.Adapters.Postgres'},
{username,<<"${DB_USER}">>},
{password,<<"${DB_PASSWORD}">>},
{database,<<"${DB_NAME}">>},
{hostname,<<"localhost">>},
{pool_size,10},
{port,<<"15432">>}]}

You’ve noticed where our special strings are in the sys.config, and if you guessed that this process can be done manually, you got it! But this replace really makes things easier for us. Otherwise, we would have to edit every option in the file. It’s very important to mention, if you change those files, you’ll have to reboot your application.

Considerations

This subject is very important if we’re going on production. It concerned us a bit when we’ve noticed that we couldn’t have more dynamic configs. This replacement solution was a relief. Make sure to keep following the discussion I mentioned before, things are probably going to change after it.

Have you already been in trouble dealing with ENV vars? How did you solve it?

The post How to config environment variables with Elixir and Exrm first appeared on Plataformatec Blog.

In the past, we have discussed:

1. how to debug your application
2. how to trace systems with Erlyberly
3. how to use the observer to introspect applications.

The examples above always connected to systems running locally. Given Elixir’s and the Erlang VM focus on distributed systems, you may have wondered: can we use the VM capabilities to trace and observe remote nodes?

Certainly!

Your application runs as part of the Erlang Runtime System, which is often called a node, as it may connect to other machines. Before we establish such connections, let’s get to know some concepts and then configure our applications.

EPMD

Erlang Port Mapper Daemon, EPMD, acts as a name server on all hosts involved in distributed Erlang communications. When an Erlang node starts, the node has a name and it obtains an address from the host OS kernel. The default port the daemon runs on is 4369 but you can change it with the ERL_EPMD_PORT environment variable .

You can run epmd -names to check the port and the nodes connected:

user@localhost:~$ epmd -names
epmd: up and running on port 4369 with data:
name myapp at port 43316

SSH Port Forwarding

Depending on your firewall configuration, the port 4369 from EPMD is blocked by default. We will use port forwarding to redirect our local EPMD port to the remote EPMD with ssh: ssh user@myapp.com -L4369:localhost:4369.

Therefore, when we start a node locally, it will attempt to register itself to the EPMD running on port 4369, which is effectively forwarded to the remote EPMD. Once our local node registers itself to the remote EPMD, it will be able to find all remote nodes running on the remote EPMD.

Configuring the Phoenix application

Imagine we want to trace or observe a Phoenix project. In our case, our project was released using exrm and our release path in production has a directory called running-config. In this directory we can find the files sys.config and vm.args.

The file vm.args is responsible for configuring our application when it starts. Let’s change it as follows:

## Name of the node
-name myapp@127.0.0.1
-kernel inet_dist_listen_min 9001 inet_dist_listen_max 9001

## Cookie for distributed erlang (you want a really long cookie)
-setcookie my_cookie

We added a name to your application, set a port range where remote nodes may connect to and chose a cookie secret. If your server was already running, you will need to restart it after changing vm.args.

After restarting our application, we should see it registered in the remote EPMD:

user@localhost:~$ epmd -names
epmd: up and running on port 4369 with data:
name myapp at port 9001

Tracing application

After our application is started, we need to change our ssh command to forward to EPMD and our application ports: ssh user@myapp.com -L4369:localhost:4369 -L9001:localhost:9001.

Now let’s start the tracing tool locally with the proper cookie options. The tracing tool will register itself to the remote EPMD, via port forwarding, and find our remote application. Once the Erlyberly is started, you should see the following in the remote EPMD:

user@localhost:~$ epmd -names
epmd: up and running on port 4369 with data:
name myapp at port 9001
name erlyberly-1460466831146 at port 54420

Observing application

We can also observe a remote system using ssh port forwarding. One option is to establish a remote shell, as explained in the IEx documentation:

$ iex --name mylocalmachine@127.0.0.1 --cookie my_cookie --remsh myapp@127.0.0.1

Now you are connected directly to a remote node and you can introspect it as well as start tools like Observer.

Alternatively, you can start a new local shell with the same cookie as the remote node:

$ iex --name mylocalmachine@127.0.0.1 --cookie my_cookie
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.2.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(mylocalmachine@127.0.0.1)1> :observer.start()
:ok

The local shell should be registered in the remote EPMD alongside the remote system:

user@localhost:~$ epmd -names
epmd: up and running on port 4369 with data:
name mylocalmachine at port 50055
name myapp at port 9001

With Observer open, we can now change the inspected node using the menu ‘Nodes > Connect node’. In the prompt we can fill in the node name. In our example the node is myapp@127.0.0.1.

Troubleshooting

You may receive an error similar to the one below when you try to connect through Observer:

16:38:44.278 [error] [node: :"mylocalmachine@127.0.0.1", call: {:observer_backend, :sys_info, []}, reason: {:badrpc, {:EXIT, {:undef, [{:observer_backend, :sys_info, [], []}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 206]}]}}}]

This occurs because the :observer_backend is disabled. You can enable it by adding the :runtime_tools to your application mix.exs file. You can get more details in the Runtime tools documentation.

Do you use other techniques to connect to remote nodes? Share your tips with a comment below.

The post Tracing and observing your remote node first appeared on Plataformatec Blog.