After practicing deploy and tracing through nodes with Exrm, we got more comfortable knowing that there is a tool we can count on for managing production releases. Our next biggest concern was how could we make the deploy process more manageable. We couldn’t stop thinking about Capistrano, which we normally use for our Rails projects, then we found Edeliver. From Edeliver’s README description:

edeliver is based on deliver and provides a bash script to build and deploy Elixir and Erlang applications and perform hot-code upgrades.

Trying the whole deploy process manually was a bit harsh with some repetitive tasks. Using Edeliver for our first script/deploy was awkwardly easy! In the end, the whole manual process was simplified to:

    #!/bin/bash -ex

    BRANCH=${1:-master};

    mix edeliver build release --branch=BRANCH --verbose
    mix edeliver deploy release to production --verbose
    mix edeliver start production --verbose
    mix edeliver migrate production up --verbose

You’re probably going to need to customize this script, adapting it for your needs. In this case, we’re using this script only for production deploys, but you can customize it for staging servers pretty easily. We’ll explain how environments work further along.

How it works

As we saw before in the README quote, Edeliver makes pretty much everything with bash scripts. The Mix tasks we saw above will be executed with Elixir, but they’ll result in bash script instructions. Part of the instructions are executed in the scripts locally, which will build new instructions that will run remotely via RPC (Remote procedure call).

Let’s go deeper in some aspects of the lib.

Environments

Edeliver is a cool option for launching and distributing releases in multiple environments. It has a concept of three environments: build, staging and production. Among these, only the build environment should get a bit more of attention.

For a release to work in a server, it must have been built in a machine with the same architecture where the release will run. That’s because Edeliver uses Exrm for building its releases. Exrm will internally use its local NIFs (C functions used by Erlang) which may vary in a different architecture, thus causing, for example, an OSX release not working on Linux. You can read more about it in this Phoenix issue where people are discussing cross-compiling issues and there are some other issues in Exrm as well.

In order to use the build environment in our own development machine, it needs to use the same architecture of our staging and production servers, otherwise it won’t work.

To configure our environments, we’ll need to create a .deliver directory in our project and add a config file. Let’s see the suggested configs from Edeliver’s README for this file:

#!/usr/bin/env bash

APP="your-erlang-app" # name of your release

BUILD_HOST="build-system.acme.org" # host where to build the release
BUILD_USER="build" # local user at build host
BUILD_AT="/tmp/erlang/my-app/builds" # build directory on build host

STAGING_HOSTS="test1.acme.org test2.acme.org" # staging / test hosts separated by space
STAGING_USER="test" # local user at staging hosts
TEST_AT="/test/my-erlang-app" # deploy directory on staging hosts. default is DELIVER_TO

PRODUCTION_HOSTS="deploy1.acme.org deploy2.acme.org" # deploy / production hosts separated by space
PRODUCTION_USER="production" # local user at deploy hosts
DELIVER_TO="/opt/my-erlang-app" # deploy directory on production hosts

It’s pretty easy to configure our environments, we only need to make sure we have ssh permission for these servers specified. A cool thing about this whole configuration, as mentioned before, is that it’s possible to distribute the releases through several servers.

How can I include extra tasks to my deploy process?

What Edeliver does is generic for Elixir and Erlang applications. When we’re using Phoenix, for example, we need to run some tasks before generating the release. The most important tasks are brunch build --production and mix phoenix.digest so we can have our assets working on our release.

To make these work, we’ll need to define a hook in our .deliver/config file:

pre_erlang_clean_compile() {
  status "Preparing assets with: brunch build and phoenix.digest"
  __sync_remote "
    # runs the commands on the build host
    [ -f ~/.profile ] && source ~/.profile # load profile (optional)

    # fail if any command fails (recommended)
    set -e

    # enter the build directory on the build host (required)
    cd '$BUILD_AT'

    mkdir -p priv/static # required by the phoenix.digest task

    # installing npm dependencies
    npm install

    # building brunch
    brunch build --production

    # run your custom task
    APP='$APP' MIX_ENV='$TARGET_MIX_ENV' $MIX_CMD phoenix.digest $SILENCE
  "
} 

This was extracted from an Edeliver doc sample, which explains all the possibilities of hooks.

What about my environment variables?

We shared a tip on dealing with environment variables with Exrm in order to avoid exporting them in our build environment and it’s still up! Although, there’s an important detail we’ll need to pay attention.

In order to make the environments replaceable we needed to set RELX_REPLACE_OS_VARS=true before our start command. But that’s not possible with Edeliver because the start task runs locally.

mix edeliver start production

Then a possible solution is to export the RELX_REPLACE_OS_VARS in your production environment.

Considerations

Edeliver seems like a cool option for dealing with our releases and deploy process, I found it really easy to use. I didn’t enter in implementation details in this post, so make sure to read its README and docs, they’re very useful and well-explained.

This was a solution we found to ease our deploy process. How have you been managing your process? Did this post help you?

The post Deploying Elixir applications with Edeliver first appeared on Plataformatec Blog.

The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

In an Elixir project, the config goes in Mix.Config files. Some examples are: config.exs and environment config files (dev.exs, test.exs and prod.exs). These files are generally used by frameworks and libraries, but they have already proven useful for using mocks in our tests.

Let’s take an Ecto config as example:

# config/dev.exs
config :myapp, MyApp.Repo,
adapter: Ecto.Adapters.Postgres,
username: "postgres",
password: "postgres",
database: "myapp_dev",
hostname: "localhost",
pool_size: 10

A well-known approach is using Environment variables to hide and scope these values through different environments. To use it, we just need to have a configured variable and get it in our application. In Elixir we do this easily with System.get_env("ENV_VAR").

We could configure our last example with this approach:

# config/dev.exs
config :myapp, MyApp.Repo,
adapter: Ecto.Adapters.Postgres,
username: System.get_env("DB_USER"),
password: System.get_env("DB_PASSWORD"),
database: System.get_env("DB_NAME"),
hostname: System.get_env("DB_HOST"),
pool_size: 10

This way you won’t expose your database configs and will actually make things more dynamic. In development this is useful because the developers won’t need to make changes on this file, they’ll just need to export these vars.

So far this isn’t much different from what we do in other languages. However, things start to happen differently when we try to generate an Exrm release to deploy our app in production.

ENV vars need to be present during compile time

We all already know that Elixir is a compiled language. And in order to deploy or generate a release we need to compile our application. So everything is compiled, even our config files! Then, there’s an interesting behavior while compiling our config files.

Our System.get_env() calls will be evaluated during the compilation, so the binaries will be generated with the current value of the ENV var. Because of this, we need all of our environment variables to be exported during compiling. When we don’t have them, their value will be nil and we won’t be able to connect to our database, for example. This way, to build a release we’d need all our environment variables where we’re building it (our own machine or a build server).

If we’re working with Phoenix, there is an exception. Phoenix has a special way of configuring an HTTP port with ENV vars that evaluates it during runtime.

config :myapp, MyApp.Endpoint,
http: [port: {:system, "PORT"}],
# ...

It works great and data won’t be fixed in the release, but it’s specific for this Phoenix config. But don’t be sad! There are already some mature discussions around this in the Exrm repo, take a look, you may be able to help!

There’s a way when using Exrm release

I was chatting around Elixir Slack channel when our friend Ranelli mentioned that there was a simple technique that we could use to solve this when we build an Exrm release. Instead of using System.get_env in our configs, we must use "${ENV_VAR}". Then, we just need to run our release with RELX_REPLACE_OS_VARS=true.

RELX_REPLACE_OS_VARS=true rel/myapp/bin/myapp start

This will make our release to use the values represented by these special strings. I’ll explain.

An Exrm release has two important files: sys.config and vm.args. These files are responsible by the data used in production (usually what’s in config.exs and prod.exs) and specific configs that we can make of the Erlang VM respectively.

sys.config

[{sasl,[{errlog_type,error}]},
{logger,
[{console,
[{format,<<"$time $metadata[$level] $message\n">>},
{metadata,[request_id]}]},
{level,info}]},
{myapp,
[{'Elixir.MyApp.Endpoint',
[{root,<<"/Users/igorffs/src/myapp">>},
{render_errors,[{accepts,[<<"html">>,<<"json">>]}]},
{pubsub,
[{name,'Elixir.MyApp.PubSub'},
{adapter,'Elixir.Phoenix.PubSub.PG2'}]},
{http,[{port,<<"${PORT}">>}]},
{url,[{host,<<"localhost">>}]},
{cache_static_manifest,<<"priv/static/manifest.json">>},
{server,true},
{secret_key_base,
<<"${SECRET_KEYBASE}">>}]},
{'Elixir.MyApp.Repo',
[{adapter,'Elixir.Ecto.Adapters.Postgres'},
{username,<<"${DB_USER}">>},
{password,<<"${DB_PASSWORD}">>},
{database,<<"${DB_NAME}">>},
{hostname,<<"localhost">>},
{pool_size,10},
{port,<<"15432">>}]}]},
{phoenix,[{generators,[{migration,true},{binary_id,false}]}]}].

vm.args

## Name of the node
-sname myapp

## Cookie for distributed erlang
-setcookie myapp

## Heartbeat management; auto-restarts VM if it dies or becomes unresponsive
## (Disabled by default..use with caution!)
##-heart

## Enable kernel poll and a few async threads
##+K true
##+A 5

## Increase number of concurrent ports/sockets
##-env ERL_MAX_PORTS 4096

## Tweak GC to run more often
##-env ERL_FULLSWEEP_AFTER 10

Exrm is using a lib called relx under the hood to build its releases. When we exported RELX_REPLACE_OS_VARS=true relx will make a replace of the strings by their correspondent ENV var values in the config files.

{'Elixir.MyApp.Repo',
[{adapter,'Elixir.Ecto.Adapters.Postgres'},
{username,<<"${DB_USER}">>},
{password,<<"${DB_PASSWORD}">>},
{database,<<"${DB_NAME}">>},
{hostname,<<"localhost">>},
{pool_size,10},
{port,<<"15432">>}]}

You’ve noticed where our special strings are in the sys.config, and if you guessed that this process can be done manually, you got it! But this replace really makes things easier for us. Otherwise, we would have to edit every option in the file. It’s very important to mention, if you change those files, you’ll have to reboot your application.

Considerations

This subject is very important if we’re going on production. It concerned us a bit when we’ve noticed that we couldn’t have more dynamic configs. This replacement solution was a relief. Make sure to keep following the discussion I mentioned before, things are probably going to change after it.

Have you already been in trouble dealing with ENV vars? How did you solve it?

The post How to config environment variables with Elixir and Exrm first appeared on Plataformatec Blog.