This sharing often leads to confusion. Do they provide distinct behaviors? Do they overlap? For instance, is there any purpose to Elixir’s fault tolerance if Kubernetes also provides self-healing?

In this article, I will go over many of these topics and show how they are mostly complementary and discuss the rare case where they do overlap.

Self-healing

Kubernetes automatically restarts or replaces containers that fail. It can also kill containers that don’t respond to your user-defined health check. Similarly, in Erlang and Elixir, you structure your code with the help of supervisors, which automatically restart parts of your application in case of failures.

Kubernetes provides fault-tolerance within the cluster, Erlang/Elixir provide it within your application. To understand this better, let’s take an application that has to talk to a database (or any other external system). Most languages handle this by keeping a pool of database connections.

If your database goes offline, because of a bad configuration or a hardware failure, both the database and the Erlang/Elixir systems will respond negatively to health checks, which would cause Kubernetes to act and potentially relocate them. This is a node-wide failure and Kubernetes got your back.

However, what happens when part of your connections to the database are sporadically failing? For example, imagine your system is under load and you suddenly started running into connection limits, such as MySQL’s prepared statement limit. This failure likely won’t cause any health check to fail but your code will fail whenever one of its many connections reach said limit. Can you reason about this error today in your applications? Can you confidently say that the faulty connection will be dropped? Will another connection be started in place of the faulty one? Can you comfortably say this error won’t cascade in the application bringing the remaining of the connection pool down?

Erlang/Elixir’s abstractions for fault tolerance allow you to reason about those questions at the language level. It provides a mechanism for you to reason about connections, resources, in-memory state, background workers, etc. You can explicitly say how they are started, how they are shut down, and what should happen when things go wrong. These features can also be extremely helpful in face of partial failures. For example, imagine you have a news website and the live stock ticker is down. Should the website continue running, potentially serving stale data, or should everything crash down? The mental model provided by Erlang/Elixir allows us to reason about these scenarios. And of course, you can always let failures bubble up after a few retries, or even immediately, so it becomes a node-wide failure to be handled by K8s.

In a nutshell, Kubernetes and containers provide isolation and an ability to restart individual nodes when they fail, but it is not a replacement for isolation and fault handling within your own software, regardless of your language of choice. Using K8s and Erlang/Elixir allow you to apply similar self-healing and fault-tolerance principles in the large (cluster) and in the small (language/instance).

Service discovery and Distributed Erlang

The Erlang VM also provides Distributed Erlang, which allows you to exchange messages between different instances running on the same or different machines. In Elixir, this is as easy as:

for node <- Node.list() do
  # Send :hello_world message to named process "MyProcess" in each node
  send {node, MyProcess}, :hello_world
end
Code language: PHP (php)

When running in distributed mode (which is not a requirement in any way and you need to explicitly enable it), the Erlang VM will automatically serialize and deserialize the data as well as make sure the connection between nodes is alive, but it does not provide any node discovery. It is the programmer responsibility to say exactly where each node is located and connect the nodes together.

Luckily, Kubernetes provides service discovery out of the box. This means that, K8s allows us to fully automate the node discovery, which would otherwise be manual and error prone. Libraries like libcluster do exactly that (and rolling your own wouldn’t be complicated either). This is another great example of where Kubernetes and the Erlang VM complement each other!

However, you may still be wondering, is there a benefit to running Distributed Erlang when Kubernetes’ Service Discovery makes it relatively easy to have systems communicating with each other? Especially when considering RPC protocols such as Thrift, gRPC, and others?

When we are talking about different languages and different systems communicating with each other, picking one of the existing RPC mechanisms is likely the best choice, and they will also work fine with Erlang/Elixir. The scenario where the Erlang VM really shines, in my opinion, is for building homogeneous systems, i.e. when you have multiple deployments of the same container and they exchange information. For example, imagine you are building a real-time application when you want to track which users are in the same chat room, or in the same city block, or in the same mountain track. As users connect and disconnect and as nodes are brought up and down, you could somehow update the database or communicate via a complex RPC mechanism, while carefully watching the cluster for topology changes.

With the Erlang VM, you can just broadcast or exchange this information directly, without having to worry about serialization protocols, connection management, etc, as everything is provided by the VM. All without external dependencies. This is one of the many features that makes Phoenix a breeze to build distributed web-realtime systems.

Automated rollouts vs Hot code swapping

When it comes to deployment, Kubernetes automatically rolls out changes to your application or its configuration, avoiding changing all instances at the same time. At the same time, the Erlang VM supports hot code swapping, which allows you to change the code that is running in production within a single instance without shutting said instance down.

Those two deployments techniques are obviously conflicting. In fact, hot code swapping does not go well in general with the whole idea of immutable containers. Does it mean that Kubernetes and the Erlang VM are a poor fit? Not really, because you don’t have to use hot code swapping. In fact, most people do not. Most Elixir applications are deployed using blue-green, canary, or similar techniques.

The truth about hot code swapping is that it is actually complicated to pull off in practice. Let’s use the database as an example once again. When you are deploying a new version of your software, whenever you update your database, you should never perform destructive changes. For example, if you want to rename a column, you have to add a new column, migrate the data over, and then remove the column. If you just rename the column, then you will have failures whenever doing rollouts, because you will have two versions of the software running at the same (one using the old column and the other using the new one). In hot code swapping, we have precisely the same issue, except it applies to all states inside your application. Companies that use hot code swapping often report they spend as much time developing the software as testing the upgrades themselves.

Of course, it doesn’t mean hot code swapping is useless. The Erlang VM development is mostly driven by business needs and there was a legitimate need for hot code swapping. In particular, when building telephone switches, there is never an appropriate moment to shut down an instance for updates, because at any given time a system is full of long running connections, perhaps days or even weeks. So being able to upgrade a live system is extremely helpful. If you have a similar need, then hot code swapping may be an option. Another option is to have smarter clients and migrate client connections between nodes when deploying.

Hot code swapping can also be used under other circumstances, such as during development to provide live code loading, without a need to restart your server, or to replace smaller components in production that don’t require replacing the whole instance.

Configuration management and Configuration providers

Another feature provided by both Elixir and Kubernetes is configuration management. However, as seen before, they work at very distinct levels. While Elixir provides a unified API for configuring applications, it is relatively low-level. In a production system, you often want both configuration and secrets to be managed by higher level tools, such as the ones provided by Kubernetes. Luckily, you can incorporate said configuration tools into your deployment workflow with the help of Configuration Providers. This functionality is part of Elixir releases, which were officially made part of the Elixir language in version 1.9.

Stay alert: pod resources

When provisioning Erlang and Elixir with Kubernetes, it is important to stay alert to one particular configuration: pod resources.

When using other technologies, it is common practice to break a large node into a bunch of small pods/containers. For example, if you have a node with 8 cores, you could allocate half of each CPU to a pod and split the memory equally between them, on a total of 16 pods.

This approach makes sense in many technologies that cannot exploit CPU and I/O concurrency simultaneously. However, the Erlang VM excels at managing system resources and your system will most likely be more efficient if you assign large pods to your Erlang and Elixir applications instead of breaking it apart into a bunch of small ones.

If the Erlang VM is sharing a machine with other applications you may want to consider reducing busy waiting. By doing so, the VM will optimize for lower CPU usage, making it a better neighbor, but with slightly higher latencies.

Summing up

Kubernetes and the Erlang VM work at distinct levels. Kubernetes orchestrates within a cluster, the Erlang VM orchestrates at the language level within an instance. Fred Hebert summed up this distinction well in a tweet:

Still seeing bad comparisons between kubernetes and #Erlang/OTP. K8s is to OTP what region failover is to k8s. They operate on different layers of abstraction and impact distinct components.

OTP allows handling partial failures WITHIN an instance, something k8s can’t help with.— Fred Hebert (@mononcqc) April 29, 2019

If you are using Erlang/Elixir and you wonder how Kubernetes applies compared to other languages, you can use Kubernetes for the Erlang VM as you would with any other technology. Given that Erlang/Elixir software can typically scale both horizontally and vertically, it gives you many options on how you want to allocate your resources within K8s.

On other areas, Kubernetes and the Erlang VM can nicely complement each other, such as using K8s Service Discovery to connect Erlang VM instances. Of course, Distributed Erlang is not a requirement and Erlang/Elixir are great languages even for stateless apps, thanks to its scalability and reliability.

If you are one of the few who really need hot code swapping in production, then the Erlang VM may be one of the best platforms to do so, but keep in mind you will be straying away from the common path in both technologies.

Finally, if you appreciate Kubernetes and its concepts, you may enjoy working with Erlang and Elixir, as they will give you an opportunity to apply similar idioms on the small and on the large.

Thanks to Fernando Tapia Rico, Fred Hebert, George Guimarães, Tristan Sloughter, and Wojtek Mach for reviewing this article.

The post Kubernetes and the Erlang VM: orchestration on the large and the small first appeared on Plataformatec Blog.

After practicing deploy and tracing through nodes with Exrm, we got more comfortable knowing that there is a tool we can count on for managing production releases. Our next biggest concern was how could we make the deploy process more manageable. We couldn’t stop thinking about Capistrano, which we normally use for our Rails projects, then we found Edeliver. From Edeliver’s README description:

edeliver is based on deliver and provides a bash script to build and deploy Elixir and Erlang applications and perform hot-code upgrades.

Trying the whole deploy process manually was a bit harsh with some repetitive tasks. Using Edeliver for our first script/deploy was awkwardly easy! In the end, the whole manual process was simplified to:

    #!/bin/bash -ex

    BRANCH=${1:-master};

    mix edeliver build release --branch=BRANCH --verbose
    mix edeliver deploy release to production --verbose
    mix edeliver start production --verbose
    mix edeliver migrate production up --verbose

You’re probably going to need to customize this script, adapting it for your needs. In this case, we’re using this script only for production deploys, but you can customize it for staging servers pretty easily. We’ll explain how environments work further along.

How it works

As we saw before in the README quote, Edeliver makes pretty much everything with bash scripts. The Mix tasks we saw above will be executed with Elixir, but they’ll result in bash script instructions. Part of the instructions are executed in the scripts locally, which will build new instructions that will run remotely via RPC (Remote procedure call).

Let’s go deeper in some aspects of the lib.

Environments

Edeliver is a cool option for launching and distributing releases in multiple environments. It has a concept of three environments: build, staging and production. Among these, only the build environment should get a bit more of attention.

For a release to work in a server, it must have been built in a machine with the same architecture where the release will run. That’s because Edeliver uses Exrm for building its releases. Exrm will internally use its local NIFs (C functions used by Erlang) which may vary in a different architecture, thus causing, for example, an OSX release not working on Linux. You can read more about it in this Phoenix issue where people are discussing cross-compiling issues and there are some other issues in Exrm as well.

In order to use the build environment in our own development machine, it needs to use the same architecture of our staging and production servers, otherwise it won’t work.

To configure our environments, we’ll need to create a .deliver directory in our project and add a config file. Let’s see the suggested configs from Edeliver’s README for this file:

#!/usr/bin/env bash

APP="your-erlang-app" # name of your release

BUILD_HOST="build-system.acme.org" # host where to build the release
BUILD_USER="build" # local user at build host
BUILD_AT="/tmp/erlang/my-app/builds" # build directory on build host

STAGING_HOSTS="test1.acme.org test2.acme.org" # staging / test hosts separated by space
STAGING_USER="test" # local user at staging hosts
TEST_AT="/test/my-erlang-app" # deploy directory on staging hosts. default is DELIVER_TO

PRODUCTION_HOSTS="deploy1.acme.org deploy2.acme.org" # deploy / production hosts separated by space
PRODUCTION_USER="production" # local user at deploy hosts
DELIVER_TO="/opt/my-erlang-app" # deploy directory on production hosts

It’s pretty easy to configure our environments, we only need to make sure we have ssh permission for these servers specified. A cool thing about this whole configuration, as mentioned before, is that it’s possible to distribute the releases through several servers.

How can I include extra tasks to my deploy process?

What Edeliver does is generic for Elixir and Erlang applications. When we’re using Phoenix, for example, we need to run some tasks before generating the release. The most important tasks are brunch build --production and mix phoenix.digest so we can have our assets working on our release.

To make these work, we’ll need to define a hook in our .deliver/config file:

pre_erlang_clean_compile() {
  status "Preparing assets with: brunch build and phoenix.digest"
  __sync_remote "
    # runs the commands on the build host
    [ -f ~/.profile ] && source ~/.profile # load profile (optional)

    # fail if any command fails (recommended)
    set -e

    # enter the build directory on the build host (required)
    cd '$BUILD_AT'

    mkdir -p priv/static # required by the phoenix.digest task

    # installing npm dependencies
    npm install

    # building brunch
    brunch build --production

    # run your custom task
    APP='$APP' MIX_ENV='$TARGET_MIX_ENV' $MIX_CMD phoenix.digest $SILENCE
  "
} 

This was extracted from an Edeliver doc sample, which explains all the possibilities of hooks.

What about my environment variables?

We shared a tip on dealing with environment variables with Exrm in order to avoid exporting them in our build environment and it’s still up! Although, there’s an important detail we’ll need to pay attention.

In order to make the environments replaceable we needed to set RELX_REPLACE_OS_VARS=true before our start command. But that’s not possible with Edeliver because the start task runs locally.

mix edeliver start production

Then a possible solution is to export the RELX_REPLACE_OS_VARS in your production environment.

Considerations

Edeliver seems like a cool option for dealing with our releases and deploy process, I found it really easy to use. I didn’t enter in implementation details in this post, so make sure to read its README and docs, they’re very useful and well-explained.

This was a solution we found to ease our deploy process. How have you been managing your process? Did this post help you?

The post Deploying Elixir applications with Edeliver first appeared on Plataformatec Blog.

The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

In an Elixir project, the config goes in Mix.Config files. Some examples are: config.exs and environment config files (dev.exs, test.exs and prod.exs). These files are generally used by frameworks and libraries, but they have already proven useful for using mocks in our tests.

Let’s take an Ecto config as example:

# config/dev.exs
config :myapp, MyApp.Repo,
adapter: Ecto.Adapters.Postgres,
username: "postgres",
password: "postgres",
database: "myapp_dev",
hostname: "localhost",
pool_size: 10

A well-known approach is using Environment variables to hide and scope these values through different environments. To use it, we just need to have a configured variable and get it in our application. In Elixir we do this easily with System.get_env("ENV_VAR").

We could configure our last example with this approach:

# config/dev.exs
config :myapp, MyApp.Repo,
adapter: Ecto.Adapters.Postgres,
username: System.get_env("DB_USER"),
password: System.get_env("DB_PASSWORD"),
database: System.get_env("DB_NAME"),
hostname: System.get_env("DB_HOST"),
pool_size: 10

This way you won’t expose your database configs and will actually make things more dynamic. In development this is useful because the developers won’t need to make changes on this file, they’ll just need to export these vars.

So far this isn’t much different from what we do in other languages. However, things start to happen differently when we try to generate an Exrm release to deploy our app in production.

ENV vars need to be present during compile time

We all already know that Elixir is a compiled language. And in order to deploy or generate a release we need to compile our application. So everything is compiled, even our config files! Then, there’s an interesting behavior while compiling our config files.

Our System.get_env() calls will be evaluated during the compilation, so the binaries will be generated with the current value of the ENV var. Because of this, we need all of our environment variables to be exported during compiling. When we don’t have them, their value will be nil and we won’t be able to connect to our database, for example. This way, to build a release we’d need all our environment variables where we’re building it (our own machine or a build server).

If we’re working with Phoenix, there is an exception. Phoenix has a special way of configuring an HTTP port with ENV vars that evaluates it during runtime.

config :myapp, MyApp.Endpoint,
http: [port: {:system, "PORT"}],
# ...

It works great and data won’t be fixed in the release, but it’s specific for this Phoenix config. But don’t be sad! There are already some mature discussions around this in the Exrm repo, take a look, you may be able to help!

There’s a way when using Exrm release

I was chatting around Elixir Slack channel when our friend Ranelli mentioned that there was a simple technique that we could use to solve this when we build an Exrm release. Instead of using System.get_env in our configs, we must use "${ENV_VAR}". Then, we just need to run our release with RELX_REPLACE_OS_VARS=true.

RELX_REPLACE_OS_VARS=true rel/myapp/bin/myapp start

This will make our release to use the values represented by these special strings. I’ll explain.

An Exrm release has two important files: sys.config and vm.args. These files are responsible by the data used in production (usually what’s in config.exs and prod.exs) and specific configs that we can make of the Erlang VM respectively.

sys.config

[{sasl,[{errlog_type,error}]},
{logger,
[{console,
[{format,<<"$time $metadata[$level] $message\n">>},
{metadata,[request_id]}]},
{level,info}]},
{myapp,
[{'Elixir.MyApp.Endpoint',
[{root,<<"/Users/igorffs/src/myapp">>},
{render_errors,[{accepts,[<<"html">>,<<"json">>]}]},
{pubsub,
[{name,'Elixir.MyApp.PubSub'},
{adapter,'Elixir.Phoenix.PubSub.PG2'}]},
{http,[{port,<<"${PORT}">>}]},
{url,[{host,<<"localhost">>}]},
{cache_static_manifest,<<"priv/static/manifest.json">>},
{server,true},
{secret_key_base,
<<"${SECRET_KEYBASE}">>}]},
{'Elixir.MyApp.Repo',
[{adapter,'Elixir.Ecto.Adapters.Postgres'},
{username,<<"${DB_USER}">>},
{password,<<"${DB_PASSWORD}">>},
{database,<<"${DB_NAME}">>},
{hostname,<<"localhost">>},
{pool_size,10},
{port,<<"15432">>}]}]},
{phoenix,[{generators,[{migration,true},{binary_id,false}]}]}].

vm.args

## Name of the node
-sname myapp

## Cookie for distributed erlang
-setcookie myapp

## Heartbeat management; auto-restarts VM if it dies or becomes unresponsive
## (Disabled by default..use with caution!)
##-heart

## Enable kernel poll and a few async threads
##+K true
##+A 5

## Increase number of concurrent ports/sockets
##-env ERL_MAX_PORTS 4096

## Tweak GC to run more often
##-env ERL_FULLSWEEP_AFTER 10

Exrm is using a lib called relx under the hood to build its releases. When we exported RELX_REPLACE_OS_VARS=true relx will make a replace of the strings by their correspondent ENV var values in the config files.

{'Elixir.MyApp.Repo',
[{adapter,'Elixir.Ecto.Adapters.Postgres'},
{username,<<"${DB_USER}">>},
{password,<<"${DB_PASSWORD}">>},
{database,<<"${DB_NAME}">>},
{hostname,<<"localhost">>},
{pool_size,10},
{port,<<"15432">>}]}

You’ve noticed where our special strings are in the sys.config, and if you guessed that this process can be done manually, you got it! But this replace really makes things easier for us. Otherwise, we would have to edit every option in the file. It’s very important to mention, if you change those files, you’ll have to reboot your application.

Considerations

This subject is very important if we’re going on production. It concerned us a bit when we’ve noticed that we couldn’t have more dynamic configs. This replacement solution was a relief. Make sure to keep following the discussion I mentioned before, things are probably going to change after it.

Have you already been in trouble dealing with ENV vars? How did you solve it?

The post How to config environment variables with Elixir and Exrm first appeared on Plataformatec Blog.

We have been using Exrm here at Plataformatec. It is the chosen tool for deploying our projects and the contributors are doing a great job maintaining and developing new features. However, since Exrm is in its early days, it’s natural that there are some features that aren’t completely covered yet. One of these features was the possibility to run migration tasks on a release.

An Exrm release is a package with binaries, dependencies and tools so you can run and manage it independently. All needed dependencies are compiled within the releases. Even Erlang and Elixir binaries are compiled within the release, so we don’t need to install it in the machine. It’s a self-contained package! As we don’t have the source code included in our release, we can’t run Mix tasks because it needs a mix.exs file.

To run our migrations without Mix, an approach is using the Exrm console. There you can call Ecto Migrator.

However, since the 1.0.3 version, we have a command task available for Exrm releases. With this interface, we can call a module defined within our Elixir application.

Example:

rel/my_app/bin/my_app command <Module> <function> <args>

How does it work?

Well, under the hood, what this task does is basically the same we needed to do before. It opens a console and runs the code we’ve sent. Although, it runs an Erlang console (through the erlexec binary contained in the release).

To the migration!

We’ve defined an Elixir module in our app and a migrate function:

defmodule Release.Tasks do
  def migrate do
    {:ok, _} = Application.ensure_all_started(:my_app)

    path = Application.app_dir(:my_app, "priv/repo/migrations")

    Ecto.Migrator.run(MyApp.Repo, path, :up, all: true)

    :init.stop()
  end
end

Notice that in order to run the migrations, we’ll need to ensure that our app or the supervisor tree that Ecto counts in. In this example, we’ve just ensured that the entire app has started, but you could get only ecto, postgrex and logger. Also, we need to call :init.stop in order to close the console, otherwise our buffer will remain opened, waiting for something.

To call this function we’ll need to:

rel/my_app/bin/my_app command Elixir.Release.Tasks migrate

We need to include the Elixir namespace because we’re running it in an Erlang console, as explained above. Otherwise, we can’t reach the module.

Another possible approach is defining an Erlang module instead of Elixir’s:

defmodule :release_tasks do
  def migrate do
    {:ok, _} = Application.ensure_all_started(:my_app)

    path = Application.app_dir(:my_app, "priv/repo/migrations")

    Ecto.Migrator.run(MyApp.Repo, path, :up, all: true)

    :init.stop()
  end
end

This way our call gets a bit prettier:

rel/my_app/bin/my_app command release_tasks migrate

Conclusion

Since Elixir’s early days, several people are contributing and developing great tools. We’re certain that this is one of the reasons that makes Elixir so attractive. And because of this, things are getting better for us in Elixir when dealing with releases and deployments.

This blog post is highly inspired by an Exrm issue discussion. Make sure to read or cooperate with it if you feel like it. There are plenty of great tips in it and they’ll probably have tons more as this idea grows.

How are you currently dealing with these tasks? Are there any tips to share?

The post Running migration in an Exrm release first appeared on Plataformatec Blog.

Depending on which deployment decision you take, it will impact your team and how you use your version control system.

As a consultancy, we have worked in lots of projects, and together with our customers we have devised many ways to deliver their product to their customers. We have seen some patterns, advantages and challenges on each way, and today I would like to discuss some of them:

1. The open-source way
2. The pipeline way
3. The support branch way
4. The feature toggle way

The open-source way

In the open-source world most of the times we should maintain many versions of a same product. For example, Ruby on Rails has many versions released, like 2, 3.2, 4.0, 4.1. Bugs happens, new features are created, so news releases must be delivered, but in a set of supported released versions. Still on RoR example, the supported releases are 4.1, 4.0 and 3.2 (http://guides.rubyonrails.org/maintenance_policy.html). But how this releasing works?

The most recent version of the product is maintained on the master branch, the previous major releases have their own branches. On RoR we have the master for the new 4.2 version release, and we still have 4-1-stable, 4-0-stable, 3-2-stable branches. By following this organization we can easily apply changes on the desired versions.

For each release a tag must be created. For example, there’s a tag for RoR 4.0.0, one for 4.1.0 and so on. With tags it is possible to navigate between the released versions and if the worst happens, like losing the “version-stable” branch, it’s easy to create another one from the last released version.

Usually, a web product has just one version to be maintained, so usually we don’t need the “version-stable” branches. We can keep the new product releases on the master branch and generate a tag when we want to package and release a new product version.

When we need a “hotfix” or an urgent feature and the master is not ready yet for production, we can easily create a branch from the latest tagged version, apply the desired changes, create a new tag and release a new version. By the way, using this way you can release any branch you want. All that manipulation of applying commits, merging and creating branches and tags, can be simplified with a powerful version control system like “Git”.

Strong points

• The flexible package creation and release.
• It works for large teams, primarily when there are planned releases.

The challenges

• It requires the infra to be flexible enough to support it.
• It requires time to control what can be merged on master before the package creation.
• It will require good ability with the version control system.
• Manage the release versions.

Common phrases with this approach

• “Sorry pals, I forgot to apply that hot fix patch on master”. – A developer after releasing a new product version.

The pipeline way

Using a pipeline in your deployment process means you have well defined steps and all steps must be accomplished in order to do a deployment.

Usually the steps are: run the automated tests, release on test/qa environment, create the release tag, release on production. After the steps are defined, you need some software that allows the team to automate some steps and to add the option of requiring a approval for the next step. For example, you only want to release the package to production when your QA team and PO have approved the version on QA.

Having a pipeline means your master branch is always production ready. Any new code inserted on master branch must pass the pipeline, then it is very important that the team and the pipeline to be able to quickly respond to changes.

One important precaution is to be sure that only wanted features are on master, because all code on master will always be deployed on the next software release. I have seen some confusion in this aspect, because some companies are a bit more bureaucratic and have some strict deployment rules.

Per example, a feature can only go to production when the QA team and PO approves. Placing the QA process on pipeline means you’ll put the feature on master that is not ready yet for production. It generates a problem I see regularly with this approach, I’ll call it for now the “release lock”.

Release lock

The release lock can be better understood with an example:

1. The developers have released on master the Feature A and B.
2. The QA team finds a BUG on Feature A.
3. The developers release a Feature C on master
4. The developers fix the BUG.
5. PO approves Feature A and B and wants a deploy.

Can we deploy a release with Feature C untested and unapproved by PO? Most of the time the companies answer is no.

Some approaches we can do here are: revert Feature C commits, or simply lock code changes on master and the entire team focus on finishing the release with Feature C.

Of course there are other approaches we can incorporate in the pipeline process, and we’ll see a further discussion about it later in this post.

Strong points

• With the pipeline it is easier for everyone in the team to understand how the deployment works.
• The pipeline gives accessibility for anyone in the team to launch a release.
• Less time managing versioning.

The challenges

• You lose the flexibility to deploy any branches.
• In large teams and some companies rules can produce release locks often.

Common phrases with this approach

• “What? This feature is already in production?” – A member from QA team looking at the version in production.
• “Hey, stop merging on master! We need a release today!” – The product manager after receiving pressure from stakeholders.

The support branch way

You define a branch as QA or test branch. This branch is useful to test features which aren’t ready for production. For example, if your deployment process needs QA/PO features approval.

With this approach you’ll send the features to support branch. When the feature is approved you send them to the master branch and use the normal deployment process flow. It is important to do a regression test of the merged features on master. When a regression test finds a defect, it is easy to apply them on master, since the master is clear of unwanted features.

While using this approach you should be aware that now you have two points of integration. Resolving the merging conflicts twice is a problem that can happen often, but the most troublesome issue is when the integration on the support branch breaks the application functionality.

When the support branch integration is broken you need to analyze when and where the patch with the fix will be applied.

If you apply the fix on support branch, you must remember to apply it on master again.

The other option is to find out which changes made the features incompatible together. When you find that, you can apply those changes to the branch that doesn’t have those changes. Be aware that depending on the way you do this, you may end up needing to release both features together.

Strong points

• You can easily apply the support branch in any deployment process you choose.
• You mitigate the release lock problem.

The challenges

• Two points of features integration.
• Requires good abilities with version control system.
• The support branch maintenance.

Common phrases with this approach

• “Gosh! I need to fix that merge conflict again.” – Developer merging on master the QA approved feature.

The feature toggle way

Sometimes you are using the feature toggle without knowing you are doing it. For example: when you enable some features only for beta users, or enable some application routes only for some network IPs, or create A/B tests for the users. In general, you are using a feature toggle when your application is restricting in some way the access for some features.

To solve the release lock problem, some teams apply the feature toggle in every feature that needs approval. In this way the team can send unapproved features to production. When the feature is approved, the feature can be turned on without a new deploy. Be aware that sending turned off features to production also means that unapproved feature code will be sended too.

Creating toggles for your features means more code and tests to control what your software does with and without the toggles. Each feature toggle you add increases the complexity and the maintenance cost of your codebase.

Thus, it is important to remove them after the feature approval before it starts damaging your software. I know what you’re thinking, yes, it is true, most of the times the QA/PO team want to test the toggle removal and you might face the release lock problem again.

Strong points

• You can use the pipeline with only one point of integration.
• You reduce the release lock problem.

The Challenges

• You may increase the cost of development because of maintenance and removal of feature toggles.
• The feature toggle management.

Common phrases with this approach

• “What this method does?” – A developer asking for his pal.
• “It depends, which toggles are on?” – The answer of the first question.

Conclusion

Most of the challenges of each deployment process requires team engagement and organization. It’s hard to decide which one is better, because it fully depends on how your team adapt to the process.

Each person perceive problems in different ways. What is a huge matter for someone, for the other is just a small itch. But if you still want answers for “What is the best option?”, I would say it is the same answer for “Which challenge your team will endure more?”.

Given that you prefer one deployment process over the others, I still think that no one should be attached totally with one process forever. Your problems can change, your team can change, your company rules can change, your application can change. Therefore, your deployment process should change together to deal with the new challenges. You can change your deployment in many ways, for example mixing ideas from each the of processes we discussed.

I’m curious to know how your team deliver features. If you use one of these options, if you mix them or if you do something different. If you want share this knowledge, please leave a comment below.

The post The pros and cons of 4 deployment process techniques first appeared on Plataformatec Blog.