In the past, we have discussed:

1. how to debug your application
2. how to trace systems with Erlyberly
3. how to use the observer to introspect applications.

The examples above always connected to systems running locally. Given Elixir’s and the Erlang VM focus on distributed systems, you may have wondered: can we use the VM capabilities to trace and observe remote nodes?

Certainly!

Your application runs as part of the Erlang Runtime System, which is often called a node, as it may connect to other machines. Before we establish such connections, let’s get to know some concepts and then configure our applications.

EPMD

Erlang Port Mapper Daemon, EPMD, acts as a name server on all hosts involved in distributed Erlang communications. When an Erlang node starts, the node has a name and it obtains an address from the host OS kernel. The default port the daemon runs on is 4369 but you can change it with the ERL_EPMD_PORT environment variable .

You can run epmd -names to check the port and the nodes connected:

user@localhost:~$ epmd -names
epmd: up and running on port 4369 with data:
name myapp at port 43316

SSH Port Forwarding

Depending on your firewall configuration, the port 4369 from EPMD is blocked by default. We will use port forwarding to redirect our local EPMD port to the remote EPMD with ssh: ssh user@myapp.com -L4369:localhost:4369.

Therefore, when we start a node locally, it will attempt to register itself to the EPMD running on port 4369, which is effectively forwarded to the remote EPMD. Once our local node registers itself to the remote EPMD, it will be able to find all remote nodes running on the remote EPMD.

Configuring the Phoenix application

Imagine we want to trace or observe a Phoenix project. In our case, our project was released using exrm and our release path in production has a directory called running-config. In this directory we can find the files sys.config and vm.args.

The file vm.args is responsible for configuring our application when it starts. Let’s change it as follows:

## Name of the node
-name myapp@127.0.0.1
-kernel inet_dist_listen_min 9001 inet_dist_listen_max 9001

## Cookie for distributed erlang (you want a really long cookie)
-setcookie my_cookie

We added a name to your application, set a port range where remote nodes may connect to and chose a cookie secret. If your server was already running, you will need to restart it after changing vm.args.

After restarting our application, we should see it registered in the remote EPMD:

user@localhost:~$ epmd -names
epmd: up and running on port 4369 with data:
name myapp at port 9001

Tracing application

After our application is started, we need to change our ssh command to forward to EPMD and our application ports: ssh user@myapp.com -L4369:localhost:4369 -L9001:localhost:9001.

Now let’s start the tracing tool locally with the proper cookie options. The tracing tool will register itself to the remote EPMD, via port forwarding, and find our remote application. Once the Erlyberly is started, you should see the following in the remote EPMD:

user@localhost:~$ epmd -names
epmd: up and running on port 4369 with data:
name myapp at port 9001
name erlyberly-1460466831146 at port 54420

Observing application

We can also observe a remote system using ssh port forwarding. One option is to establish a remote shell, as explained in the IEx documentation:

$ iex --name mylocalmachine@127.0.0.1 --cookie my_cookie --remsh myapp@127.0.0.1

Now you are connected directly to a remote node and you can introspect it as well as start tools like Observer.

Alternatively, you can start a new local shell with the same cookie as the remote node:

$ iex --name mylocalmachine@127.0.0.1 --cookie my_cookie
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.2.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(mylocalmachine@127.0.0.1)1> :observer.start()
:ok

The local shell should be registered in the remote EPMD alongside the remote system:

user@localhost:~$ epmd -names
epmd: up and running on port 4369 with data:
name mylocalmachine at port 50055
name myapp at port 9001

With Observer open, we can now change the inspected node using the menu ‘Nodes > Connect node’. In the prompt we can fill in the node name. In our example the node is myapp@127.0.0.1.

Troubleshooting

You may receive an error similar to the one below when you try to connect through Observer:

16:38:44.278 [error] [node: :"mylocalmachine@127.0.0.1", call: {:observer_backend, :sys_info, []}, reason: {:badrpc, {:EXIT, {:undef, [{:observer_backend, :sys_info, [], []}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 206]}]}}}]

This occurs because the :observer_backend is disabled. You can enable it by adding the :runtime_tools to your application mix.exs file. You can get more details in the Runtime tools documentation.

Do you use other techniques to connect to remote nodes? Share your tips with a comment below.

The post Tracing and observing your remote node first appeared on Plataformatec Blog.

In this blog post, we’ll show you other 2 options: IEx.pry/0 and :debugger.

IEx.pry

The name “pry” is an old friend in the Ruby ecosystem but it has a different behavior in Elixir. Let’s create a new project with mix to try it out:

$ mix new example
$ cd example

Now let’s write some sample code in lib/example.ex:

require IEx;

defmodule Example do
  def double_sum(x, y) do
    IEx.pry
    hard_work(x, y)
  end

  defp hard_work(x, y) do
    2 * (x + y)
  end
end

Now start a new IEx session and invoke our new function:

$ iex -S mix
Interactive Elixir (1.2.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Example.double_sum(1, 2)

IEx.pry/0 is built on top of IEx. Although it isn’t a traditional debugger since you can’t step, add breakpoints and so forth, it’s a good tool for non-production debugging. It runs in the caller process, blocking the caller and allowing us to access its binding (variables), verify its lexical information and access the process information. You can finish your “pry” session by calling respawn, which starts a new IEx shell.

You can find more information at IEx.pry doc.

Debugger

If you need a breakpoint feature, we can use the :debugger module that ships with Erlang. Let’s make a change in our example to be more didactic:

defmodule Example do
  def double_sum(x, y) do
    hard_work(x, y)
  end

  defp hard_work(x, y) do
    x = 2 * x
    y = 2 * y

    x + y
  end
end

Now we can start our debugger:

$ iex -S mix
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Compiled lib/example.ex
Interactive Elixir (1.2.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :debugger.start()
{:ok, #PID<0.87.0>}
iex(2)> :int.ni(Example)
{:module, Example}
iex(3)> :int.break(Example, 3)
:ok
iex(4)> Example.double_sum(1,2)

When you started the debugger, a Graphical User Interface must have opened in your machine. We called :int.ni(Example) to prepare our module for debugging and then added a breakpoint to line 3 with :int.break(Example, 3). After we call our function, we can see our process with break status in the debugger:

The process is blocked as in IEx.pry/0. We can add a new breakpoint in the monitor window, inspect the code, see the variables and navigate it in steps.

Debugger has more options and command instructions that you can use. Take a look at Debbuger doc for more information.

Troubleshooting

You may have some problems when executing :int.ni(Example) in the example above:

iex(2)> :int.ni(Example)
** Invalid beam file or no abstract code: 'Elixir.Example'

Before the upcoming Erlang 19 version, the debugger did not have the heuristic that traverses the module source attribute applied. If you are not on the latest Erlang version, you can update the debugger manually with the following steps:

1. Download the file int.erl from the PR.
2. Compile it with erlc -o . int.erl.
3. Overwrite lib/debugger/ebin/int.beam in your Erlang installation with the new compiled file.

In the next post, we will see a tracing technique that doesn’t block the caller process.

What about you? What are the tools that you are using to debug your Elixir applications?

The post Debugging techniques in Elixir first appeared on Plataformatec Blog.

One of the things I really like in Elixir projects is that its dependencies are all explicitly included in the deps/ directory. Every time we’re curious about how a dependency works, we can just look at deps/lib-name.

After working on a project for a while, I’ve noticed a strange behavior in one of our dependencies. I’ve opened the deps/lib-name directory in my editor, inserted a couple IO.inspect calls, and recompiled my dependencies with:

mix deps.compile

That seems logical, doesn’t it? For my surprise, I didn’t get the expected behavior. The reason is straight-forward. Elixir projects (and its dependencies) are by default compiled for dev, test and prod environments. These compiled files are located in _build/, in a directory that holds the environment name (_build/dev/, _build/test/ and _build/prod/). Accessing the applied changes in the source code depends on which environment we’re running it.

The kind of task we execute is deterministic. Running a test will make our _build/test/ compiled files to be used, just like iex -S mix would use build/dev/ files and so on.

What I did before would work if I tried to call the function I was inspecting through iex -S mix. That’s only because Elixir’s default environment is dev. In order to make those changes to be visible in my tests, I’d have to:

MIX_ENV=test mix deps.compile

There’s a simple test we can run to understand this better:

1. Create a simple Elixir project with mix new project
2. Run iex -S mix
3. Take a look at the _build directory and notice that we now have a _build/dev directory with that project’s compiled file
4. Run mix test and notice that we now have a _build/test

Using :path option

Instead of running mix deps.compile for every change, there is a more convenient alternative.

When we declare our dependencies in the mix.exs file we need to give it the library name and its version. We can also give it some extra options, among them there is a specific option that will help us when debugging dependencies: path.

  defp deps do
    [{:plug, path: "deps/plug"},
     ...]
  end

When we set :path, our dependency will be automatically recompiled by our project, as mentioned in Mix.Tasks.Deps docs (we can also access it through mix help deps).

Path and in umbrella dependencies are automatically recompiled by the parent project whenever they change.

This way, every time I make a change and run a Mix task like mix test or iex -S mix, the dependency will be recompiled without having to run the compile task over and over again. With the path option we can also omit the version because it’ll be retrieved from the project being addressed.

An important consideration here is that we can inform any path in this option, it doesn’t need to be a source code in the deps/ directory. It could, for example, point to a checkout of a dependency in our machine. This is an excellent option when we’re working on an open source project. We can test the changes we intend to submit (or to find an issue) inside a real project more easily.

A dependency of other dependencies

We know that Phoenix Framework uses Plug for managing its requests but it’s not listed in our project dependencies. That’s because Plug is a Phoenix dependency.

In order to inspect Plug we’d have to include it on our deps function and use the override: true option. Otherwise Mix will warn us that there is a dependency conflict.

  defp deps do
    [{:phoenix, "~> 1.1.4"},
     {:postgrex, ">= 0.0.0"},
     {:phoenix_ecto, "~> 2.0"},
     {:phoenix_html, "~> 2.4"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:cowboy, "~> 1.0"},
     {:plug, path: "deps/plug", override: true}]
  end

Conclusion

I believe that Elixir has already proven that explicitness is a great asset. Keeping project’s dependencies in deps/ has already proved useful when searching for code and documentation.

In case you find a bug in any of your dependencies, we strongly recommend that you submit a pull request back to its repository. You’ll help other developers that are going through the same issue and you’ll he the community growth.

What about you? Are there any tips you’d like to share for debugging Elixir dependencies?

The post Inspecting, changing and debugging Elixir project dependencies first appeared on Plataformatec Blog.