Ecto v1.1 has been released. Ecto v1.1 brings improvements and bug fixes to Ecto but, more importantly, paves the way for the upcoming Ecto v2.0 release by deprecating functionality that has been said to be error prone or confusing by developers. This article will highlight both improvements and what to expect by Ecto 2.0.

For detailed information on the release, please check the CHANGELOG.

Let’s get started!

Better pipelines

The pipeline operator in Elixir is a great way to express a series of computations on top of a data structure. Given Ecto queries are just data structures, they are a perfect fit to be modified as part of pipelines.

Ecto has always supported both keyword and function queries. Let’s start with a keyword query:

from p in Post,
  where: p.author == "José",
  order_by: [desc: p.published_at],
  limit: 5

In Ecto v1.0, it could have be written using pipelines as follows:

Post
|> where([p], p.author == "José")
|> order_by([p], desc: p.published_at)
|> limit(5)

Ecto v1.1 improve pipelines by making the binding argument required only when working with associations and by allowing dynamic data to be given on more places. In v1.1, we can rewrite the example above as:

Post
|> where(author: "José")
|> order_by(desc: :published_at)
|> limit(5)

Most query operations like where, distinct, having support the syntax above. The only exceptions are select and group_by which will be tackled on Ecto v2.0.

No more models

Ecto.Model is being deprecated on Ecto v1.1. This aims to solve both conceptual and practical issues. Let’s discuss them.

What are models?

The big question imposed by Ecto.Model is: what is a model?

One thing is clear, Ecto did not provide models in the “traditional” sense. In OO languages, you would say a model can be instantiated and it would have methods that contain business logic. However, the data that comes from the database in Ecto is just data. It is an Elixir struct. It is not an Ecto model.

Working closely on Phoenix applications and on the Programming Phoenix book made it clear that, similar to controllers and views, models are not an entity. A model, a controller or a view (from the MVC pattern) are just group of functions that share similar responsibilities. They are just guidelines on how to group code towards a common purpose.

For those reasons, Ecto.Model is being deprecated in Ecto. At first, this implies Ecto data structures are now defined directly with Ecto.Schema. In Ecto v1.0:

defmodule MyApp.Post do
  use Ecto.Model

  schema "posts" do
    # ...
  end
end

From Ecto v1.1:

defmodule MyApp.Post do
  use Ecto.Schema

  schema "posts" do
    # ...
  end
end

Not only that, many of the functions in the Ecto.Model module have been moved to Ecto.

However, the biggest change with the deprecation of models is that model callbacks are being removed. To understand why this matters, let’s look at one Ecto feature that relied on callbacks and was rewritten to be a simple function.

Optimistic lock

Ecto provides optimistic locks on top of your schema. A simple implementation of optimistic lock uses an integer column, usually named lock_version, to store the current version of a given row. On update, Ecto would do a “compare and increase” operation. If the entry being updated had the same lock_version as in the database, the update operation succeeds and the lock_version is incremented. Otherwise, the update fails because the entry is stale.

On Ecto v1.0, optimistic_lock was enabled for the whole model:

defmodule MyApp.Post do
  use Ecto.Model

  schema "posts" do
    # ...
  end

  optimistic_lock :lock_version
end

This reveals the awkwardness behind callbacks. We are suddenly adding “behaviour” to our data structures. Not only that, because callbacks are enabled on all operations, we have no control over its use.

For example, what if you also provide an admin interface. Do you want the admin to be under the same lock constraints as regular users? More importantly, what if you want to trigger the lock only if some fields are changing? The only way to add this functionality is by growing the complexity of the optimistic_lock implementation by providing an ever growing set of complex options.

It happens Ecto has the perfect solution to this problem: changesets. For example, instead of defining validations in the model, you define per changeset:

@required_params [:title, :body]
@optional_params [:metadata]

def changeset(post, params \\ :empty) do
  post
  |> cast(params, @required_params, @optional_params)
  |> validate_length(:title, min: 3)
  |> validate_length(:metadata, min: 3)
end

In other words, a changeset is a data structure that controls the changes being sent to the database. This means that, if you have different roles in your application that work on different facets of the same data, you define different changesets for every operation.

Ecto v1.1 has replaced the optimistic_lock/1 macro implementation by a simple function that works on the changeset. If you want to add optimistic locking, just pipe your changeset in the optimistic_lock/2 with the lock column name:

def changeset(post, params \\ :empty) do
  post
  |> cast(params, @required_params, @optional_params)
  |> validate_length(:title, min: 3)
  |> validate_length(:metadata, min: 3)
  |> optimistic_lock(:lock_version)
end

Because it is only a function call, you have control of exactly when and where you can apply the lock. And ultimately that’s the fundamental problem with callbacks: it makes developers write functionality that is hard to compose.

Goodbye callbacks

After a quick search on GitHub, we quickly noticed that many developers relied on callbacks in many cases where changesets would suffice, introducing exactly the same problems we saw with optimistic_lock.

Furthermore, after_* callbacks provide their own set of issues. Because after_insert and after_update callbacks would still run inside a transaction, there is no guarantee the transaction that wraps both insert and update would actually commit. So someone would rely on such callbacks to index data or write to the filesystem while the transaction could rollback afterwards. Those mistakes are always bound to happen with callbacks because the execution flow is hidden from developers.

For all the reasons mentioned above, callbacks are deprecated in Ecto and will be removed by Ecto v2.0. Meanwhile we are working on solutions like Ecto.Multi that will give developers a data-driven approach to work with transactions.

Looking forward to 2.0

Besides the improvements already listed above, we are looking forward to many exciting new features on Ecto v2.0:

• Streamlined syntax for select and group_by
• A more efficient way of working with transactions via Ecto.Multi
• Many to many associations
• Automatic handling of associations and embeds on insert
• Automatic handling of both belongs_to and the upcoming many_to_many associations in insert and update
• An ownership system that allow tests that rely on the database to run concurrently by managing connection access

Furthermore, James Fish is working on a project called db_connection that will simplify adapter implementations and speed-up many operations by removing the amount of process communication and by providing client-side decoding. Early experiments showed performance improvements of ~25% when loading data. Such changes will also lead the way for running queries in parallel. For example, we will be able to preload associations in parallel instead of sequentially like today.

The best news is that we expect Ecto v2.0 to be simpler and smaller in size than Ecto v1.1 thanks to the removal of callbacks and the support being brought by db_connection.

We are really excited about future versions of Ecto and the improvements it will bring to everyday applications!