Plataformatec Blog http://blog.plataformatec.com.br Plataformatec's place to talk about Ruby, Ruby on Rails and software engineering Thu, 16 Apr 2015 12:06:00 +0000 en-US hourly 1 http://wordpress.org/?v=4.1.1 Plataformatec at Erlang Factory San Francisco 2015 http://blog.plataformatec.com.br/2015/04/plataformatec-at-erlang-factory-san-francisco-2015/ http://blog.plataformatec.com.br/2015/04/plataformatec-at-erlang-factory-san-francisco-2015/#comments Thu, 16 Apr 2015 12:00:38 +0000 http://blog.plataformatec.com.br/?p=4480 »]]> Erlang Factory

Two weeks ago José Valim and I went to Erlang Factory San Francisco and we had a great time. Erlang Factory San Francisco is one of the biggest events in the Erlang community. One of the things that got me most excited about it was how many people were excited and talking about Elixir throughout the event.

Elixir talks

The event itself had a whole track dedicated to Elixir. I may be biased but the talk that I liked the most was Valim’s one. In his talk, he explains what Elixir is about and how the language foundation is enabling Elixir maintainers to explore different strategies for laziness, parallelism and distribution. You should definitely watch it:

Besides Valim’s talk, there were many others. Jamie Winsor shared his experience in developing a whole Massively Multiplayer Online Game platform using Elixir. In another room, Frank Hunleth introduced the Nerves Project to run Elixir in embedded devices.

I’ll not comment on each one of the talks because this week’s Elixir Radar will be a special edition focusing only on the Elixir talks at Erlang Factory. Subscribe to Elixir Radar and stay tuned! Also, you can go ahead and watch all of the those Elixir talks on Youtube.

People using Elixir in production

The highlight of the event was chatting with other Elixir developers. It was awesome to see how many developers are falling in love with Elixir.

Not only that, many of them are already running Elixir in production! So Valim and I decided to grab our phones, order a microphone from Amazon and record some video interviews to show their experience running Elixir in different environments, from marketing and video companies to messaging systems and embedded devices.

We’re still editing those videos, but as soon as we get them done, we’re going to publish them in our Youtube channel and will announce them here and in our Twitter account.

Stay tuned for more Elixir news!

Subscribe here to Elixir Radar

]]>
http://blog.plataformatec.com.br/2015/04/plataformatec-at-erlang-factory-san-francisco-2015/feed/ 0
Build embedded and start permanent in Elixir 1.0.4 http://blog.plataformatec.com.br/2015/04/build-embedded-and-start-permanent-in-elixir-1-0-4/ http://blog.plataformatec.com.br/2015/04/build-embedded-and-start-permanent-in-elixir-1-0-4/#comments Fri, 10 Apr 2015 12:00:54 +0000 http://blog.plataformatec.com.br/?p=4454 »]]> Elixir v1.0.4 ships with two new important options for new projects. If you generate a new application with mix new, you will see in your mix.exs:

[build_embedded: Mix.env == :prod,
 start_permanent: Mix.env == :prod]

Although those options were originally meant to be in Elixir v1.1, we have decided to bring them into v1.0.4 and do a new release. In this post, we will explain why.

Protocol consolidation

One of Elixir’s most important features are protocols. Protocols allow developers to write code that accept any data type, dispatching to the appropriate implementation of the protocol at runtime. For example:

defprotocol JSON do
  def encode(data)
end

defimpl JSON, for: List do
  def encode(list) do
    # ...
  end
end

We have written about protocols before and I recently explored on my Erlang Factory talk the foundation protocols have allowed us to build.

However, in order to play nicely with the dynamic nature of the Erlang VM where modules can be loaded at any time by the VM, as well as any protocol implementation, protocols need to check on every dispatch if a new implementation is available for any given data type.

While we would gladly pay this price in development as it gives developers flexibility, we would like to avoid such in production as deployments gives us a consolidated view of all modules in the system allowing us to skip those runtime checks. For this reason, Elixir provides a feature called protocol consolidation, that consolidates all protocols with their implementations, giving protocols a fast dispatch to use in production.

Prior to Elixir v1.0.4, protocol consolidation had to be manually invoked by calling mix compile.protocols, which would consolidate protocols into a predefined directory, and this directory had to be explicitly added to your load path when starting your project. Due to the manual nature of such commands, a lot of developers ended-up not running them in production, or were often confused when doing so.

For this reason, Elixir v1.0.4 introduces a :consolidate_protocols option to your projects which will take care of consolidating and loading all protocols before your application starts. This option is also set to true when :build_embedded is true.

Build embedded

When compiling your projects, Elixir will place all compiled artifacts into the _build directory:

_build/
  dev/
    lib/
      your_app/
        ebin/
        priv/
      dep1/
      dep2/
  test/
  prod/

Many of those applications and dependencies have artifacts in their source that are required during runtime. Such artifacts are placed in the priv directory in Elixir applications. By default, Elixir will symlink to their source directories during development.

In production, though, we could copy those contents to avoid symlink traversal, embedding all relevant files to run your application into the _build directory, without a need for their sources.

That’s what the :build_embedded option does and it defaults to true in production for new applications.

Start permanent

Elixir code is packaged into applications. For example, each entry we saw under _build/dev/lib above is a different application. When an application is started, it can be started in one of the three following modes:

  • permanent – if app terminates, all other applications and the entire node are also terminated
  • transient – if app terminates with :normal reason, it is reported but no other applications are terminated. If a transient application terminates abnormally, all other applications and the entire node are also terminated
  • temporary – if app terminates, it is reported but no other applications are terminated

The default mode is temporary, which again, makes sense for development. For example, our test library called ExUnit, is also an application. If the application being tested crashes, we still want the ExUnit application to continue running in order to finish all tests and generate the proper reports. In this case, you definitely do not want your application to run as permanent.

However, in production, once your application crashes permanently, beyond recovery, we want the whole node to terminate, otherwise whatever you have monitoring your application at the operating system level won’t notice any change.

The :start_permanent option starts your application as :permanent and it defaults to true in production for new applications.

Summing up

Those new options have been introduced into Elixir v1.0.4 because they are very important for running Elixir in production. They bring more performance and stability to your Elixir-based systems.

There are other smaller changes in this release, like support for Erlang 17.5 and 18.0-rc1, as well as bug fixes. Check the release notes for more information and enjoy!

Subscribe here to Elixir Radar

]]>
http://blog.plataformatec.com.br/2015/04/build-embedded-and-start-permanent-in-elixir-1-0-4/feed/ 2
Continuous communication http://blog.plataformatec.com.br/2015/03/continuous-communication/ http://blog.plataformatec.com.br/2015/03/continuous-communication/#comments Wed, 11 Mar 2015 12:00:45 +0000 http://blog.plataformatec.com.br/?p=4443 »]]> After continuous integration, which evolved to discrete integration, and continuous delivery, why not try continuous communication to avoid misleading messages inside your team?

Why communication matters?

It’s known that communication issues results in many software development problems.

Some agile frameworks, such as Scrum, have well defined communication activities like daily meetings and sprint plannings.

Communication is an important subject, not only for software development, but also in many other areas. There are some social frameworks, such as colletive impact and collaboration for impact, that use continuous communication to help people achieve their goals.

Here at Plataformatec, we take this quite seriously as well. We’re always evolving our communication practices, to achieve our goals and to keep our culture strong.

Continuous communication at Plataformatec

We talk a lot to each other. We use Campfire for team chatting, Basecamp for persistent messages and hangouts or skype for face-to-face calls.

Besides the day-to-day communication, we use some agile practices such as daily meetings and retrospectives. Some of these meetings/tools are used inside projects, others inter-projects and finally, we have company-wide meetings. Let’s see how these practices evolved to fit our current Modus Operandi.

Inside projects

Dailies

In order to keep everyone inside the project on the same page we use daily meetings. An important detail here is the client presence. The client’s daily presence helps to avoid re-working tasks, since they know what is happening on a day-to-day basis.

Weekly project meetings

Once a week, every project team also has a meeting with the Account Manager. Our Account Manager is the person who helps the team solve problems at a higher level, typically overseeing multiple projects at a time, so they are not directly involved in the project’s day-to-day tasks.

We have an open communication channel with them, but it’s important to make a follow-up with all the team together. This is to ensure that the team exchanges more information and keep the Account Manager better informed about the project. Then he can provide the team with better advice on how to solve day-to-day challenges.

Inter-projects

Dashboard meetings

Every Friday our company daily meeting gets a new attribution. In addition to company announcements, we also share the projects status, what happened in the current week, new technical challenges, applied techniques, releases delivered, and so on.

Everyone in projects assigns a grade to the project, which can be a value between -1 and 2. A -1 grade means “We are performing really bad” and 2 means “We are performing really well”. Together with the grade we write a brief explanation justifying it.

We start the dashboard meeting analyzing the current and past grades so we can check how the project is evolving. This is important because everyone can have a basic knowledge of how the other projects are moving on and new things that our colleagues are using. Having a big picture of each project paves a way for advice and knowledge sharing.

Company-wide

Company Dailies

Back when the Plataformatec team was smaller, we used to do dailies to exchange projects information and other company announcements, such as new employees or new clients. But as the team got bigger, we had more and more projects teams, and exchanging all that information became complex.

Our dailies couldn’t involve sharing project information anymore, so we changed its purpose, and today we use it to share information at a company level. Information about projects is now shared in the dashboard meetings, weekly.

Weekly reports

We also have a weekly summary email, where we point our project highlights, new leads, new employees, next events. It’s a very good tool for those who lost some meeting during the week.

Monthly retrospectives

Back when our team was smaller, we used to have biannual retrospectives, where we listed our good points, things to be maintained, and points we needed to improve. We also used the retrospectives meetings to do team appreciation, where everyone has the opportunity to congratulate or appreciate someone else’s work in the team.

We used to have one day for this meeting, but as new members joined the team, it was not enough anymore. So we changed to have a monthly retrospective, focused on the good points and the points to improve. Every month we select one or more subjects to be discussed, and quarterly we re-prioritize and list new subjects, if necessary. The appreciation now is done in the Biannual reviews.

Biannual reviews

With the retrospectives occurring monthly, now we have more time in our biannual reviews. We keep doing the team appreciation, and we also:

  • check our goal status (Mid-Year review, usually in June or July) and
  • present the year analysis in the ending of the year, usually in November.

Kick-Off

At the very beginning of each year, the partners share the company goals and the strategic vision. This is the moment to review where we were last year, where we are now and where we want to be in the near future. It is also at this meeting that we get last year’s balance and financial projections.

One-on-one conversations

The main reason of all these meetings is to give and receive feedback. The communication problem occurs when some information is not crystal clear; the feedback mitigates this problem since everyone has more opportunities to solve one’s doubts.

However, some people have difficulty talking in crowded places, so we created an opportunity for those people too. We have a face-to-face meeting with the HR bi-monthly. Also, all partners have a weekly hour slot on their agendas to receive anyone who wants to talk about any subject.

How to evolve

It’s part of our culture to have good and clear communication. The practices we are using now certainly will not stay the same way forever. Our team is still growing; every year we have more and more projects and people joining us. Adapting and improving these practices are required to keep the company communication evolving.

It may appear that we have a lot of meetings, but they are organized and prepared to be efficient and avoid problems that having only ad-hoc communication could cause. All cited meetings here are timeboxed and have a specific goal. We work hard to achieve the meetings goals in the given timeframe for each meeting.

But beware! This is not about creating new formal meetings per se. You should focus on promoting a healthy culture of good and clear communication, and keeping feedback channels wide open within the whole company is crucial.

And what about you? What are your communication practices? Do you ever changed them?

]]>
http://blog.plataformatec.com.br/2015/03/continuous-communication/feed/ 3
Introducing Discrete Integration http://blog.plataformatec.com.br/2015/02/introducing-discrete-integration/ http://blog.plataformatec.com.br/2015/02/introducing-discrete-integration/#comments Tue, 10 Feb 2015 11:00:11 +0000 http://blog.plataformatec.com.br/?p=4420 »]]> Continuous Integration has been out there for some years by now. The book, Continuous Integration: Improving Software Quality and Reducing Risk dates from 2007, but the first mention of the term has came up earlier in 1994.

So, it’s been nearly 20 years since this idea is present in the software industry. During this time new tools, new practices and new possibilities appeared.

“Continuous Integration is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early”(1).

Its main goal is to detect errors earlier so they can be fixed. It helps in delivering software faster, because it allows you to deploy small changes with higher confidence due to automated tests.

Continuous Integration forces committing to the master branch and for each commit, tests are run in a production-like environment, a build is generated and outputs a deployable package.

Here at Plataformatec we do not follow Continuous Integration, although we’ve inherited some of its best practices. Some time ago, Rodrigo Flores came out with the name of what we do, and since then, we’re calling it Discrete Integration.

Discrete, as in discrete mathematics, means we’re working with fundamentally discrete structures rather than continuous, in other words, the objects studied by discrete mathematics do not vary smoothly, but have distinct, separated values(2).

One of those objects studied by discrete mathematics are graphs which can be representations of git branches. Git branches are the core of discrete integration. But in order to elucidate some important points, let me give you some context of Continuous Integration.

Continuous Integration adopters push code to master branch on a daily basis. Because of that, you’re going to need a mechanism to hide incomplete code that is being pushed, such as Feature Toggles. In other words, you will need to prevent pieces of code to be executed because they’re not yet ready for production.

Using toggles increases the maintainability costs and adds complexity to the code and to the tests. It’ll add technical debts along the development, since it needs to be removed. You’ll need toggles in your front-end code too, like in Javascript and CSS. One of the problems here is that CSS doesn’t support conditionals, so you end up with a lot of styles that will need to be removed in the end.

So, at Plataformatec, we tend to use toggles just when it is really required. For instance in an incremental rollout of a feature or when someone wants to control, for some reason, the software behaviour. And that’s because for Discrete Integration feature toggles are not necessary. Only fully implemented code is merged into the master branch.

In order to keep your work tidy, branches need to be merged as soon as possible. I personally call it the “minimum mergeable state”. You can think of when a task is complete or a user story is done – actually, it can be done anytime -, but the idea is to reach this minimum state so that you avoid integration problems.

To accomplish that, we use to shrink user stories to its smallest mergeable state. For instance, you could deliver firstly a page with a blog post form, then, you add a draft support, then an autosave feature and so on. Although autosave may be part of the draft feature, since autosave will be saved like a draft, it makes sense to be implemented afterwards. The three user stories are independent and deliverable, we’re just iteratively incrementing.

Sometimes, when pulling master’s code, conflicts will happen, but there’s nothing to do about it. What I mean is that Discrete Integration does not fear conflicts, since it is part of a developer work, although we do mitigate conflicts to happen.

As stated, shrinking user stories is not enough, that’s why Discrete Integration encourage developers to communicate. It can be done by a Chat tool, like Campfire, face to face, through Pull Requests or by a bit of each one. We even wrote a guideline where you can read more about its philosophies.

For us, Pull Requests are an import source of knowledge sharing, it also spreads collective code ownership, allows asynchronous code architecture discussions, decreases bug ratios and increases engagement and business knowledge.

It’s worth pointing out that commits into the master branch still happens. Sometimes you need to fix a typo, change a configuration line or other small changes. Although we would still send the commit link or ask for review before pushing, we’ve learned that it’s important to communicate all the changes, even small ones.

So, if you don’t push code directly into the master branch, you’re not using Continuous Integration by definition.

How close are you to Discrete Integration? Which practices did you miss? Please, let us know in the comments below!


  1. Fragment from http://www.thoughtworks.com/continuous-integration. 
  2. From http://en.wikipedia.org/wiki/Discrete_mathematics. 












]]>
http://blog.plataformatec.com.br/2015/02/introducing-discrete-integration/feed/ 3
Introducing Elixir Radar: the weekly email newsletter about Elixir http://blog.plataformatec.com.br/2015/01/introducing-elixir-radar-the-weekly-email-newsletter-about-elixir/ http://blog.plataformatec.com.br/2015/01/introducing-elixir-radar-the-weekly-email-newsletter-about-elixir/#comments Wed, 28 Jan 2015 11:00:15 +0000 http://blog.plataformatec.com.br/?p=4405 »]]> Tl;DR: We’re launching a weekly email newsletter about Elixir. It’s called Elixir Radar. We would like to invite you to subscribe to it.

A little bit of Elixir’s history

You may know José Valim. You may know Elixir. You may know Plataformatec. But you may not know the story of how Elixir was created and how that story connects all of those dots. Let me tell you the story.

José is one of the founders of Plataformatec and a Ruby on Rails Core team member. Back in 2010, he was working with improving Rails thread safety guarantees. It was a big challenge, and one where he learned a lot.

Because of that experience, he felt solving concurrency problems was too hard. So hard that it was difficult to maintain the Ruby’s philosophy of programmer happiness while solving that kind of problem. That was when he decided to study other paradigms and technologies.

During that journey, he fell in love with the Erlang VM. Its concurrency model brought productivity when tackling simple or complex problems. The failure semantics showed him how to write robust code without being defensive. The sheer amount of instrumentation the VM provides reflected well the fact companies have been using it for decades in production and he was confident he could do it too.

At the beginning of 2011, after some time working with Erlang VM, Valim decided to explore it further and build a new programming language, borrowing ideas from other technologies and ecosystems he has explored during that same journey.

In January of 2012 he presented to the other Plataformatec partners the first draft of Elixir. He told us what he envisioned Elixir could be. He wanted a productive language with great tooling. He also wanted the language to be extensible while encouraging practices that improve maintainability. And, of course, it would have the built-in concurrency model and robustness that the Erlang VM offers.

Valim said all of those features could make Elixir a relevant language for our work. Being it when building custom TCP servers, complex web applications or tackling distributed systems.

We were excited and decided to invest. We agreed Valim could spend 100% of his time working on Elixir.

For months, we were uncertain if we took the right decision. However that uncertainty was gone at the moment when Dave Thomas (founder at Pragmatic Programmers) and Simon St. Laurent (senior editor at O’Reilly) announced that they were writing books about Elixir. After that, everything changed.

Lots of Elixir meetups started to pop around the globe. Hundreds of libraries have been published to hex.pm. The first ElixirConf happened and at least two other conferences are planned for 2015. We have seen plenty of books and other educational materials around!

Those are signs of a strong open source community and we finally felt that Elixir was in the right direction.

Why Elixir Radar

We’re a small company and we knew back in 2012 that investing in a new programming language was a long term shot and with high risks. But today, Elixir is showing lots of signs that it’s on the right track. Like how its ecosystem is growing on its own!

As the company behind Elixir, we’re now looking into other ways to contribute to the ecosystem besides building the language itself. That’s why we decided to create an email newsletter with curated content about the community: Elixir Radar.

As the community grows, we believe both newcomers and experienced Elixir developers will benefit from a newsletter that shares and promotes the exciting projects, ideas and events happening all around.

If you want to receive a weekly email with blog posts, latest libraries, talks, events and job opportunities, please go to Elixir Radar’s page and subscribe.

We hope you’ll enjoy it.

]]>
http://blog.plataformatec.com.br/2015/01/introducing-elixir-radar-the-weekly-email-newsletter-about-elixir/feed/ 5
Organizing microservices in a single git repository http://blog.plataformatec.com.br/2015/01/organizing-microservices-in-a-single-git-repository/ http://blog.plataformatec.com.br/2015/01/organizing-microservices-in-a-single-git-repository/#comments Thu, 22 Jan 2015 10:44:30 +0000 http://blog.plataformatec.com.br/?p=4397 »]]> Microservices has gained popularity recently and some projects I’ve worked on had followed this approach. Basically, it’s an approach of software architecture that allows breaking monolithic applications into smaller decoupled, business-oriented and isolated deployable applications.

Each microservice normally is hosted in its own git repository, since it has very defined business boundaries and the code must be isolated from other microservices to ensure decoupling and deploy independance.

It may work greatly if you organize a team by each microservice. So, if a team is responsible for a given microservice and won’t work on other microservices, this organization may be good enough.

During project developments, we at Plataformatec understood that it is not so productive to focus on specific parts of a feature. Instead, we design and develop features by perceiving it as a whole, as it would be perceived by the end user. We don’t work with application specialists, we work with generalists and a lot of communication through pull requests.

So the best fit for the way we work, as our experience has shown us, is to put all the microservices and the clients that consume them into a single git repository. It may sound weird or semantically wrong for some, but after all, those microservices are small parts of a whole, something bigger, that is called software ecosystem. Since they share or exchange information among them, they’re somehow connected to each other.

This pragmatic approach is not exclusively ours, many people out there apply it. Two very nice examples are Facebook and Google. Of course their codebase is far larger than a normal application. They’re an exception. Google’s codebase, for instance, keeps really low level information like operating system configurations.

Using a single repository has proven to be a very good practice for us, because we can keep track of relevant pull requests easier; we can refactor, create and test new features throughout all the microservices faster; and test its integration without leaving the current context. Also, project gardening is way simpler: upgrading Ruby, Rails version, gem updates, using shared paths as gems, tests and deploy all of them can be automated and run across all microservices.

Have you worked with a single or multiple repositories? Please share your thoughts about it in the comments below!












]]>
http://blog.plataformatec.com.br/2015/01/organizing-microservices-in-a-single-git-repository/feed/ 7