As software developers, we have the bias to see the systems we create from a technical perspective. We have been taught to think like that since the beginning of our careers, and unfortunately it’s the dominant mindset in our industry. The perception that software development is an activity that requires few interpersonal interactions is still strong, and the myth about the “genius loner coder” persists to this day. With microservices this is not different.

I’ve been working with microservices for some time now, and for most of that time my reasoning about this architecture was aligned with what I just described. When designing new services or making tough architectural decisions, my bias was analysing each alternative with a technical mindset. Very important questions like “How we gonna extract this functionality from the monolith?” or “How we gonna handle this interaction? Event-based or through a REST endpoint?” were always analysed in a technical-first perspective.

And everything went well for a while, until we started having some problems. In one of the projects I’ve worked on, we constantly faced features that were supposed to be simple to ship but took a lot longer than expected. These same features also required a great deal of alignment with other developers, reducing our speed. Decisions we made in one service ended up impacting the whole organization, and bugs started to appear in unexpected places.

These problems made me question if something was wrong in the way the services were being designed. This thought seemed odd at first, our client was a mature organization with a great expertise in microservices. But the more I thought about it, the more I got the feeling that even mature organizations can still have their share of problems. Very subtle problems, that are not as obvious to see as a common programming mistake. Weren’t microservices supposed to make us ship faster and in a fault-tolerant way?

After a lot of thinking and research, I brought together some opinions about building microservices that I’m gonna share with you. But first, let’s expand more the problems of that misguiding mindset…

The problems of the service-as-code mindset

From now on we gonna talk about Booble, a fictitious car rental company. Booble adopted the microservices architecture a few years ago, after migrating from a standard Rails monolithic application.

Booble’s architectural direction was that services should be reusable. Each service was created to hold a piece of the platform’s functionality, such as: mailing service, document service, pricing service and vehicle service. Each one of them was extracted in a way as not to duplicate any code or data with other services. If one service needed data from another service, it would have to use a REST API for that, as each service was the authoritative data-source of its domain.

This architectural decision is what I would call the “service-as-code” mindset: when designing a new service, we tend to define them in a similar way to how we would define modules – or classes – in a monolith. And by doing that, we may end up with services that, if combined, would pretty much result in the monolith they were extracted from: a “distributed monolith”. This led Booble to have a myriad of services, and as many times more dependencies between them.

In time, a series of problems started appearing:

• Lack of resilience in the platform: if one service went down, many could become inoperant;
• Some services concentrated the bulk of requests, leading to performance and scalability problems. In extreme circumstances, this could even cause a service to crash under load;
• Changes made by one team impacted other ones, sometimes in unexpected ways;
• Changes in one service’s data-model could impact the whole platform, slowing down the development speed.

The problems listed above are almost the exact opposite of what we would expect from a microservices architecture: robustness and speed. When facing these problems, we lose all the advantages that this architecture can provide us, and risk having the same ones that a monolith would have. But this time, distributed in a network of services. How did Booble end up like this?

This may be due to the fact that the microservices architecture is quite new, and there’s still a lot of confusion about what constitutes a service. In face of that ambiguity, the developers responsible for the decision about what services should be created focus on the technical side of it. Services start to be reasoned about as something close to a code library.

I believe that this is the result of many years applying the DRY principle in the code. We are so accustomed to it that it seems natural to apply it into the architectural layer as well, resulting in services that don’t have any redundancy between then – in the code or in the data. The same DRY principle, if taken further, may lead to the construction of shared artifacts between services, like libraries and shared databases, resulting in even more coupling.

“The evils of too much coupling between services are far worse than the problems caused by code duplication.” – Sam Newman, Building Microservices

So how can we design a service, or many services for that matter, that would avoid the problems we listed?

Designing a robust service

Every time a service is designed, it should increase the related team’s capacity to deliver value faster and in a robust way. And, to be able to do so, you need to think beyond its technical aspects. In fact, even though software architecture may appear similar to coding, when designing systems and their interactions one should be careful to not let your developer mindset misguide your decisions.

Take the Booble example: there, the teams were divided in accordance with the company’s business areas: car rental team, finance team, analytics team, mobile team etc. But the services did not follow that division. This led to a situation where only a handful of the services were properly owned by a team, as each one of them held responsibilities that were needed across the company. Because of that, many features of the services were shared between different teams, sometimes for completely different use cases.

This taught me that when deciding if a new service needs to be created, it’s paramount to consider how it will fit into the company’s teams structure. It’s too easy to fall into the entity service antipattern and start modeling our services according to our business entities, forgetting the teams that will actually use them and how this shared dependency is going to affect their workflow.

Ideally, teams should have as fewer dependencies as possible with each other in order to do their job. Every time a team has to sync with another one, it’s a friction point that can cause delays and headaches. By keeping the services limited to the interested team, we get closer to the micro of microservices: instead of big services that fulfill a role for all the company, we have smaller services that fulfill a specific need for a business segment. Changes made in these services will only impact a fraction of the architecture, as opposed to the big services approach.

In order to attain the microservices architecture advantages, things that we usually avoid should actually be embraced:

• data redundancy
• code redundancy

If one service has all the data it needs to perform its business rules, we reduce its dependency from other services, increasing the system’s resilience and performance – no need to transfer data, after all. The service is also free to store this data in a format that makes sense to it.

But developers avoid writing applications with these traits for a good reason: when used badly, they can lead to disaster. But from a systems architecture point of view, if used correctly, these traits can boost the productivity of the whole organization as it decreases the dependency between teams and services.

Microservices is about team independence and alignment, not DRYness and entanglement. The team should suppress the urge to create a new service for each new feature that appears, and instead think about how it will fit into the broader picture of the architecture.

A service is not an end in itself, and vanity metrics such as

• number of services
• number of different technologies
• number of deploys per day

do not count much at the end of the day if it’s harder to ship code.

Your business will not survive based on the number of services your platform has, but instead in your organization’s capacity to stay ahead of the competition.

Some good rules of thumb are:

• Think about where/when the service will be used. Does it make sense to put this logic into an existing service?
• Avoid services with generic responsibilities, break them into smaller ones related to the business context where they will be used;
• Foment code locality. Code should be close to where it will be used.

These points are very difficult to ponder and, to do so properly, one needs experience. I, like most developers, started building microservices with a mindset that may hinder the microservices strong points. After facing big problems, I changed my mind about some dogmas that seemed right at the time. Today, if I would start a new project, I would try to focus on increasing the teams independency and alignment, and not worry too much about DRYness in the services.

Another lesson that I learned was that in order to guarantee that the architecture grows in a healthy manner, we need people who can take these kinds of decisions and have a broad view of the organization. But I’m gonna talk about this in a future post. Stay tuned!

Have you ever faced similar issues? Have any comment or suggestion? Please leave them in the comments section below.