That is what we are trying to achieve using a gathering of metrics and running Monte Carlo simulation over them.<\/p>\n
To understand how we collect our metrics, I recommend you read these posts:<\/p>\n
- \n
- Why we love metrics? Learning with lead time<\/a><\/li>\n
- Why we love metrics? Throughput and burnup charts<\/a><\/li>\n
- Why we love metrics? Cumulative flow diagrams<\/a><\/li>\n<\/ul>\n
Most people don\u2019t use any metrics to manage their projects. You can then imagine how many use complex statistical models to forecast a project completion. However, having this knowledge on your current project will help you deliver faster and better.<\/p>\n
In this post, you will see how we went from a naive forecast approach to a more precise and robust one using Monte Carlo Simulation. Moreover, we\u2019ll explain how Monte Carlo simulation works in an easy and straightforward way and then make an analogy between the simple example and our real world approach. Knowing when your project will likely end can help you manage the project\u2019s budget, give better feedbacks to stakeholders and give you the confidence that you are indeed in control of your project.<\/p>\n
Why not other methods?<\/h2>\n
We have been playing with other methods of forecasting for a long time, and all of them had some drawbacks. Here we list some of them and how and why we opted to use Monte Carlo.<\/p>\n
Mean Throughput<\/h3>\n
Some of you might have thought of a simple method in which we would use the average throughput to forecast the end of the project. However, as you can see on this blog post Power of the metrics: Don\u2019t use average to forecast deadlines<\/em><\/a>, that would be an extremely naive method, and it doesn\u2019t work.<\/p>\n
Linear Regression Approach<\/h3>\n
As a first approach, we used linear regression, based on our accumulated throughput, to forecast when a project would end:<\/p>\n
![\"Linear](\"http:\/\/blog.plataformatec.com.br\/wp-content\/uploads\/2016\/08\/linear-regression-project-completion-date.png\")
<\/p>\nHowever, we identified two different problems that we were not considering when doing that analysis:<\/p>\n
- \n
- We had to maintain the number of items in our backlog constant for the forecasting to work, which is very<\/strong> biasing.<\/li>\n
- We discovered, after some analysis, that both throughput and lead times do not follow a normal distribution, thus it would not make sense to calculate a linear regression (See on Looking at Lead Time in a different way<\/em><\/a> and Assumptions of Linear Regression<\/em><\/a>).<\/li>\n<\/ul>\n
Manual Setting Approach<\/h3>\n
Firstly, we focused on avoiding problem number 2, and we started using a more parsimonious model, which used methods that we knew were somehow skewed, but we could better control them.<\/p>\n
![\"Percentile](\"http:\/\/blog.plataformatec.com.br\/wp-content\/uploads\/2016\/08\/percentile-chart-project-completion-date.png\")
<\/p>\nWe manually added the throughput performance that would cause each different forecast to reach the backlog in a certain date. This forecasting approach comes from the study of percentiles of our throughput history.<\/p>\n
We had better results with that, since we could change the forecast manually and, therefore, tune it according to the current state of the project. But the problem number 1, the fact that we were not considering the change in the backlog, made us take a step further and try Monte Carlo Simulation.<\/p>\n
Monte Carlo Simulation<\/h2>\n
Wikipedia defines Monte Carlo<\/a> method as<\/p>\n
\nMonte Carlo methods vary, but tend to follow a particular pattern:<\/p>\n
- \n
- Define a domain of possible inputs.<\/li>\n
- Generate inputs randomly from a probability distribution over the domain.<\/li>\n
- Perform a deterministic computation on the inputs.<\/li>\n
- Aggregate the results.<\/li>\n<\/ul>\n<\/blockquote>\n
It may seem difficult or complex at first but it is actually much simpler than it looks. You can look at it as the brute force method of forecasting. We\u2019ll present a simple example and then show how we did in the real world. There\u2019s a very good simulation made by Larry Maccherone<\/a> that can elucidate it even more.<\/p>\n
Dice Game<\/h3>\n
Imagine you are playing a dice game in which the goal is to reach a sum of 12 points with the least number of rolls. The best play here would be 2 consecutive rolls in which you get a 6 in both of them, and the worst would be 12 rolls getting a 1. What we want to calculate is the probability of ending the game after N runs.<\/p>\n
Considering that we have 6 possible outcomes for each dice roll, the 6 faces of the dice, what is the probability of finishing the game in 1 roll?<\/p>\n
Zero. Since you cannot get to 12 points if the dice goes only up to 6 points.<\/p>\n
What about in the second roll?<\/p>\n
It is the probability of getting two consecutive 6, which is basic statistics:<\/p>\n
![\"Dice](\"http:\/\/blog.plataformatec.com.br\/wp-content\/uploads\/2016\/08\/dice-game-eq1.png\")
<\/p>\nWhat about in the third roll?<\/p>\n
You can achieve 12 points in many ways now, some of them are: (3,3,6), (3,4,5), (3,5,4), (3,4,6), (5,5,2), (4,4,4) and etc. Now the statistics behind that calculation are not that straightforward, right? That is when Monte Carlo simulation comes handy.<\/p>\n
What Monte Carlo does is to simulate thousands dice rolls and then analyze the outcome. For example, to know the probability of finishing the game on the third round, it would roll the dice three times, sum the points and store that result. After that, it would repeat those steps 5000 times and summarize how many rolls each sum of points got:<\/p>\n
- We discovered, after some analysis, that both throughput and lead times do not follow a normal distribution, thus it would not make sense to calculate a linear regression (See on Looking at Lead Time in a different way<\/em><\/a> and Assumptions of Linear Regression<\/em><\/a>).<\/li>\n<\/ul>\n
- Why we love metrics? Throughput and burnup charts<\/a><\/li>\n
![\"Dice](\"http:\/\/blog.plataformatec.com.br\/wp-content\/uploads\/2016\/08\/dice-game-eq2.png\")
<\/p>\n![\"Monte](\"http:\/\/blog.plataformatec.com.br\/wp-content\/uploads\/2016\/08\/monte-carlo-project-completion-date.png\")
<\/p>\n![\"Download:](\"http:\/\/blog.plataformatec.com.br\/wp-content\/uploads\/2016\/08\/forecasting-software-project-cta.png\")
<\/a>\n<\/div>\n","protected":false},"excerpt":{"rendered":"