Thursday, April 1, 2010

Test automation cost

Well, as I promised in the previous post, I am here to provide my thoughts on what the cost of automation is. I will try to prioritize parameters that in my opinion may increase the efforts and resources needed to make your automated tests work.

This is well known nowadays that the most important factor of test automation usage is Return Оn the Investment (ROI). In the traditional cost model ROI is determined as:

ROI = benefit / cost

If ROI is above 1 then you are in good shape. If it's below 1 then you have spent more than you gained.

Despite it looks pretty simple the difficulties came into the action when you try to understand what is beyond the terms of benefit and cost.

Benefit

This is not easy to measure what you gain in result of implementing you tests as automated. The problem is that not all of the benefits can be calculated and not all of the positive signs can be directly attributed to test automation. For example, you may discover more defects doing ad-hoc manual testing in time testers have available due to having some tests automated. Or you have found defects that could possibly be difficult to find manually. In both those cases you can't say for sure that this is automation that brought those benefits up.

The good news is that there is a cost parameter that we can directly calculate. I mean the cost of test execution. If you are tracking down what it takes to execute the tests manually (at least with some level of approximation), you can easily tell what you will save having those tests automated.

Let's pretend we have 100 manual tests, which execution usually takes 2 days. After having those tests automated you will save 2 days every time you need to do this. The most important here is "every time" because the more often you need those tests, the more cost effective their automation will be. But do not confuse "the need" to execute with "the desire" to execute. This is easy to fall a victim of self-deceive and draw a wrong conclusion about test automation effectiveness just by confusing the intentions.

So, I would stick to determining the benefit of automation as the sum of benefits you gain from test execution plus a little bit more (feel free to decide what this is going to be for your project: 10%, 20%, or even 50%).

benefit = execution*iterations + a little bit more

Cost

When people think of the cost they usually get confined to implementation. This is all natural because implementation is the first thing that comes to mind. The trick is to be able to see a bigger picture. Implementation is just one of parameters. There are also other factors that may severely affect the cost of automation. Below is a list of factors that can also affect automation ROI.

Experience in test automation

One who starts automation project with no experience in it at all puts himself in a big peril of the failure. Few survived having pursuing this goal without proper skills and knowledge. All the worse, those who aren't become paranoid of the automation and rarely try doing it again.

So if you have no experience in automation, try to get some. Hire a person who will move it from zero point or find a mentor who will team and guide the team. Otherwise there are too many pitfalls to fall into, so I would bet on the success by no means.

I would rate the factor of experience as the bigger in the cost formula. It may lead to more than 100% cost increase.


Maintenance


It's easy to forget because we rarely think of support until we face the need in it. But we will have to sooner than you can imagine. The changes will fire from all the cannons into the hull of you suite attesting to sink it down. The only way of staying afloat is to be able to quickly handle them.

This is not just about the code that you write. Of course it must be supportable. In this sense writing automated test is no different from writing a program code. This is also about the process that you follow. You can build a direct informational channel between developers and test engineers, so as the latter know about the upcoming changes and have time to prepare. Or you can let things go as they are, facing the consequences of massive changes 1 week before release, when all the tests you have automated appear to be broken and you have no idea why.

Maintenance factor from my experience is about 40% for large projects. You may have it down to 20% in a small project though.

Experience with tools

This is not s big as the previous ones but it counts too. Tools are so different. I worked with several and each has something that made me think "Common it cannot be that... weird!" So, you need to keep in mind that every tool has its own weirdness that you will face and you will need to go about.

The most important this parameter is for big projects. Test automation project architecture is the most important in that case. Architecture is built upon limitations that a tool had and there is almost no possibility to work around it. So be prepared to inventing new architectural patterns.

I would rate the increase this parameter may bring as 30%.

Suite debugging cost


Having a test implemented and passed is not all. Tests should work in the suites as well. If a test passes when executed alone does not guarantee it does the same when executed in a suite, on another machine and with other credentials.

So the tests should be designed to run smoothly in the suites, for which they are created. And it takes some additional efforts to be accounted for.

Test suites debugging is +10% in my experience.

The cost formula

cost = implementation cost + implementation cost*(experience% + tool experience% + maintenance% + suite debugging%)

The conclusion

As you see from the above test automation can only be successful when all positive factors are maximized having all the negative ones mitigated. This is in your hands to organize the process so as to get the biggest possible benefit.

I never use the scheme above as I learned how to incorporate all those factors into decision making many times when in my professional career. But if you find it useful I will be happy to know that I have spent my time here not in vain =)

2 comments:

  1. "Isn't the implementation cost as in your cost formular redundant with the [implementation cost*] ?

    ReplyDelete
  2. No, it's not! :) Let's take a look at the example:

    You have implementation cost of 1 man month, but you have identified that you lack experience with tool to be used. You also lend some time for debugging and maintenance. So the final figure will be:

    cost = 1 man month + 1 man month * (0%+30%+20%+10%) = 1 man month + 60% of a man month = 1.6 man month.

    ReplyDelete