Wednesday, April 28, 2010

"Look into the roots"

One of my colleagues who wanted to leave has claimed that the company has done something wrong to him, so he could not stand being in it any more. After 30 minutes of talking to him I could not make an idea what the reason of his complaints is as well as the complaints themselves remained the mystery to me.

He named such things like advertising that we are looking for a senior specialist and did not bother to suggest that position to him. That's was unfortunate because we considered him on very that position several months and he did not manage to impress the customer well enough. I asked "why didn't you come to me to discuss it?" He responded that "Anyways, this is not as big a problem as..."

The next reason he called out was the lack of work in the crisis times. Well I accepted that was the case and many people were sitting on the bench whereas the company tried to find the work for them. They were underpaid but this was done to keep the team. Those who could not stand such conditions have found another job. But that colleague of mine didn’t. I asked him why and didn't get the response either. He used the same tactics and jumped to the next complain which was about his personal attitude to work. He insisted that it was not exciting enough. So this might be a case. Nonetheless this is not the type of a problem one could not decide with a manager. But he never came to me with this.

In the end I decided that he complains not about the conditions but about his own behaviors in those conditions. I quickly figured that I can do nothing about it and let him go to find himself in another place (I purposely did not use word "better" here).

What the wisdom we could elicit from this short story? Obviously, if we cannot stand something or don't like something about our works or in more general sense about our lives them go an change it. Do not wait until the boomerang to hit you in the back.

And never fake the reason why you leave the company. Aside from not giving a chance for the company to fix the problem you make a bad impression of yourslef, so no one will miss you after all.

Friday, April 16, 2010

Estimations in software testing

Today I happened to read through the training course on estimates on testing. The method described was based on assuming the quantity of test cases. Knowing test cases you can relatively easily predict how much time you would need to execute them. But...

The trainer lacked some underlying knowledge of making the estimates. I tried to pretend I am a newbie who came to the training with goal to learn new things. After such a training I would risk to get further from the truth than I was before. I could have my own undertsnading of the matter, which is more correct than the provided.

Trainers beware! Your first commandment is "do not harm"! =)

Some tips of estimates:

- Consider all tasks including preparation, communication etc.
- Weight all factors influencing the tasks
- Break down task list into smaller tasks (2-3 days)
- Consider all possibilities what can go wrong
- Think of the risks
- Watch out of the optimism
- Know your resources
- Do estimates with more than one method and analyze the difference
- If you ask an expert, do not refrain to only one
- Never provide point estimates
- Provide estimation in a range
- Check out the estimate against other projects and analyze the differece
- Re-think risks
- Provide assumptions within the estimates
- Provide estimate with the level of confidence you have in them
- Present what you need to make the estimation more precise
- Validate estimation with other managers
- Check your estimate against estimates of other teams (testing vs. development, documentation vs. testing, etc.) and analyze the difference
- Add buffers for illness, turnover and other force major
- Beware of faulty level of precision (3 man days vs. 2.997 man days)

Good luck! :)

Thursday, April 1, 2010

Test automation cost

Well, as I promised in the previous post, I am here to provide my thoughts on what the cost of automation is. I will try to prioritize parameters that in my opinion may increase the efforts and resources needed to make your automated tests work.

This is well known nowadays that the most important factor of test automation usage is Return –ěn the Investment (ROI). In the traditional cost model ROI is determined as:

ROI = benefit / cost

If ROI is above 1 then you are in good shape. If it's below 1 then you have spent more than you gained.

Despite it looks pretty simple the difficulties came into the action when you try to understand what is beyond the terms of benefit and cost.


This is not easy to measure what you gain in result of implementing you tests as automated. The problem is that not all of the benefits can be calculated and not all of the positive signs can be directly attributed to test automation. For example, you may discover more defects doing ad-hoc manual testing in time testers have available due to having some tests automated. Or you have found defects that could possibly be difficult to find manually. In both those cases you can't say for sure that this is automation that brought those benefits up.

The good news is that there is a cost parameter that we can directly calculate. I mean the cost of test execution. If you are tracking down what it takes to execute the tests manually (at least with some level of approximation), you can easily tell what you will save having those tests automated.

Let's pretend we have 100 manual tests, which execution usually takes 2 days. After having those tests automated you will save 2 days every time you need to do this. The most important here is "every time" because the more often you need those tests, the more cost effective their automation will be. But do not confuse "the need" to execute with "the desire" to execute. This is easy to fall a victim of self-deceive and draw a wrong conclusion about test automation effectiveness just by confusing the intentions.

So, I would stick to determining the benefit of automation as the sum of benefits you gain from test execution plus a little bit more (feel free to decide what this is going to be for your project: 10%, 20%, or even 50%).

benefit = execution*iterations + a little bit more


When people think of the cost they usually get confined to implementation. This is all natural because implementation is the first thing that comes to mind. The trick is to be able to see a bigger picture. Implementation is just one of parameters. There are also other factors that may severely affect the cost of automation. Below is a list of factors that can also affect automation ROI.

Experience in test automation

One who starts automation project with no experience in it at all puts himself in a big peril of the failure. Few survived having pursuing this goal without proper skills and knowledge. All the worse, those who aren't become paranoid of the automation and rarely try doing it again.

So if you have no experience in automation, try to get some. Hire a person who will move it from zero point or find a mentor who will team and guide the team. Otherwise there are too many pitfalls to fall into, so I would bet on the success by no means.

I would rate the factor of experience as the bigger in the cost formula. It may lead to more than 100% cost increase.


It's easy to forget because we rarely think of support until we face the need in it. But we will have to sooner than you can imagine. The changes will fire from all the cannons into the hull of you suite attesting to sink it down. The only way of staying afloat is to be able to quickly handle them.

This is not just about the code that you write. Of course it must be supportable. In this sense writing automated test is no different from writing a program code. This is also about the process that you follow. You can build a direct informational channel between developers and test engineers, so as the latter know about the upcoming changes and have time to prepare. Or you can let things go as they are, facing the consequences of massive changes 1 week before release, when all the tests you have automated appear to be broken and you have no idea why.

Maintenance factor from my experience is about 40% for large projects. You may have it down to 20% in a small project though.

Experience with tools

This is not s big as the previous ones but it counts too. Tools are so different. I worked with several and each has something that made me think "Common it cannot be that... weird!" So, you need to keep in mind that every tool has its own weirdness that you will face and you will need to go about.

The most important this parameter is for big projects. Test automation project architecture is the most important in that case. Architecture is built upon limitations that a tool had and there is almost no possibility to work around it. So be prepared to inventing new architectural patterns.

I would rate the increase this parameter may bring as 30%.

Suite debugging cost

Having a test implemented and passed is not all. Tests should work in the suites as well. If a test passes when executed alone does not guarantee it does the same when executed in a suite, on another machine and with other credentials.

So the tests should be designed to run smoothly in the suites, for which they are created. And it takes some additional efforts to be accounted for.

Test suites debugging is +10% in my experience.

The cost formula

cost = implementation cost + implementation cost*(experience% + tool experience% + maintenance% + suite debugging%)

The conclusion

As you see from the above test automation can only be successful when all positive factors are maximized having all the negative ones mitigated. This is in your hands to organize the process so as to get the biggest possible benefit.

I never use the scheme above as I learned how to incorporate all those factors into decision making many times when in my professional career. But if you find it useful I will be happy to know that I have spent my time here not in vain =)

Automation is a must!

Recently I've got some free time and implemented several automated tests for an application being in testing and development for quite a long time. I did not expect much from my tests but cutting the cost of re-testing the basics of the system functionality under different operating systems.

I have implemented only 4 tests that I have executed against only one operating system. To my sheer surprise the application crashed. It crashed on doing the same sequence of operations it has just did successfully in the previous tests. Hm... I have re-run the entire suite once again and it failed. Wow!

This is not the first time I noted that automated tests is not necessarily a replacement for manual tests. It is rather an addition to it. Automated tests help finding defects one could not find manually and vice versa.

Having said that there are issues your testers are unlikely to find manually, I dare say that automation is absolutely a must for any type of a project. Thos of you who compromise this idea to project cost simply do compromising the quality.

The cost of automation is not as bad as it may seem to be. I will write on this a little more in the following blogs. As of now, I may assure you the cost is not that big as you may imagine. For example, for those 4 tests I wrote slightly more than 200 lines of test code (just 200 simple plain lines of code!). This is all it takes!

It has taken me only one and a half day to create, debug, and run the entire suite on one platform. It just a little longer than it takes to it once manually. However, now I have a suite that I can run with a single click. I don't need to spend my time on it anymore. So I can focus on what matters instead of doing tedious routine work again and again.

When deciding whether to pursue automation, do not think of the losses. Think of what will have in return. Think of the advantages you will have. Yes it takes time and resources to build a reliable automated suite. However, it is worth of every dime spent on it when you are a little further down the road.

Good luck with your automated testing!