Note: This topic is about system test automation only. Developers (unit) testing is out of scope of this article.
I have purposely chosen such a sound name for this topic. I myself am a big believer in system test automation. But, as any other kind of tool, automation likes those who know what to do with it.
This is way too easy to be misguided by marketing stuff promising you to become more effective using their new generation super-duper automation tool. Beware, 90% of what advertisement says is a lie. The solution how it is presented may work on a very limited number of projects. What vendors don't say may hit you hard or even kill your test automation endeavor. Read on in order to get ready to concealed challenges.
Test automation is development
First thing to remember is that test automation is creating a program that tests another program. As any other kind of code it is governed by same lows. Unnecessary complexity, unclearness, duplicates, high coupling, hard-coding strings and constants have the same bad effect for test code as they do for production code. So, make sure that you know how to write code efficiently before jumping into automation boat.
Record&playback does not work
Do not believe anyone who tells you that test automation is as easy as record&playback. This is a pure lie. Following this model is undermining the biggest automation benefit - having reliable regression suite that requires little changes after changes in application. Tests generated by action recording tools are non-maintainable and you will need to re-record them afterwards. Test execution in this case will be more expensive than running those tests manually. Now think of how often you make changes to your system and try calculating whether you are going to have benefit from automation with record&playback.
Test automation is the investment
Be ready to invest. Automation is not a tool that starts working right away. There is always some leading cost involved. Yet the benefit you get down the line worth it all.
Implementation of a good automated test suite requires skilled staff, resources and time. Our suite took 4 engineers for 4 years. But the result was outstanding. It worked so well that we almost stopped testing for regressions manually. It helped us finding the issues we could never find any other way. So, our investment into automation stared paying back after about 2 years since the project start.
I concluded that automation starts working when you have at least all critical tests automation but it shines when the number of automated tests is over about 40% of your test backlog.
Automated test is strict
Automated test is a program and it can help you finding only those issues for which it is programmed. Manual testers look around, note different strange things that automation tests just pass by, on his mission to click this and read that.
Make sure you have a human being tested the feature before it is handed over to test automation.
Automation tools are as easy
I would never start automation project having no skilled professionals involved. Experience is the key. Else you are in position to pay for automation as much as 300% of its actual cost. Rephrasing famous Russian advertisement: Have too much money? So, we are going after you ;)
Tests have to be selected carefully
Ideally, you need to think of testing as a whole in the very beginning of the project. You have to get a clear picture of which tests you will have automated and which are better to be left manual. Test design should be oriented at contributing test automation using data-driven testing technique. Due test design may make your automation efforts 10 times more effective.
In the real world, we sometime face the situation when tests are already designed. Even in that case, if you have the possibility, look at test design from automation perspective. Change it the way as to get most benefit of your automation tool. Sometimes it makes sense to add tests in places where we had to compromise coverage for execution cost.
As far as implementing automated tests is an additional cost it can only be saved by execution. The best candidates for automation are those, which are executed often. Also there can be tests that are easy to automate. Doing them manually may be not reasonable.
Automated testing economics
I mentioned that automated tests pay for their creation during execution. There is a simple math example that demonstrates this postulate:
• 10 manual tests require 2 hours for execution.
• Implementation of these 10 tests would take 16 man hours.
• Execution of 10 automated tests takes 0 (zero) man hours.
• So in order to get the benefit we have to execute tests at least 16/2=8 times.
Follow this principle in assessing automation feasibility and you will have positive return on investment (ROI) at your test automation projects.
Note: Sometimes we fool ourselves wrongly assuming "perceived" frequency of test usage for the real one. The problem is that once we have tests implemented it's very easy to run them. So we run even those tests, which we would never execute manually. If you want to know the real benefit of the automation, try to get rid of desire to calculate this "perceived" frequency. However, reporting the “perceived” benefit can be good for you career ;)
Conclusion
Automation is a silver bullet but only in case when it is done by experiences professionals, on a lengthy projects (2 years and more), following strict best development practices and disciplines.
No comments:
Post a Comment