Several days ago, while working on a following test automation project for a desktop application an idea how to make it right the first time has struck me. This is the first time I am writing on this matter, so be my co-authors and provide any thoughts which happen to come to your mind while reading this. Any feedback will be greatly appreciated.
Dealing with UI looks simple at the first glance but it is rather tricky afterwards. All the time you have to do some UI testing (confirmation messages, cancel buttons, input error messages and the like) you have to step back and to change (refactor) the code that you have already wrote.
For example, you have created a function that opens a project file in your application. It takes the name of the file for the input parameter. Then it click menu File/Open, fills in file path and click button Ok.
You created the tests that rely of that function. Several tests may open different files and validate their content. But now you need to create a test, which will open an incorrect file. In result you shall see a message telling you what is wrong with it. But your original function does rely on smooth flawless file opening. It does not take into account any errors which may come up. So, we go back to the original file opening function and refactor it in order to fit both needs (usually it is done by creating a new function that does only part of the job).
This is only on example of hundreds or even thousands for a big project. And any time you have to step back, do changes and test the changes. This is tricky and error-prone.
Instead I am suggesting DOING THINGS RIGHT THE FIRST TIME - Invoker-Setter-Confirmer (ISC) test architecture.
ISC is a UI test architecture that allows you to think in terms of functional bricks of which you can build any kind of UI based tests. You will not need to go back afterward, I promise. If you do, I am gonna eat my hat :)
ISC stands for the set of functions, each with its own purpose described below.
Invokers - functions responsible for making a UI element to appear on the screen. In most cases it can be done with more than one way (File Open dialog may be caller through menu, toolbar, accelerators and hot keys).
Setters - functions responsible for setting input values into UI element. For example, find dialog requests user to set a string to find, direction of the search and other properties. All those values shall be set through a setter functions.
Confirmers - functions which do some actions with UI element. For a dialog they click a button or close it with Esc or Alt+F4. Just like invocation the confirmation can be done in different ways.
The best way to explain it is to show an example. Here is goes.
Everyone has a Notepad program (Mac users, I am very sorry). Let's create ISC-based library for Find dialog.
// code is in Java Script
// constants
notepad = Aliases.notepad;
wndNotepad = notepad.wndNotepad;
ACCELERATOR = 1;
MENU = 2;
HOTKEYS = 3;
OK = 10;
CANCEL = 11;
ESC = 12;
ALTF4 = 13;
//////////////////////////////////////
// Invoker
//////////////////////////////////////
function InvokerDlgFind(method) {
switch(method) {
case ACCELERATOR :
wndNotepad.Keys("[Ctrl]F");
break;
case MENU :
wndNotepad.MainMenu.Click("Edit|Find...");
break;
case HOTKEYS :
wndNotepad.Keys("[Alt]EF");
break;
default:
InvokerDlgFind(MENU);
break;
}
// No more code. Just show this thing on
// the screen and leave.
}
//////////////////////////////////////
// Setter
//////////////////////////////////////
function SetDlgFind(toFind, case, direction) {
dlgFind.Edit.wText = toFind;
dlgFind.checkMatchCase.ClickButton(case);
if(direction) {
dlgFind.radioUp.ClickButton();
} else {
dlgFind.radioDown.ClickButton();;
}
// that's it! Setters do not do anything else.
}
//////////////////////////////////////
// Confirmer
//////////////////////////////////////
function ConfirmDlgFind(method) {
switch(method) {
case OK :
notepad.dlgFind.btnOK.ClickButton();
break;
case CANCEL :
notepad.dlgFind.btnCancel.ClickButton();
break;
case ESC :
notepad.dlgFind.Keys("[ESC]");
break;
case ALTF4 :
notepad.dlgFind.Keys("[Alt][F4]");
break;
default:
ConfirmDlgFind(OK);
break;
}
}
// Now how it looks in the tests
function Test_Find_Succeeded() {
// open test file
InvokeDlgFind(MENU);
SetDlgFind("123", false, true);
ConfirmDlgFind(OK);
// verify find result
}
function Test_Find_NotFound() {
// open test file
InvokeDlgFind(MENU);
SetDlgFind("456", false, true);
ConfirmDlgFind(OK);
// verify message "not found"
}
function Test_Find_EmptyInput() {
// open test file
InvokeDlgFind(MENU);
SetDlgFind("", false, true);
// verify button is disabled
}
Hope this helps. Will be glad to hear from you on this matter.
Showing posts with label automated testing. Show all posts
Showing posts with label automated testing. Show all posts
Wednesday, October 13, 2010
Wednesday, September 15, 2010
TestComplete 7.X refuses to recognize .NET objects?
If you cannot see .NET objects with appropriate icon in Object Browser, then most likely you have no corresponding plug-in installed or you have .NET 4.0. Check you extension settings and install .NET plugin. In the later case uninstalling .NET 4.0 will not solve the problem. You will need to re-install whole system. Alas!
Friday, April 16, 2010
Estimations in software testing
Today I happened to read through the training course on estimates on testing. The method described was based on assuming the quantity of test cases. Knowing test cases you can relatively easily predict how much time you would need to execute them. But...
The trainer lacked some underlying knowledge of making the estimates. I tried to pretend I am a newbie who came to the training with goal to learn new things. After such a training I would risk to get further from the truth than I was before. I could have my own undertsnading of the matter, which is more correct than the provided.
Trainers beware! Your first commandment is "do not harm"! =)
Some tips of estimates:
- Consider all tasks including preparation, communication etc.
- Weight all factors influencing the tasks
- Break down task list into smaller tasks (2-3 days)
- Consider all possibilities what can go wrong
- Think of the risks
- Watch out of the optimism
- Know your resources
- Do estimates with more than one method and analyze the difference
- If you ask an expert, do not refrain to only one
- Never provide point estimates
- Provide estimation in a range
- Check out the estimate against other projects and analyze the differece
- Re-think risks
- Provide assumptions within the estimates
- Provide estimate with the level of confidence you have in them
- Present what you need to make the estimation more precise
- Validate estimation with other managers
- Check your estimate against estimates of other teams (testing vs. development, documentation vs. testing, etc.) and analyze the difference
- Add buffers for illness, turnover and other force major
- Beware of faulty level of precision (3 man days vs. 2.997 man days)
Good luck! :)
The trainer lacked some underlying knowledge of making the estimates. I tried to pretend I am a newbie who came to the training with goal to learn new things. After such a training I would risk to get further from the truth than I was before. I could have my own undertsnading of the matter, which is more correct than the provided.
Trainers beware! Your first commandment is "do not harm"! =)
Some tips of estimates:
- Consider all tasks including preparation, communication etc.
- Weight all factors influencing the tasks
- Break down task list into smaller tasks (2-3 days)
- Consider all possibilities what can go wrong
- Think of the risks
- Watch out of the optimism
- Know your resources
- Do estimates with more than one method and analyze the difference
- If you ask an expert, do not refrain to only one
- Never provide point estimates
- Provide estimation in a range
- Check out the estimate against other projects and analyze the differece
- Re-think risks
- Provide assumptions within the estimates
- Provide estimate with the level of confidence you have in them
- Present what you need to make the estimation more precise
- Validate estimation with other managers
- Check your estimate against estimates of other teams (testing vs. development, documentation vs. testing, etc.) and analyze the difference
- Add buffers for illness, turnover and other force major
- Beware of faulty level of precision (3 man days vs. 2.997 man days)
Good luck! :)
Thursday, April 1, 2010
Test automation cost
Well, as I promised in the previous post, I am here to provide my thoughts on what the cost of automation is. I will try to prioritize parameters that in my opinion may increase the efforts and resources needed to make your automated tests work.
This is well known nowadays that the most important factor of test automation usage is Return Оn the Investment (ROI). In the traditional cost model ROI is determined as:
ROI = benefit / cost
If ROI is above 1 then you are in good shape. If it's below 1 then you have spent more than you gained.
Despite it looks pretty simple the difficulties came into the action when you try to understand what is beyond the terms of benefit and cost.
Benefit
This is not easy to measure what you gain in result of implementing you tests as automated. The problem is that not all of the benefits can be calculated and not all of the positive signs can be directly attributed to test automation. For example, you may discover more defects doing ad-hoc manual testing in time testers have available due to having some tests automated. Or you have found defects that could possibly be difficult to find manually. In both those cases you can't say for sure that this is automation that brought those benefits up.
The good news is that there is a cost parameter that we can directly calculate. I mean the cost of test execution. If you are tracking down what it takes to execute the tests manually (at least with some level of approximation), you can easily tell what you will save having those tests automated.
Let's pretend we have 100 manual tests, which execution usually takes 2 days. After having those tests automated you will save 2 days every time you need to do this. The most important here is "every time" because the more often you need those tests, the more cost effective their automation will be. But do not confuse "the need" to execute with "the desire" to execute. This is easy to fall a victim of self-deceive and draw a wrong conclusion about test automation effectiveness just by confusing the intentions.
So, I would stick to determining the benefit of automation as the sum of benefits you gain from test execution plus a little bit more (feel free to decide what this is going to be for your project: 10%, 20%, or even 50%).
benefit = execution*iterations + a little bit more
Cost
When people think of the cost they usually get confined to implementation. This is all natural because implementation is the first thing that comes to mind. The trick is to be able to see a bigger picture. Implementation is just one of parameters. There are also other factors that may severely affect the cost of automation. Below is a list of factors that can also affect automation ROI.
Experience in test automation
One who starts automation project with no experience in it at all puts himself in a big peril of the failure. Few survived having pursuing this goal without proper skills and knowledge. All the worse, those who aren't become paranoid of the automation and rarely try doing it again.
So if you have no experience in automation, try to get some. Hire a person who will move it from zero point or find a mentor who will team and guide the team. Otherwise there are too many pitfalls to fall into, so I would bet on the success by no means.
I would rate the factor of experience as the bigger in the cost formula. It may lead to more than 100% cost increase.
Maintenance
It's easy to forget because we rarely think of support until we face the need in it. But we will have to sooner than you can imagine. The changes will fire from all the cannons into the hull of you suite attesting to sink it down. The only way of staying afloat is to be able to quickly handle them.
This is not just about the code that you write. Of course it must be supportable. In this sense writing automated test is no different from writing a program code. This is also about the process that you follow. You can build a direct informational channel between developers and test engineers, so as the latter know about the upcoming changes and have time to prepare. Or you can let things go as they are, facing the consequences of massive changes 1 week before release, when all the tests you have automated appear to be broken and you have no idea why.
Maintenance factor from my experience is about 40% for large projects. You may have it down to 20% in a small project though.
Experience with tools
This is not s big as the previous ones but it counts too. Tools are so different. I worked with several and each has something that made me think "Common it cannot be that... weird!" So, you need to keep in mind that every tool has its own weirdness that you will face and you will need to go about.
The most important this parameter is for big projects. Test automation project architecture is the most important in that case. Architecture is built upon limitations that a tool had and there is almost no possibility to work around it. So be prepared to inventing new architectural patterns.
I would rate the increase this parameter may bring as 30%.
Suite debugging cost
Having a test implemented and passed is not all. Tests should work in the suites as well. If a test passes when executed alone does not guarantee it does the same when executed in a suite, on another machine and with other credentials.
So the tests should be designed to run smoothly in the suites, for which they are created. And it takes some additional efforts to be accounted for.
Test suites debugging is +10% in my experience.
The cost formula
cost = implementation cost + implementation cost*(experience% + tool experience% + maintenance% + suite debugging%)
The conclusion
As you see from the above test automation can only be successful when all positive factors are maximized having all the negative ones mitigated. This is in your hands to organize the process so as to get the biggest possible benefit.
I never use the scheme above as I learned how to incorporate all those factors into decision making many times when in my professional career. But if you find it useful I will be happy to know that I have spent my time here not in vain =)
This is well known nowadays that the most important factor of test automation usage is Return Оn the Investment (ROI). In the traditional cost model ROI is determined as:
ROI = benefit / cost
If ROI is above 1 then you are in good shape. If it's below 1 then you have spent more than you gained.
Despite it looks pretty simple the difficulties came into the action when you try to understand what is beyond the terms of benefit and cost.
Benefit
This is not easy to measure what you gain in result of implementing you tests as automated. The problem is that not all of the benefits can be calculated and not all of the positive signs can be directly attributed to test automation. For example, you may discover more defects doing ad-hoc manual testing in time testers have available due to having some tests automated. Or you have found defects that could possibly be difficult to find manually. In both those cases you can't say for sure that this is automation that brought those benefits up.
The good news is that there is a cost parameter that we can directly calculate. I mean the cost of test execution. If you are tracking down what it takes to execute the tests manually (at least with some level of approximation), you can easily tell what you will save having those tests automated.
Let's pretend we have 100 manual tests, which execution usually takes 2 days. After having those tests automated you will save 2 days every time you need to do this. The most important here is "every time" because the more often you need those tests, the more cost effective their automation will be. But do not confuse "the need" to execute with "the desire" to execute. This is easy to fall a victim of self-deceive and draw a wrong conclusion about test automation effectiveness just by confusing the intentions.
So, I would stick to determining the benefit of automation as the sum of benefits you gain from test execution plus a little bit more (feel free to decide what this is going to be for your project: 10%, 20%, or even 50%).
benefit = execution*iterations + a little bit more
Cost
When people think of the cost they usually get confined to implementation. This is all natural because implementation is the first thing that comes to mind. The trick is to be able to see a bigger picture. Implementation is just one of parameters. There are also other factors that may severely affect the cost of automation. Below is a list of factors that can also affect automation ROI.
Experience in test automation
One who starts automation project with no experience in it at all puts himself in a big peril of the failure. Few survived having pursuing this goal without proper skills and knowledge. All the worse, those who aren't become paranoid of the automation and rarely try doing it again.
So if you have no experience in automation, try to get some. Hire a person who will move it from zero point or find a mentor who will team and guide the team. Otherwise there are too many pitfalls to fall into, so I would bet on the success by no means.
I would rate the factor of experience as the bigger in the cost formula. It may lead to more than 100% cost increase.
Maintenance
It's easy to forget because we rarely think of support until we face the need in it. But we will have to sooner than you can imagine. The changes will fire from all the cannons into the hull of you suite attesting to sink it down. The only way of staying afloat is to be able to quickly handle them.
This is not just about the code that you write. Of course it must be supportable. In this sense writing automated test is no different from writing a program code. This is also about the process that you follow. You can build a direct informational channel between developers and test engineers, so as the latter know about the upcoming changes and have time to prepare. Or you can let things go as they are, facing the consequences of massive changes 1 week before release, when all the tests you have automated appear to be broken and you have no idea why.
Maintenance factor from my experience is about 40% for large projects. You may have it down to 20% in a small project though.
Experience with tools
This is not s big as the previous ones but it counts too. Tools are so different. I worked with several and each has something that made me think "Common it cannot be that... weird!" So, you need to keep in mind that every tool has its own weirdness that you will face and you will need to go about.
The most important this parameter is for big projects. Test automation project architecture is the most important in that case. Architecture is built upon limitations that a tool had and there is almost no possibility to work around it. So be prepared to inventing new architectural patterns.
I would rate the increase this parameter may bring as 30%.
Suite debugging cost
Having a test implemented and passed is not all. Tests should work in the suites as well. If a test passes when executed alone does not guarantee it does the same when executed in a suite, on another machine and with other credentials.
So the tests should be designed to run smoothly in the suites, for which they are created. And it takes some additional efforts to be accounted for.
Test suites debugging is +10% in my experience.
The cost formula
cost = implementation cost + implementation cost*(experience% + tool experience% + maintenance% + suite debugging%)
The conclusion
As you see from the above test automation can only be successful when all positive factors are maximized having all the negative ones mitigated. This is in your hands to organize the process so as to get the biggest possible benefit.
I never use the scheme above as I learned how to incorporate all those factors into decision making many times when in my professional career. But if you find it useful I will be happy to know that I have spent my time here not in vain =)
Automation is a must!
Recently I've got some free time and implemented several automated tests for an application being in testing and development for quite a long time. I did not expect much from my tests but cutting the cost of re-testing the basics of the system functionality under different operating systems.
I have implemented only 4 tests that I have executed against only one operating system. To my sheer surprise the application crashed. It crashed on doing the same sequence of operations it has just did successfully in the previous tests. Hm... I have re-run the entire suite once again and it failed. Wow!
This is not the first time I noted that automated tests is not necessarily a replacement for manual tests. It is rather an addition to it. Automated tests help finding defects one could not find manually and vice versa.
Having said that there are issues your testers are unlikely to find manually, I dare say that automation is absolutely a must for any type of a project. Thos of you who compromise this idea to project cost simply do compromising the quality.
The cost of automation is not as bad as it may seem to be. I will write on this a little more in the following blogs. As of now, I may assure you the cost is not that big as you may imagine. For example, for those 4 tests I wrote slightly more than 200 lines of test code (just 200 simple plain lines of code!). This is all it takes!
It has taken me only one and a half day to create, debug, and run the entire suite on one platform. It just a little longer than it takes to it once manually. However, now I have a suite that I can run with a single click. I don't need to spend my time on it anymore. So I can focus on what matters instead of doing tedious routine work again and again.
When deciding whether to pursue automation, do not think of the losses. Think of what will have in return. Think of the advantages you will have. Yes it takes time and resources to build a reliable automated suite. However, it is worth of every dime spent on it when you are a little further down the road.
Good luck with your automated testing!
I have implemented only 4 tests that I have executed against only one operating system. To my sheer surprise the application crashed. It crashed on doing the same sequence of operations it has just did successfully in the previous tests. Hm... I have re-run the entire suite once again and it failed. Wow!
This is not the first time I noted that automated tests is not necessarily a replacement for manual tests. It is rather an addition to it. Automated tests help finding defects one could not find manually and vice versa.
Having said that there are issues your testers are unlikely to find manually, I dare say that automation is absolutely a must for any type of a project. Thos of you who compromise this idea to project cost simply do compromising the quality.
The cost of automation is not as bad as it may seem to be. I will write on this a little more in the following blogs. As of now, I may assure you the cost is not that big as you may imagine. For example, for those 4 tests I wrote slightly more than 200 lines of test code (just 200 simple plain lines of code!). This is all it takes!
It has taken me only one and a half day to create, debug, and run the entire suite on one platform. It just a little longer than it takes to it once manually. However, now I have a suite that I can run with a single click. I don't need to spend my time on it anymore. So I can focus on what matters instead of doing tedious routine work again and again.
When deciding whether to pursue automation, do not think of the losses. Think of what will have in return. Think of the advantages you will have. Yes it takes time and resources to build a reliable automated suite. However, it is worth of every dime spent on it when you are a little further down the road.
Good luck with your automated testing!
Tuesday, August 4, 2009
Scripted testing or free flying?
The more I work in testing domain, the more I learn how difficult it is to answer this question. On one hand we need to be very diligent and specific about things we are testing. We have to be ready to answer what we tested and how. We have to collect information on what was tested and how in order to do retrospective review. On another hand, many test professionals believe that the best way to find a defect is get into something like a trance, a "free flying" through the functionality. Like a dog sniffing its pray, tester searches defects following gut instincts. This is way too difficult to organize in form of a checklist or guidelines. You can hardly explain what it takes to ride a bicycle or jump with a parachute to a person new to those activities. All you can achieve is "I have no idea what you are talking about" response. Same is for the testing.
So what is the best way for you, you may ask? The answer cannot be universal for everyone because each team is unique. The way we used to work, the qualities of team members and peculiarities of a task are very important. I personally believe that a mix of both approaches is what works better for most of the organizations. But what kind of a mix should it be? One may try to script everything, including tests executed once (executed and throw away), like in medical industry. Others may not bother having such tests written down on the paper, allowing some time for their testers to do non-scripted testing fee of any bureaucracy. What mix works best for you is a matter of numerous trial and errors. Try different approaches and see what effect it may have. If it's a waste of time don't do it.
Unlike test design based on product requirements "free flying" testing is in-context activity, so it takes less efforts to generate ideas. When you do not have application to click buttons and try different combinations it is difficult to come up with ideas. Modeling software behavior in mind is resource-consuming task. Your mind simply has not enough free processor time required for the creativity.
In general, scripted testing adds more organization and clarity in testing process. It helps in making the testing process visible and controllable, thus helping to meet deadlines. Having no test scripted, you can hardly say how long it will take to run all tests, because you can only guess how many of them you are going to execute.
So, the best way of dealing about it on the level of testing schedule is generating scripted tests against requirements (and don't bother too much about very specific ways of using software at this point) and allow a day or two (depending on how big the functionality is) for "free flying" testing. These two days are better to be scheduled at the end of the testing stage. Prior scripted testing will help removing blocking issues as well it will lend your testers time to get familiar with new functionality.
All is in your hands. Only you can say what is best for your specific situation. Good luck!
So what is the best way for you, you may ask? The answer cannot be universal for everyone because each team is unique. The way we used to work, the qualities of team members and peculiarities of a task are very important. I personally believe that a mix of both approaches is what works better for most of the organizations. But what kind of a mix should it be? One may try to script everything, including tests executed once (executed and throw away), like in medical industry. Others may not bother having such tests written down on the paper, allowing some time for their testers to do non-scripted testing fee of any bureaucracy. What mix works best for you is a matter of numerous trial and errors. Try different approaches and see what effect it may have. If it's a waste of time don't do it.
Unlike test design based on product requirements "free flying" testing is in-context activity, so it takes less efforts to generate ideas. When you do not have application to click buttons and try different combinations it is difficult to come up with ideas. Modeling software behavior in mind is resource-consuming task. Your mind simply has not enough free processor time required for the creativity.
In general, scripted testing adds more organization and clarity in testing process. It helps in making the testing process visible and controllable, thus helping to meet deadlines. Having no test scripted, you can hardly say how long it will take to run all tests, because you can only guess how many of them you are going to execute.
So, the best way of dealing about it on the level of testing schedule is generating scripted tests against requirements (and don't bother too much about very specific ways of using software at this point) and allow a day or two (depending on how big the functionality is) for "free flying" testing. These two days are better to be scheduled at the end of the testing stage. Prior scripted testing will help removing blocking issues as well it will lend your testers time to get familiar with new functionality.
All is in your hands. Only you can say what is best for your specific situation. Good luck!
Monday, July 20, 2009
Automated tests: the sooner the better!
This is the most impressive experience I gained in my entire professional career. Read on and you will probably find it worth of trying.
Several years ago we faced the need to dramatically increase test automation coverage. Let the reasons be aside for now and let's focus on the problem per se. We had complex UI-driven software, full of custom controls and volatile UI elements changed from release to release. Our experience in test automation was refrained to the collection of several hundred tests that we occasionally used for smoke and regression testing. In order to get where we needed to be we had to automate ten thousand tests in a very short period of time, 1-2 years.
At the first glance the problem had no solution. It was really hard to be in that situation, when you see no way out. But we started the project having no idea if it's going to be successful. We started from optimizing of what we already can do, divided team in two sub-teams (one team focused on test design and manual execution, another took responsibility for test automation). We also introduced all the best practices that helped us to keep test production and maintenance cost low.
But this was not enough. Automation team was always behind testing process because they needed time to implement and debug test code on a working system. With insignificant exceptions, so, we could not use automated tests for functional test execution during the production cycle. We could only use it for regression testing of future versions. Tests still needed to be executed manually at least once. It had severe impact on testing schedule and drug us back because manual team was always on a critical path.
Then we’ve got an idea. It was as easy as it could only be. As far as implementation phase is what makes automation team fall behind we have to shorten it somehow. Adding resources is not an option. We already had several people working on this project and adding more people could severely increase overhead cost of communication. We went anther way.
Instead of adding resources we decided to move a part of test automation preparation task back in time, to do them in advance. We decided to do design of automated tests long before a working version is made available. Design means that we created fish bones of tests. Those tests used virtual functions that, when implemented, allow manipulating application features. Functions remained not implemented until we have a working version of a product in our hands. So, the tests could not be debugged until that time. But the most of design work is already done. And our experience indicated that this part is very significant.
When a new version of application comes to testing, automation engineers started implementation phase. They simply added code to the helper functions to make the test work. After that they run and debugged tests.
I will demonstrate how it worked in example:
1. We need to test a web search system that allows users to run the search, browse results and bookmark interesting findings.
2. Automation engineers select tests for automation. For example they have selected the following tests:
a. Different types of queries ("", "my search", "very-very-very long string").
b. Browse results (pages 1 through 10).
3. Automation team has steps of tests and test data defined in the test description created by manual team. So, thy create test architecture design like this:
test 1 - Different types of queries
test01_Search (String query, Integer expected) {
login();
doSearch(query);
assertEqual(getResults().totalCount, expected);
}
Used functions login(), doSearch(), and getResults() are not implemented so far! We have only figured out which functions we will need to enable our tests work.
Note: In order to do it safely it is recommended to insert a string of code that will fail your tests until function is not implemented, like this:
function doSearch(String query) {
fail('Not implemented');
}
Test that go through pages of results could be looking like follows:
test02_Paging (String query, Integer expected) {
String[] pageTokens = {"page1", ..."page 10"};
login();
doSearch(query);
for (int i=1; i<=10; i++) {
goToPage(i);
assertEqual(getPageToken(), pageTokens[i]);
}
}
Same way we can Design all tests that we selected for automation in advance. Thus we are saving this time from implementation and shorten the duration of automation. As practice had shown we can save up to 50% of time allocated for test automation. Roughly assessing, design phase is accounted for a half time allocated for the automation. So we shorten the second phase in a half, allowing automated tests to be ready sooner in the testing cycle.
Using this technology we have achieved the ratio of tests executed manually and automatically 1:1. This means that only a half of tests were executed manually each new release. Another half of tests were executed as automated right away. This had greatly increased automation ROI because we had no need to execute them manually at all and saved up to 40% of manual testing resources each release. Additionally, automated tests could be used for in-project regression testing much earlier. It also helped to get much of benefit from the idea.
In general, this approach had completely changed the role automated testing played in our project, making it at least as important and effective as manual testing.
Hope this helps making your test automation effort more fun! :) Feel free to comment should you have questions on details of implementation or should you have risks that may prevent you from having it work for you.
Several years ago we faced the need to dramatically increase test automation coverage. Let the reasons be aside for now and let's focus on the problem per se. We had complex UI-driven software, full of custom controls and volatile UI elements changed from release to release. Our experience in test automation was refrained to the collection of several hundred tests that we occasionally used for smoke and regression testing. In order to get where we needed to be we had to automate ten thousand tests in a very short period of time, 1-2 years.
At the first glance the problem had no solution. It was really hard to be in that situation, when you see no way out. But we started the project having no idea if it's going to be successful. We started from optimizing of what we already can do, divided team in two sub-teams (one team focused on test design and manual execution, another took responsibility for test automation). We also introduced all the best practices that helped us to keep test production and maintenance cost low.
But this was not enough. Automation team was always behind testing process because they needed time to implement and debug test code on a working system. With insignificant exceptions, so, we could not use automated tests for functional test execution during the production cycle. We could only use it for regression testing of future versions. Tests still needed to be executed manually at least once. It had severe impact on testing schedule and drug us back because manual team was always on a critical path.
Then we’ve got an idea. It was as easy as it could only be. As far as implementation phase is what makes automation team fall behind we have to shorten it somehow. Adding resources is not an option. We already had several people working on this project and adding more people could severely increase overhead cost of communication. We went anther way.
Instead of adding resources we decided to move a part of test automation preparation task back in time, to do them in advance. We decided to do design of automated tests long before a working version is made available. Design means that we created fish bones of tests. Those tests used virtual functions that, when implemented, allow manipulating application features. Functions remained not implemented until we have a working version of a product in our hands. So, the tests could not be debugged until that time. But the most of design work is already done. And our experience indicated that this part is very significant.
When a new version of application comes to testing, automation engineers started implementation phase. They simply added code to the helper functions to make the test work. After that they run and debugged tests.
I will demonstrate how it worked in example:
1. We need to test a web search system that allows users to run the search, browse results and bookmark interesting findings.
2. Automation engineers select tests for automation. For example they have selected the following tests:
a. Different types of queries ("", "my search", "very-very-very long string").
b. Browse results (pages 1 through 10).
3. Automation team has steps of tests and test data defined in the test description created by manual team. So, thy create test architecture design like this:
test 1 - Different types of queries
test01_Search (String query, Integer expected) {
login();
doSearch(query);
assertEqual(getResults().totalCount, expected);
}
Used functions login(), doSearch(), and getResults() are not implemented so far! We have only figured out which functions we will need to enable our tests work.
Note: In order to do it safely it is recommended to insert a string of code that will fail your tests until function is not implemented, like this:
function doSearch(String query) {
fail('Not implemented');
}
Test that go through pages of results could be looking like follows:
test02_Paging (String query, Integer expected) {
String[] pageTokens = {"page1", ..."page 10"};
login();
doSearch(query);
for (int i=1; i<=10; i++) {
goToPage(i);
assertEqual(getPageToken(), pageTokens[i]);
}
}
Same way we can Design all tests that we selected for automation in advance. Thus we are saving this time from implementation and shorten the duration of automation. As practice had shown we can save up to 50% of time allocated for test automation. Roughly assessing, design phase is accounted for a half time allocated for the automation. So we shorten the second phase in a half, allowing automated tests to be ready sooner in the testing cycle.
Using this technology we have achieved the ratio of tests executed manually and automatically 1:1. This means that only a half of tests were executed manually each new release. Another half of tests were executed as automated right away. This had greatly increased automation ROI because we had no need to execute them manually at all and saved up to 40% of manual testing resources each release. Additionally, automated tests could be used for in-project regression testing much earlier. It also helped to get much of benefit from the idea.
In general, this approach had completely changed the role automated testing played in our project, making it at least as important and effective as manual testing.
Hope this helps making your test automation effort more fun! :) Feel free to comment should you have questions on details of implementation or should you have risks that may prevent you from having it work for you.
Thursday, June 11, 2009
The silver bullet of test automation
Note: This topic is about system test automation only. Developers (unit) testing is out of scope of this article.
I have purposely chosen such a sound name for this topic. I myself am a big believer in system test automation. But, as any other kind of tool, automation likes those who know what to do with it.
This is way too easy to be misguided by marketing stuff promising you to become more effective using their new generation super-duper automation tool. Beware, 90% of what advertisement says is a lie. The solution how it is presented may work on a very limited number of projects. What vendors don't say may hit you hard or even kill your test automation endeavor. Read on in order to get ready to concealed challenges.
Test automation is development
First thing to remember is that test automation is creating a program that tests another program. As any other kind of code it is governed by same lows. Unnecessary complexity, unclearness, duplicates, high coupling, hard-coding strings and constants have the same bad effect for test code as they do for production code. So, make sure that you know how to write code efficiently before jumping into automation boat.
Record&playback does not work
Do not believe anyone who tells you that test automation is as easy as record&playback. This is a pure lie. Following this model is undermining the biggest automation benefit - having reliable regression suite that requires little changes after changes in application. Tests generated by action recording tools are non-maintainable and you will need to re-record them afterwards. Test execution in this case will be more expensive than running those tests manually. Now think of how often you make changes to your system and try calculating whether you are going to have benefit from automation with record&playback.
Test automation is the investment
Be ready to invest. Automation is not a tool that starts working right away. There is always some leading cost involved. Yet the benefit you get down the line worth it all.
Implementation of a good automated test suite requires skilled staff, resources and time. Our suite took 4 engineers for 4 years. But the result was outstanding. It worked so well that we almost stopped testing for regressions manually. It helped us finding the issues we could never find any other way. So, our investment into automation stared paying back after about 2 years since the project start.
I concluded that automation starts working when you have at least all critical tests automation but it shines when the number of automated tests is over about 40% of your test backlog.
Automated test is strict
Automated test is a program and it can help you finding only those issues for which it is programmed. Manual testers look around, note different strange things that automation tests just pass by, on his mission to click this and read that.
Make sure you have a human being tested the feature before it is handed over to test automation.
Automation tools are as easy
I would never start automation project having no skilled professionals involved. Experience is the key. Else you are in position to pay for automation as much as 300% of its actual cost. Rephrasing famous Russian advertisement: Have too much money? So, we are going after you ;)
Tests have to be selected carefully
Ideally, you need to think of testing as a whole in the very beginning of the project. You have to get a clear picture of which tests you will have automated and which are better to be left manual. Test design should be oriented at contributing test automation using data-driven testing technique. Due test design may make your automation efforts 10 times more effective.
In the real world, we sometime face the situation when tests are already designed. Even in that case, if you have the possibility, look at test design from automation perspective. Change it the way as to get most benefit of your automation tool. Sometimes it makes sense to add tests in places where we had to compromise coverage for execution cost.
As far as implementing automated tests is an additional cost it can only be saved by execution. The best candidates for automation are those, which are executed often. Also there can be tests that are easy to automate. Doing them manually may be not reasonable.
Automated testing economics
I mentioned that automated tests pay for their creation during execution. There is a simple math example that demonstrates this postulate:
• 10 manual tests require 2 hours for execution.
• Implementation of these 10 tests would take 16 man hours.
• Execution of 10 automated tests takes 0 (zero) man hours.
• So in order to get the benefit we have to execute tests at least 16/2=8 times.
Follow this principle in assessing automation feasibility and you will have positive return on investment (ROI) at your test automation projects.
Note: Sometimes we fool ourselves wrongly assuming "perceived" frequency of test usage for the real one. The problem is that once we have tests implemented it's very easy to run them. So we run even those tests, which we would never execute manually. If you want to know the real benefit of the automation, try to get rid of desire to calculate this "perceived" frequency. However, reporting the “perceived” benefit can be good for you career ;)
Conclusion
Automation is a silver bullet but only in case when it is done by experiences professionals, on a lengthy projects (2 years and more), following strict best development practices and disciplines.
I have purposely chosen such a sound name for this topic. I myself am a big believer in system test automation. But, as any other kind of tool, automation likes those who know what to do with it.
This is way too easy to be misguided by marketing stuff promising you to become more effective using their new generation super-duper automation tool. Beware, 90% of what advertisement says is a lie. The solution how it is presented may work on a very limited number of projects. What vendors don't say may hit you hard or even kill your test automation endeavor. Read on in order to get ready to concealed challenges.
Test automation is development
First thing to remember is that test automation is creating a program that tests another program. As any other kind of code it is governed by same lows. Unnecessary complexity, unclearness, duplicates, high coupling, hard-coding strings and constants have the same bad effect for test code as they do for production code. So, make sure that you know how to write code efficiently before jumping into automation boat.
Record&playback does not work
Do not believe anyone who tells you that test automation is as easy as record&playback. This is a pure lie. Following this model is undermining the biggest automation benefit - having reliable regression suite that requires little changes after changes in application. Tests generated by action recording tools are non-maintainable and you will need to re-record them afterwards. Test execution in this case will be more expensive than running those tests manually. Now think of how often you make changes to your system and try calculating whether you are going to have benefit from automation with record&playback.
Test automation is the investment
Be ready to invest. Automation is not a tool that starts working right away. There is always some leading cost involved. Yet the benefit you get down the line worth it all.
Implementation of a good automated test suite requires skilled staff, resources and time. Our suite took 4 engineers for 4 years. But the result was outstanding. It worked so well that we almost stopped testing for regressions manually. It helped us finding the issues we could never find any other way. So, our investment into automation stared paying back after about 2 years since the project start.
I concluded that automation starts working when you have at least all critical tests automation but it shines when the number of automated tests is over about 40% of your test backlog.
Automated test is strict
Automated test is a program and it can help you finding only those issues for which it is programmed. Manual testers look around, note different strange things that automation tests just pass by, on his mission to click this and read that.
Make sure you have a human being tested the feature before it is handed over to test automation.
Automation tools are as easy
I would never start automation project having no skilled professionals involved. Experience is the key. Else you are in position to pay for automation as much as 300% of its actual cost. Rephrasing famous Russian advertisement: Have too much money? So, we are going after you ;)
Tests have to be selected carefully
Ideally, you need to think of testing as a whole in the very beginning of the project. You have to get a clear picture of which tests you will have automated and which are better to be left manual. Test design should be oriented at contributing test automation using data-driven testing technique. Due test design may make your automation efforts 10 times more effective.
In the real world, we sometime face the situation when tests are already designed. Even in that case, if you have the possibility, look at test design from automation perspective. Change it the way as to get most benefit of your automation tool. Sometimes it makes sense to add tests in places where we had to compromise coverage for execution cost.
As far as implementing automated tests is an additional cost it can only be saved by execution. The best candidates for automation are those, which are executed often. Also there can be tests that are easy to automate. Doing them manually may be not reasonable.
Automated testing economics
I mentioned that automated tests pay for their creation during execution. There is a simple math example that demonstrates this postulate:
• 10 manual tests require 2 hours for execution.
• Implementation of these 10 tests would take 16 man hours.
• Execution of 10 automated tests takes 0 (zero) man hours.
• So in order to get the benefit we have to execute tests at least 16/2=8 times.
Follow this principle in assessing automation feasibility and you will have positive return on investment (ROI) at your test automation projects.
Note: Sometimes we fool ourselves wrongly assuming "perceived" frequency of test usage for the real one. The problem is that once we have tests implemented it's very easy to run them. So we run even those tests, which we would never execute manually. If you want to know the real benefit of the automation, try to get rid of desire to calculate this "perceived" frequency. However, reporting the “perceived” benefit can be good for you career ;)
Conclusion
Automation is a silver bullet but only in case when it is done by experiences professionals, on a lengthy projects (2 years and more), following strict best development practices and disciplines.
Subscribe to:
Posts (Atom)