Automated tests: the sooner the better!

This is the most impressive experience I gained in my entire professional career. Read on and you will probably find it worth of trying.

Several years ago we faced the need to dramatically increase test automation coverage. Let the reasons be aside for now and let's focus on the problem per se. We had complex UI-driven software, full of custom controls and volatile UI elements changed from release to release. Our experience in test automation was refrained to the collection of several hundred tests that we occasionally used for smoke and regression testing. In order to get where we needed to be we had to automate ten thousand tests in a very short period of time, 1-2 years.

At the first glance the problem had no solution. It was really hard to be in that situation, when you see no way out. But we started the project having no idea if it's going to be successful. We started from optimizing of what we already can do, divided team in two sub-teams (one team focused on test design and manual execution, another took responsibility for test automation). We also introduced all the best practices that helped us to keep test production and maintenance cost low.

But this was not enough. Automation team was always behind testing process because they needed time to implement and debug test code on a working system. With insignificant exceptions, so, we could not use automated tests for functional test execution during the production cycle. We could only use it for regression testing of future versions. Tests still needed to be executed manually at least once. It had severe impact on testing schedule and drug us back because manual team was always on a critical path.

Then we’ve got an idea. It was as easy as it could only be. As far as implementation phase is what makes automation team fall behind we have to shorten it somehow. Adding resources is not an option. We already had several people working on this project and adding more people could severely increase overhead cost of communication. We went anther way.

Instead of adding resources we decided to move a part of test automation preparation task back in time, to do them in advance. We decided to do design of automated tests long before a working version is made available. Design means that we created fish bones of tests. Those tests used virtual functions that, when implemented, allow manipulating application features. Functions remained not implemented until we have a working version of a product in our hands. So, the tests could not be debugged until that time. But the most of design work is already done. And our experience indicated that this part is very significant.

When a new version of application comes to testing, automation engineers started implementation phase. They simply added code to the helper functions to make the test work. After that they run and debugged tests.

I will demonstrate how it worked in example:

1. We need to test a web search system that allows users to run the search, browse results and bookmark interesting findings.

2. Automation engineers select tests for automation. For example they have selected the following tests:

a. Different types of queries ("", "my search", "very-very-very long string").
b. Browse results (pages 1 through 10).

3. Automation team has steps of tests and test data defined in the test description created by manual team. So, thy create test architecture design like this:

test 1 - Different types of queries

test01_Search (String query, Integer expected) {
login();
doSearch(query);
assertEqual(getResults().totalCount, expected);
}

Used functions login(), doSearch(), and getResults() are not implemented so far! We have only figured out which functions we will need to enable our tests work.

Note: In order to do it safely it is recommended to insert a string of code that will fail your tests until function is not implemented, like this:

function doSearch(String query) {
fail('Not implemented');
}

Test that go through pages of results could be looking like follows:

test02_Paging (String query, Integer expected) {
String[] pageTokens = {"page1", ..."page 10"};
login();
doSearch(query);
for (int i=1; i<=10; i++) {
goToPage(i);
assertEqual(getPageToken(), pageTokens[i]);
}
}

Same way we can Design all tests that we selected for automation in advance. Thus we are saving this time from implementation and shorten the duration of automation. As practice had shown we can save up to 50% of time allocated for test automation. Roughly assessing, design phase is accounted for a half time allocated for the automation. So we shorten the second phase in a half, allowing automated tests to be ready sooner in the testing cycle.

Using this technology we have achieved the ratio of tests executed manually and automatically 1:1. This means that only a half of tests were executed manually each new release. Another half of tests were executed as automated right away. This had greatly increased automation ROI because we had no need to execute them manually at all and saved up to 40% of manual testing resources each release. Additionally, automated tests could be used for in-project regression testing much earlier. It also helped to get much of benefit from the idea.

In general, this approach had completely changed the role automated testing played in our project, making it at least as important and effective as manual testing.

Hope this helps making your test automation effort more fun! :) Feel free to comment should you have questions on details of implementation or should you have risks that may prevent you from having it work for you.

Ho to obtain good estimates?

Today, we will speak about eliciting estimates from people. It would make the task easier if your peers who oppose you in this matter would read "how to make good estimates?" first. Even if they didn't this is in your hands to lead them the right path.

First of all make sure the task is understood by the performer. Ask him or her to paraphrase the assignment to you, so you can correct it until it's too late. This is also important to help the performer understand what other task performed in the past could be used for comparison or verification of the estimate. Sit down together and find what you did before that can be measured up against a new task (similar project, task). However, be careful to use examples more than 3 times bigger or smaller in size.

If you don't have examples and the task is completely new to a performer then define the plan how to analyze the problem (investigation, prototyping, consulting gurus, etc.). One of most important parts of this stage is having a concrete goal, a milestone when the investigation will complete, or at least a date of its completion will be known.

Once you have got first estimate do not accept it as is. Question it. People tend to take estimates too optimistic, so they have troubled in meeting them afterwards. In order not to fall prey of wishful thinking, start asking "what if?" question. What is that part will not be delivered to you for testing at the date you assumed? What is if it will take longer to test your code? I have seen that testing of such modules takes up to 30% of development time. What if you have no product requirements finished at the desired date? What else you can do to mitigate that risk? Asking such questions will help you analyze different scenarios of "how it may go wrong". Such scenarios are usually neglected by people who used to think overly optimistically.

This is also helpful to build your own estimates or to ask another expert to provide a variant of the estimation. Comparing alternatives is a key to finding out what is missed from consideration. Creating estimates using different parameters is another way of drawing alternatives for the analysis (calculate time needed to test out of development hours vs. the same parameter calculated from the number of requirement). Never use average, or some percent of several estimates. This is completely wrong because our goal is to find why the estimates are different, so we can correct them with new information that has been missed from consideration during the initial estimation.

This is it! It's pretty easy and straightforward process :)

As for the methods of estimation I recommend you reading "How much does a project cost?" by Steve McConnel.