Estimation of testing

If you need to ground testing estimates on development estimates then read on.

This is definitely not the best way of producing testing estimation. It would be more correct to ask development and testing do their estimates independently. Later on you can use both estimates to analyze the difference and to arrange those properly.

In most of cases testing takes from 30% to 35% of development estimates. Taking 35% you will lend enough time for your team to complete the goals on time and with due quality.

But be careful! There can be tasks, which execution may run out of the initial estimates. For example:

• Performance testing
• Load testing
• Compatibility testing
• Testing on a real (big) amount of data or in real hardware
• Reliability testing
• Test automation
• Complex environment setup
• Complex business area (business context)

All above as well as the risks and any kind of exotic testing should be estimated separately and the result should be added to the initial rough estimates.


Here is the full algorithm as I see it:

1. Both developers and tester do their estimates separately.
2. Then estimates are compared and the difference is being explained.
3. If the ratio in estimates is as expected you are done.
4. If not - the difference is explained and a corrective actions are taken.


Notes:
• Do not take the biggest "just in case" - this is inefficient.
• Do not take the "most trustful" - explain.


The biggest ratio I have ever met in my career was 41% of development efforts. It was due to heavy use of test automation and with complex test environment.

Happy estimation! :)

Performance Appraisals

Many of you have heard of this, some were even appraised and few where directly involved in creation of such reviews. No doubt, reviewing someone's contribution helps to develop skills required to drive the team to strategic goals as well as to avoid silly mistakes. But today I am not going to judge upon the process of performance reviews, nor I am here to express ideas on how to do it right. The mere fact I am writing this is to resolve a potential risk for those of you who just start to implement this process.

Some believe that performance appraising (PA) is a great tool in managing someone's... expectations of the compensation. So, it's... wrong!

We should not mess together things that are better to keep apart (herring and milk so to say), things that serve different purposes. Compensation is a way to keep employee from leaving the company. PA is a tool in hands of a manager but it is written for the employee. These goals are not the same. If you pretend they are, you will end up having a logical inconsistency because if you introduce salary parameter into equation then you will inevitably write a review which will serve your goals on keeping the salaries down or keeping the employee. Anyways there will be no "written for employee" thing anymore.

So, don't mess it up. Write reviews in order to help people to develop the required skills and fix the problems. Use salary to keep and motive people to move ahead. No need to blend it all together if you want to stay mentally healthy ;)

If I confused things for you just leave me a note. It's my bad and I will do my best to help you out :)

ISC UI testing architecture

Several days ago, while working on a following test automation project for a desktop application an idea how to make it right the first time has struck me. This is the first time I am writing on this matter, so be my co-authors and provide any thoughts which happen to come to your mind while reading this. Any feedback will be greatly appreciated.

Dealing with UI looks simple at the first glance but it is rather tricky afterwards. All the time you have to do some UI testing (confirmation messages, cancel buttons, input error messages and the like) you have to step back and to change (refactor) the code that you have already wrote.

For example, you have created a function that opens a project file in your application. It takes the name of the file for the input parameter. Then it click menu File/Open, fills in file path and click button Ok.

You created the tests that rely of that function. Several tests may open different files and validate their content. But now you need to create a test, which will open an incorrect file. In result you shall see a message telling you what is wrong with it. But your original function does rely on smooth flawless file opening. It does not take into account any errors which may come up. So, we go back to the original file opening function and refactor it in order to fit both needs (usually it is done by creating a new function that does only part of the job).

This is only on example of hundreds or even thousands for a big project. And any time you have to step back, do changes and test the changes. This is tricky and error-prone.

Instead I am suggesting DOING THINGS RIGHT THE FIRST TIME - Invoker-Setter-Confirmer (ISC) test architecture.

ISC is a UI test architecture that allows you to think in terms of functional bricks of which you can build any kind of UI based tests. You will not need to go back afterward, I promise. If you do, I am gonna eat my hat :)

ISC stands for the set of functions, each with its own purpose described below.

Invokers - functions responsible for making a UI element to appear on the screen. In most cases it can be done with more than one way (File Open dialog may be caller through menu, toolbar, accelerators and hot keys).

Setters - functions responsible for setting input values into UI element. For example, find dialog requests user to set a string to find, direction of the search and other properties. All those values shall be set through a setter functions.

Confirmers - functions which do some actions with UI element. For a dialog they click a button or close it with Esc or Alt+F4. Just like invocation the confirmation can be done in different ways.

The best way to explain it is to show an example. Here is goes.

Everyone has a Notepad program (Mac users, I am very sorry). Let's create ISC-based library for Find dialog.

// code is in Java Script

// constants
notepad = Aliases.notepad;
wndNotepad = notepad.wndNotepad;

ACCELERATOR = 1;
MENU = 2;
HOTKEYS = 3;

OK = 10;
CANCEL = 11;
ESC = 12;
ALTF4 = 13;

//////////////////////////////////////
// Invoker
//////////////////////////////////////
function InvokerDlgFind(method) {
switch(method) {
case ACCELERATOR :
wndNotepad.Keys("[Ctrl]F");
break;
case MENU :
wndNotepad.MainMenu.Click("Edit|Find...");
break;
case HOTKEYS :
wndNotepad.Keys("[Alt]EF");
break;
default:
InvokerDlgFind(MENU);
break;
}
// No more code. Just show this thing on
// the screen and leave.
}

//////////////////////////////////////
// Setter
//////////////////////////////////////
function SetDlgFind(toFind, case, direction) {
dlgFind.Edit.wText = toFind;
dlgFind.checkMatchCase.ClickButton(case);
if(direction) {
dlgFind.radioUp.ClickButton();
} else {
dlgFind.radioDown.ClickButton();;
}
// that's it! Setters do not do anything else.
}

//////////////////////////////////////
// Confirmer
//////////////////////////////////////
function ConfirmDlgFind(method) {
switch(method) {
case OK :
notepad.dlgFind.btnOK.ClickButton();
break;
case CANCEL :
notepad.dlgFind.btnCancel.ClickButton();
break;
case ESC :
notepad.dlgFind.Keys("[ESC]");
break;
case ALTF4 :
notepad.dlgFind.Keys("[Alt][F4]");
break;
default:
ConfirmDlgFind(OK);
break;
}
}

// Now how it looks in the tests

function Test_Find_Succeeded() {
// open test file

InvokeDlgFind(MENU);
SetDlgFind("123", false, true);
ConfirmDlgFind(OK);

// verify find result
}

function Test_Find_NotFound() {
// open test file

InvokeDlgFind(MENU);
SetDlgFind("456", false, true);
ConfirmDlgFind(OK);

// verify message "not found"
}

function Test_Find_EmptyInput() {
// open test file

InvokeDlgFind(MENU);
SetDlgFind("", false, true);

// verify button is disabled
}

Hope this helps. Will be glad to hear from you on this matter.

Defect tracking workflow

Lately I was involved in creating yet another defects tracking workflow for a big outsourcing company. The goal was to define a standard way of tracking defects keeping in mind the support for defect analysis and prevention.

The task per se does not look too difficult. In most of cases, the standard workflow that your DTS has embedded will do the job four you. But it does not in our case. The workflow for defects defined as default in JIRA is too minimalistic. We used to have additional states and transitions that we want to be noticed and filed by the system.

So we have come up with several additional states. Some default states were renamed. Some transition rules were added and permissions changed. But I do not want to get into a boring description of how we were doing it. I want to outline what I think is most important in building workflows:

1. Keep it simple, stupid!

The all times rule of thumb is more than just applicable here. We can ideate dozens of states and hundreds of names for transitions in pursue for the perfection. But do we really need to get perfect in this case? get back to the intention that you had starting all this. The intentions were: have means to automate defect tracking to make sure nothing is wasted or lost; have data records to support learning from mistakes; have means to control the project effectively. That's it! So, if you have enough states and transitions to satisfy those requirements - stop right there, don't go any further.

2. More flexibility, more problems

If you allow to much flexibility in sense of who does what in the DTS you lend yourself and your colleagues for numerous mistakes. This is always better to let people do only those actions that they are supposed to, no more. In terms of defining workflows it gets down to defining user groups and assigning permitted actions for those groups. Once you've done it you are sure that no issue will go a wrong path.

3. Self explanatory, intuitive names

States, transitions, priority and severity of issues - all this must not require additional explanation. Name "Deferred" or "Postponed" is better than "Not for fixing" or "Not important".

4. Test it twice before going live!

Have somebody else to go through the workflow before letting others use it. I did for my last installment and I wasn't surprised to fix 6 critical issues in it despite I was sure I passed it through myself.

Happy workflow engineering! :)

S60 emulator connection problem

If you experience this problem the first thing to try is:

- In Eclipse go to Window / Preferences / Debug and set timeouts to 100000.

If it does not help then try...

- to start emulator manually in Dedug mode and then start debugging from Eclipse.


In case the latter did not help and you receive something like "You could start only one instance of..." then close emulator and start it again.

TestComplete 7.X refuses to recognize .NET objects?

If you cannot see .NET objects with appropriate icon in Object Browser, then most likely you have no corresponding plug-in installed or you have .NET 4.0. Check you extension settings and install .NET plugin. In the later case uninstalling .NET 4.0 will not solve the problem. You will need to re-install whole system. Alas!

LoadRunner crashes on recording?

Here is the cure...

1. Control Panel / System
2. Advanced
3. Performance Settings
4. Data Execution Prevention
5. Turn on DEP for essential Windows programs and services only.
6. Restart!

That's it :)