ISC UI testing architecture

Several days ago, while working on a following test automation project for a desktop application an idea how to make it right the first time has struck me. This is the first time I am writing on this matter, so be my co-authors and provide any thoughts which happen to come to your mind while reading this. Any feedback will be greatly appreciated.

Dealing with UI looks simple at the first glance but it is rather tricky afterwards. All the time you have to do some UI testing (confirmation messages, cancel buttons, input error messages and the like) you have to step back and to change (refactor) the code that you have already wrote.

For example, you have created a function that opens a project file in your application. It takes the name of the file for the input parameter. Then it click menu File/Open, fills in file path and click button Ok.

You created the tests that rely of that function. Several tests may open different files and validate their content. But now you need to create a test, which will open an incorrect file. In result you shall see a message telling you what is wrong with it. But your original function does rely on smooth flawless file opening. It does not take into account any errors which may come up. So, we go back to the original file opening function and refactor it in order to fit both needs (usually it is done by creating a new function that does only part of the job).

This is only on example of hundreds or even thousands for a big project. And any time you have to step back, do changes and test the changes. This is tricky and error-prone.

Instead I am suggesting DOING THINGS RIGHT THE FIRST TIME - Invoker-Setter-Confirmer (ISC) test architecture.

ISC is a UI test architecture that allows you to think in terms of functional bricks of which you can build any kind of UI based tests. You will not need to go back afterward, I promise. If you do, I am gonna eat my hat :)

ISC stands for the set of functions, each with its own purpose described below.

Invokers - functions responsible for making a UI element to appear on the screen. In most cases it can be done with more than one way (File Open dialog may be caller through menu, toolbar, accelerators and hot keys).

Setters - functions responsible for setting input values into UI element. For example, find dialog requests user to set a string to find, direction of the search and other properties. All those values shall be set through a setter functions.

Confirmers - functions which do some actions with UI element. For a dialog they click a button or close it with Esc or Alt+F4. Just like invocation the confirmation can be done in different ways.

The best way to explain it is to show an example. Here is goes.

Everyone has a Notepad program (Mac users, I am very sorry). Let's create ISC-based library for Find dialog.

// code is in Java Script

// constants
notepad = Aliases.notepad;
wndNotepad = notepad.wndNotepad;

ACCELERATOR = 1;
MENU = 2;
HOTKEYS = 3;

OK = 10;
CANCEL = 11;
ESC = 12;
ALTF4 = 13;

//////////////////////////////////////
// Invoker
//////////////////////////////////////
function InvokerDlgFind(method) {
switch(method) {
case ACCELERATOR :
wndNotepad.Keys("[Ctrl]F");
break;
case MENU :
wndNotepad.MainMenu.Click("Edit|Find...");
break;
case HOTKEYS :
wndNotepad.Keys("[Alt]EF");
break;
default:
InvokerDlgFind(MENU);
break;
}
// No more code. Just show this thing on
// the screen and leave.
}

//////////////////////////////////////
// Setter
//////////////////////////////////////
function SetDlgFind(toFind, case, direction) {
dlgFind.Edit.wText = toFind;
dlgFind.checkMatchCase.ClickButton(case);
if(direction) {
dlgFind.radioUp.ClickButton();
} else {
dlgFind.radioDown.ClickButton();;
}
// that's it! Setters do not do anything else.
}

//////////////////////////////////////
// Confirmer
//////////////////////////////////////
function ConfirmDlgFind(method) {
switch(method) {
case OK :
notepad.dlgFind.btnOK.ClickButton();
break;
case CANCEL :
notepad.dlgFind.btnCancel.ClickButton();
break;
case ESC :
notepad.dlgFind.Keys("[ESC]");
break;
case ALTF4 :
notepad.dlgFind.Keys("[Alt][F4]");
break;
default:
ConfirmDlgFind(OK);
break;
}
}

// Now how it looks in the tests

function Test_Find_Succeeded() {
// open test file

InvokeDlgFind(MENU);
SetDlgFind("123", false, true);
ConfirmDlgFind(OK);

// verify find result
}

function Test_Find_NotFound() {
// open test file

InvokeDlgFind(MENU);
SetDlgFind("456", false, true);
ConfirmDlgFind(OK);

// verify message "not found"
}

function Test_Find_EmptyInput() {
// open test file

InvokeDlgFind(MENU);
SetDlgFind("", false, true);

// verify button is disabled
}

Hope this helps. Will be glad to hear from you on this matter.

Defect tracking workflow

Lately I was involved in creating yet another defects tracking workflow for a big outsourcing company. The goal was to define a standard way of tracking defects keeping in mind the support for defect analysis and prevention.

The task per se does not look too difficult. In most of cases, the standard workflow that your DTS has embedded will do the job four you. But it does not in our case. The workflow for defects defined as default in JIRA is too minimalistic. We used to have additional states and transitions that we want to be noticed and filed by the system.

So we have come up with several additional states. Some default states were renamed. Some transition rules were added and permissions changed. But I do not want to get into a boring description of how we were doing it. I want to outline what I think is most important in building workflows:

1. Keep it simple, stupid!

The all times rule of thumb is more than just applicable here. We can ideate dozens of states and hundreds of names for transitions in pursue for the perfection. But do we really need to get perfect in this case? get back to the intention that you had starting all this. The intentions were: have means to automate defect tracking to make sure nothing is wasted or lost; have data records to support learning from mistakes; have means to control the project effectively. That's it! So, if you have enough states and transitions to satisfy those requirements - stop right there, don't go any further.

2. More flexibility, more problems

If you allow to much flexibility in sense of who does what in the DTS you lend yourself and your colleagues for numerous mistakes. This is always better to let people do only those actions that they are supposed to, no more. In terms of defining workflows it gets down to defining user groups and assigning permitted actions for those groups. Once you've done it you are sure that no issue will go a wrong path.

3. Self explanatory, intuitive names

States, transitions, priority and severity of issues - all this must not require additional explanation. Name "Deferred" or "Postponed" is better than "Not for fixing" or "Not important".

4. Test it twice before going live!

Have somebody else to go through the workflow before letting others use it. I did for my last installment and I wasn't surprised to fix 6 critical issues in it despite I was sure I passed it through myself.

Happy workflow engineering! :)

S60 emulator connection problem

If you experience this problem the first thing to try is:

- In Eclipse go to Window / Preferences / Debug and set timeouts to 100000.

If it does not help then try...

- to start emulator manually in Dedug mode and then start debugging from Eclipse.


In case the latter did not help and you receive something like "You could start only one instance of..." then close emulator and start it again.

TestComplete 7.X refuses to recognize .NET objects?

If you cannot see .NET objects with appropriate icon in Object Browser, then most likely you have no corresponding plug-in installed or you have .NET 4.0. Check you extension settings and install .NET plugin. In the later case uninstalling .NET 4.0 will not solve the problem. You will need to re-install whole system. Alas!

LoadRunner crashes on recording?

Here is the cure...

1. Control Panel / System
2. Advanced
3. Performance Settings
4. Data Execution Prevention
5. Turn on DEP for essential Windows programs and services only.
6. Restart!

That's it :)

Load testing is not easy

Recently I have convinced in it myself yet another time. The task looked as simple as generating load on a site accepting several files for parameters. As usual it was easy to record and parametrize scripts, debug and plan the first load.

The difficulties appear during the testing. Every time when I approach load testing the results kind of amaze me. They are not what I would expect. This time was no different. I was amazed and did at least 5 times more test execution than planned originally.

Thanks to my managerial experience I have lend myself some time for contingencies. So it worked well and I met the deadline :)

Every time you plan for load testing keep in mind that this is investigation rather than determined work in sense that we put in it when we approach planning.

What a developer wants?

After so many years of driving testing and QA processes I have some possibilities to realize and rights to publish here some of ideas on how things are working from the inside. One of such things is what developers ask of a QA manager.

My top 5 are following:

1. Change defect tracking system

2. Stop reporting duplicates and non-defects

3. Do no find defects too late

4. Stop reporting minor/cosmetic defects in such a number

5. Test design is not review-able

Now back to the details =)

Change the defects tracking system I am always happy when people come to me with their ideas which they think will help them doing their work better. However, I don't like when it stems from the "I don't know what is wrong but I feel that something needs to be changed" intention.

In the past, developers came to me complaining that it takes too much time to work with defects. The only solution they saw was to change the tracking system. I asked in respond what exactly does take time when they work with it. As it turned out the reason of their complaints is not in the system per se but in the time they had to spend understanding the defect and trying to reproduce it. No defect tracking system can change that, so they went out in peace with the idea to leave the old system. Others developers came after... period =)

Stop reporting duplicates and non-defects Yes, this is a problem. But let's see how big it is. On my memory we reported not more than 1% of bad defects. Is that so huge to be whining about at each managers meeting? Nope. Then what was it?... I'll tell you. This is an excuse for not doing own job right. Kind of "hey! they also have problems". It makes them look better in they eyes. What a shame! =)

Do no find defects too late Another problem for testers. We have to find serious problems as soon as they appear in the code. There is no excuse... Wait a minute! What issue are speaking about? That one that developers introduced in the same build? How do you think testers could find it earlier?... We hear such acquisitions too many times. Make sure you take the blame for your own mistakes.

Stop reporting minor/cosmetic defects in such a number The answer is "stop reading them". It's easy =)

Test design is not review-able In majority of cases they just don't care to put some efforts in this. This is merely a laziness that cannot be an excuse :) However, providing a summary of your tests design is always helpful. Remember, those who read your design help you making it better, so it's in your interest to make it easier to them.