Test design: do it right the first time!

If you are in software testing field you probably noted how difficult it is to grasp the idea behind the test cases when all you see is a collection of steps presented for review. This is too easy to get bogged down into step details. You may have even got this impression "Wait a minute! I wanted to review what is supposed to be tested instead of reading individual steps trying to compile them into logical test sequences!" If you did – then read on. Some advises may prove to be useful for you. If you never experienced such issue read on anyway. The fact you never noted the problem does not mean it cannot not hit you ;) Remember, this is not what you know will kill you.

The solution to the problem is of the same sort as we used to apply in other types of activities - early validation. So, you need the means to early identify problems with your design, until it gets "hardcoded" into written form, with steps and test data. The question remains – how to present testing ideas in a short and convenient for review form? There are a couple of ideas. Both proved to be quite successful in my experience.

The first idea is to generate a list of test conditions or test requirements before starting to generate test cases. The list can be presented in form of a tree:

- file types:
---- doc
---- txt
---- pdf
---- xml
- file content:
---- text
---- pictures
---- text + pictures
- file size:
---- large
---- empty

The time needed to generate such a list is negligible. You don't even need to bother about test data at this point. The whole idea you put behind the testing may be wrong, so time spent for creating complex test files will be a total waste.

Tree forms allows for easy review. It helps providing your thoughts in a structured way and keeps reader's attention at one context at a time, thus improving the quality of analysis. Seeing such a list one may easily find out that there are no tests for specific conditions (text in UT8 format for example). The only problem with this approach is that you do not tell how you are going to validate those conditions.

Another way is providing an overview along with your test design. It works as well to better review process, but it does not help against rework. If you have chosen wrong way in your design, you will need to re-develop your test cases.

Overview is written for the reviewer. It shall answer the following questions:

- What is supposed to be tested?
- How is it supposed to be tested?
- What conditions are to be tried out?
- How the results are going to verified?
- Whether or not automated testing will be used?
- Risks and limitations of selected testing strategy.

Having the answers on these questions will allow a reviewer to easily grasp the ideas behind the testing and to validate them.

Test design review is a hard task because there is a lot of a detail. The main goal of review is not to make sure details are correct. We are much more interested to see if testing design is adequate to get confidence that, once it pass, we have a quality product. There is no need to provide obscuring details to review. The early you present your work to review and the easiest it will be to understand it - the best quality and productivity you can achieve.

Scripted testing or free flying?

The more I work in testing domain, the more I learn how difficult it is to answer this question. On one hand we need to be very diligent and specific about things we are testing. We have to be ready to answer what we tested and how. We have to collect information on what was tested and how in order to do retrospective review. On another hand, many test professionals believe that the best way to find a defect is get into something like a trance, a "free flying" through the functionality. Like a dog sniffing its pray, tester searches defects following gut instincts. This is way too difficult to organize in form of a checklist or guidelines. You can hardly explain what it takes to ride a bicycle or jump with a parachute to a person new to those activities. All you can achieve is "I have no idea what you are talking about" response. Same is for the testing.

So what is the best way for you, you may ask? The answer cannot be universal for everyone because each team is unique. The way we used to work, the qualities of team members and peculiarities of a task are very important. I personally believe that a mix of both approaches is what works better for most of the organizations. But what kind of a mix should it be? One may try to script everything, including tests executed once (executed and throw away), like in medical industry. Others may not bother having such tests written down on the paper, allowing some time for their testers to do non-scripted testing fee of any bureaucracy. What mix works best for you is a matter of numerous trial and errors. Try different approaches and see what effect it may have. If it's a waste of time don't do it.

Unlike test design based on product requirements "free flying" testing is in-context activity, so it takes less efforts to generate ideas. When you do not have application to click buttons and try different combinations it is difficult to come up with ideas. Modeling software behavior in mind is resource-consuming task. Your mind simply has not enough free processor time required for the creativity.

In general, scripted testing adds more organization and clarity in testing process. It helps in making the testing process visible and controllable, thus helping to meet deadlines. Having no test scripted, you can hardly say how long it will take to run all tests, because you can only guess how many of them you are going to execute.

So, the best way of dealing about it on the level of testing schedule is generating scripted tests against requirements (and don't bother too much about very specific ways of using software at this point) and allow a day or two (depending on how big the functionality is) for "free flying" testing. These two days are better to be scheduled at the end of the testing stage. Prior scripted testing will help removing blocking issues as well it will lend your testers time to get familiar with new functionality.

All is in your hands. Only you can say what is best for your specific situation. Good luck!