Friday, August 28, 2009

Test design: do it right the first time!

If you are in software testing field you probably noted how difficult it is to grasp the idea behind the test cases when all you see is a collection of steps presented for review. This is too easy to get bogged down into step details. You may have even got this impression "Wait a minute! I wanted to review what is supposed to be tested instead of reading individual steps trying to compile them into logical test sequences!" If you did – then read on. Some advises may prove to be useful for you. If you never experienced such issue read on anyway. The fact you never noted the problem does not mean it cannot not hit you ;) Remember, this is not what you know will kill you.

The solution to the problem is of the same sort as we used to apply in other types of activities - early validation. So, you need the means to early identify problems with your design, until it gets "hardcoded" into written form, with steps and test data. The question remains – how to present testing ideas in a short and convenient for review form? There are a couple of ideas. Both proved to be quite successful in my experience.

The first idea is to generate a list of test conditions or test requirements before starting to generate test cases. The list can be presented in form of a tree:

- file types:
---- doc
---- txt
---- pdf
---- xml
- file content:
---- text
---- pictures
---- text + pictures
- file size:
---- large
---- empty

The time needed to generate such a list is negligible. You don't even need to bother about test data at this point. The whole idea you put behind the testing may be wrong, so time spent for creating complex test files will be a total waste.

Tree forms allows for easy review. It helps providing your thoughts in a structured way and keeps reader's attention at one context at a time, thus improving the quality of analysis. Seeing such a list one may easily find out that there are no tests for specific conditions (text in UT8 format for example). The only problem with this approach is that you do not tell how you are going to validate those conditions.

Another way is providing an overview along with your test design. It works as well to better review process, but it does not help against rework. If you have chosen wrong way in your design, you will need to re-develop your test cases.

Overview is written for the reviewer. It shall answer the following questions:

- What is supposed to be tested?
- How is it supposed to be tested?
- What conditions are to be tried out?
- How the results are going to verified?
- Whether or not automated testing will be used?
- Risks and limitations of selected testing strategy.

Having the answers on these questions will allow a reviewer to easily grasp the ideas behind the testing and to validate them.

Test design review is a hard task because there is a lot of a detail. The main goal of review is not to make sure details are correct. We are much more interested to see if testing design is adequate to get confidence that, once it pass, we have a quality product. There is no need to provide obscuring details to review. The early you present your work to review and the easiest it will be to understand it - the best quality and productivity you can achieve.

3 comments:

  1. Vladimir,

    Nice post.

    You wrote "we are much more interested to see if testing design is adequate to get confidence that, once it passes we have a quality product." I agree. From my experience, testing every test condition at least once is not adequate to accomplish this goal.

    To achieve a reasonable level of confidence in quality, from my experience, it is very often advisable to design tests to achieve AT LEAST pairwise coverage, if not 3-way coverage. Pairwise coverage, for example, would guarantee that not only would you test "text" as an input type but that "text" would be tested in at least one test case against every other variable (e.g., "text" with "doc file type" in one test case; "text" with "txt file type" in another test case; "text" with "pdf file type" in another test case; "text" with "xml file type" in another test case and (in the test cases listed above, being sure to include "text" with "big file size" and "text" with "small file size".

    For anyone interested in finding out more, tools like our Hexawise test case generator can be extremely useful to accomplish this. We have a free version available at http://www.hexawise.com/users/new


    - Justin

    ReplyDelete
  2. Vladimir,

    Nice thoughts.

    The idea of preparing list of test conditions or test requirements & presenting in tree format is simply great.

    It surely helps a lot when we design testcase manually or using any Test Case Generator (as mentioned by Justin)

    ReplyDelete
  3. to hexawise

    Yep! What you have mentioned is called dependent test requirements. This effecte comes to action when one requirement's existance or features may depend on another one. For example, it may matter what size of files of different types you try to open with your text editor. If it support up to 1MB files then txt file will contain more textual information than Word or HTML.

    Another example of dependent requirements is search in Notepad. It may go up and down, depending on the choosen options. So, if you test finding first and finiding last symbols you will need to test both directions. Test requriements in this case will look like follows:

    - find
    -- up
    --- find first
    --- find last
    -- down
    --- find first
    --- find last

    ReplyDelete