Tuesday, August 4, 2009

Scripted testing or free flying?

The more I work in testing domain, the more I learn how difficult it is to answer this question. On one hand we need to be very diligent and specific about things we are testing. We have to be ready to answer what we tested and how. We have to collect information on what was tested and how in order to do retrospective review. On another hand, many test professionals believe that the best way to find a defect is get into something like a trance, a "free flying" through the functionality. Like a dog sniffing its pray, tester searches defects following gut instincts. This is way too difficult to organize in form of a checklist or guidelines. You can hardly explain what it takes to ride a bicycle or jump with a parachute to a person new to those activities. All you can achieve is "I have no idea what you are talking about" response. Same is for the testing.

So what is the best way for you, you may ask? The answer cannot be universal for everyone because each team is unique. The way we used to work, the qualities of team members and peculiarities of a task are very important. I personally believe that a mix of both approaches is what works better for most of the organizations. But what kind of a mix should it be? One may try to script everything, including tests executed once (executed and throw away), like in medical industry. Others may not bother having such tests written down on the paper, allowing some time for their testers to do non-scripted testing fee of any bureaucracy. What mix works best for you is a matter of numerous trial and errors. Try different approaches and see what effect it may have. If it's a waste of time don't do it.

Unlike test design based on product requirements "free flying" testing is in-context activity, so it takes less efforts to generate ideas. When you do not have application to click buttons and try different combinations it is difficult to come up with ideas. Modeling software behavior in mind is resource-consuming task. Your mind simply has not enough free processor time required for the creativity.

In general, scripted testing adds more organization and clarity in testing process. It helps in making the testing process visible and controllable, thus helping to meet deadlines. Having no test scripted, you can hardly say how long it will take to run all tests, because you can only guess how many of them you are going to execute.

So, the best way of dealing about it on the level of testing schedule is generating scripted tests against requirements (and don't bother too much about very specific ways of using software at this point) and allow a day or two (depending on how big the functionality is) for "free flying" testing. These two days are better to be scheduled at the end of the testing stage. Prior scripted testing will help removing blocking issues as well it will lend your testers time to get familiar with new functionality.

All is in your hands. Only you can say what is best for your specific situation. Good luck!


  1. Scripted testing is good at regression testing, Performance and load testing. But free fly can find bugs more fastly in the start but , after sometime its waste more time then find new bugs. I thing their is a need to adopt the formal approaches in testing and do the pragmatic testing from at the unit level.

  2. Yes. Both approached have advantages and disadvantages. I doubt one may free-fly-test something for more than 2 days in a row. He or she will simply be out of wits. Scripted testing, in contrary, is very limited because this is not possible to effectively define testing strategy having no working software at hand. So, the article above speaks of a mix of both that works best. How much of both to include is up to specific business conditions. Thanks for the comment!

  3. Vladimir,

    Another nice post. I like how you point out the pro's and cons of scripted vs. free flying.

    Another good blog post to check out was written by Jared Quinert and is found at: http://www.software-testing.com.au/blog/2009/06/23/test-automation-models-dehumanising-testers-and-agile-dysfunctions/

    Like your post, it takes a balanced view on pro's and con's of scripted testing (defending scripted testing against a blunt, overgeneralized and and inaccurate criticism of scripted manual testing that Jared came across on Twitter.

    - Justin Hunter
    Founder of Hexawise

  4. to Justin

    Thanks for your comments! =) Did you try both or any of above yourself? What are your impressions?