Today I partook in Beta program for a test generation tool. This tool accepts the list of parameters with possible values for the input and generates combinations, which represented virtually complete coverage (all values of all parameters represented at least once).
Despite the idea is good, the actual use of such tool is in doubt. Of dozens systems that we tested I could name one or two where we could apply this idea. The idea of cutting testing cost by combining test input is not new. I have got acquainted with it while reading Marick's "The craft of software testing" in the end of 20th century. Author tried to minimize test count by carefully combining input values, taking into account dependencies between them. It was beautiful but it proved to be useless in the real practice. The time spent on optimizing tests can be easily spent running more tests (including those, removed with optimization).
Another problem with the idea is that combinations cannot be seen from mathematical point of view only. There are other factors that can affect selection of combinations to try. We may know that testing under a specific browser is more important because it is very likely that application can fail in it. Whereas testing under another browser can be minimal because we know that all development is performed using it. So, the context does matter and test generation tools shall take it into account or they will stay as useless.
No comments:
Post a Comment