Decreasing Response Time of Failing Automated Tests by Applying Test Case Prioritization

Abstract

Running automated tests can be a time-consuming task, especially when doing regression testing. If the sequence of the execution of the test cases is arbitrary, there is a good chance that many of the defects are not detected until the end of the test run. If the developer could get the failing tests first, he would almost immediately be able to get back to coding or correcting mistakes. In order to achieve this, we designed and analyzed a set of test case prioritization techniques. The prioritization techniques were compared in an experiment, and evaluated against two existing techniques for prioritizing test cases.Our implementation of the prioritization techniques resulted in a tool called Pritest, built according to good design principles for performance, adaptability and maintainability. This tool was compared to an existing similar tool through a discussion.The problem we address is relevant for the increased popularity of agile software methods, where rapid regression testing is of high importance. The experiment indicates that some prioritization techniques perform better than others, and that techniques based on code analysis is outperformed by techniques analyzing code changes, in the context of our experiment.