Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software

TitleStudying the fault-detection effectiveness of GUI test cases for rapidly evolving software
Publication TypeJournal Articles
Year of Publication2005
AuthorsMemon AM, Xie Q
JournalSoftware Engineering, IEEE Transactions on
Pagination884 - 896
Date Published2005/10//
ISBN Number0098-5589
Keywordsdaily automated regression tester, Fault diagnosis, fault-detection, formal specification, formal verification, Graphical user interfaces, GUI test cases, program testing, quality assurance mechanism, rapidly evolving software, smoke regression testing technique, software development, software fault tolerance, Software maintenance, software prototyping, Software quality, test oracles

Software is increasingly being developed/maintained by multiple, often geographically distributed developers working concurrently. Consequently, rapid-feedback-based quality assurance mechanisms such as daily builds and smoke regression tests, which help to detect and eliminate defects early during software development and maintenance, have become important. This paper addresses a major weakness of current smoke regression testing techniques, i.e., their inability to automatically (re)test graphical user interfaces (GUIs). Several contributions are made to the area of GUI smoke testing. First, the requirements for GUI smoke testing are identified and a GUI smoke test is formally defined as a specialized sequence of events. Second, a GUI smoke regression testing process called daily automated regression tester (DART) that automates GUI smoke testing is presented. Third, the interplay between several characteristics of GUI smoke test suites including their size, fault detection ability, and test oracles is empirically studied. The results show that: 1) the entire smoke testing process is feasible in terms of execution time, storage space, and manual effort, 2) smoke tests cannot cover certain parts of the application code, 3) having comprehensive test oracles may make up for not having long smoke test cases, and 4) using certain oracles can make up for not having large smoke test suites.