Skoll: A Process and Infrastructure for Distributed Continuous Quality Assurance

TitleSkoll: A Process and Infrastructure for Distributed Continuous Quality Assurance
Publication TypeJournal Articles
Year of Publication2007
AuthorsPorter A, Yilmaz C, Memon AM, Schmidt DC, Natarajan B
JournalSoftware Engineering, IEEE Transactions on
Pagination510 - 525
Date Published2007/08//
ISBN Number0098-5589
KeywordsACE+TAO communication software package, component assembly, component deployment, distributed continuous quality assurance, distributed development teams, distributed processing, end-user customization, flexible product design, incremental development, object-oriented programming, Skoll, software engineering, Software quality, systems analysis

Software engineers increasingly emphasize agility and flexibility in their designs and development approaches. They increasingly use distributed development teams, rely on component assembly and deployment rather than green field code writing, rapidly evolve the system through incremental development and frequent updating, and use flexible product designs supporting extensive end-user customization. While agility and flexibility have many benefits, they also create an enormous number of potential system configurations built from rapidly changing component implementations. Since today's quality assurance (QA) techniques do not scale to handle highly configurable systems, we are developing and validating novel software QA processes and tools that leverage the extensive computing resources of user and developer communities in a distributed, continuous manner to improve software quality significantly. This paper provides several contributions to the study of distributed, continuous QA (DCQA). First, it shows the structure and functionality of Skoll, which is an environment that defines a generic around-the-world, around-the-clock QA process and several sophisticated tools that support this process. Second, it describes several novel QA processes built using the Skoll environment. Third, it presents two studies using Skoll: one involving user testing of the Mozilla browser and another involving continuous build, integration, and testing of the ACE+TAO communication software package. The results of our studies suggest that the Skoll environment can manage and control distributed continuous QA processes more effectively than conventional QA processes. For example, our DCQA processes rapidly identified problems that had taken the ACE+TAO developers much longer to find and several of which they had not found. Moreover, the automatic analysis of QA results provided developers information that enabled them to quickly find the root causes of problems