TY - JOUR T1 - Skoll: A Process and Infrastructure for Distributed Continuous Quality Assurance JF - Software Engineering, IEEE Transactions on Y1 - 2007 A1 - Porter, Adam A1 - Yilmaz,C. A1 - Memon, Atif M. A1 - Schmidt,D. C A1 - Natarajan,B. KW - ACE+TAO communication software package KW - component assembly KW - component deployment KW - distributed continuous quality assurance KW - distributed development teams KW - distributed processing KW - end-user customization KW - flexible product design KW - incremental development KW - object-oriented programming KW - Skoll KW - software engineering KW - Software quality KW - systems analysis AB - Software engineers increasingly emphasize agility and flexibility in their designs and development approaches. They increasingly use distributed development teams, rely on component assembly and deployment rather than green field code writing, rapidly evolve the system through incremental development and frequent updating, and use flexible product designs supporting extensive end-user customization. While agility and flexibility have many benefits, they also create an enormous number of potential system configurations built from rapidly changing component implementations. Since today's quality assurance (QA) techniques do not scale to handle highly configurable systems, we are developing and validating novel software QA processes and tools that leverage the extensive computing resources of user and developer communities in a distributed, continuous manner to improve software quality significantly. This paper provides several contributions to the study of distributed, continuous QA (DCQA). First, it shows the structure and functionality of Skoll, which is an environment that defines a generic around-the-world, around-the-clock QA process and several sophisticated tools that support this process. Second, it describes several novel QA processes built using the Skoll environment. Third, it presents two studies using Skoll: one involving user testing of the Mozilla browser and another involving continuous build, integration, and testing of the ACE+TAO communication software package. The results of our studies suggest that the Skoll environment can manage and control distributed continuous QA processes more effectively than conventional QA processes. For example, our DCQA processes rapidly identified problems that had taken the ACE+TAO developers much longer to find and several of which they had not found. Moreover, the automatic analysis of QA results provided developers information that enabled them to quickly find the root causes of problems VL - 33 SN - 0098-5589 CP - 8 M3 - 10.1109/TSE.2007.70719 ER - TY - JOUR T1 - Techniques and processes for improving the quality and performance of open-source software JF - Software Process: Improvement and Practice Y1 - 2006 A1 - Porter, Adam A1 - Yilmaz,Cemal A1 - Memon, Atif M. A1 - Krishna,Arvind S. A1 - Schmidt,Douglas C. A1 - Gokhale,Aniruddha KW - distributed continuous quality assurance KW - open-source software development and testing AB - Open-source development processes have emerged as an effective approach to reduce cycle-time and decrease design, implementation, and quality assurance (QA) costs for certain types of software, particularly systems infrastructure software, such as operating systems (OS), compilers and language processing tools, text and drawing editors, and middleware. This article presents two contributions to the study of open-source software processes. First, we describe key challenges of open-source software and illustrate how QA processes—specifically those tailored to open-source development—help mitigate these challenges better than traditional closed-source processes do. Second, we summarize results of empirical studies that evaluate how our Skoll distributed continuous quality assurance (DCQA) techniques and processes help to resolve key challenges in developing and validating open-source software. Our results show that: (a) using models to configure and guide the DCQA process improves developer understanding of open-source software, (b) improving the diversity of platform configurations helps QA engineers find defects missed during conventional testing, and (c) centralizing control of QA activities helps to eliminate redundant work. Copyright © 2006 John Wiley & Sons, Ltd. VL - 11 SN - 1099-1670 UR - http://onlinelibrary.wiley.com/doi/10.1002/spip.260/abstract CP - 2 M3 - 10.1002/spip.260 ER - TY - CONF T1 - Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems T2 - Proceedings of the 27th international conference on Software engineering Y1 - 2005 A1 - Yilmaz,Cemal A1 - Krishna,Arvind S. A1 - Memon, Atif M. A1 - Porter, Adam A1 - Schmidt,Douglas C. A1 - Gokhale,Aniruddha A1 - Natarajan,Balachandran KW - design of experiment theory KW - distributed continuous quality assurance KW - performance-oriented regression testing AB - Developers of highly configurable performance-intensive software systems often use a type of in-house performance-oriented "regression testing" to ensure that their modifications have not adversely affected their software's performance across its large configuration space. Unfortunately, time and resource constraints often limit developers to in-house testing of a small number of configurations and unreliable extrapolation from these results to the entire configuration space, which allows many performance bottlenecks and sources of QoS degradation to escape detection until systems are fielded. To improve performance assessment of evolving systems across large configuration spaces, we have developed a distributed continuous quality assurance (DCQA) process called main effects screening that uses in-the-field resources to execute formally designed experiments to help reduce the configuration space, thereby allowing developers to perform more targeted in-house QA. We have evaluated this process via several feasibility studies on several large, widely-used performance-intensive software systems. Our results indicate that main effects screening can detect key sources of performance degradation in large-scale systems with significantly less effort than conventional techniques. JA - Proceedings of the 27th international conference on Software engineering T3 - ICSE '05 PB - ACM CY - New York, NY, USA SN - 1-58113-963-2 UR - http://doi.acm.org/10.1145/1062455.1062515 M3 - 10.1145/1062455.1062515 ER - TY - CONF T1 - Skoll: distributed continuous quality assurance T2 - Software Engineering, 2004. ICSE 2004. Proceedings. 26th International Conference on Y1 - 2004 A1 - Memon, Atif M. A1 - Porter, Adam A1 - Yilmaz,C. A1 - Nagarajan,A. A1 - Schmidt,D. A1 - Natarajan,B. KW - 1MLOC+ software package KW - ACE+TAO KW - around-the-clock QA process KW - around-the-world QA process KW - distributed continuous QA KW - distributed continuous quality assurance KW - distributed programming KW - program verification KW - Quality assurance KW - Skoll KW - software performance evaluation KW - software profiling KW - Software quality KW - Software testing AB - Quality assurance (QA) tasks, such as testing, profiling, and performance evaluation, have historically been done in-house on developer-generated workloads and regression suites. Since this approach is inadequate for many systems, tools and processes are being developed to improve software quality by increasing user participation in the QA process. A limitation of these approaches is that they focus on isolated mechanisms, not on the coordination and control policies and tools needed to make the global QA process efficient, effective, and scalable. To address these issues, we have initiated the Skoll project, which is developing and validating novel software QA processes and tools that leverage the extensive computing resources of worldwide user communities in a distributed, continuous manner to significantly and rapidly improve software quality. This paper provides several contributions to the study of distributed continuous QA. First, it illustrates the structure and functionality of a generic around-the-world, around-the-clock QA process and describes several sophisticated tools that support this process. Second, it describes several QA scenarios built using these tools and process. Finally, it presents a feasibility study applying these scenarios to a 1MLOC+ software package called ACE+TAO. While much work remains to be done, the study suggests that the Skoll process and tools effectively manage and control distributed, continuous QA processes. Using Skoll we rapidly identified problems that had taken the ACE+TAO developers substantially longer to find and several of which had previously not been found. Moreover, automatic analysis of QA task results often provided developers information that quickly led them to the root cause of the problems. JA - Software Engineering, 2004. ICSE 2004. Proceedings. 26th International Conference on M3 - 10.1109/ICSE.2004.1317468 ER -