%0 Journal Article %J Software Engineering, IEEE Transactions on %D 2007 %T Skoll: A Process and Infrastructure for Distributed Continuous Quality Assurance %A Porter, Adam %A Yilmaz,C. %A Memon, Atif M. %A Schmidt,D. C %A Natarajan,B. %K ACE+TAO communication software package %K component assembly %K component deployment %K distributed continuous quality assurance %K distributed development teams %K distributed processing %K end-user customization %K flexible product design %K incremental development %K object-oriented programming %K Skoll %K software engineering %K Software quality %K systems analysis %X Software engineers increasingly emphasize agility and flexibility in their designs and development approaches. They increasingly use distributed development teams, rely on component assembly and deployment rather than green field code writing, rapidly evolve the system through incremental development and frequent updating, and use flexible product designs supporting extensive end-user customization. While agility and flexibility have many benefits, they also create an enormous number of potential system configurations built from rapidly changing component implementations. Since today's quality assurance (QA) techniques do not scale to handle highly configurable systems, we are developing and validating novel software QA processes and tools that leverage the extensive computing resources of user and developer communities in a distributed, continuous manner to improve software quality significantly. This paper provides several contributions to the study of distributed, continuous QA (DCQA). First, it shows the structure and functionality of Skoll, which is an environment that defines a generic around-the-world, around-the-clock QA process and several sophisticated tools that support this process. Second, it describes several novel QA processes built using the Skoll environment. Third, it presents two studies using Skoll: one involving user testing of the Mozilla browser and another involving continuous build, integration, and testing of the ACE+TAO communication software package. The results of our studies suggest that the Skoll environment can manage and control distributed continuous QA processes more effectively than conventional QA processes. For example, our DCQA processes rapidly identified problems that had taken the ACE+TAO developers much longer to find and several of which they had not found. Moreover, the automatic analysis of QA results provided developers information that enabled them to quickly find the root causes of problems %B Software Engineering, IEEE Transactions on %V 33 %P 510 - 525 %8 2007/08// %@ 0098-5589 %G eng %N 8 %R 10.1109/TSE.2007.70719 %0 Journal Article %J Software Process: Improvement and Practice %D 2006 %T Techniques and processes for improving the quality and performance of open-source software %A Porter, Adam %A Yilmaz,Cemal %A Memon, Atif M. %A Krishna,Arvind S. %A Schmidt,Douglas C. %A Gokhale,Aniruddha %K distributed continuous quality assurance %K open-source software development and testing %X Open-source development processes have emerged as an effective approach to reduce cycle-time and decrease design, implementation, and quality assurance (QA) costs for certain types of software, particularly systems infrastructure software, such as operating systems (OS), compilers and language processing tools, text and drawing editors, and middleware. This article presents two contributions to the study of open-source software processes. First, we describe key challenges of open-source software and illustrate how QA processes—specifically those tailored to open-source development—help mitigate these challenges better than traditional closed-source processes do. Second, we summarize results of empirical studies that evaluate how our Skoll distributed continuous quality assurance (DCQA) techniques and processes help to resolve key challenges in developing and validating open-source software. Our results show that: (a) using models to configure and guide the DCQA process improves developer understanding of open-source software, (b) improving the diversity of platform configurations helps QA engineers find defects missed during conventional testing, and (c) centralizing control of QA activities helps to eliminate redundant work. Copyright © 2006 John Wiley & Sons, Ltd. %B Software Process: Improvement and Practice %V 11 %P 163 - 176 %8 2006/// %@ 1099-1670 %G eng %U http://onlinelibrary.wiley.com/doi/10.1002/spip.260/abstract %N 2 %R 10.1002/spip.260 %0 Conference Paper %B Proceedings of the 27th international conference on Software engineering %D 2005 %T Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems %A Yilmaz,Cemal %A Krishna,Arvind S. %A Memon, Atif M. %A Porter, Adam %A Schmidt,Douglas C. %A Gokhale,Aniruddha %A Natarajan,Balachandran %K design of experiment theory %K distributed continuous quality assurance %K performance-oriented regression testing %X Developers of highly configurable performance-intensive software systems often use a type of in-house performance-oriented "regression testing" to ensure that their modifications have not adversely affected their software's performance across its large configuration space. Unfortunately, time and resource constraints often limit developers to in-house testing of a small number of configurations and unreliable extrapolation from these results to the entire configuration space, which allows many performance bottlenecks and sources of QoS degradation to escape detection until systems are fielded. To improve performance assessment of evolving systems across large configuration spaces, we have developed a distributed continuous quality assurance (DCQA) process called main effects screening that uses in-the-field resources to execute formally designed experiments to help reduce the configuration space, thereby allowing developers to perform more targeted in-house QA. We have evaluated this process via several feasibility studies on several large, widely-used performance-intensive software systems. Our results indicate that main effects screening can detect key sources of performance degradation in large-scale systems with significantly less effort than conventional techniques. %B Proceedings of the 27th international conference on Software engineering %S ICSE '05 %I ACM %C New York, NY, USA %P 293 - 302 %8 2005/// %@ 1-58113-963-2 %G eng %U http://doi.acm.org/10.1145/1062455.1062515 %R 10.1145/1062455.1062515 %0 Conference Paper %B Software Engineering, 2004. ICSE 2004. Proceedings. 26th International Conference on %D 2004 %T Skoll: distributed continuous quality assurance %A Memon, Atif M. %A Porter, Adam %A Yilmaz,C. %A Nagarajan,A. %A Schmidt,D. %A Natarajan,B. %K 1MLOC+ software package %K ACE+TAO %K around-the-clock QA process %K around-the-world QA process %K distributed continuous QA %K distributed continuous quality assurance %K distributed programming %K program verification %K Quality assurance %K Skoll %K software performance evaluation %K software profiling %K Software quality %K Software testing %X Quality assurance (QA) tasks, such as testing, profiling, and performance evaluation, have historically been done in-house on developer-generated workloads and regression suites. Since this approach is inadequate for many systems, tools and processes are being developed to improve software quality by increasing user participation in the QA process. A limitation of these approaches is that they focus on isolated mechanisms, not on the coordination and control policies and tools needed to make the global QA process efficient, effective, and scalable. To address these issues, we have initiated the Skoll project, which is developing and validating novel software QA processes and tools that leverage the extensive computing resources of worldwide user communities in a distributed, continuous manner to significantly and rapidly improve software quality. This paper provides several contributions to the study of distributed continuous QA. First, it illustrates the structure and functionality of a generic around-the-world, around-the-clock QA process and describes several sophisticated tools that support this process. Second, it describes several QA scenarios built using these tools and process. Finally, it presents a feasibility study applying these scenarios to a 1MLOC+ software package called ACE+TAO. While much work remains to be done, the study suggests that the Skoll process and tools effectively manage and control distributed, continuous QA processes. Using Skoll we rapidly identified problems that had taken the ACE+TAO developers substantially longer to find and several of which had previously not been found. Moreover, automatic analysis of QA task results often provided developers information that quickly led them to the root cause of the problems. %B Software Engineering, 2004. ICSE 2004. Proceedings. 26th International Conference on %P 459 - 468 %8 2004/05// %G eng %R 10.1109/ICSE.2004.1317468