Principles for designing data-/compute-intensive distributed applications and middleware systems for heterogeneous environments

TitlePrinciples for designing data-/compute-intensive distributed applications and middleware systems for heterogeneous environments
Publication TypeJournal Articles
Year of Publication2007
AuthorsKim J-S, Andrade H, Sussman A
JournalJournal of Parallel and Distributed Computing
Volume67
Issue7
Pagination755 - 771
Date Published2007/07//
ISBN Number0743-7315
KeywordsComputational science applications, Data-/compute-intensive applications, Heterogeneous environments, Middleware systems
Abstract

The nature of distributed systems is constantly and steadily changing as the hardware and software landscape evolves. Porting applications and adapting existing middleware systems to ever changing computational platforms has become increasingly complex and expensive. Therefore, the design of applications, as well as the design of next generation middleware systems, must follow a set of guiding principles in order to insure long-term “survivability” without costly re-engineering. From our practical experience, the key determinants to success in this endeavor are adherence to the following principles: (1) Design for change; (2) Provide for storage subsystem I/O coordination; (3) Employ workload partitioning and load balancing techniques; (4) Employ caching; (5) Schedule the workload; and (6) Understand the workload. In order to support these principles, we have collected extensive experimental results comparing three middleware systems targeted at data- and compute-intensive applications implemented by our research group during the course of the last decade, on a single data- and compute-intensive application. The main contribution of this work is the analysis of a level playing field, where we discuss and quantify how adherence to these guiding principles impacts overall system throughput and response time.

URLhttp://www.sciencedirect.com/science/article/pii/S0743731507000603
DOI10.1016/j.jpdc.2007.04.006