Query planning for the grid: adapting to dynamic resource availability

TitleQuery planning for the grid: adapting to dynamic resource availability
Publication TypeConference Papers
Year of Publication2005
AuthorsZhang K, Andrade H, Raschid L, Sussman A
Conference NameCluster Computing and the Grid, IEEE International Symposium on
Date Published2005///
PublisherIEEE Computer Society
Conference LocationLos Alamitos, CA, USA
ISBN Number0-7803-9074-1

The availability of massive datasets, comprising sensor measurements or the results of scientific simulations, has had a significant impact on the methodology of scientific reasoning. Scientists require storage, bandwidth and computational capacity to query and analyze these datasets, to understand physical phenomena or to test hypotheses. This paper addresses the challenge of identifying and selecting resources to develop an evaluation plan for large scale data analysis queries when data processing capabilities and datasets are dispersed across nodes in one or more computing and storage clusters. We show that generating an optimal plan is hard and we propose heuristic techniques to find a good choice of resources. We also consider heuristics to cope with dynamic resource availability; in this situation we have stale information about reusable cached results (datasets) and the load on various nodes.