Optimizing retrieval and processing of multi-dimensional scientific datasets

TitleOptimizing retrieval and processing of multi-dimensional scientific datasets
Publication TypeConference Papers
Year of Publication2000
AuthorsChang C, Kurc T, Sussman A, Saltz J
Conference NameParallel and Distributed Processing Symposium, 2000. IPDPS 2000. Proceedings. 14th International
Date Published2000///
ISBN Number0-7695-0574-0
Keywordsactive data repository, Area measurement, Computer science, Data analysis, distributed memory parallel machines, Educational institutions, Information retrieval, Information Storage and Retrieval, infrastructure, Microscopy, Microwave integrated circuits, multi-dimensional scientific datasets retrieval, PARALLEL PROCESSING, Pathology, range queries, regular d-dimensional array, Satellites, Tomography

We have developed the Active Data Repository (ADR), an infrastructure that integrates storage, retrieval, and processing of large multi-dimensional scientific datasets on distributed memory parallel machines with multiple disks attached to each node. In earlier work, we proposed three strategies for processing range queries within the ADR framework. Our experimental results show that the relative performance of the strategies changes under varying application characteristics and machine configurations. In this work we investigate approaches to guide and automate the selection of the best strategy for a given application and machine configuration. We describe analytical models to predict the relative performance of the strategies where input data elements are uniformly distributed in the attribute space of the output dataset, restricting the output dataset to be a regular d-dimensional array