Using uncertainty as a model selection and comparison criterion

TitleUsing uncertainty as a model selection and comparison criterion
Publication TypeConference Papers
Year of Publication2009
AuthorsSarcia' S A, Basili VR, Cantone G
Conference NameProceedings of the 5th International Conference on Predictor Models in Software Engineering
Date Published2009///
Conference LocationNew York, NY, USA
ISBN Number978-1-60558-634-2
Keywordsaccuracy, Bayesian prediction intervals, Calibration, cost estimation, cost model, model evaluation, model selection, prediction interval, Uncertainty

Over the last 25+ years, software estimation research has been searching for the best model for estimating variables of interest (e.g., cost, defects, and fault proneness). This research effort has not lead to a common agreement. One problem is that, they have been using accuracy as the basis for selection and comparison. But accuracy is not invariant; it depends on the test sample, the error measure, and the chosen error statistics (e.g., MMRE, PRED, Mean and Standard Deviation of error samples). Ideally, we would like an invariant criterion. In this paper, we show that uncertainty can be used as an invariant criterion to figure out which estimation model should be preferred over others. The majority of this work is empirically based, applying Bayesian prediction intervals to some COCOMO model variations with respect to a publicly available cost estimation data set coming from the PROMISE repository.