Robustness, fidelity and prediction-looseness of models

Source abstract

Assessment of the credibility of a mathematical or numerical model of a complex system must combine three components: (i) the fidelity of the model to test data, e.g. as quantified by a mean-squared error; (ii) the robustness, of model fidelity, to lack of understanding of the underlying processes; and (iii) the prediction-looseness of the model. ‘Prediction-looseness’ is the range of predictions of models that are equivalent in terms of fidelity. The main result of this paper asserts that fidelity, robustness and prediction-looseness are mutually antagonistic. A change in the model that enhances one of these attributes will cause deterioration of another. In particular, increasing the fidelity to test data will decrease the robustness to imperfect understanding of the process. Likewise, increasing the robustness will increase the predictive looseness . The conclusion is that focusing only on fidelity-to-data is not a sound decision-making strategy for model building and validation. A better strategy is to explore the trade-offs between robustness-to-uncertainty, fidelity to data and tightness of predictions. Our analysis is based on info-gap models of uncertainty, which can be applied to cases of severe uncertainty and lack of knowledge.