False precision, surprise and improved uncertainty assessment

Source abstract

An uncertainty report describes the extent of an agent’s uncertainty about some matter. We identify two basic requirements for uncertainty reports, which we call faithfulness and completeness . We then discuss two pitfalls of uncertainty assessment that often result in reports that fail to meet these requirements. The first involves adopting a one-size-fits-all approach to the representation of uncertainty, while the second involves failing to take account of the risk of surprises. In connection with the latter, we respond to the objection that it is impossible to account for the risk of genuine surprises. After outlining some steps that both scientists and the bodies who commission uncertainty assessments can take to help avoid these pitfalls, we explain why striving for faithfulness and completeness is important.