Uncertainty exists and is generally unavoidable. This is neither surprising not intrinsically problematic. But having to make predictions given uncertainty about the world as it has been observed can be difficult. There is a potential to too closely follow observed results and overfit to noise, and there is potential to disbelieve observations in favor of only slightly informed prior beliefs. The ideal models, generally, are somewhere in the middle.
In this blog post, we briefly consider the role of uncertainty in building models. The discussion will focus on the balance between building models that respect the data and models that are well-behaved (according to a pre-defined sense of what that means). In follow-up blog posts we will discuss how these decisions play a role in Bayesian optimization.