But forecasters often resist considering these out-of-sample problems. When we expand our sample to include events further apart from us in time and space, it often means that we will encounter cases in which the relationships we are studying did not hold up as well as we are accustomed to. The model will seem to be less powerful. It will look less impressive in a PowerPoint presentation (or a journal article or a blog post). We will be forced to acknowledge that we know less about the world than we thought we did. Our personal and professional incentives almost always discourage us from doing this.
…
We forget—or we willfully ignore—that our models are simplifications of the world. We figure that if we make a mistake, it will be at the margin.
Out of Sample Problems – “The Signal and the Noise: The Art and Science of Prediction”, Nate Silver
November 16th, 2012 · No Comments · Causation and Correlation, Data, Prediction, Statistics
Tags: Bayesian Analysis·Bias·Incentives·prediction·Statistics