## Monday, May 4, 2015

### Measuring Predictability

A friend writes the following.  (I have edited very slightly for clarity.)
Based on forecasts you've seen, what would you say is a "reasonable" ratio of the standard deviation of the forecast error to the standard deviation of a covariance-stationary series being forecast? ... It would be great if you can tell me "I'd consider x reasonable and y too high."
The problem is that the premise underlying the question (namely, that there is such a "reasonable" value of the ratio $$r$$ of innovation variance to unconditional  variance) is false.  That is, there's no small value $$c$$ of $$r$$ such that $$r<c$$ means that we've done a good forecasting job.  Equivalently, there's no large value $$c'$$ of the predictive $$R^2~ (R^2 = 1 - r^2)$$ such that $$R^2 > c'$$ means that we've done a good forecasting job.  Instead, "good" $$c$$ or $$c'$$ values depend critically on the dynamic nature of the series being forecast.  Consider, for example, a covariance-stationary AR(1) process, $$y_t = \phi y_{t-1} + \varepsilon_t$$, where $$\varepsilon_t \sim iid (0, \sigma^2)$$. The innovation variance is $$\sigma^2$$ and the unconditional variance is $$\sigma^2 / (1 - \phi^2)$$, so the lower bound on  $$r$$ (and hence the upper bound on  $$R^2$$) depends entirely on $$\phi$$ and can be anywhere in the unit interval! This is an important lesson: "predictability" can (and does) differ greatly across economic series. For more than you ever wanted to know, see Diebold and Kilian (2001), "Measuring Predictability: Theory and Macroeconomic Applications".