Monday, May 4, 2015

Measuring Predictability

A friend writes the following.  (I have edited very slightly for clarity.)
Based on forecasts you've seen, what would you say is a "reasonable" ratio of the standard deviation of the forecast error to the standard deviation of a covariance-stationary series being forecast? ... It would be great if you can tell me "I'd consider x reasonable and y too high." 
The problem is that the premise underlying the question (namely, that there is such a "reasonable" value of the ratio r of innovation variance to unconditional  variance) is false.  That is, there's no small value c of r such that r<c means that we've done a good forecasting job.  Equivalently, there's no large value c of the predictive R2 (R2=1r2) such that R2>c means that we've done a good forecasting job.  Instead, "good" c or c values depend critically on the dynamic nature of the series being forecast.  Consider, for example, a covariance-stationary AR(1) process, yt=ϕyt1+εt, where εtiid(0,σ2). The innovation variance is σ2 and the unconditional variance is σ2/(1ϕ2), so the lower bound on  r (and hence the upper bound on  R2) depends entirely on ϕ and can be anywhere in the unit interval! This is an important lesson: "predictability" can (and does) differ greatly across economic series. For more than you ever wanted to know, see Diebold and Kilian (2001), "Measuring Predictability: Theory and Macroeconomic Applications".

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.