Becker Friedman Expectations Conference

I just returned from a great BFI Conference at U Chicago, Developing and Using Business Expectations Data, organized by Nick Bloom and Steve Davis.

Wonderfully, density as opposed to point survey forecasts were featured throughout. There was the latest on central bank surveys (e.g., Binder et al.), but most informative (to me) was the emphasis on surveys that I'm less familiar with, typically soliciting density expectations from hundreds or thousands of C-suite types at major firms. Examples include Germany's important IFO survey (e.g., Bachman et al.), the U.S. Census Management and Organizational Practices Survey (e.g., Bloom et al.)., and fascinating work in progress at FRB Atlanta. 

The Census survey is especially interesting due to its innovative structuring of histogram bins. There are no fixed bins. Instead users give 5 bins of their own choice, and five corresponding probabilities (which add to 1). This solves the problem in fixed-bin surveys of  (lazy? behaviorally-biased?) respondents routinely and repeatedly assigning 0 probability to subsequently-realized events.

Peter Christoffersen and Forecast Evaluation

For obvious reasons Peter Christoffersen has been on my mind. Here's an example of how his influence extended in important ways. Hopefully it's also an entertaining and revealing story.

Everyone knows Peter's classic 1998 "Evaluating Interval Forecasts" paper, which was part of his Penn dissertation. The key insight was that correct conditional calibration requires not only that the 0-1 "hit sequence" of course have the right mean ((1-\(\alpha\)) for a nominal 1-\(\alpha\) percent interval), but also that it be iid (assuming 1-step-ahead forecasts). More precisely, it must be iid Bernoulli(1-\(\alpha\)).

Around the same time I naturally became interested in going all the way to density forecasts and managed to get some more students interested (Todd Gunther and Anthony Tay). Initially it seemed hopeless, as correct density forecast conditional calibration requires correct conditional calibration of all possible intervals that could be constructed from the density, of which there are uncountably infinitely many.

Then it hit us. Peter had effectively found the right notion of an optimal forecast error for interval forecasts. And just as optimal point forecast errors generally must be independent, so too must optimal interval forecast errors (the Christoffersen hit sequence). Both the point and interval versions are manifestations of "the golden rule of forecast evaluation": Errors from optimal forecasts can't be forecastable. The key to moving to density forecasts, then, would be to uncover the right notion of forecast error for a density forecast. That is, to uncover the function of the density forecast and realization that must be independent under correct conditional calibration. The answer turns out to be the Probability Integral Transform, \(PIT_t=\int_{-\infty}^{y_t} p_t(y_t)\), as discussed in Diebold, Gunther and Tay (1998), who show that correct density forecast conditional calibration implies \(PIT \sim iid U(0,1)\). 

The meta-result that emerges is coherent and beautiful: optimality of point, interval, and density forecasts implies, respectively, independence of forecast error, hit, and \(PIT\) sequences. The overarching point is that a large share of the last two-thirds of the three-part independence result -- not just the middle third -- is due to Peter. He not only cracked the interval forecast evaluation problem, but also supplied key ingredients for cracking the density forecast evaluation problem.

Wonderfully and appropriately, Peter's paper and ours were published together, indeed contiguously, in the International Economic Review. Each is one of the IER's ten most cited since its founding in 1960, but Peter's is clearly in the lead!

Ghysels and Marcellino on Time-Series Forecasting

If you're teaching a forecasting course and want a good text, or if you're just looking for an informative and modern treatment, see Applied Economic Forecasting Using Time Series Methods, by Eric Ghysels and Massimilliano Marcellino. It will be published this week by Oxford University Press. It has a very nice modern awareness of Big Data with emphasis on reduced-rank structure, regularization methods -- LASSO appears as early as p. 23! -- , structural change, mixed-frequencies, etc. It's also very tastefully done in terms of what's included and what's excluded, emphasizing what's most important and de-emphasizing the rest. As regards non-linearity, for example, volatility dynamics and regime-switching are in, and most of the rest is out.

The History of Forecasting Competitions

Check out Rob Hyndman's "Brief History of Time Series Forecasting Competitions". I'm not certain whether the title's parallel to Hawking's Brief History of Time is intentional. At any rate, even if Hyndman's focus is rather more narrow than the origin and fate of the universe, his post is still fascinating and informative. Thanks to Ross Askanasi for bring it to my attention.

More on Neural Nets and ML

I earlier mentioned Matt Taddy's "The Technological Elements of Artificial Intelligence" (ungated version here).

Among other things the paper has good perspective on the past and present of neural nets. (Read:  his views mostly, if not exactly, match mine...)  

Here's my personal take on some of the history vis a vis econometrics:

Econometricians lost interest in NN's in the 1990's. The celebrated Hal White et al. proof of NN non-parametric consistency as NN width (number of neurons) gets large at an appropriate rate was ultimately underwhelming, insofar as it merely established for NN's what had been known for decades for various other non-parametric estimators (kernel, series, nearest-neighbor, trees, spline, etc.). That is, it seemed that there was nothing special about NN's, so why bother? 

But the non-parametric consistency focus was all on NN width; no one thought or cared much about NN depth. Then, more recently, people noticed that adding NN depth (more hidden layers) could be seriously helpful, and the "deep learning" boom took off. 

Here are some questions/observations on the new "deep learning":

1.  Adding NN depth often seems helpful, insofar as deep learning often seems to "work" in various engineering applications, but where/what are the theorems? What can be said rigorously about depth?

2. Taddy emphasizes what might be called two-step deep learning. In the first step, "pre-trained" hidden layer nodes are obtained based on unsupervised learning (e.g., principle components (PC)) from various sets of variables. And then the second step proceeds as usual. That's very similar to the age-old idea of PC regression. Or, in multivariate dynamic environments and econometrics language, "factor-augmented vector autoregression" (FAVAR), as in Bernanke et al. (2005). So, are modern implementations of deep NN's effectively just nonlinear FAVAR's? If so, doesn't that also seem underwhelming, in the sense of -- dare I say it -- there being nothing really new about deep NN's?

3. Moreover, PC regressions and FAVAR's have issues of their own relative to one-step procedures like ridge or LASSO.  See this and this. 

Comparing Interval Forecasts

Here's a new one, "On the Comparison of Interval Forecasts".  

You'd think that interval forecast evaluation would be easy.  After all, point forecast evaluation is (more or less) well understood and easy, and density forecast evaluation is (more or less) well understood and easy, and interval forecasts seem  somewhere in between, so by some sort of continuity argument you'd think that their evaluation would also be well understood and easy.  But no.  In fact it's quite difficult, maybe impossible...

Central Bank Forecast Accuracy

[Sorry for being AWOL.  Like everyone else, I'm generally three feet underwater and breathing through a straw, but in March and April it seems that even the straw goes under.  Anyway, lots of stuff in the pipeline, so let's try to get going again.]

Rummaging around in the basement I just found this Bloomberg piece on comparative central bank forecasting performance.  A friend emailed it a little more than six months ago (ouch), but it's still interesting.  Maybe you missed it.

Bloomberg's first-ever ranking of central bank forecasting, which is relied on by business and finance, turns up winners and losersBloomberg's first-ever ranking of central bank forecasting, which is relied on by business and finance, turns up winners and losers

Superforecasting

A gratis copy of Philip Tetlock and Dan Gardner's new book, Superforecasting, arrived a couple months ago, just before it was published. It's been sitting on my desk until now. With a title like "Superforcasting," perhaps I subconsciously thought it would be pop puffery and delayed looking at it. If so, I was wrong. It's a winner.

Superforecasting is in the tradition of Nate Silver's The Signal and the Noise, but whereas Silver has little expertise (except in politics, baseball and poker, which he knows well) and goes for breadth rather than depth, Tetlock has significant expertise (his own pioneering research, on which his book is built) and goes for depth. Tetlock's emphasis throughout is on just one question: What makes good forecasters good?

Superforecasting is mostly about probabilistic event forecasting, for events much more challenging than those that we econometricians and statisticians typically consider, and for which there is often no direct historical data (e.g., conditional on information available at this moment, what is the probability that Google files for bankruptcy by December 31, 2035?). Nevertheless it contains many valuable lessons for us in forecast construction, evaluation, combination, updating, etc.

You can expect several posts on aspects of Superforecasting in the new year as I re-read it. For now I just wanted to bring it to your attention in case you missed it. Really nice.

New Review of Forecasting at Bank of England

Check it out here. It's thorough and informative.  

It's interesting and unfortunate that even the Bank of England, the great "fan chart pioneer," produces density forecasts for only three of eleven variables forecasted (p. 15). In my view, the most important single forecasting improvement that the Bank of England -- and all central banks -- could implement is a complete switch from point to density forecast construction, evaluation and combination.

Climatology and Predictive Modeling

A notice about this paper just arrived.

Climate Engineering Economics

Garth Heutel, Juan Moreno-Cruz, Katharine Ricke

NBER Working Paper No. 21711
Issued in November 2015
NBER Program(s):   EEE 

Very cool, I thought. So I clicked on the EEE above, to see more systematically what the NBER's Environmental and Energy Economics group is doing these days. In general it has a very interesting list, and in particular it has an interesting list from a predictive modeling viewpoint. Check this, for example:

Modeling Uncertainty in Climate Change: A Multi-Model Comparison

Kenneth Gillingham, William D. Nordhaus, David Anthoff, Geoffrey Blanford, Valentina Bosetti, Peter Christensen, Haewon McJeon, John Reilly, Paul Sztorc

NBER Working Paper No. 21637
Issued in October 2015
NBER Program(s):   EEE 

The economics of climate change involves a vast array of uncertainties, complicating both the analysis and development of climate policy. This study presents the results of the first comprehensive study of uncertainty in climate change using multiple integrated assessment models. The study looks at model and parametric uncertainties for population, total factor productivity, and climate sensitivity. It estimates the pdfs of key output variables, including CO2 concentrations, temperature, damages, and the social cost of carbon (SCC). One key finding is that parametric uncertainty is more important than uncertainty in model structure. Our resulting pdfs also provide insights on tail events.

There's lots of great stuff in GNABBCMRS. (Sorry for the tediously-long acronym.) Among other things, it is correct in noting that "It is conceptually clear that the ensemble approach is an inappropriate measure of uncertainty of outcomes," and it takes a much broader approach. [The "ensemble approach" means different things in different meteorological / climatological contexts, but in this paper's context it means equating forecast error uncertainty with the dispersion of point forecasts across models.] The fact is that point forecast dispersion and forecast uncertainty are very different things. History is replete with examples of tight consensuses that turned out to be wildly wrong.

Unfortunately, however, the "ensemble approach" remains standard in meteorology / climatology. The standard econometric/statistical taxonomy, in contrast, includes not only model uncertainty, but also parameter uncertainty and innovation (stochastic shock) uncertainty. GNABBCMRS focus mostly on parameter uncertainty vs. model uncertainty and find that parameter uncertainty is much more important. That's a major advance.

But more focus is still needed on the third component of forecast error uncertainty, innovation uncertainty. The deterministic Newton / Lorenz approach embodied in much of meteorology / climatology needs thorough exorcising. I have long believed that traditional time-series econometric methods have much to offer in that regard.

Predictive Accuracy Rankings by MSE vs. MAE

We've all ranked forecast accuracy by mean squared error (MSE) and mean absolute error (MAE), the two great workhorses of relative accuracy comparison. MSE-rankings and MAE-rankings often agree, but they certainly don't have to -- they're simply different loss functions -- which is why we typically calculate and examine both.

Here's a trivially simple question: Under what conditions will MSE-rankings and MAE-rankings agree? It turns out that the answer it is not at all trivial -- indeed it's unknown. Things get very difficult, very quickly. 

With \(N(\mu, \sigma^2)\) forecast errors we have that
\( E(|e|) = \sigma \sqrt{2/\pi} \exp\left( -\frac{\mu^{2}}{2 \sigma^{2}}\right) + \mu \left[1-2 \Phi\left(-\frac{\mu}{\sigma} \right) \right], \)
where \(\Phi(\cdot)\) is the standard normal cdf. This relates MAE to the two components of MSE, bias (\(\mu\)) and variance (\(\sigma^2\)), but the relationship is complex. In the unbiased Gaussian case (\(\mu=0\) ), the result collapses to \(MAE \propto \sigma \), so that MSE-rankings and MAE-rankings must agree. But the unbiased Gaussian case is very, very special, and little else is known.

Some energetic grad student should crack this nut, giving necessary and sufficient conditions for identical MSE and MAE rankings in general environments. Two leads: See section 5 of Diebold-Shin (2014), who give a numerical characterization in the biased Gaussian case, and section 2 of Ardakani et al. (2015), who make analytical progress using the SED representation of expected loss.