Interval Prediction

Last time I blogged on Serena's amazing presentation from Per's Chicago meeting,
https://fxdiebold.blogspot.com/2019/10/large-dimensional-factor-analysis-with.html

But I was equally blown away by Rina's amazing "Predictive inference with the jackknife+".
Rina Foygel Barber∗ , Emmanuel J. Canes† , Aaditya Ramdas‡ , Ryan J. Tibshirani‡§
https://arxiv.org/pdf/1905.02928.pdf.

Correctly calibrated prediction intervals despite arbitrary model misspecification!

Of course I'm left with lots of questions.  They have nice correct-coverage theorems. What about length?  I would like theorems (not just simulations) as regards shortest length intervals with guaranteed correct coverage. Their results seem to require iid or similar exchangability environments. What about heteroskedastic environments where prediction error variance depends on covariates? What about time series environments?

Then, quite amazingly, "Distributional conformal prediction" by Victor Chernozukov et al., arrived in my mailbox.
https://arxiv.org/pdf/1909.07889.pdf
It is similarly motivated and may address some of my questions.

Anyway, great developments for interval prediction!

Big Data in Dynamic Predictive Modeling

Our Journal of Econometrics issue, Big Data in Dynamic Predictive Econometric Modeling, is now in press.  It is partly based on a Penn conference, generously supported by Penn's Warren Center for Network and Data Sciences, University of Chicago's Stevanovich Center for Financial Mathematics, and Penn's Institute for Economic Research.  The intro is here and the paper list is here.

Machine Learning and Alternative Data for Predicting Economic Indicators

I discussed an interesting paper by Chen et al. today at the CRIW.  My slides are here.

Causality as Robust Prediction

I like thinking about causal estimation as a type of prediction (e.g., here). Here's a very nice slide deck from Peter Buhlmann at ETH Zurich detailing his group's recent and ongoing work in that tradition.

Dual Regression and Prediction

Richard Spady and Sami Stouli have an interesting new paper, “Dual Regression". They change the usual OLS loss function from quadratic to something related but different, as per their equation (2.2), and they get impressive properties for estimation under correct specification. They also have some results under misspecification.

I'd like to understand more regarding dual regression's properties for prediction under misspecification. Generally we're comfortable with quadratic loss, in which case OLS delivers the goods (the conditional mean or linear projection) in large samples under great generality (e.g., see here). The dual regression estimator, in contrast, has a different probability limit under misspecification -- it's not providing a KLIC-optimal approximation.

If the above sounds negative, note well that the issue raised may be an opportunity, not a pitfall! Certainly there is nothing sacred about quadratic loss, even if the conditional mean is usually a natural predictor. We sometimes move to absolute-error loss (conditional median predictor), check-function loss (conditional quantile predictor), or all sorts of other predictive loss functions depending on the situation. But movements away from conditional mean or median prediction generally require some justification and interpretation. Equivalently, movements away from quadratic or absolute predictive loss generally require some justification and interpretation. I look forward to seeing that for the loss function that drives dual regression.

Nearest-Neighbor Prediction

The beautiful idea has been around for ages. Find the N closest H-histories to the current H-history (you choose/tune N and H), for each H-history see what followed, take an average, and use that as your forecast. Of course there are many variations and extensions. Interesting new work by Dendramis, Kapetanios, and Marcellino is in exactly that tradition, except that Dendramis et al.  don't show much awareness of the tradition, or attempt to stand on its shoulders, which I find odd. I find myself hungry for tighter connections, for example to my favorite old nearest-neighbor prediction piece, Sid Yakowitz's well-known "Nearest-Neighbor Methods for Time Series Analysis,” Journal of Time Series Analysis, 1987.

Machine Learning for Forecast Combination

How could I have forgotten to announce my latest paper, "Machine Learning for Regularized Survey Forecast Combination: Partially-Egalitarian Lasso and its Derivatives"? (Actually a heavily-revised version of an earlier paper, including a new title.) Came out as an NBER w.p. a week or two ago.

The Coming Storm

The role of time-series statistics / econometrics in climate analyses is expanding (e.g., here).  Related -- albeit focusing on shorter-term meteorological aspects rather than longer-term climatological aspects -- it's worth listening to Michael Lewis' latest, The Coming Storm.  (You have to listen rather than read, as it's only available as an audiobook, but it's only about two hours.)  It's a fascinating story, well researched and well told by Lewis, just as you'd expect.  There are lots of interesting insights on (1) the collection, use, and abuse of public weather data, including ongoing, ethically-dubious, and potentially life-destroying attempts to privatize public weather data for private gain, (2) the clear and massive improvements in weather forecasting in recent decades, (3) behavioral aspects of how best to communicate forecasts so people understand them, believe them, and take appropriate action before disaster strikes. 

Peter Christoffersen and Forecast Evaluation

For obvious reasons Peter Christoffersen has been on my mind. Here's an example of how his influence extended in important ways. Hopefully it's also an entertaining and revealing story.

Everyone knows Peter's classic 1998 "Evaluating Interval Forecasts" paper, which was part of his Penn dissertation. The key insight was that correct conditional calibration requires not only that the 0-1 "hit sequence" of course have the right mean ((1-\(\alpha\)) for a nominal 1-\(\alpha\) percent interval), but also that it be iid (assuming 1-step-ahead forecasts). More precisely, it must be iid Bernoulli(1-\(\alpha\)).

Around the same time I naturally became interested in going all the way to density forecasts and managed to get some more students interested (Todd Gunther and Anthony Tay). Initially it seemed hopeless, as correct density forecast conditional calibration requires correct conditional calibration of all possible intervals that could be constructed from the density, of which there are uncountably infinitely many.

Then it hit us. Peter had effectively found the right notion of an optimal forecast error for interval forecasts. And just as optimal point forecast errors generally must be independent, so too must optimal interval forecast errors (the Christoffersen hit sequence). Both the point and interval versions are manifestations of "the golden rule of forecast evaluation": Errors from optimal forecasts can't be forecastable. The key to moving to density forecasts, then, would be to uncover the right notion of forecast error for a density forecast. That is, to uncover the function of the density forecast and realization that must be independent under correct conditional calibration. The answer turns out to be the Probability Integral Transform, \(PIT_t=\int_{-\infty}^{y_t} p_t(y_t)\), as discussed in Diebold, Gunther and Tay (1998), who show that correct density forecast conditional calibration implies \(PIT \sim iid U(0,1)\). 

The meta-result that emerges is coherent and beautiful: optimality of point, interval, and density forecasts implies, respectively, independence of forecast error, hit, and \(PIT\) sequences. The overarching point is that a large share of the last two-thirds of the three-part independence result -- not just the middle third -- is due to Peter. He not only cracked the interval forecast evaluation problem, but also supplied key ingredients for cracking the density forecast evaluation problem.

Wonderfully and appropriately, Peter's paper and ours were published together, indeed contiguously, in the International Economic Review. Each is one of the IER's ten most cited since its founding in 1960, but Peter's is clearly in the lead!

Machines Learning Finance

FRB Atlanta recently hosted a meeting on "Machines Learning Finance". Kind of an ominous, threatening (Orwellian?) title, but there were lots of (non-threatening...) pieces. I found the surveys by Ryan Adams and John Cunningham particularly entertaining. A clear theme on display throughout the meeting was that "supervised learning" -- the main strand of machine learning -- is just function estimation, and in particular, conditional mean estimation. That is, regression. It may involve high dimensions, non-linearities, binary variables, etc., but at the end of the day it's still just regression. If you're a regular No Hesitations reader, the "insight" that supervised learning = regression will hardly be novel to you, but still it's good to see it disseminating widely.

Pockets of Predictability

Some months ago I blogged on "Pockets of Predictability," here. The Farmer-Schmidt-Timmermann paper that I mentioned is now available, here.

Ghysels and Marcellino on Time-Series Forecasting

If you're teaching a forecasting course and want a good text, or if you're just looking for an informative and modern treatment, see Applied Economic Forecasting Using Time Series Methods, by Eric Ghysels and Massimilliano Marcellino. It will be published this week by Oxford University Press. It has a very nice modern awareness of Big Data with emphasis on reduced-rank structure, regularization methods -- LASSO appears as early as p. 23! -- , structural change, mixed-frequencies, etc. It's also very tastefully done in terms of what's included and what's excluded, emphasizing what's most important and de-emphasizing the rest. As regards non-linearity, for example, volatility dynamics and regime-switching are in, and most of the rest is out.

The History of Forecasting Competitions

Check out Rob Hyndman's "Brief History of Time Series Forecasting Competitions". I'm not certain whether the title's parallel to Hawking's Brief History of Time is intentional. At any rate, even if Hyndman's focus is rather more narrow than the origin and fate of the universe, his post is still fascinating and informative. Thanks to Ross Askanasi for bring it to my attention.

Big Data and Economic Nowcasting

Check out this informative paper from the Federal Reserve Bank of New York: "Macroeconomic Nowcasting and Forecasting with Big Data", by Brandyn Bok, Daniele Caratelli, Domenico Giannone, Argia Sbordone, and Andrea Tambalotti.

Key methods for confronting big data include (1) imposition of restrictions (for example, (a) zero restrictions correspond to "sparsity", (b) reduced-rank restrictions correspond to factor structure, etc.), and (2) shrinkage (whether by formal Bayesian approaches or otherwise).

Bok et al. provide historical perspective on use of (1)(b) for macroeconomic nowcasting; that is, for real-time analysis and interpretation of hundreds of business-cycle indicators using dynamic factor models. They also provide a useful description of FRBNY's implementation and use of such models in policy deliberations.

It is important to note that the Bok et al. approach nowcasts current-quarter GDP, which is different from nowcasting "the business cycle" (as done using dynamic factor models at FRB Philadelphia, for example), because GDP alone is not the business cycle. Hence the two approaches are complements, not substitutes, and both are useful.

STILL MORE on NN's and ML

I recently discussed how the nonparametric consistency of wide NN's proved underwhelming, which is partly why econometricians lost interest in NN's in the 1990s.

The other thing was the realization that NN objective surfaces are notoriously bumpy, so that arrival at a local optimum (e.g., by the stochastic gradient descent popular in NN circles) offered little comfort.

So econometricians' interest declined on both counts. But now both issues are being addressed. The new focus on NN depth as opposed to width is bearing much fruit. And recent advances in "reinforcement learning" methods effectively promote global as opposed to just local optimization, by experimenting (injecting randomness) in clever ways. (See, e.g., Taddy section 6, here.)

All told, it seems like quite an exciting new time for NN's. I've been away for 15 years. Time to start following again...

ML, Forecasting, and Market Design

Nice stuff from Milgrom and Tadelis. Improved forecasting via improved machine learning in turn helps improve our ability to design effective markets -- better anticipating consumer/producer demand/supply movements, more finely segmenting and targeting consumers/producers, more accurately setting auction reserve prices, etc. Presumably full density forecasts, not just the point forecasts on which ML tends to focus, should soon move to center stage.
http://www.nber.org/chapters/c14008.pdf

Averaging for Prediction in Econometrics and ML

Random thought. At the risk of belaboring the obvious, it's interesting to heighten collective awareness by thinking about the many appearances of averaging in forecasting, particularly in forecast combination. Some averages are weighted, and some are not. Most are linear, some are not.

• The "equal weights puzzle" in forecast combination 
• Random forests, and ensemble averaging algorithms more generally
• Bootstrap aggregation ("bagging") 
• Boosting 
• Best subset averaging
• Survey averages
• k-nearest-neighbor forecasts
• Amisano-Geweke equally-weighted prediction pools
• "1/N" portfolios
• Bayesian model averaging
• Bates-Granger-Ramanathan frequentist model averaging
• Any forecasts extracted from markets (the ultimate information aggregator), ranging from "standard" markets (e.g., volatility forecasts extracted from options prices, interest rate forecasts extracted from the current yield curve, etc.), to explicit so-called "prediction markets" (e.g., sports betting markets).

Comparing Interval Forecasts

Here's a new one, "On the Comparison of Interval Forecasts".  

You'd think that interval forecast evaluation would be easy.  After all, point forecast evaluation is (more or less) well understood and easy, and density forecast evaluation is (more or less) well understood and easy, and interval forecasts seem  somewhere in between, so by some sort of continuity argument you'd think that their evaluation would also be well understood and easy.  But no.  In fact it's quite difficult, maybe impossible...

More on the Problem with Bayesian Model Averaging

I blogged earlier on a problem with Bayesian model averaging (BMA) and gave some links to new work that chips away at it. The interesting thing about that new work is that it stays very close to traditional BMA while acknowledging that all models are misspecified.

But there are also other Bayesian approaches to combining density forecasts, such as prediction pools formed to optimize a predictive score. (See, e.g. Amisano and Geweke, 2017, and the references therein.  Ungated final draft, and code, here.)

Another relevant strand of new work, less familiar to econometricians, is "Bayesian predictive synthesis" (BPS), which builds on the expert opinions analysis literature. The framework, which traces to Lindley et al. (1979), concerns a Bayesian faced with multiple priors coming from multiple experts, and explores how to get a posterior distribution utilizing all of the information available. Earlier work by Genest and Schervish (1985) and West and Crosse (1992) develops the basic theory, and new work (McAlinn and West, 2017), extends it to density forecast combination.

Thanks to Ken McAlinn for reminding me about BPS. Mike West gave a nice presentation at the FRBSL forecasting meeting. [Parts of this post are adapted from private correspondence with Ken.]

The Problem With Bayesian Model Averaging...

The problem is that one of the models considered is traditionally assumed true (explicitly or implicitly) since the prior model probabilities sum to one. Hence all posterior weight gets placed on a single model asymptotically -- just what you don't want when constructing a portfolio of surely-misspecified models. The earliest paper I know that makes and explores this point is one of mine, here. Recent and ongoing research is starting to address it much more thoroughly, for example here and here. (Thanks to Veronika Rockova for sending.)