Standard Errors for Things that Matter

Many times in applied / empirical seminars I have seen something like this:

The paper estimates a parameter vector b and dutifully reports asymptotic s.e.'s.  But then the ultimate object of interest turns out not to be b, but rather some nonlinear but continuous function of the elements of b, say c = f(b). So the paper calculates and reports an estimate of c as c_hat = f(b_hat).  Fine, insofar as c_hat is consistent if b_hat is consistent.  But then the paper forgets to calculate an asymptotic s.e. for c_hat.

So c is the object of interest, and hundreds, maybe thousands, of person-hours are devoted to producing a point estimate of c, but then no one remembers (cares?) to assess its estimation uncertainty.  Geez.  Of course one could do delta method, simulation, etc.

Score-Driven and Nonlinear Time-Series Models

Check out the upcoming conference here.  Definitely worth reading through the program.  Earlier related post here.

Nearest-Neighbor Prediction

The beautiful idea has been around for ages. Find the N closest H-histories to the current H-history (you choose/tune N and H), for each H-history see what followed, take an average, and use that as your forecast. Of course there are many variations and extensions. Interesting new work by Dendramis, Kapetanios, and Marcellino is in exactly that tradition, except that Dendramis et al.  don't show much awareness of the tradition, or attempt to stand on its shoulders, which I find odd. I find myself hungry for tighter connections, for example to my favorite old nearest-neighbor prediction piece, Sid Yakowitz's well-known "Nearest-Neighbor Methods for Time Series Analysis,” Journal of Time Series Analysis, 1987.

Gu-Kelly-Xiu and Neural Nets in Economics

I'm on record as being largely unimpressed by the contributions of neural nets (NN's) in economics thus far. In many economic environments the relevant non-linearities seem too weak and the signal/noise ratios too low for NN's to contribute much. 

The Gu-Kelly-Xiu paper that I mentioned earlier may change that. I mentioned their success in applying machine-learning methods to forecast equity risk premia out of sample. NN's, in particular, really shine. The paper is thoroughly and meticulously done. 

This is potentially a really big deal.

Machine Learning, Volatility, and the Interface

Just got back from the NBER Summer Institute. Lots of good stuff happening in the Forecasting and Empirical Methods group. The program, with links to papers, is here.

Lots of room for extensions too. Here's a great example. Consider the interface of the Gu-Kelly-Xiu and Bollerslev-Patton-Quagvleg papers. At first you might think that there is no interface. 

Kelly-Xiu is about using off-the-shelf machine-learning methods to model risk premia in financial markets; that is, to construct portfolios that deliver superior performance. (I had guessed they'd get nothing, but I was massively wrong.) Bollerslev et al. is about predicting realized covariance by exploiting info on past signs (e.g., was yesterday's covariance cross-product pos-pos, neg-neg, pos-neg, or neg-pos?). (They also get tremendous results.)

But there's actually a big interface.

Note that Kelly-Xiu is about conditional mean dynamics -- uncovering the determinants of expected excess returns. You might expect even better results for derivative assets, as the volatility dynamics that drive options prices may be nonlinear in ways missed by standard volatility models. And that's exactly the flavor of the Bollerslev et al. results -- they find that a tree structure conditioning on sign is massively successful.

But Bollerslev et al. don't do any machine learning. Instead they basically stumble upon their result, guided by their fine intuition. So here's a fascinating issue to explore: Hit the Bollerslev et al. realized covariance data with machine learning (in particular, tree methods like random forests) and see what happens. Does it "discover" the Bollerslev et al. result? If not, why not, and what does it discover? Does it improve upon Bollerslev et al.?

Deep Neural Nets for Volatility Dynamics

There doesn't seem to be much need for nonparametric nonlinear modeling in empirical macro and finance. Not that lots of smart people haven't tried. The two key nonlinearities (volatility dynamics and regime switching) just seem to be remarkably well handled by tightly-parametric customized models (GARCH/SV and Markov-switching, respectively). 

But the popular volatility models are effectively linear (ARMA) in squares. Maybe that's too rigidly constrained. Volatility dynamics seem like something that could be nonlinear in ways much richer than just ARMA in squares. 

Here's an attempt using deep neural nets. I'm not convinced by the paper -- much more thorough analysis and results are required than the 22 numbers reported in the "GARCH" and "stocvol" columns of its Table 1 -- but I'm intrigued.

It's quite striking that neural nets, which have been absolutely transformative in other areas of predictive modeling, have thus far contributed so little in economic / financial contexts. Maybe the "deep" versions will change that, at least for volatility modeling. Or maybe not. 

The History of Forecasting Competitions

Check out Rob Hyndman's "Brief History of Time Series Forecasting Competitions". I'm not certain whether the title's parallel to Hawking's Brief History of Time is intentional. At any rate, even if Hyndman's focus is rather more narrow than the origin and fate of the universe, his post is still fascinating and informative. Thanks to Ross Askanasi for bring it to my attention.

STILL MORE on NN's and ML

I recently discussed how the nonparametric consistency of wide NN's proved underwhelming, which is partly why econometricians lost interest in NN's in the 1990s.

The other thing was the realization that NN objective surfaces are notoriously bumpy, so that arrival at a local optimum (e.g., by the stochastic gradient descent popular in NN circles) offered little comfort.

So econometricians' interest declined on both counts. But now both issues are being addressed. The new focus on NN depth as opposed to width is bearing much fruit. And recent advances in "reinforcement learning" methods effectively promote global as opposed to just local optimization, by experimenting (injecting randomness) in clever ways. (See, e.g., Taddy section 6, here.)

All told, it seems like quite an exciting new time for NN's. I've been away for 15 years. Time to start following again...

More on Neural Nets and ML

I earlier mentioned Matt Taddy's "The Technological Elements of Artificial Intelligence" (ungated version here).

Among other things the paper has good perspective on the past and present of neural nets. (Read:  his views mostly, if not exactly, match mine...)  

Here's my personal take on some of the history vis a vis econometrics:

Econometricians lost interest in NN's in the 1990's. The celebrated Hal White et al. proof of NN non-parametric consistency as NN width (number of neurons) gets large at an appropriate rate was ultimately underwhelming, insofar as it merely established for NN's what had been known for decades for various other non-parametric estimators (kernel, series, nearest-neighbor, trees, spline, etc.). That is, it seemed that there was nothing special about NN's, so why bother? 

But the non-parametric consistency focus was all on NN width; no one thought or cared much about NN depth. Then, more recently, people noticed that adding NN depth (more hidden layers) could be seriously helpful, and the "deep learning" boom took off. 

Here are some questions/observations on the new "deep learning":

1.  Adding NN depth often seems helpful, insofar as deep learning often seems to "work" in various engineering applications, but where/what are the theorems? What can be said rigorously about depth?

2. Taddy emphasizes what might be called two-step deep learning. In the first step, "pre-trained" hidden layer nodes are obtained based on unsupervised learning (e.g., principle components (PC)) from various sets of variables. And then the second step proceeds as usual. That's very similar to the age-old idea of PC regression. Or, in multivariate dynamic environments and econometrics language, "factor-augmented vector autoregression" (FAVAR), as in Bernanke et al. (2005). So, are modern implementations of deep NN's effectively just nonlinear FAVAR's? If so, doesn't that also seem underwhelming, in the sense of -- dare I say it -- there being nothing really new about deep NN's?

3. Moreover, PC regressions and FAVAR's have issues of their own relative to one-step procedures like ridge or LASSO.  See this and this. 

Neural Nets, ML and AI

"The Technological Elements of Artificial Intelligence", by Matt Taddy, is packed with insight on the development of neural nets and ML as related to the broader development of AI. I have lots to say, but it will have to wait until next week. For now I just want you to have the paper. Ungated version at http://www.nber.org/chapters/c14021.pdf.

Abstract:

We have seen in the past decade a sharp increase in the extent that companies use data to optimize their businesses.  Variously called the `Big Data' or `Data Science' revolution, this has been characterized by massive amounts of data, including unstructured and nontraditional data like text and images, and the use of fast and flexible Machine Learning (ML) algorithms in analysis.  With recent improvements in Deep Neural Networks (DNNs) and related methods, application of high-performance ML algorithms has become more automatic and robust to different data scenarios.  That has led to the rapid rise of an Artificial Intelligence (AI) that works by combining many ML algorithms together - each targeting a straightforward prediction task - to solve complex problems.  

We will define a framework for thinking about the ingredients of this new ML-driven AI.  Having an understanding of the pieces that make up these systems and how they fit together is important for those who will be building businesses around this technology. Those studying the economics of AI can use these definitions to remove ambiguity from the conversation on AI's projected productivity impacts and data requirements.  Finally, this framework should help clarify the role for AI in the practice of modern business analytics and economic measurement.

ML and Metrics VI: A Key Difference Between ML and TS Econometrics

[Click on "Machine Learning" at right for earlier "Machine Learning and Econometrics" posts.]

Continuing:

So then, statistical machine learning (ML) and time series econometrics (TS) have lots in common. But there's also an interesting difference: ML's emphasis on flexible nonparametric modeling of conditional-mean nonlinearity doesn't play a big role in TS. 

Of course there are the traditional TS conditional-mean nonlinearities: smooth non-linear trends, seasonal shifts, and so on. But there's very little evidence of important conditional-mean nonlinearity in the covariance-stationary (de-trended, de-seasonalized) dynamics of most economic time series. Not that people haven't tried hard -- really hard -- to find it, with nearest neighbors, neural nets, random forests, and lots more. 

So it's no accident that things like linear autoregressions remain overwhelmingly dominant in TS. Indeed I can think of only one type of conditional-mean nonlinearity that has emerged as repeatedly important for (at least some) economic time series: Hamilton-style Markov-switching dynamics.

[Of course there's a non-linear elephant in the room:  Engle-style GARCH-type dynamics. They're tremendously important in financial econometrics, and sometimes also in macro-econometrics, but they're about conditional variances, not conditional means.]

So there are basically only two important non-linear models in TS, and only one of them speaks to conditional-mean dynamics. And crucially, they're both very tightly parametric, closely tailored to specialized features of economic and financial data.

Now let's step back and assemble things:

ML emphasizes approximating non-linear conditional-mean functions in highly-flexible non-parametric fashion. That turns out to be doubly unnecessary in TS: There's just not much conditional-mean non-linearity to worry about, and when there occasionally is, it's typically of a highly-specialized nature best approximated in highly-specialized (tightly-parametric) fashion.

Randomization Tests for Regime Switching

I have always been fascinated by distribution-free non-parametric tests, or randomization tests, or Monte Carlo tests -- whatever you want to call them.  (For example, I used some in ancient work like Diebold-Rudebusch 1992.)  They seem almost too good to be true: exact finite-sample tests without distributional assumptions!  They also still seem curiously underutilized in econometrics, notwithstanding, for example, the path-breaking and well-known contributions over many decades by Jean-Marie Dufour, Marc Hallin, and others.

For the latest, see the fascinating new contribution by Jean-Marie Dufour and Richard Luger. They show how to use randomization to perform simple tests of the null of linearity against the alternative of Markov switching in dynamic environments.  That's a very hard problem (nuisance parameters not identified under the null, singular information matrix under the null), and several top researchers have wrestled with it (e.g., Garcia, Hansen, Carasco-Hu-Ploberger). Randomization delivers tests that are exact, distribution-free, and simple. And power looks pretty good too.