Predictive Modeling and Causal Estimation

Here are the slides, "Econometrics, Predictive Modeling, and Causal Estimation", from my talk at the recent conference at the IMF in honor of Kajal Lahiri's 70th (group photo above). They build on an earlier No Hesitations post.

ML and Metrics VII: Cross-Section Non-Linearities

[Click on "Machine Learning" at right for earlier "Machine Learning and Econometrics" posts.]

The predictive modeling perspective needs not only to be respected and embraced in econometrics (as it routinely is, notwithstanding the Angrist-Pischke revisionist agenda), but also to be enhanced by incorporating elements of statistical machine learning (ML). This is particularly true for cross-section econometrics insofar as time-series econometrics is already well ahead in that regard.  For example, although flexible non-parametric ML approaches to estimating conditional-mean functions don't add much to time-series econometrics, they may add lots to cross-section econometric regression and classification analyses, where conditional mean functions may be highly nonlinear for a variety of reasons.  Of course econometricians are well aware of traditional non-parametric issues/approaches, especially kernel and series methods, and they have made many contributions, but there's still much more to be learned from ML.

Machine Learning and Econometrics V: Similarities to Time Series

[Notice that I changed the title from "Machine Learning vs. Econometrics" to "Machine Learning and  Econometrics", as the two are complements, not competitors, as this post will begin to emphasize. But I've kept the numbering, so this is number five.  For others click on Machine Learning at right.]

Thanks for the overwhelming response to my last post, on Angrist-Pischke (AP).  I'll have more to say on AP a few posts from now, but first I need to set the stage.

A key observation is that statistical machine learning (ML) and time-series econometrics/statistics (TS) are largely about modeling, and they largely have the same foundational perspective. Some of the key ingredients are:

-- George Box got it right: "All models are false; some are useful", so search for good approximating models, not "truth".

-- Be explicit about the loss function, that is, about what defines a "good approximating model" (e.g., 1-step-ahead out-of-sample mean-squared forecast error)

-- Respect and optimize that loss function in model selection (e.g., BIC)

-- Respect and optimize that loss function in estimation (e.g., least squares)

-- Respect and optimize that loss function in forecast construction (e.g., Wiener-Kolmogorov-Kalman)

-- Respect and optimize that loss function in forecast evaluation, comparison, and combination (e.g., Mincer-Zarnowitz evaluations, Diebold-Mariano comparisons, Granger-Ramanathan combinations).

So time-series econometrics should embrace ML -- and it is.  Just look at recent work like this.

Econometrics: Angrist and Pischke are at it Again

Check out the new Angrist-Pischke (AP), "Undergraduate Econometrics Instruction: Through Our Classes, Darkly".

I guess I have no choice but to weigh in. The issues are important, and my earlier AP post, "Mostly Harmless Econometrics?", is my all-time most popular.

Basically AP want all econometrics texts to look a lot more like theirs. But their books and their new essay unfortunately miss (read: dismiss) half of econometrics.

Here's what AP get right:

(Goal G1) One of the major goals in econometrics is predicting the effects of exogenous "treatments" or "interventions" or "policies". Phrased in the language of estimation, the question is "If I intervene and give someone a certain treatment \({\partial x}, x \in X\), what is my minimum-MSE estimate of her \(\ \partial y\)?" So we are estimating the partial derivative \({\partial y / \partial x}\).

AP argue the virtues and trumpet the successes of a "design-based" approach to G1. In my view they make many good points as regards G1: discontinuity designs, dif-in-dif designs, and other clever modern approaches for approximating random experiments indeed take us far beyond "Stones'-age" approaches to G1. (AP sure turn a great phrase...). And the econometric simplicity of the design-based approach is intoxicating: it's mostly just linear regression of \(y\) on \(x\) and a few cleverly-chosen control variables -- you don't need a full model -- with White-washed standard errors. Nice work if you can get it. And yes, moving forward, any good text should feature a solid chapter on those methods.

Here's what AP miss/dismiss:

(Goal G2) The other major goal in econometrics is predicting \(y\). In the language of estimation, the question is "If a new person \(i\) arrives with covariates \(X_i\), what is my minimum-MSE estimate of her \(y_i\)? So we are estimating a conditional mean \(E(y | X) \), which in general is very different from estimating a partial derivative \({\partial y / \partial x}\).

The problem with the AP paradigm is that it doesn't work for goal G2. Modeling nonlinear functional form is important, as the conditional mean function \(E(y | X) \) may be highly nonlinear in \(X\); systematic model selection is important, as it's not clear a priori what subset of \(X\) (i.e., what model) might be most useful for approximating \(E(y | X) \); detecting and modeling heteroskedasticity is important (in both cross sections and time series), as it's the key to accurate interval and density prediction; detecting and modeling serial correlation is crucially important in time-series contexts, as "the past" is the key conditioning information for predicting "the future"; etc., etc, ... 

(Notice how often "model" and "modeling" appear in the above paragraph. That's precisely what AP dismiss, even in their abstract, which very precisely, and incorrectly, declares that "Applied econometrics ...[now prioritizes]... the estimation of specific causal effects and empirical policy analysis over general models of outcome determination".)

The AP approach to goal G2 is to ignore it, in a thinly-veiled attempt to equate econometrics exclusively with G1, which nicely feathers the AP nest. Sorry guys, but no one's buying it. That's why the textbooks continue to feature G2 tools and techniques so prominently, as well they should.

DDH Now in Chinese

For my Chinese readers:
A Chinese version of the Diebold-Doherty-Herring risk management book just appeared. Interesting surprise. I knew nothing about it until it arrived in the snail mail, just as with the earlier Chinese version of the Diebold-Rudebusch yield curve book. Ya gotta love Princeton University Press. They take care of business, with minimal hassle.

The Black Swan Spectrum

Speaking of the newly-updated draft of Econometrics, now for some fun. Here's a question from the Chapter 6 EPC (exercises, problems and complements). Where does your reaction fall on the A-B spectrum below?

Nassim Taleb is a financial markets trader (and Wharton graduate) turned pop author. His book, The Black Swan, deals with many of the issues raised in this chapter. "Black swans"' are seemingly impossible or very low-probability events -- after all, swans are supposed to be white -- that occur with annoying regularity in reality. Read his book. Where does your reaction fall on the A-B spectrum below? 

A. Taleb offers crucial lessons for econometricians, heightening awareness in ways otherwise difficult to achieve. After reading Taleb, it's hard to stop worrying about non-normality, model misspecification, and so on.

B. Taleb belabors the obvious for hundreds of pages, arrogantly "informing"' us that non-normality is prevalent, that all models are misspecified, and so on. Moreover, it takes a model to beat a model, and Taleb offers nothing new.

The book is worth reading, regardless of where your reaction falls on the A-B spectrum.

Open Econometrics Text Updated for Fall Use

I have just posted an update of my introductory undergraduate Econometrics (book, slides, R code, EViews code, data, etc.). Warning: although it is significantly improved, it nevertheless remains highly (alas, woefully) preliminary and incomplete.

I intend to keep everything permanently "open," freely available on the web, continuously evolving and improving.

If you use the materials in your teaching this fall (and even if you don't), I would be grateful for feedback.

Comparing Predictive Accuracy, Twenty Years Later

I have now posted the final pre-meeting draft of the "Use and Abuse" paper (well, more-or-less "final").

I'll present it as the JBES Lecture, January 2014 ASSA meetings, Philadelphia. Please join if you're around. It's Friday January 3, 2:30, Pennsylvania Convention Center Room 2004-C (I think).

By the way, the 2010 Peter Hansen paper that I now cite in my final paragraph, "A Winners Curse for Econometric Models: On the Joint Distribution of In-Sample Fit and Out-of-Sample Fit and its Implications for Model Selection," is tremendously insightful. I saw Peter present it a few years ago at a Stanford summer workshop, but I didn't fully appreciate it and had forgotten about it until he reminded me when he visited Penn last week. He's withheld the 2010 and later revisions from general circulation evidently because one section still needs work. Let's hope that he gets it revised and posted soon! (A more preliminary 2009 version remains online from a University of Chicago seminar.) One of Peter's key points is that although split-sample model comparisons can be "tricked" by data mining in finite samples, just as can all model comparison procedures, split-sample comparisons appear to be harder to trick, in a sense that he makes precise. That's potentially a very big deal.

Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests

Abstract: The Diebold-Mariano (DM) test was intended for comparing forecasts; it has been, and remains, useful in that regard. The DM test was not intended for comparing models. Much of the large ensuing literature, however, uses DM-type tests for comparing models, in (pseudo-) out-of-sample environments. In that case, simpler yet more compelling full-sample model comparison procedures exist; they have been, and should continue to be, widely used. The hunch that (pseudo-) out-of-sample analysis is somehow the ``only," or ``best," or even necessarily a ``good" way to provide insurance against in-sample over-fitting in model comparisons proves largely false. On the other hand, (pseudo-) out-of-sample analysis remains useful for certain tasks, most notably for providing information about comparative predictive performance during particular historical episodes.