Machine Learning vs. Econometrics, I

[If you're reading this in email, remember to click through on the title to get the math to render.]

Machine learning (ML) is almost always centered on prediction; think "\(\hat{y}\)".   Econometrics (E) is often, but not always, centered on prediction.  Instead it's also often interested on estimation and associated inference; think "\(\hat{\beta}\)".

Or so the story usually goes. But that misses the real distinction. Both ML and E as described above are centered on prediction.  The key difference is that ML focuses on non-causal prediction (if a new person \(i\) arrives with covariates \(X_i\), what is my minimium-MSE guess of her \(y_i\)?), whereas the part of econometrics highlighted above focuses on causal prediction (if I intervene and give person \(i\) a certain treatment, what is my minimum-MSE guess of \(\Delta y_i\)?).  It just happens that, assuming linearity, a "minimum-MSE guess of \(\Delta y_i\)" is the same as a "minimum-MSE estimate of \(\beta_i\)".

So there is a ML vs. E distinction here, but it's not "prediction vs. estimation" -- it's all prediction.  Instead, the issue is non-causal prediction vs. causal prediction.

But there's another ML vs. E difference that's even more fundamental.  TO BE CONTINUED...

Machine Learning vs. Econometrics, IV

Some of my recent posts on this topic emphasized that (1) machine learning (ML) tends to focus on non-causal prediction, whereas econometrics and statistics (E/S) has both non-causal and causal parts, and (2) E/S tends to be more concerned with probabilistic assessment of forecast uncertainty. Here are some related thoughts.

As for (1), it's wonderful to see the ML and E/S literatures beginning to cross-fertilize, driven in significant part by E/S. Names like Athey, Chernozukov, and Imbens come immediately to mind. See, for example, the material here under "Econometric Theory and Machine Learning", and here under "Big Data: Post-Selection Inference for Causal Effects" and "Big Data: Prediction Methods". 

As for (2) but staying with causal prediction, note that the traditional econometric approach treats causal prediction as an estimation problem (whether by instrumental variables, fully-structural modeling, or whatever...) and focuses not only on point estimates, but also on inference (standard errors, etc.) and hence implicitly on interval prediction of causal effects (by inverting the test statistics).  Similarly, the financial-econometric "event study" approach, which directly compares forecasts of what would have happened in the absence of an intervention to what happened with the intervention, also focuses on inference for the treatment effect, and hence implicitly on interval prediction.

Machine Learning vs. Econometrics, III

I emphasized here that both machine learning (ML) and econometrics (E) prominently feature prediction, one distinction being that ML tends to focus on non-causal prediction, whereas a significant part of E focuses on causal prediction. So they're both focused on prediction, but there's a non-causal vs. causal distinction.  [Alternatively, as Dean Foster notes, you can think of both ML and E as focused on estimation, but with different estimands.  ML tends to focus on estimating conditional expectations, whereas the causal part of E focuses on estimating partial derivatives.]

In any event, there's another key distinction between much of ML and Econmetrics/Statistics (E/S):   E/S tends to be more concerned with probabilistic assessment of uncertainty.  Whereas ML is often satisfied with point forecasts, E/S often wants interval, and ultimately density, forecasts.

There are at least two classes of reasons for the difference.  

First, E/S recognizes that uncertainty is often of intrinsic economic interest.  Think market risk, credit risk, counter-party risk, systemic risk, inflation risk, business cycle risk, etc.

Second, E/S is evidently uncomfortable with ML's implicit certainty-equivalence approach of simply plugging point forecasts into decision rules obtained under perfect foresight.  Evidently the linear-quadratic-Gaussian world in which certainty equivalence holds resonates less than completely with E/S types.  That sounds right to me.  [By the way, see my earlier piece on optimal prediction under asymmetric loss.]

Machine Learning vs. Econometrics, II

My last post focused on one key distinction between machine learning (ML) and econometrics (E):   non-causal ML prediction vs. causal E prediction.  I promised later to highlight another, even more important, distinction.  I'll get there in the next post.

But first let me note a key similarity.  ML vs. E in terms of non-causal vs. causal prediction is really only comparing ML to "half" of E (the causal part).  The other part of E (and of course statistics, so let's call it E/S), going back a century or so, focuses on non-causal prediction, just like ML.  The leading example is time-series E/S.  Just take a look at an E/S text like Elliott and Timmermann (contents and first chapter here; index here).  A lot of it looks like parts of ML.  But it's not "E/S people chasing ML ideas"; rather, E/S has been in the game for decades, often well ahead of ML.

For this reason the E/S crowd sometimes wonders whether "ML" and "data science" are just the same old wine in a new bottle.  (The joke goes, Q: What is a "data scientist"?  A: A statistician who lives in San Francisco.)  ML/DataScience is not the same old wine, but it's a blend, and a significant part of the blend is indeed E/S.

To be continued...

Ensemble Methods for Causal Prediction

Great to see ensemble learning methods (i.e., forecast combination) moving into areas of econometrics beyond time series / macro-econometrics, where they have thrived ever since Bates and Granger (1969), generating a massive and vibrant literature.  (For a recent contribution, including historical references, see Diebold and Shin, 2019.)  In particular, the micro-econometric / panel / causal literature is coming on board.  See for example this new and interesting paper by Susan Athey et al.

Athey and Imbens on Machine Learning and Econometrics

Check out Susan Athey and Guido Imbens' NBER Summer Institute 2015 "Lectures on Machine Learning". (Be sure to scroll down, as there are four separate videos.) I missed the lectures this summer, and I just remembered that they're on video. Great stuff, reflecting parts of an emerging blend of machine learning (ML), time-series econometrics (TSE) and cross-section econometrics (CSE).

The characteristics of ML are basically (1) emphasis on overall modeling, for prediction (as opposed, for example, to emphasis on inference), (2) moreover, emphasis on non-causal modeling and prediction, (3) emphasis on computationally-intensive methods and algorithmic development, and (4) emphasis on large and often high-dimensional datasets.

Readers of this blog will recognize the ML characteristics as closely matching those of TSE! Rob Engle's V-Lab at NYU Stern's Volatility Institute, for example, embeds all of (1)-(4). So TSE and ML have a lot to learn from each other, but the required bridge is arguably quite short.

Interestingly, Athey and Imbens come not from the TSE tradition, but rather from the CSE tradition, which typically emphasizes causal estimation and inference. That makes for a longer required CSE-ML bridge, but it may also make for a larger payoff from building and crossing it (in both directions).

In any event I share Athey and Imbens' excitement, and I welcome any and all cross-fertilization of ML, TSE and CSE.

Causality as Robust Prediction

I like thinking about causal estimation as a type of prediction (e.g., here). Here's a very nice slide deck from Peter Buhlmann at ETH Zurich detailing his group's recent and ongoing work in that tradition.

Econometrics: Angrist and Pischke are at it Again

Check out the new Angrist-Pischke (AP), "Undergraduate Econometrics Instruction: Through Our Classes, Darkly".

I guess I have no choice but to weigh in. The issues are important, and my earlier AP post, "Mostly Harmless Econometrics?", is my all-time most popular.

Basically AP want all econometrics texts to look a lot more like theirs. But their books and their new essay unfortunately miss (read: dismiss) half of econometrics.

Here's what AP get right:

(Goal G1) One of the major goals in econometrics is predicting the effects of exogenous "treatments" or "interventions" or "policies". Phrased in the language of estimation, the question is "If I intervene and give someone a certain treatment \({\partial x}, x \in X\), what is my minimum-MSE estimate of her \(\ \partial y\)?" So we are estimating the partial derivative \({\partial y / \partial x}\).

AP argue the virtues and trumpet the successes of a "design-based" approach to G1. In my view they make many good points as regards G1: discontinuity designs, dif-in-dif designs, and other clever modern approaches for approximating random experiments indeed take us far beyond "Stones'-age" approaches to G1. (AP sure turn a great phrase...). And the econometric simplicity of the design-based approach is intoxicating: it's mostly just linear regression of \(y\) on \(x\) and a few cleverly-chosen control variables -- you don't need a full model -- with White-washed standard errors. Nice work if you can get it. And yes, moving forward, any good text should feature a solid chapter on those methods.

Here's what AP miss/dismiss:

(Goal G2) The other major goal in econometrics is predicting \(y\). In the language of estimation, the question is "If a new person \(i\) arrives with covariates \(X_i\), what is my minimum-MSE estimate of her \(y_i\)? So we are estimating a conditional mean \(E(y | X) \), which in general is very different from estimating a partial derivative \({\partial y / \partial x}\).

The problem with the AP paradigm is that it doesn't work for goal G2. Modeling nonlinear functional form is important, as the conditional mean function \(E(y | X) \) may be highly nonlinear in \(X\); systematic model selection is important, as it's not clear a priori what subset of \(X\) (i.e., what model) might be most useful for approximating \(E(y | X) \); detecting and modeling heteroskedasticity is important (in both cross sections and time series), as it's the key to accurate interval and density prediction; detecting and modeling serial correlation is crucially important in time-series contexts, as "the past" is the key conditioning information for predicting "the future"; etc., etc, ... 

(Notice how often "model" and "modeling" appear in the above paragraph. That's precisely what AP dismiss, even in their abstract, which very precisely, and incorrectly, declares that "Applied econometrics ...[now prioritizes]... the estimation of specific causal effects and empirical policy analysis over general models of outcome determination".)

The AP approach to goal G2 is to ignore it, in a thinly-veiled attempt to equate econometrics exclusively with G1, which nicely feathers the AP nest. Sorry guys, but no one's buying it. That's why the textbooks continue to feature G2 tools and techniques so prominently, as well they should.