Evaluating Interval Forecasts

Evaluation of point and density forecasts is more-or-less straightforward.  Interval forecasts are another matter entirely.  We raised and tried to resolve (some of) the issues here.  Now, in a new paper, "Scoring Interval Forecasts: Equal-Tailed, Shortest, and Modal Interval", Brehmer and Gneiting make major progress. The shortest interval fails to be elicitable!  Their eventual conclusion (p. 14) precisely matches my own view: "In many ways, interval forecasts can be seen as an intermediate stage in the ongoing, transdiciplinary transition from point forecasts to fully probabilistic or distribution forecasts...Indeed, probabilistic forecasts in the form of predictive distributions are the gold standard, as they allow for full-fledged decision making and well-understood, powerful evaluation methods are available..."

Scoring Interval Forecasts: Equal-Tailed, Shortest, and Modal Interval

Jonas Brehmer, Tilmann Gneiting

We consider different types of predictive intervals and ask whether they are elicitable, i.e. are unique minimizers of a loss or scoring function in expectation. The equal-tailed interval is elicitable, with a rich class of suitable loss functions, though subject to either translation invariance, or positive homogeneity and differentiability, the Winkler interval score becomes a unique choice. The modal interval also is elicitable, with a sole consistent scoring function, up to equivalence. However, the shortest interval fails to be elicitable relative to practically relevant classes of distributions. These results provide guidance in interval forecast evaluation and support recent choices of performance measures in forecast competitions.

Comments:	22 pages
Subjects:	Statistics Theory (math.ST)
MSC classes:	62C05, 91B06
Cite as:	arXiv:2007.05709 [math.ST]
	(or arXiv:2007.05709v1 [math.ST] for this version)