Evaluating forecasts and training forecasting models

Thordis Thorarinsdottir (University of Oslo)

Wednesday 7th October 11:00-13:00
Maths 311B

Abstract

In forecast evaluation, a scoring rule provides an evaluation metric for probabilistic predictions or forecasts. That is, the scoring rule assigns a numerical score to the forecast by comparing the forecast and the realised observation. A scoring rule is called proper if it has the property that the score is optimised in expectation when the true data distribution is issued as the forecast. This property is considered a necessary condition for a decision-theoretically principled forecast evaluation. The class of proper scoring rules is large and diverse, so that proper scoring rules may evaluate different aspects of the forecast. For example, it has been stated that the aim of probabilistic forecasting should be to maximise the sharpness of the forecast subject to calibration. That is, there should be statistical compatibility between the predictive distribution and the observation while, at the same time, the forecast should provide as much information on the observation as possible. Different proper scoring rules are able to assess these properties to a different degree. On the other hand, if we want the forecast to possess certain properties that are well measured by a certain proper scoring rule, it seems natural to also consider this same scoring rule as a loss function when estimating the forecast. This line of thinking is increasingly being used in machine learning, in particular in applications such as meteorology where there is a long tradition for forecast evaluation with proper scoring rules. We will cover the foundations of forecast evaluation with a focus on proper scoring rules and related evaluation metrics, discuss some recent developments and, in particular, connections to the training of machine learning algorithms.

Add to your calendar

Download event information as iCalendar file (only this event)