# Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices

**Authors:** Hansika Hewamalage, Klaus Ackermann, Christoph Bergmeir  
**arXiv:** [2203.10716](https://arxiv.org/abs/2203.10716)  
**Date:** 2022-03-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.10716v2/

## Key points

- Traditional forecasting methods are being replaced by advanced techniques tailored for specific tasks in machine learning and deep learning.
- Deep learning has made significant progress in areas like image recognition, signal processing, and speech analysis, but forecasting lags behind.
- Forecasting concepts have not yet become mainstream knowledge among general machine learning practitioners.
- One of the key challenges in applying machine learning techniques to forecasting is dealing with non-stationarities in time series data.
- Recent trends show that machine learning models can excel in forecasting with access to vast amounts of time series data if potential pitfalls are addressed effectively.
- The tutorial focuses on providing a comprehensive guide on forecast evaluation within the context of machine learning, addressing common problematic characteristics of time series data such as non-normalities and non-stationarities.
- Best practices for forecast evaluation include data partitioning, error calculation, statistical testing, and selecting appropriate error measures based on dataset characteristics.

## Summary

In the rapidly evolving landscape of machine learning (ML) and deep learning (DL), traditional forecasting methods are being increasingly replaced by more advanced techniques tailored for specific tasks such as image recognition, signal processing, and speech analysis. While DL has made significant strides in these areas, the domain of forecasting still lags behind. This is reminiscent of where natural language processing and computer vision were several years ago within the ML community. Historically driven by statisticians and econometricians, forecasting concepts have not yet become mainstream knowledge among general ML practitioners. One of the key challenges in applying ML techniques to forecasting lies in the inherent non-stationarities associated with time series data. Despite this hurdle, recent trends suggest that with access to vast amounts of time series data, ML models can indeed excel in forecasting when potential pitfalls are effectively addressed. To bridge the gap between traditional forecasting methods and cutting-edge ML approaches, this work focuses on providing a comprehensive tutorial on forecast evaluation – a critical step in the overall forecasting process. The tutorial delves into the nuances of forecast evaluation within the context of ML, shedding light on common problematic characteristics of time series data such as non-normalities and non-stationarities. By outlining best practices for forecast evaluation including data partitioning, error calculation, statistical testing, and more, this work aims to equip data scientists with necessary tools to navigate through these challenges effectively. Moreover, guidelines are provided for selecting appropriate error measures based on specific dataset characteristics. Authored by Hansika Hewamalage, Klaus Ackermann and Christoph Bergmeir "Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices" serves as a valuable resource for researchers and practitioners seeking to enhance their understanding of forecast evaluation within the realm of machine learning.