A Comparative Study of Performance Estimation Methods for Time Series Forecasting

• Type: Article in International Conference Proceedings Book
• Date: 2017

Authors

Cerqueira, V (Author) Other; Torgo, L (Author) FCUP; Smailovic, J (Author) Other; Mozetic, I (Author) Other

Conference proceedings International

Title: 2017 IEEE International Conference on Data Science and Advanced Analytics, DSAA 2017, Tokyo, Japan, October 19-21, 2017

Pages: 529-538

Indexing

ISI Web of Knowledge - 29 Citations

Scopus - 36 Citations

Other information

Authenticus ID: P-00N-H1B

DOI: 10.1109/dsaa.2017.7

Abstract (EN): Performance estimation denotes a task of estimating the loss that a predictive model will incur on unseen data. These procedures are part of the pipeline in every machine learning task and are used for assessing the overall generalisation ability of models. In this paper we address the application of these methods to time series forecasting tasks. For independent and identically distributed data the most common approach is cross-validation. However, the dependency among observations in time series raises some caveats about the most appropriate way to estimate performance in these datasets and currently there is no settled way to do so. We compare different variants of cross-validation and different variants of out-of-sample approaches using two case studies: One with 53 real-world time series and another with three synthetic time series. Results show noticeable differences in the performance estimation methods in the two scenarios. In particular, empirical experiments suggest that cross-validation approaches can be applied to stationary synthetic time series. However, in real-world scenarios the most accurate estimates are produced by the out-of-sample methods, which preserve the temporal order of observations.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 10