Estimating reliability for assessing and correcting individual streaming predictions

• Type: Chapter or Part of a Book
• Date: 2012

Authors

Pedro Pereira Rodrigues (Author) FMUP; Bosni¿, Z (Author) Other; João Gama (Author) FEP; Kononenko, I (Author) Other

Book

Title: Reliable Knowledge Discovery

Pages: 29-49

ISBN: 9781461419037; 9781461419020

Indexing

Scopus - 4 Citations

Other information

Authenticus ID: P-00K-DXF

DOI: 10.1007/978-1-4614-1903-7

Abstract (EN): Several predictive systems are nowadays vital for operations and decision support. The quality of these systems is most of the time defined by their average accuracy which has low or no information at all about the estimated error of each individual prediction. In these cases, users should be allowed to associate a measure of reliability to each prediction. However, with the advent of data streams, batch state-of-the-art reliability estimates need to be redefined. In this chapter we adapt and evaluate five empirical measures for online reliability estimation of individual predictions: similarity-based (k-NN) error, local sensitivity (bias and variance) and online bagging predictions (bias and variance). Evaluation is performed with a neural network base model on two different problems, with results showing that online bagging and k-NN estimates are consistently correlated with the error of the base model. Furthermore, we propose an approach for correcting individual predictions based on the CNK reliability estimate. Evaluation is done on a real-world problem (prediction of the electricity load for a selected European geographical region), using two different regression models: neural network and the k nearest neighbors algorithm. Comparison is performed with corrections based on the Kalman filter. The results show that our method performs better than the Kalman filter, significantly improving the original predictions to more accurate values.

Language: English

Type (Professor's evaluation): Scientific