Statistically Robust Evaluation of Stream-Based Recommender Systems

• Type: Article in International Scientific Journal
• Date: 2021

Authors

Vinagre, J (Author) Other; Jorge, AM (Author) FCUP; Rocha, C (Author) Other; João Gama (Author) FEP

Journal

Title: IEEE Transactions on Knowledge and Data Engineering

Vol. 33

Pages: 2971-2982

ISSN: 1041-4347

Publisher: IEEE

Indexing

ISI Web of Knowledge - 8 Citations

Scopus - 11 Citations

Other information

Authenticus ID: P-00R-FP4

DOI: 10.1109/tkde.2019.2960216

Abstract (EN): Online incremental models for recommendation are nowadays pervasive in both the industry and the academia. However, there is not yet a standard evaluation methodology for the algorithms that maintain such models. Moreover, online evaluation methodologies available in the literature generally fall short on the statistical validation of results, since this validation is not trivially applicable to stream-based algorithms. We propose a k-fold validation framework for the pairwise comparison of recommendation algorithms that learn from user feedback streams, using prequential evaluation. Our proposal enables continuous statistical testing on adaptive-size sliding windows over the outcome of the prequential process, allowing practitioners and researchers to make decisions in real time based on solid statistical evidence. We present a set of experiments to gain insights on the sensitivity and robustness of two statistical tests-McNemar's and Wilcoxon signed rank-in a streaming data environment. Our results show that besides allowing a real-time, fine-grained online assessment, the online versions of the statistical tests are at least as robust as the batch versions, and definitely more robust than a simple prequential single-fold approach.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 12