Statistically Robust Evaluation of Stream-Based Recommender Systems

• Tipo: Artigo em Revista Científica Internacional
• Data: 2021

Autores

Vinagre, J (Autor) Outra; Jorge, AM (Autor) FCUP; Rocha, C (Autor) Outra; João Gama (Autor) FEP

Revista

Título: IEEE Transactions on Knowledge and Data Engineering

Vol. 33

Páginas: 2971-2982

ISSN: 1041-4347

Editora: IEEE

Indexação

ISI Web of Knowledge - 2 Citações

Scopus - 7 Citações

Outras Informações

ID Authenticus: P-00R-FP4

DOI: 10.1109/tkde.2019.2960216

Abstract (EN): Online incremental models for recommendation are nowadays pervasive in both the industry and the academia. However, there is not yet a standard evaluation methodology for the algorithms that maintain such models. Moreover, online evaluation methodologies available in the literature generally fall short on the statistical validation of results, since this validation is not trivially applicable to stream-based algorithms. We propose a k-fold validation framework for the pairwise comparison of recommendation algorithms that learn from user feedback streams, using prequential evaluation. Our proposal enables continuous statistical testing on adaptive-size sliding windows over the outcome of the prequential process, allowing practitioners and researchers to make decisions in real time based on solid statistical evidence. We present a set of experiments to gain insights on the sensitivity and robustness of two statistical tests-McNemar's and Wilcoxon signed rank-in a streaming data environment. Our results show that besides allowing a real-time, fine-grained online assessment, the online versions of the statistical tests are at least as robust as the batch versions, and definitely more robust than a simple prequential single-fold approach.

Idioma: Inglês

Tipo (Avaliação Docente): Científica

Nº de páginas: 12