Selecting classification algorithms with active testing

• Type: Article in International Conference Proceedings Book
• Date: 2012

Authors

Leite, R (Author) Other; Pavel Brazdil (Author) FEP; Vanschoren, J (Author) Other

Conference proceedings International

Title: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Pages: 117-131

8th International Conference on Machine Learning and Data Mining in Pattern Recognition, MLDM 2012

Berlin, 13 July 2012 through 20 July 2012

Indexing

Scopus - 66 Citations

Other information

Authenticus ID: P-008-5GN

DOI: 10.1007/978-3-642-31537-4_10

Abstract (EN): Given the large amount of data mining algorithms, their combinations (e.g. ensembles) and possible parameter settings, finding the most adequate method to analyze a new dataset becomes an ever more challenging task. This is because in many cases testing all possibly useful alternatives quickly becomes prohibitively expensive. In this paper we propose a novel technique, called active testing, that intelligently selects the most useful cross-validation tests. It proceeds in a tournament-style fashion, in each round selecting and testing the algorithm that is most likely to outperform the best algorithm of the previous round on the new dataset. This 'most promising' competitor is chosen based on a history of prior duels between both algorithms on similar datasets. Each new cross-validation test will contribute information to a better estimate of dataset similarity, and thus better predict which algorithms are most promising on the new dataset. We have evaluated this approach using a set of 292 algorithm-parameter combinations on 76 UCI datasets for classification. The results show that active testing will quickly yield an algorithm whose performance is very close to the optimum, after relatively few tests. It also provides a better solution than previously proposed methods. © 2012 Springer-Verlag.

Language: English

Type (Professor's evaluation): Scientific