Combining Feature and Algorithm Hyperparameter Selection using some Metalearning Methods

• Type: Article in International Conference Proceedings Book
• Date: 2017

Authors

Cachada, M (Author) Other; Abdulrahman, SM (Author) Other; Pavel Brazdil (Author) FEP

Conference proceedings International

Title: AutoML@PKDD/ECML

Pages: 69-83

2017 International Workshop on Automatic Selection, Configuration and Composition of Machine Learning Algorithms, AutoML 2017

22 September 2017

Indexing

Scopus - 4 Citations

Other information

Authenticus ID: P-00N-AQQ

Abstract (EN): Machine learning users need methods that can help them identify algorithms or even workflows (combination of algorithms with preprocessing tasks, using or not hyperparameter configurations that are different from the defaults), that achieve the potentially best performance. Our study was oriented towards average ranking (AR), an algorithm selection method that exploits meta-data obtained on prior datasets. We focused on extending the use of a variant of AR¿ that takes A3R as the relevant metric (combining accuracy and run time). The extension is made at the level of diversity of the portfolio of workflows that is made available to AR. Our aim was to establish whether feature selection and different hyperparameter configurations improve the process of identifying a good solution. To evaluate our proposal we have carried out extensive experiments in a leave-one-out mode. The results show that AR¿ was able to select workflows that are likely to lead to good results, especially when the portfolio is diverse. We additionally performed a comparison of AR¿ with Auto-WEKA, running with different time budgets. Our proposed method shows some advantage over Auto-WEKA, particularly when the time budgets are small.

Language: English

Type (Professor's evaluation): Scientific