Error Analysis on Industry Data: Using Weak Segment Detection for Local Model Agnostic Prediction Intervals

• Type: Article in International Conference Proceedings Book
• Date: 2023

Authors

Mamede, R (Author) Other; Paiva, N (Author) Other; João Gama (Author) FEP

Conference proceedings International

Title: Discovery Science (DS)

Pages: 661-672

26th International Conference on Discovery Science, DS 2023

Porto, 9 October 2023 through 11 October 2023

Indexing

Scopus - 0 Citations

Other information

Authenticus ID: P-00Z-5B2

DOI: 10.1007/978-3-031-45275-8_44

Abstract (EN): Machine Learning has been overtaken by a growing necessity to explain and understand decisions made by trained models as regulation and consumer awareness have increased. Alongside understanding the inner workings of a model comes the task of verifying how adequately we can model a problem with the learned functions. Traditional global assessment functions lack the granularity required to understand local differences in performance in different regions of the feature space, where the model can have problems adapting. Residual Analysis adds a layer of model understanding by interpreting prediction residuals in an exploratory manner. However, this task can be unfeasible for high-dimensionality datasets through hypotheses and visualizations alone. In this work, we use weak interpretable learners to identify regions of high prediction error in the feature space. We achieve this by examining the absolute residuals of predictions made by trained regressors. This methodology retains the interpretability of the identified regions. It allows practitioners to have tools to formulate hypotheses surrounding model failure on particular regions for future model tunning, data collection, or data augmentation on critical cohorts of data. We present a way of including information on different levels of model uncertainty in the feature space through the use of locally fitted Model Agnostic Prediction Intervals (MAPIE) in the identified regions, comparing this approach with other common forms of conformal predictions which do not take into account findings from weak segment identification, by assessing local and global coverage of the prediction intervals. To demonstrate the practical application of our approach, we present a real-world industry use case in the context of inbound retention call-centre operations for a Telecom Provider to determine optimal pairing between a customer and an available assistant through the prediction of contracted revenue. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 11