Probabilistic change detection and visualization methods for the assessment of temporal stability in biomedical data quality

• Type: Article in International Scientific Journal
• Date: 2015

Authors

Carlos Saez (Author) Other; Pedro Pereira Rodrigues (Author) FMUP; João Gama (Author) FEP; Montserrat Robles (Author) Other; Juan M Garcia Gomez (Author) Other

Journal

Title: Data Mining and Knowledge Discovery

Vol. 29

Pages: 950-975

ISSN: 1384-5810

Publisher: Springer Nature

Indexing

ISI Web of Knowledge - 17 Citations

Scopus - 18 Citations

Scientific classification

FOS: Natural sciences > Computer and information sciences

Other information

Authenticus ID: P-00G-6BX

DOI: 10.1007/s10618-014-0378-6

Abstract (EN): Knowledge discovery on biomedical data can be based on on-line, data-stream analyses, or using retrospective, timestamped, off-line datasets. In both cases, changes in the processes that generate data or in their quality features through time may hinder either the knowledge discovery process or the generalization of past knowledge. These problems can be seen as a lack of data temporal stability. This work establishes the temporal stability as a data quality dimension and proposes new methods for its assessment based on a probabilistic framework. Concretely, methods are proposed for (1) monitoring changes, and (2) characterizing changes, trends and detecting temporal subgroups. First, a probabilistic change detection algorithm is proposed based on the Statistical Process Control of the posterior Beta distribution of the Jensen-Shannon distance, with a memoryless forgetting mechanism. This algorithm (PDF-SPC) classifies the degree of current change in three states: In-Control, Warning, and Out-of-Control. Second, a novel method is proposed to visualize and characterize the temporal changes of data based on the projection of a non-parametric information-geometric statistical manifold of time windows. This projection facilitates the exploration of temporal trends using the proposed IGT-plot and, by means of unsupervised learning methods, discovering conceptually-related temporal subgroups. Methods are evaluated using real and simulated data based on the National Hospital Discharge Survey (NHDS) dataset.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 26