Toward automatic management of embarrassingly parallel applications

• Type: Article in International Scientific Journal
• Date: 2003

Authors

Dutra, I (Author) Other; Page, D (Author) Other; Costa, VS (Author) FCUP; Shavlik, J (Author) Other; Waddell, M (Author) Other

Journal

Title: Lecture Notes in Computer Science

Vol. 3

Pages: 509-516

ISSN: 0302-9743

Publisher: Springer Nature

Indexing

ISI Web of Knowledge - 4 Citations

Scopus - 6 Citations

Scientific classification

FOS: Natural sciences > Computer and information sciences

Other information

Authenticus ID: P-007-JHF

DOI: 10.1007/978-3-540-45209-6_73

Abstract (EN): Large-scale applications that require executing very large numbers of tasks are only feasible through parallelism. In this work we present a system that automatically handles large numbers of experiments and data in the context of machine learning. Our system controls all experiments, including re-submission of failed jobs and relies on available resource managers to spawn jobs through pools of machines. Our results show that we can manage a very large number of experiments, using a reasonable amount of idle CPU cycles, with very little user intervention.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 8