Evaluating and improving fault localization

• Type: Article in International Conference Proceedings Book
• Date: 2017

Authors

Pearson, S (Author) Other; José Campos (Author) Other; Just, R (Author) Other; Fraser, G (Author) Other; Rui Abreu (Author) FEUP; Ernst, MD (Author) Other; Pang, D (Author) Other; Keller, B (Author) Other

Conference proceedings International

Title: 2017 IEEE/ACM 39TH INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING (ICSE)

Pages: 609-620

39th IEEE/ACM International Conference on Software Engineering (ICSE)

Buenos Aires, ARGENTINA, MAY 20-28, 2017

Indexing

ISI Web of Knowledge - 134 Citations

Scopus - 355 Citations

Other information

Authenticus ID: P-00M-T5K

DOI: 10.1109/icse.2017.62

Abstract (EN): Most fault localization techniques take as input a faulty program, and produce as output a ranked list of suspicious code locations at which the program may be defective. When researchers propose a new fault localization technique, they typically evaluate it on programs with known faults. The technique is scored based on where in its output list the defective code appears. This enables the comparison of multiple fault localization techniques to determine which one is better. Previous research has evaluated fault localization techniques using artificial faults, generated either by mutation tools or manually. In other words, previous research has determined which fault localization techniques are best at finding artificial faults. However, it is not known which fault localization techniques are best at finding real faults. It is not obvious that the answer is the same, given previous work showing that artificial faults have both similarities to and differences from real faults. We performed a replication study to evaluate 10 claims in the literature that compared fault localization techniques (from the spectrum-based and mutation-based families). We used 2995 artificial faults in 6 real-world programs. Our results support 7 of the previous claims as statistically significant, but only 3 as having non-negligible effect sizes. Then, we evaluated the same 10 claims, using 310 real faults from the 6 programs. Every previous result was refuted or was statistically and practically insignificant. Our experiments show that artificial faults are not useful for predicting which fault localization techniques perform best on real faults. In light of these results, we identified a design space that includes many previously-studied fault localization techniques as well as hundreds of new techniques. We experimentally determined which factors in the design space are most important, using an overall set of 395 real faults. Then, we extended this design space with new techniques. Several of our novel techniques outperform all existing techniques, notably in terms of ranking defective code in the top-5 or top-10 reports.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 12