| Summary: |
From the international monetary system and power plants to TVs and cell phones, almost everything runs on (embedded) software. With the pervasion of software at all levels of society the impact of software defects is becoming increasingly problematic. A 2002 landmark study on this problem estimated that software defects cost the US economy $60 billion per year (0.6% GDP). Apart from economic loss during deployment, defects also account for an important part of the development costs. With respect to development costs residual defect density is very much dependent on the effort invested in testing, defect finding, and fixing. For embedded software development cost estimates are around $15-30 per line of code. The US Defense Dept. and Carnegie Mellon¿s Software Engineering Institute estimate that defect density ranges from 5 to 15 bugs per KLOC. Error Detection and fault diagnosis are a major cost factor in both the development and the deployment of software. The above data clearly suggests that even a large investment in development does not yield zero-defect code, despite recent advances in software engineering. In fact, for many applications the optimum investment set point (minimizing development and deployment costs due to defects) will correspond to a much higher residual defect density. The fact that in many domains it is more pragmatic to simply cope with failures, rather than invest asymptotic effort at the development phase, is increasingly being recognized. In this new paradigm fault diagnosis is a central focus, since providing the system (and/or operator) with the crucial insight when the system fails and what part(s) of the system cause(d) the failure(s) is key to timely recovery and/or repair. As the earlier data suggests that a large fraction of the costs incurred at development and deployment is related to error detection fault diagnosis, investments in better fault diagnosis have a dramatic effect on development cost and costs incurred at deploymen  |
Summary
From the international monetary system and power plants to TVs and cell phones, almost everything runs on (embedded) software. With the pervasion of software at all levels of society the impact of software defects is becoming increasingly problematic. A 2002 landmark study on this problem estimated that software defects cost the US economy $60 billion per year (0.6% GDP). Apart from economic loss during deployment, defects also account for an important part of the development costs. With respect to development costs residual defect density is very much dependent on the effort invested in testing, defect finding, and fixing. For embedded software development cost estimates are around $15-30 per line of code. The US Defense Dept. and Carnegie Mellon¿s Software Engineering Institute estimate that defect density ranges from 5 to 15 bugs per KLOC. Error Detection and fault diagnosis are a major cost factor in both the development and the deployment of software. The above data clearly suggests that even a large investment in development does not yield zero-defect code, despite recent advances in software engineering. In fact, for many applications the optimum investment set point (minimizing development and deployment costs due to defects) will correspond to a much higher residual defect density. The fact that in many domains it is more pragmatic to simply cope with failures, rather than invest asymptotic effort at the development phase, is increasingly being recognized. In this new paradigm fault diagnosis is a central focus, since providing the system (and/or operator) with the crucial insight when the system fails and what part(s) of the system cause(d) the failure(s) is key to timely recovery and/or repair. As the earlier data suggests that a large fraction of the costs incurred at development and deployment is related to error detection fault diagnosis, investments in better fault diagnosis have a dramatic effect on development cost and costs incurred at deployment due to residual defects. At development time pass/fail information is typically derived from existing test oracles. However, in many situations, in particular at deployment time, no information exists on nominal system behavior unless models are introduced. While at a high behavioral level the use of models (and/or specifications) cannot be avoided, much information on (impending) errors can be inferred from low-level errors within the code. Recent research on error detection as well as preliminary research by the PI¿s group has indicated that automatic instrumentation with simple, generic program invariants (aka fault screeners, probes) provides useful pass/fail information to the state-of-the-art, automatic fault diagnosis technique SFL such that the diagnostic precision that can be achieved is comparable to the use of test oracles. Eliminating the need for (error) modeling paves the way for fully automatic software diagnosis, thus reducing the time needed to pinpoint a defect. In this project we propose to investigate the use of various generic invariants in the value and time domain, their effect on SFL¿s diagnostic precision, their relation with existing test oracles, and their runtime overhead, in particular, the density required or strategic placement (trading off overhead vs precision). Evaluation will be based on data from previous research projects, the SIR benchmark suites used in the community, as well as using two case studies provided by our industrial partner (viz. Critical Software). In summary, the main contribution of this project is a set of novel techniques and tools to automatically detect software errors. The error detection and fault diagnosis capabilities of these techniques will be validated with case studies provided by our industrial partner. A broader impact of this research is the possibility of integrating these techniques with test case generation approaches, achieving total automation of the rather manual,expensive testing phase |