I am delighted to announce that our paper titled “FDG: A Precise Measurement of Fault Diagnosability Gain of Test Cases” has been accepted at ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2022)! This work proposes a novel measure FDG to quantify the fault diagnosability gain of individual test cases. Using the test coverage and the ongoing Fault Localisation (FL) results, FDG can precisely measure how much diagnosability gain each test can bring to the current test suite i.e., the fault-finding capability of a test case.
FDG help developers and testers to prioritize the most effective test cases for enhancing the test suite. This is especially helpful in the following fault localization scenarios:
- Iterative Fault Localization: When the current test suite’s fault diagnosability is low, and traditional SBFL (Spectrum-Based Fault Localization) techniques are not yielding satisfactory results, additional test cases can be generated using automated test generation tools. In that case, whether each of the auto-gen test cases is revealing the fault or not (= oracle problem) should be checked by developers or testers before they are used for fault localization. However, manually inspecting all the generated test cases can be time-consuming and impractical. In this situation, FDG can help prioritize the most valuable auto-generated test cases that are expected to improve the FL accuracy the most, thus saving valuable developer time.
- Selective Test Case Addition: During regression testing, a subset of test cases is often selected to minimize test execution costs while maintaining fault detection ability. However, when test failures occur, it may be necessary to add not-executed test cases to improve fault localization accuracy. FDG can aid in this process by using past coverage information of not-executed test cases as a proxy for their actual coverage and selecting the most promising test cases within a given budget.
I’m excited about sharing our findings at the conference and look forward to discussing our work with the SE community!