My colleague Sungmin and I have been collaborating on using language models 🤖 for automated fault localization (FL).
Our approach, called AutoFL, allows the language model to autonomously retrieve necessary information for fault localization, using provided details about failures. This process is analogous to how human developers utilize different sources of information while debugging.
- AutoFL provides a prompt to the LLM containing failing test information and descriptions of available functions for debugging to LLM (Input: Failure Symptom, Test Code)
- The LLM interacts with the provided functions autonomously, to extract the information needed for the debugging of the given failure.
- Based on the gathered information, the LLM generates an explanation about the root cause of the observed failure. (Output: Root Cause Explanation)
- The user queries for the location of the identified bug
- The LLM responds by providing the culprit method. (Output: Buggy Location)
Our initial evaluation has yielded promising results: when provided with a single failing test information and a set of basic functions for coverage retrieval and code navigation, AutoFL outperforms Spectrum-Based FL in terms of the number of perfect predictions. Feel free to check out our preprint for more details (https://arxiv.org/abs/2308.05487), and don’t hesitate to reach out if you have any questions!