GPT-3-Powered Type Error Debugging: Investigating the Use of Large Language Models for Code Repair

• Type: Article in International Conference Proceedings Book
• Date: 2023

Authors

Ribeiro, F (Author) Other; de Macedo, JNC (Author) Other; Tsushima, K (Author) Other; Rui Abreu (Author) FEUP; Saraiva, J (Author) Other

Conference proceedings International

Title: PROCEEDINGS OF THE 16TH ACM SIGPLAN INTERNATIONAL CONFERENCE ON SOFTWARE LANGUAGE ENGINEERING, SLE 2023

Pages: 111-124

16th ACM SIGPLAN International Conference on Software Language Engineering (SLE) as part of SPLASH Conference

Cascais, PORTUGAL, OCT 23-24, 2023

Indexing

ISI Web of Knowledge - 0 Citations

Scopus - 0 Citations

Other information

Authenticus ID: P-00Z-B11

DOI: 10.1145/3623476.3623522

Abstract (EN): Type systems are responsible for assigning types to terms in programs. That way, they enforce the actions that can be taken and can, consequently, detect type errors during compilation. However, while they are able to flag the existence of an error, they often fail to pinpoint its cause or provide a helpful error message. Thus, without adequate support, debugging this kind of errors can take a considerable amount of effort. Recently, neural network models have been developed that are able to understand programming languages and perform several downstream tasks. We argue that type error debugging can be enhanced by taking advantage of this deeper understanding of the language's structure. In this paper, we present a technique that leverages GPT-3's capabilities to automatically fix type errors in OCaml programs. We perform multiple source code analysis tasks to produce useful prompts that are then provided to GPT-3 to generate potential patches. Our publicly available tool, Mentat, supports multiple modes and was validated on an existing public dataset with thousands of OCaml programs. We automatically validate successful repairs by using Quickcheck to verify which generated patches produce the same output as the user-intended fixed version, achieving a 39% repair rate. In a comparative study, Mentat outperformed two other techniques in automatically fixing ill-typed OCaml programs.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 14