Security company Vulcan has warned software developers about the potential risks associated with ChatGPT’s tendency to “hallucinate” or generate false information. Researchers Ortal Keizman and Yair Divinsky discovered that ChatGPT, a large language model, can generate URLs, references, code libraries, and functions that do not actually exist. This phenomenon, known as “hallucinations,” could be exploited by attackers to spread malicious packages in software development environments.
The researchers highlighted that attackers could leverage these hallucinations to distribute malicious packages without resorting to traditional techniques like typosquatting or masquerading. By offering a package that replaces the hallucinated content, an attacker could deceive a victim into downloading and using the malicious package. This technique, termed “AI package hallucination,” poses a significant threat if ChatGPT recommends non-existent packages that are not published in legitimate package repositories. Attackers can take advantage of the hallucinated names to post malicious packages, which may be subsequently recommended by ChatGPT to unsuspecting users.
To demonstrate the potential risks, the researchers conducted tests using popular questions from forums like StackOverflow, focusing on programming languages such as Python and Node.js. Their experiments revealed that a significant number of ChatGPT’s responses included references to non-existent packages. For instance, out of 201 questions about Node.js, 40 answers referred to more than 50 non-existent packages. Similarly, 227 questions about Python drew answers mentioning over 100 non-existent packages.
The findings highlight the importance of addressing the hallucination issue in language models like ChatGPT to mitigate potential risks and protect software development environments from malicious actors.
