The information security company Rezilion held study which showed that the use of solutions based on large language models (Large Language Model, LLM) carries certain risks for business.

Rezilion experts ranked the 50 most popular LLM projects on GitHub using the Scorecard tool from the Open Source Security Foundation, which takes into account various aspects of projects, such as the number of vulnerabilities, methods of support, and other factors.

Since public launch ChatGPT over 30,000 projects based on GPT-3.5 have been created on GitHub. They have been widely used in various software solutions. The researchers built a “map” of these projects, where the y-axis is the level of popularity, and the x-axis is the level of security (based on the OpenSSF Scorecard). As a result, none of the analyzed projects received more than 6.1 points out of 10. Thus, all the most popular LLM solutions are associated with a high level of risk, and the average score is only 4.6.

The most popular project was Auto GPT , which has earned almost 140k stars on GitHub in less than 3 months since its launch. However, his Scorecard rating was only 3.7, indicating a high level of risk. As Rezilion notes, new projects are often characterized by rapid growth in popularity, but developers and IT professionals need to be aware of the risks they carry.

Rezilion experts emphasize that when a new project is launched, it is impossible to say with certainty how it will develop and be supported. Most of the projects that quickly reached the top of popularity have a low level of protection. When the researchers compared the age of projects to their Scorecard rating, they found that the most common projects were 2 months old and rated 4.5–5 out of 10.