OpenAI's confirmation of harmful software distribution

OpenAI has recently confirmed that the responses of ChatGPT and similar AI systems have unintentionally contributed to the distribution of harmful software packages. This situation has occurred globally, including in Europe, where users have been guided towards malicious software through installation tips and code examples. The company claims to have taken technical countermeasures and is investigating this issue, which also affects Dutch organizations under the European AI Regulation.

According to OpenAI, the AI models are sometimes responsible for providing advice that leads developers to infected packages. This often happens via installation commands like pip or npm. The AI systems themselves do not host software, but by making these recommendations, malicious packages can be integrated into projects unnoticed. The problem arises from a combination of model hallucinations, where the AI generates non-existent or altered package names that are exploited by attackers.

OpenAI claims to have taken actions to mitigate this issue, such as tightening detections and filters to discourage risky installations. Warnings and instructions have also been added to the installation guidelines generated by the models. Additionally, collaboration is ongoing with external partners to limit the misuse of the AI systems. The company urges organizations to take extra precautions, especially now that the European AI Regulation will soon come into force, outlining responsibilities for providers of AI systems.

The European AI Regulation imposes obligations on companies to mitigate systemic risks and transparently report any cases of abuse. This is crucial for organizations in sectors covered by NIS2, as stricter requirements apply for supply chain security. Under the Cyber Resilience Act, companies must also demonstrate that they are effectively managing vulnerabilities and risks. Organizations are advised to consider AI output as suggestive, thoroughly check dependencies, and build internal guidelines and controls to ensure safety.

Read the full article from AI Insider.