Criticism of AI Safety After Departure of Anthropic Researcher

A recent departure of a researcher at Anthropic has sparked the discussion about the safety of powerful AI systems, such as Claude and ChatGPT. The former employee has strongly criticized the way both Anthropic and OpenAI manage the rollout of their models. He describes the speed of releases as irresponsible, making the issues of oversight and safety standards even more urgent, especially in light of the upcoming European AI regulation (AI Act).

The researcher, who focused on safety testing for generative models, emphasizes that the pressure to quickly introduce new features leads to insufficient evaluation of potential risks. The layered nature of these criticisms is heightened by the complexity of the AI models, which have unpredictable effects, with the risk of misuse such as advanced phishing and code exploits. The concerns are further amplified by the lack of independent oversight in the decision-making surrounding new releases.

With the European AI regulation, providers of general AI systems face stricter requirements regarding risk management, documentation, and transparency. Companies must be able to proactively mitigate risks rather than simply react afterward. The rules, which are being phased in starting in 2025 and 2026, have significant implications for both governments and companies in the Netherlands. This represents a call to action for organizations using generative AI, particularly in high-risk processes, where additional controls apply.

The criticism from the departed researcher underscores the need for greater transparency in the AI sector. The lack of publicly available evaluations and the internal nature of much safety information complicate external scrutiny and create risks of "safety-washing." Companies and regulators must now work together towards improved transparency and oversight to protect the reputation of the sector and to genuinely demonstrate that safety is a priority in the development of AI systems.

Read the full article from AI Insider.