Anthropic employee warns: Claude could pose a risk to humanity

An employee of Anthropic, the American AI company behind the Claude model, has publicly warned this week of a real risk that artificial intelligence, such as the Claude system, could annihilate humanity. These statements come amid a growing discussion about the existential risks of AI that could have potentially global consequences. With the new AI regulation in Europe, stricter rules for generative models arise, highlighting the need for responsible handling of such technology.

The employee of Anthropic emphasizes that advanced algorithms have the capacity to develop new strategies and improve themselves, which can lead to unintended or even dangerous situations. The core concern is not the likelihood of such a scenario, but the potential impact it could have. A small chance of catastrophic damage still requires strict precautions, which also involve the misuse of these technologies, such as cyberattacks. This issue is exacerbated as AI models become more accessible and cheaper.

Anthropic is actively working on safety through methodologies like 'Constitutional AI', aimed at minimizing harmful outcomes. The company has also implemented safety protocols and independent audits to evaluate the risks of their models. However, the company notes that there are still gaps in the control and explainability of these technologies. This is crucial, as the likelihood of accidents is estimated between 10 and 25 percent, according to CEO Dario Amodei.

The EU's AI regulation imposes additional steps for generative AI systems, such as Claude, setting high demands for technical documentation and safety requirements. Manufacturers are required to conduct in-depth evaluations and engage robust red teams to better manage risks. This new regulation also places demands on government institutions using AI applications, increasing the responsibilities surrounding data protection and quality management. This requires organizations to not only handle the technology carefully but also to maintain ongoing oversight after going live.

Read the full article from AI Insider.