Safety first: AI models by OpenAI, Anthropic to undergo testing before US rollouts

Upcoming frontier AI models by OpenAI and Anthropic will be tested for safety before being released to the public, as part of an agreement negotiated between the two AI startups and the US AI Safety Institute.

The US AI Safety Institute will get “access to major new models from each company prior to and following their public release..” and this “will enable collaborative research on how to evaluate capabilities and safety risks, as well as methods to mitigate those risks,” read a press release dated Thursday, August 29.

Additionally, “the US AI Safety Institute plans to provide feedback to Anthropic and OpenAI on potential safety improvements to their models, in close collaboration with its partners at the UK AI Safety Institute,” the press release read.

The US AI Safety Institute is part of the National Institute of Standards and Technology (NIST) that falls under the US Department of Commerce. It was established after US President Joe Biden issued an executive order on artificial intelligence in October 2023, mandating safety assessments of AI models, among other requirements.

Sam Altman, the CEO of OpenAI, welcomed the move and said, “We are happy to have reached an agreement with the US AI Safety Institute for pre-release testing of our future models.” “We strongly support the U.S. AI Safety Institute’s mission and look forward to working together to inform safety best practices and standards for AI models,” Jason Kwon, the chief strategy officer of the ChatGPT parent company, was quoted as saying by CNBC.

Meanwhile, Anthropic co-founder Jack Clark said that the Amazon-backed company’s “collaboration with the US AI Safety Institute leverages their wide expertise to rigorously test our models before widespread deployment” and “strengthens our ability to identify and mitigate risks, advancing responsible AI development.”

Why does it matter?

This is probably the first time that tech companies have agreed to subject their models for inspection by a government body. It could also open the door for governments in other countries like India to demand that AI models be evaluated for safety risks and ethical concerns before being released to the public.

Earlier this year, the Indian government sparked uproar after the Union Ministry of Electronics and Information Technology issued an advisory stating that untested or unreliable AI models need to receive explicit approval from the government before being rolled out to the public. It later clarified that such models can be made available to users in the country “after appropriately labelling the possible inherent fallibility or unreliability of the output generated.”

More recently, lawmakers in California voted to pass legislation that would make it mandatory to carry out safety tests for AI models of a certain cost or computing power. The AI bill called SB 1047, which is yet to receive final approval by California Governor Gavin Newsom, has been opposed by some tech companies on the grounds that it could hinder growth and innovation.

.