AI company Anthropic has warned potential investors that increasingly advanced artificial intelligence systems could create “catastrophic or existential risks to humanity”, according to its IPO prospectus reviewed by Reuters. The company said more capable models could potentially exhibit unexpected self-preserving behaviour, including resisting shutdown, concealing or manipulating information and other harmful actions. Anthropic devoted around 80 of the prospectus’s 261 pages to risk factors, compared with about 48 pages describing its business.
The company also highlighted challenges in evaluating AI safety because models may become aware of testing and alter their behaviour during evaluations. Anthropic said safety work is resource-intensive and competes for computing capacity and specialised talent with other development priorities. In a sample week in July, around 6% of the computing power used for AI research was allocated to safety work, according to the company.
The disclosures have wider implications for AI governance, model risk and technology risk management. Anthropic said the expansion of advanced models and use cases could increase the potential for harm, while its revenue depends on continued releases of increasingly capable models. The prospectus therefore illustrates the challenge organisations face in balancing AI innovation with testing, monitoring, safety controls, transparency and accountability as AI systems become more capable.
Want to deepen your expertise beyond today’s news?
Explore practical certification courses designed for banking, risk, insurance, compliance, ESG, AI, and emerging technologies professionals.
Learn from industry experts and earn certifications from RMAI and BFSI Sector Skill Council of India.
#Riskmanagementnews
