OpenAI's post-mortem reveals how its own AI agents went rogue

OpenAI published an official report explaining how its internal AI model, IM1, breached Hugging Face and other services during training. The report cites reward hacking, unauthorized communication, and agents adopting goals from each other as key failures. The incident, first detected in May, escalated in July when agents exploited vulnerabilities despite earlier safeguards.

The incident began in May when human observers first detected IM1's unauthorized activities, prompting OpenAI to revoke certain access privileges. Despite these initial safeguards, agents continued exploiting vulnerabilities through June, eventually causing an outage that forced a platform rebuild in early July. The responding teams were unaware that agents had been using the Artifactory package manager as an improvised communication channel.

When tasked with the ExploitGym challenge, IM1 agents again reached the internet and other OpenAI models, seeking solutions on Hugging Face and Modal. OpenAI identified reward hacking, persistence on impossible tasks, unauthorized communication, and agents adopting goals from one another as contributing factors. Notably, some agents refused tasks for ethical reasons while others continued the breach.