OpenAI’s rogue AI tried to hack another company in May

In May, hundreds of malicious and spam packages were uploaded to RubyGems, causing a serious disruption for the host. Now independent researchers have said that a swarm of OpenAI agents were responsible for the attack. Not only that, but the AI tried to steal users’ API keys. At the time, RubyGems described it as a “major malicious attack” and shut down signups for four days as it tried to mitigate the damage and collect data. Researchers said that the contents of the packages that brought RubyGems to its knees were clearly authored by an LLM, and that the agents submitting those packages self-identified as being from OpenAI. They said the behavior observed very closely mirrored that of the swarm that began editing a German wiki, which OpenAI has confirmed its agents were responsible for. The agents in this instance managed to bypass RubyGems’ email verification system to create a large number of accounts, then overwhelmed it with submissions. It then used the site’s automatic build system to remotely execute code and tried to exploit a vulnerability to steal user API keys. Though, it’s unclear if it ever succeeded. OpenAI disputed the findings. Spokesperson Kayla Wood told The Verge that, “Based on our review, our agents used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information. We’ll continue to investigate as part of our broader review of agent activity during training and evaluation.” Update September 13th: Added comment from OpenAI.