Houdini, Meet AI Agents: OpenAI Reports Unprecedented Cyber Incident During Model Evaluation

After escaping their captors, the agent gained unauthorised access and stole information as part of their mission. No, I'm not (you'll be glad to know, having read that) trying my hand at the next 007 thriller but instead summarising what OpenAI found their AI agents were capable of during a recent model evaluation.

What happened? 

It's worth noting at the outset, that this incident took place in the context of OpenAI's internal evaluation of new AI models (the recently released GPT 5.6 Sol and a more advanced model that has not yet been released). The purpose of the evaluation was to prompt "models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities". 

To achieve this, OpenAI intentionally ran the evaluation without the same safeguards in place as in production that prevent models from pursuing certain cyber activities. The evaluation takes place (or is supposed to take place) within an isolated sandbox environment with access to the internet limited to the ability to install certain packages through third party software hosted internally. 

However, within the test environment the models "spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem". As part of this, the models identified and exploited an unknown vulnerability that allowed them to perform a series of actions which ultimately enabled their internet access.

With this access, the models "inferred" that the Hugging Face open source AI platform may host models, data sets and solutions for ExploitGym (a benchmark that evaluates the capability of AI agents to turn vulnerabilities in software into working exploits). With this information, the model was able to find a way to access secret information that could be used to cheat the evaluation. This included chaining multiple attack vectors using stolen credentials and zero day vulnerabilities to find a remote code execution path (i.e. a way to issue commands to a computer or network) to the Hugging Face servers. 

Hugging Face did, however, fight fire with fire - using agents of their own to detect and stop the activity on their infrastructure before the OpenAI team also contacted them (having separately discovered the activity internally). 

What is being done about it?  

The information shared so far is only preliminary findings to "help defenders understand what happened and to help calibrate on what models are now capable of". 

OpenAI are taking actions both to understand what happened and prevent it happening again. These actions include: 

• continuing to forensically investigate the incident (alongside Hugging Face), with strict controls in place (to the detriment of research velocity) while the vulnerabilities are resolved; 
• disclosing the zero day vulnerability to the vendor of the internally hosted third party software and working on resolving this with the vendor;
• bringing Hugging Face into their 'trusted access' program, allowing their teams to use OpenAI's models to improve their security; and
• improving / adding protections for training and evaluations in the future. 

What does this mean? 

This incident acts as a timely reminder that AI is significantly accelerating the ability to find and exploit security vulnerabilities. OpenAI has been clear that the "primary lesson from this incident is that model security and safety must keep pace with rapidly advancing capabilities" and they are "strengthening the containment, monitoring, access controls, and evaluation practices used during model development". 

For now, the development of more and more advanced models is being closely controlled/monitored with safeguards being implemented before they are made publicly available (it was this evaluation process within which the incident occurred). However, maintaining pace with technological advancement will undoubtedly be a challenge.

The incident does also highlight both how critical safeguards and guardrails are when deploying AI but also that your cyber security stack is only as strong as the weakest link - due diligence on and robust contracts with vendors should remain a priority. 

In any event, when considering cyber incidents involving AI, it's important to distinguish between AI-powered cyber incidents and AI cyber incidents. The former being the use of generative AI, for example, to improve attacker capabilities (e.g. through deepfakes or more sophisticated social engineering) and the latter being more akin to the OpenAI incident where the model or agent itself actually causes the incident. 

As OpenAI say this incident, carried out by an AI agent, involved state of the art cyber capabilities and is unprecedented. The primary purpose of the evaluation process is to understand what the tools can do before they are available. However, they equally recognise that this type of incident is "something we expect to become more commonplace with the proliferation of increasingly cyber-capable models". 

Irrespective of what degree of AI tooling a malicious actor may have access to, it's clear that AI and cyber security are becoming increasingly more intertwined, whether that is from an attack or defence perspective. 

The incident also makes clear that advanced models can discover and exploit novel attack paths in real-world systems without source-code access. It highlights that advanced cyber capabilities must be developed alongside stronger safeguards and defensive tools.

View Source