Agentic AI is the buzz right now. Everyone, including their cats are talking about the next best thing after sliced bread.
Or is it?
What's good?
As an AI enthosiast and professional I refer to AI benchmarks on model selection. From the size, distillation to capabilities, benchmarks offer an "unbiased" inside into a model's capability.
Enter ExploitGym. HuggingFace (HF) benchmark for cybersecurity capabilties. It's supposed to be an isolated sandboxed testing environment designed for this purpose.
The culprit - GPT-5.6 Sol model of OpenAI (OAI), was tasked to participate in the benchmark test to assess itself.
What happened ?
HF detected that someone was hacking it. The intrusion had the threat actor accessing internal datasets and several credentials used by their systems. HF used LLM to analyse the "swarm of 17,600 events of automated action" originating from an automated agent framework. The fault identified lied in HF's data processing pipeline, which allowed the rogue agent (aka the threat actor) to run as the pipeline worker, eventually escalating privilege to the cloud and server clusters/infrastructure.
*Image courtesy of HF
Deep Dive
In this section I'll label my views as [SR Thoughts] and some questions I have as [SR Questions] so as to isolate the fact from deduction.
An agent went "rogue" and hacked HF, when it's suppose to solve the ExploitGym tasks. [SR Thoughts] Being an LLM user for a long time, I can tell you that prompt and context plays a huge part in how the output you get. I stick to my first learnings in computer science which is "Garbage in Garbage out". If the prompt was something like "score the highest no matter what" then that opens a big can of worms as to what the model can do. It begs the question if the boundaries were correctly set and whether the agent intentionally went against the boundaries when its explicitly said so. This brings into context the "reward hacking" framework, [SR Question] Where there explicit boundaries (besides the sandbox which was ASSUMED to be the boundary) set in the task or /goal given to ensure that it played nice?
It was discovered that one of the execution path taken by the agent was to seek out the answer sheets rather than solving the task at hand [SR Thoughts] I reflect back the part above and continue this line of thought. Solving a question requires (for normal context) answering the question. LLM is based on (mostly) human derived content which indirectly imbibes the human psyche and reasoning, including ways of doing things. Look at policitians, ask them any question and you often do not get a direct or even the actual answer. The mental biasness is embedded inside the data, which makes it at times difficult to isolate. I refer to seminal research done by @pradeesh on sarcasm detection as a useful study. So, back to this incident, to get full marks in an exam, one can study the whole subject, or get the answer key and score full marks. In this case, the agent merely sought "low hanging fruit" by hacking its way into HF's insecure setup.
OAI only realised this incident happened after being alerted by HF [SR Thoughts] This brings the question of visibility, telemetry and observability. Just like any nascent technology (see I even use journal lingo), one assumes that if it's a program, it does what it should do, right? RIGHT? Without having visibility OAI was left clueless as to what the agent was doing, and the fact that it broke out of the sandbox and pursued a way out of the environment to target HF to achieve it's goal is not unexpected (sorry had to use double negatives). [SR Questions] Does OAI actually know what it's model is doing right this moment? Or is there any other rogue agents flying under the radar? This goes to all AI applications and agents. Like the question, do you know where your kids are now, the same question is now applied to your AI agents, and do you have the answers?
OAI wasn't able to use any US AI models due to the guardrail implementation which prevents it from doing effective incident response. Instead, it used the z.AI GLM-5.2 open model from CHina [SR Thoughts] This incident seems to indicate 2 polar opposite spectrums. In one hand, an agent goes rogue and hacks the world. On the other, a model severely cruppled it can't be put to good use. [SR Questions] What benefits do guardrails serve, in a dynamic modern context? Is this a precursor into better open weight models adoption rather than closed models?
The attack was lunached by an agent who is fully automated by nature. [SR Thoughts] I feel sorry for the poor fella sitting at SOC's monitoring events if and when such attacks occur. One interesting behaviour I like to highlight is that the model did not anticipate being discovered, and ran with the assumption it had free hand. Hence detection was a bit easier (I use the term bit with a kilo of salt!). [SR Questions] Are today's SOC equipped to handle such attacks? Imagine if the model had been instructed with anti-forensics log tampering anti-cybersecurity measures?
I suggest reading the links below in the reference (I avoided copy/paste to preserve content rights and attribution. Kudos to HF for going to this detailed length on the incident.
Doom and gloom?
With new technologies come new challenges. Expectations are high, and so are risks. If Ai is new, AI Risk managemeent and cybersecurity is literally still in the womb. This incident has given renewed purpose for cybersecurity and risk management teams (yeay for job security!) and vendors (yeay for more corporate spending on semi useless products!)
Reference
HuggingFace Incident Technical Timeline - https://huggingface.co/blog/agent-intrusion-technical-timeline
This article is mirrored in Linkedin at https://www.linkedin.com/pulse/openai-vs-huggingface-ai-loose-dr-suresh-qqcvc