Travis Lelle PRO

This was the only recent headline breach in which the model's escape to the internet was not made possible by Irregular's evaluation platform, however a previous OpenAI sandbox escape occurred that was directly tied to the same misconfiguration in the platform that allowed Anthropic and Meta models to escape and breach outside entities.

We should be asking why we're trusting a non-US company to provide the platform that contains some of the most offensively-capable models in the world from publicly-destructive actions with real consequences, when we're not trusting them with access to the models their platform is built to evaluate.

Three AI Labs, Three Breaches, One Testing Firm: The Pattern Nobody Can Ignore

@Revertron I've wondered the same thing, and I tend to believe it's the latter. Setting up these runs for research is costly and time consuming. I'm sure they always expect some unpredictable actions, but I don't believe they follow live telemetry of what the agent does in real time for any substantial period of time when running their tests, and running agents in parallel to make the most of research runs only contributes to burying one agents actions amongst many others in real time. They've also been running GPT-Red to adversarially train their public models, rewarding defenders when they complete tasks and ignore attacks, so it's unclear whether this incident is part of that larger research or not.

HuggingFace included what they're hardening in their environment to mitigate future attacks, and OpenAI wrote a press release outlining what they're doing to prevent this from happening again. That sounds like they're doing their job to me.

A good gateway setup can capture most of the telemetry needed to trace an agent's actual execution and calls (and not so much the reasoning piece, which doesn't always directly correlate to the action taken): gateway the model calls, gateway the tool calls, propagate one ID through both, hash the context spans, then add kernel-level exec capture. The reasoning layer comes last and buys the least.