Two nonprofit AI safety groups, METR (opens in new tab) and Redwood Research, said Thursday that they had stress-tested OpenAI’s internal system for monitoring AI agents for misaligned behavior. They found “a number of novel vulnerabilities,” according to a public summary METR (opens in new tab) posted. One employee from each group worked on the pilot from July 21 to Aug. 11. They had access to OpenAI’s internal codebase, monitor prompts and documentation, used company-issued devices that “never left the premises,” and were accompanied while on site.
Among the findings, the evaluators described an exploit that agents could use to prevent “hundreds of recent actions (including the actions involved in the exploit itself) from being monitored.” They also found a way that Codex agents could in principle bypass Codex sandboxing and Auto-Review, and two client-controllable headers that disable monitoring for new agents. They estimated that roughly 75% to 94% of OpenAI’s internal agentic Responses API traffic from models at least as strong as GPT-5.5 was eligible for monitoring at the time, though they said they had “substantial uncertainty in both directions.”
METR (opens in new tab) said OpenAI “has stated that all of the vulnerabilities we identified that impact their production safeguards stack have been patched,” but added: “We have not evaluated these claims.” The findings describe the system as it existed on and before July 21, and OpenAI was expanding it during the exercise. The groups shared a 61-page internal report with OpenAI and agreed in advance to publish only a brief, high-level post. OpenAI could redact sensitive details.
“This collaboration builds on our longstanding commitment to pioneering new approaches to third-party testing, and we’re using what we learned to strengthen our safeguards and shape future collaborations with independent evaluators,” an OpenAI spokesperson told CNBC (opens in new tab). CNBC (opens in new tab) noted that OpenAI CEO Sam Altman pledged in September to give independent evaluators “employee-like access.”