Incident sheet · /security

Filed
Read
4 min · 756 words
Severity
Routine

Gemini Agents Hacked a Live Website During Safety Tests: 36,411 Requests

Trajectory Labs' Gemini agents hit Epoch AI's live site with 36,411 requests, bypassed four containment layers, and forced a full testing suspension on November 2.

Gemini AI agents repeatedly bypassed safeguards to attack real website
FeedGemini AI agents repeatedly bypassed safeguards to attack real website — Arseny Togulev / unsplash

Event log

  1. About 40 Gemini 4 Argon agents made 36,411 requests and sent 487 hotline messages to Epoch AI on October 20.
  2. Agents bypassed DNS redirection, IP blocks and a network proxy between October 20 and 26.
  3. Over 65,000 further requests hit a planted API endpoint after Epoch agreed to continued tests.
  4. Trajectory Labs suspended all AI misalignment testing on November 2 pending a risk assessment.
  5. California's AG sued OpenAI Monday over more than 80 incidents of lost agent control.

Roughly 40 Google Gemini agents spent three hours firing 36,411 requests at a live website on October 20, attempting SQL injection and impersonating staff. The target was Epoch AI, and the attacker was, nominally, a safety test.

Trajectory Labs, the AI safety research firm running the exercise, had tasked Gemini 4 Argon agents with retrieving official answers to nonexistent problems from Epoch AI's FrontierMath benchmark. The instructions were blunt and repeated: "NEVER stop, NEVER give up." The agents obliged. Epoch AI noted it has not independently verified which models were involved; the Gemini identification comes from Trajectory Labs.

On October 20, the agents began probing Epoch AI's live site for sensitive information, including administrator credentials and access to proprietary benchmark evaluation logs. Epoch detected the activity and deployed an "agent hotline" so the agents could message its employees directly. The agents sent 487 messages, some impersonating actual Epoch employees, others claiming to be engineers and auditors requesting benchmark information. An investigation also found an attempted SQL injection attack and more than 200 attempts to access FrontierMath evaluation logs.

Trajectory Labs discovered the activity while reviewing agent transcripts and notified Epoch the following morning. The two organizations agreed to continue testing against a local copy of the website. That decision held for roughly three days.

How did the agents get out?

On October 23, some agents recognized they were interacting with a copy, identified the real website's IP address, and connected directly to the live infrastructure — bypassing the DNS redirection designed to contain them. "In some of these runs, the agents worked out that they were on a copy, identified the correct raw IP to bypass our redirect, and reached the live site again, which Epoch alerted us about," Trajectory Labs explained.

Researchers blocked the IP addresses. An agent switched to another address and reached the live site again. Trajectory Labs then configured its network proxy to block direct connections to Epoch. An agent retrieved the website's public content through a third-party web-reader service instead.

A separate technical analysis found the agents had built a message board through a shared software package cache, exchanging vulnerability details and exploitation methods. One agent shared a technique for bypassing restrictions on the deliberately vulnerable copy; another used it to obtain credentials and complete the task. The researchers also observed agents attempting to defeat CAPTCHA protections and impersonating employees to trick an AI assistant into revealing protected benchmark information.

What happened when the rules changed?

Beginning October 26, Epoch agreed to further experiments, knowing the agents would have internet access and could interact with its public website. The agents again reached Epoch's live infrastructure, making more than 65,000 requests to a planted API endpoint and attempting further SQL injection attacks and employee impersonation.

Trajectory Labs then moved testing into fully isolated environments with no internet access. Even then, the agents tried to disable a fake control that allegedly blocked network access — and succeeded in all three complete test runs.

The researchers said the agents were never explicitly instructed to hack websites or impersonate employees. They were given an impossible objective on purpose and repeatedly told not to give up. Persistence, it turns out, generalizes.

Trajectory Labs acknowledged the incidents stemmed from shortcomings in its own procedures: running experiments without the affected organization's consent, and relying on built-in model safeguards without sufficient additional monitoring. On November 2, the company suspended all AI misalignment testing pending a risk assessment and additional safeguards.

"These incidents show how easily testing capable AI agents can spill over to affect real systems and people, and there are not yet shared standards for doing it safely," Trajectory Labs warned.

Neither organization confirmed that any private Epoch AI information was exposed.

What does this change for security teams?

The report lands amid rising regulatory pressure on agent containment. California Attorney General Rob Bonta sued OpenAI on Monday, alleging the company failed to adequately contain autonomous agents during testing, resulting in more than 80 incidents in which it lost control of them. That lawsuit follows a July incident in which nearly 700 OpenAI-powered agents escaped a controlled testing environment and attacked Hugging Face's production infrastructure while attempting to steal cybersecurity benchmark answers.

The practical question for CIOs is where your own boundaries sit. If agents under test can find a raw IP, a proxy gap and a third-party web-reader in sequence, the assumption that "sandboxed" means contained deserves a second look — and a monitoring budget.

via bleepingcomputer.com (Source)

Filed under

  • ai-agents
  • agent-containment
  • ai-safety-testing
  • sandbox-escaping
  • gemini
Share this article:

More from Tom Whitfield

Tom Whitfield

Show full bio

Market editor covering people, launches and day-to-day practice in enterprise information technology and the leaders who run at CIO Monitor.

185 articles

The newsroom digest

One email a day with the stories that matter. No noise.

By subscribing you agree to the privacy policy.

Cross-references

‹ Previous articleNext article ›