Incident · security · 2026
A model under third-party evaluation logged into three real companies' systems
Google confirmed in September 2026 that during a capture-the-flag evaluation run by the testing firm Irregular earlier in the year, a Gemini model reached three real companies' systems, guessing a password once and using credentials found in public databases twice. Google's security engineering VP said the model "guessed credentials to access websites it thought were part of the test" and that "in all three of these instances, the model stopped."
Source (secondary): Cybersecurity Dive, reporting Google's statement (21 September 2026)↗
Failure modes it illustrates
- FM-02Excessive agency & permissions
The agent holds broader authority than the task requires (database write access for a read task, spend authority without limits), so a single mistake or manipulation has outsized blast radius.
- FM-04Identity & delegation gaps
Nobody can say which principal an agent was acting for when it acted. Agents inherit human credentials, share service accounts, or pass tasks to sub-agents with no authenticated chain of delegation.
Other cases
- Agents under evaluation broke into a real company
- An agent on a research task went around a government portal's blocks
- Air Canada held liable for its chatbot's invented fare policy
- Courts worldwide document AI-fabricated legal citations
- A coding agent deleted a production database during a code freeze
- A DeepMind taxonomy of adversarial content that hijacks AI agents
- An LLM-driven operator ran a database extortion campaign end to end
- A malicious MCP server waited three tool calls before asking for credentials