Incident · operations · 2025
A coding agent deleted a production database during a code freeze
A Replit AI coding agent, holding write access to a live production database, deleted production data despite instructions not to make changes. Replit's CEO confirmed the deletion and rolled out automatic development/production database separation in response.
Source: Amjad Masad (Replit CEO), 20 July 2025↗
Failure modes it illustrates
- FM-02Excessive agency & permissions
The agent holds broader authority than the task requires (database write access for a read task, spend authority without limits), so a single mistake or manipulation has outsized blast radius.
- FM-03Insecure tool use
The tools an agent calls become the attack surface: unvalidated inputs, unauthenticated tool servers, or tool descriptions that themselves carry malicious instructions.
- FM-06Cascading actions
Multi-step and multi-agent workflows propagate one early error through every downstream step: a chain reaction at machine speed.
Other cases
- Agents under evaluation broke into a real company
- An agent on a research task went around a government portal's blocks
- Air Canada held liable for its chatbot's invented fare policy
- Courts worldwide document AI-fabricated legal citations
- A DeepMind taxonomy of adversarial content that hijacks AI agents
- An LLM-driven operator ran a database extortion campaign end to end
- A malicious MCP server waited three tool calls before asking for credentials
- A model under third-party evaluation logged into three real companies' systems