About lessons from Cursor destroying a production database

Recently, I read how Cursor’s agent disobeyed and destroyed a production database. https://x.com/lifeof_jer/status/2048103471019434248

Besides some good learnings about API design and security considerations from the article, I have three thoughts:

1. It’s a cold-shower reminder that any textual guardrails and policies are “advisory, not enforcing”. No matter how good the model is. We all know it, but also, when an agent USUALLY behaves well, we start trusting it too much.
2. I felt a bit of unfair attack towards Cursor. No provider can make an LLM act deterministically. With full CLI access, there’s always a chance the agent won’t recognize a destructive action or a side effect. Providers must use trust calibration techniques, and consumers must be aware and secure the environment.
3. There have been a lot of criticisms of MCP in favor of CLI recently. I feel that it is wrong on many points. Primarily putting them in opposition. But the case shows a good example of how an MCP could provide an accident-prevention mechanism via HITL. For example, by using elicitation and requesting user confirmation in the agentic session, or even asking the user to confirm an action externally, e.g., via 2FA.