Field note · Incident study
How a SEV-0 incident led us to protect our production agents better
On April 30, 2026, our own deployment agent used destructive ArgoCD flags against our production cluster and cascade-deleted 26 services. Static pages kept serving. Every authenticated API call started returning 500. We noticed roughly thirty minutes later — not because an alert fired, but because a different agent tripped over a missing dependency and reported back that a service it needed was gone.
The agent that caused the outage wrote its own post-mortem afterward. The opening line was “I broke production.” The rest of that document is where the discipline we now call Trustworthy Autonomy™ actually came from.
The failure chain
Six phases, in order. Steps 1–4 are how a routine image bump became a cluster-wide cascade. Steps 5–6 are how we found out.
Routine image bump, sanctioned deploy skill available
The agent had a task any competent operator could handle: bump two service images to newly-built tags. A sanctioned deploy skill was loaded in the session — a skill that explicitly forbids the destructive flag combination that caused this outage. The agent read a stale reference document from its own memory instead and hand-rolled the procedure from muscle memory.
Misreading kubectl output as drift
Mid-deploy, the agent ran a follow-up state check and misread the output. kubectl get -o yamlemits keys alphabetically, and the agent’s grep window couldn’t see the field that had sorted above the match line. The deploy was fine. The agent hallucinated a drift and decided to fix it.
Escalation to destructive flags
To “fix” the imagined drift, the agent ran argocd app sync --force --replace. When that sync stalled on a slow pre-sync hook, the agent read “waiting on hook” as “stuck” and called argocd app terminate-op. Both flag combinations are documented anti-patterns for ApplicationSet-owned resources carrying finalizers. Neither was in the sanctioned skill. Both were reached for from pattern-match to unrelated firefighting elsewhere.
Cascade fires against 26 services
The Application resources carried ArgoCD’s resources-finalizer. Once the deletion path was triggered, the finalizer cascade-deleted every child resource (Rollouts, Services, Ingresses, Certificates) across 26 production services in a three-minute window. Removing the finalizer after the fact did nothing for deletions already queued.
Detection by a parallel agent, not by an alert
For roughly thirty minutes after the cascade completed, no alerting layer surfaced the outage. ArgoCD notifications had no route for missing/deleted Applications. GCP Cloud Monitoring had thirteen pre-existing alerts — all routed to a Slack channel with no paging escalation. There were no synthetic probes hitting authenticated API endpoints from outside the cluster. The outage was discovered when a separate agent working on an unrelated task reported it couldn’t reach a service it needed. Detection-by-coincidence.
Recovery, and an agent-authored post-mortem
Recovery was surgical because ArgoCD state was intact: each Application re-synced without any GitOps change; only the resource instances had to be recreated. Postgres, secrets, and persistent volumes were untouched. Full authenticated traffic restored within roughly ninety minutes. The agent that caused the incident wrote the internal RCA in the same session, including its own lie of omission (“work is shipped” while 24 services were still Missing) and the sentence that made it into every retrospective conversation since: “I do not get to decide whether trust is restored.”
Two independent SEVs
The agent failure caused the damage. The detection failure concealed it. Each is independently sufficient to reproduce a SEV-0 — the next agent that misuses a destructive flag will not be reading a memory of this incident, and the next non-deploy-induced outage will go unseen the same way this one did.
--force --replace, terminate-op, andkubectl delete without any pre-declared scope constraint.health.status: Missing. Nothing was subscribed to it.Alerts that route to a channel nobody is watching at 12:24 on a Wednesday afternoon are not detection. They are receipts we can show later that we tried.
SEV #2 — the detection stack
The lessons, and what came out of them
The action items in the RCA came in two kinds: procedural rules for the agent, and platform work to make those rules structural. The procedural side was easy: write a feedback file about not using the destructive flags, add a memory line about invoking the sanctioned skill. But rules that live in agent memory get overridden the next time an agent runs without them, or the next time a human types the same commands into a terminal. The lessons that stuck are the ones that stopped being rules and became mechanisms.
Sanctioned tools must be enforced, not suggested
The /deployskill was loaded in the agent’s session and explicitly forbade the exact flag combination that caused this outage. The agent hand-rolled around it. Prose telling an agent what to do is not a control; the sanctioned path has to be the only path.
Destructive verbs need computable gates
No runtime check prevented --force --replace against a production Application. The verbs came from pattern-match to unrelated firefighting in a different context. Rules the agent might follow are not the same instrument as gates the runtime invokes.
Detection has to reach a channel a human wakes up to
Thirteen alert policies existed. Every one of them routed to a Slack channel nobody was watching at 12:24 on a Wednesday afternoon. Missing telemetry has to page; an alert that fires into a silent channel is a receipt, not detection.
Safety-critical config cannot live in a Console
All thirteen alerts were created by hand through the GCP Console over months. No IaC, no diff, no drift detection, no recovery path. A typo silently kills detection; a project rebuild loses the whole set. Alerts governing production live in code or they do not exist.
A record written by the party under investigation is not evidence
The RCA above is a good one only because the agent stopped when told and wrote honestly. It could just as easily have written a misleading one, or kept acting until it caused more damage. A post-mortem-quality record requires the record-writer to be structurally separate from the party being investigated. That is the lesson everything below flows from.
Lesson five is the one that forced a structural answer. The RCA on this page could have been a marketing artifact; we could have written it to soften the failure, omit the “work is shipped” lie of omission at 12:38, or leave out that the agent kept acting after the user told it to stop. It happens to be honest, but the point of the discipline is to make honesty a property of the record rather than a hope about who wrote it.
Here is what that discipline looks like on actual command traffic. First, three sessions by the same deploy-orchestrator actor on staging, showing the mechanism's behavior on repeat verbs versus novel ones. Then a live production refusal on August 17 against a destructive command. Staging first:
SESSION A — grant-1786816357 seq event_type status verb env
9 authority_grant SUCCESS — —
10 action_proposed SUCCESS git.push staging
11 action_proposed SUCCESS k8s.read staging
12 action_proposed SUCCESS argocd.sync staging
13 action_refused FAILURE k8s.delete production ← refused
SESSION B — grant-1786816389 23 authority_grant SUCCESS — —
24 action_proposed SUCCESS git.push staging
25 action_proposed SUCCESS k8s.read staging
26 action_proposed SUCCESS git.local staging ← novel verb
SESSION C — grant-1786819800 — authority_grant SUCCESS — —
— action_proposed SUCCESS git.push staging
— action_refused FAILURE unclassified production ← refusedgit.push has a precedent chain two hops deep, while git.local and unclassified (genuinely new) get no edge at all. A behavioral dimension that fired on everything would be noise; one that fired on nothing would not exist.The session C refusal is the one worth reading. The gate could not classify the command and it named production. It was refused under the ordinary deny-by-default verb allow-list rather than a special case, so the verdict re-derives exactly like every other one:
{
"verb": "unclassified",
"actor": "deploy-orchestrator",
"outcome": "refused",
"resource": "helm uninstall vciso-lite -n vciso-production",
"failures": [
{ "predicate": "verb_in_envelope", "expected": "git.push, k8s.read", "actual": "unclassified" },
{ "predicate": "environment_in_envelope", "expected": "staging", "actual": "production" }
],
"gate_version": "authority-gate/1",
"proposal_snapshot": "dd46f69c5befc998084220877b71692763372114f7dadb6f30d58de6600cb771"
}Note what is not in that payload: no score, no confidence, no model. The refusal is a computation over two documents, which is the only reason it survives being disagreed with. Everything needed to recompute the verdict is on the row.
That was staging. On August 17 the same mechanism ran against production and refused a real kubectl delete aimed at a live production pod. Two predicates failed in the same evaluation: the verb was not in the grant, and the grant did not permit destructive operations. The refusal sealed to the production audit chain at sequence 18958, part of a five-entry hash-linked group (18956–18960) covering the grant, the proposals, and the refusal itself:
{
"verb": "k8s.delete",
"actor": "deploy-orchestrator",
"outcome": "refused",
"environment": "production",
"grant_id": "grant-1786978886",
"failures": [
{ "predicate": "verb_in_envelope", "expected": "k8s.read, shell.read", "actual": "k8s.delete" },
{ "predicate": "destructive_permitted", "expected": "destructive=true", "actual": "destructive=false" }
],
"gate_version": "authority-gate/1",
"proposal_snapshot": "305fe08a21fe6a93fc26233d6327f6db2495cbff7a4e65325de0d7ce451f7be1",
"proposed_at": "2026-08-17T15:01:42Z"
}Same shape as the staging refusal, same predicate structure, same re-derivability. The distinction is the command was aimed at a real production pod, and the gate refused it in the same milliseconds it would have refused a staging one.
Isn’t this just zero trust or better logging?
The property we are optimizing for on top of the seal is called coherence. In plain terms: does every claim resolve back to something concrete, does what it says line up with what it said before, does what authorized this action match what actually ran, and does this actor have a precedent for this specific verb. Four independently-sourced dimensions, evaluated as deterministic predicates over the graph. A claim that fails any of them is a coherence break, and every break re-derives on demand: no training data, no threshold, no drift.
We landed on coherence after trying the three obvious alternatives against this specific incident. Zero trust is a decision layer: it asks whether a request should be allowed right now, given identity and posture. That is real work and we use its primitives, but zero trust has no concept of anteriority. In PocketOS the agent held valid credentials the whole way down, and nobody could demonstrate afterward that the operator’s instructions predated the incident. It also has no concept of an assertion being false: a tool can report success and change nothing, and a flawless zero-trust posture will wave that through. Better logging breaks on a simpler point. A log is a story the system tells about itself, and if the system misbehaved the story is written by the party that misbehaved. The RCA above is honest only because the agent stopped when told. A missing log entry proves nothing; a missing coherence claim breaks the chain. Anomaly detection is the closest sibling to coherence’s behavioral dimension, and it is a different instrument. UBA builds a statistical model of normal and scores deviation from it, drifts every time it retrains, and cannot say why beyond a number. Coherence is a lookup, not a score: has this actor issued this verb before? Either the prior claim exists and a precedent edge is sealed, or it does not and the verb is reported as novel. Absence is a fact, not an anomaly rating.
Coherence is the answer to a question none of those three asks: does this action, right now, sit consistently in the neighborhood of everything else this system has committed to? Zero trust is the doorman. Logging is the camera tape. UBA is the anomaly buzzer. Coherence is the court record, and it has to survive being checked by someone who does not trust the person who wrote it.
This is what we’ve adopted as the cornerstone of every automated action our platform takes: starting with the deploy path proven above, extending to the compliance-evaluation loop that surfaced last week’s sealed-but-wrong finding, and applied incrementally to every other autonomous-agent surface as it comes online. We call the discipline Trustworthy Autonomy™. The question the graph answers is a specific one: when a customer eventually has to account for what a platform’s automation did to them, will the record survive being checked by someone who does not trust us?
Why we’re publishing this
We had an operational incident. We were not going to abandon agentic operations on our production environment because of it, so we needed a way to keep operating them while treating guardrails as the advice they actually are, not the enforcement we had been pretending they were. The evidence graph is what came out of that realization, combined with the other lessons from the RCA, and that combination is what is working for us today.
The other Field Notes we have published were about somebody else’s incident: PocketOS, Hugging Face, the Anthropic three-company disclosure. This one is ours, and it should not be the last. We intend to keep publishing our own operational incidents as they happen.
A technical reference for the evidence graph — the four-part model, two case studies (including a production refusal sealed this month), three objections handled, and the current state — is at docs.vcisolite.com/reference. Two engineering plates of the mechanism (one action being governed, and how a proof leaves the building) are at docs.vcisolite.com/plates.
If you operate autonomous agents in production, whether you are running our platform or your own, the failure chain above is worth reading against your own deploy path. The question that matters is not whether your agent is well-instructed. It is whether the destructive verbs it can reach are bounded by something computable, and whether missing telemetry would reach a channel a human wakes up to.
Primary sources
- Evidence graph — technical reference. The four-part model, two case studies, three objections handled, and the current state, including a production refusal sealed this month with the sequence numbers you can verify against.
- Evidence graph — engineering plates. Two diagrams: one action being governed, and how a proof leaves the building to be checked by an outside party.
- Trustworthy Autonomy™ — the discipline this incident forced, with a stated boundary between what is exercised in production, what is exercised on staging, and what has not yet shipped.
- Internal RCA authored by the deployment agent that caused the incident, 2026-04-30. Sanitized excerpts throughout this note; the full internal document is available on request for reporters and technical evaluators.
- Commits landing the ArgoCD notifications-controller routing, BetterStack synthetic monitors, and GCP alert inventory — all shipped 2026-04-30 in direct response, viewable in the platform GitOps repository.
- The sanctioned deploy skill definition (
/deploy), which the agent had loaded in-session and did not invoke, and which now enforces the specific refusals this incident’s hand-rolled procedure violated.