The category called “AI governance” contains two very different products. You cannot tell which one a vendor built by reading their homepage; the language collapses. You can tell by asking one question in a live demo: what does your platform do when nothing is attacking my AI system, but the outputs are quietly wrong?
Half the vendors in this category will show you a screen. The other half will pivot back to threat models, red-team reports, and prompt-injection detection coverage. Both halves are selling something valuable. They are not selling the same thing. The buyer who does not force this distinction ends up with a platform that solves the wrong half of the exposure.
Two frames, two lineages, two platforms
Every AI governance product is built around one of two organizing frames. The frames come from different engineering traditions. They produce different feature lists, different alerts, and different definitions of “working correctly.”
The two frames are not opposed. A serious operating program needs both. The problem is that most vendors pick one frame, build against it, and market as if they cover the other. The buyer who reads “AI governance” on two homepages and assumes the two products overlap is buying blind.
Why the security frame produces the platforms it does
Vendors from the AppSec tradition see AI as a new attack surface. The instinct is correct as far as it goes. Prompt injection, model theft, training-data poisoning, jailbreak workflows, tool-call credential exfiltration. These are real attack categories, and the SDLC tradition knows how to scan for them, block them, alert on them, and write red-team reports about them.
The platforms that emerge from this frame look like other SDLC tools. They plug into the CI pipeline. They gate deployments. They maintain a threat catalog and a dashboard of blocked attempts. The mental model is the one the CISO’s org already runs for application code: scan, block, alert, remediate.
This is a coherent product. It maps to a coherent buyer, usually the security engineering team. It solves a coherent problem, which is a threat actor is trying to manipulate our AI system.
It does not solve the other problem.
Where the security frame silently misses
Three failure patterns pass every security check and cost customers real money.
The action is authorized, the tool call is on the allowlist, the prompt is clean, no attack classifier fires. A migration agent runs against production instead of staging because a tool description was ambiguous or a resource ID was templated wrong. No security control was violated. No integrity control was satisfied either, because none was defined. The output is a nine-second deletion and a customer conference call. The security-frame platform records the incident as “no attack detected” and closes the ticket.
A finance agent classifies 400 invoices per week. It does the work well enough for six months. A silent model version bump changes the labeling behavior on a specific vendor class. The mistake compounds into the quarterly close and gets discovered in review. The security-frame platform has nothing to say about this because nobody attacked anything and the classifier fired on nothing; the model simply drifted. The integrity-frame platform would have caught it in week one because the measured classification rate on the invoice ground-truth sample no longer matched the certified baseline for that category, and the category would have been auto-paused with a human notification.
A regulator asks “show me every action your compliance agent took last quarter, the policy version that governed each action, and the verifier result for each.” The security-frame platform can show every blocked attack. It cannot show any of the above, because the primitive was never built. The customer discovers, in the middle of the exam, that they have a scan report and no operating record. What follows is a $200K engineering-week to reconstruct the record from raw logs, and a finding either way.
None of these are attacks. All of them are integrity failures. A platform that watches only for attacks will not see them coming and will not see them going.
What “do no harm” looks like as a product
The integrity frame has different primitives. It does not throw out the security frame; it operates on top of it. Four primitives an integrity-first platform is built on:
- Operating scope, stated in policy-as-code, per category of action: Each category of autonomous action has a written policy expressed in a machine-checkable language (Rego, Cedar, JSON Schema, or a purpose-built DSL). The policy names which actions are in scope, which are out of scope, which require escalation, and what the input/output invariants are. Every action is bound to the policy version that governed it at the moment of execution — not the version that happens to be current when the auditor asks.
- A verifier that fires before the action commits: The action produces its intended output, a deterministic verifier tool checks the output against the policy invariants, and the action commits only when the verifier passes. Verifier failure produces a structured refusal, not a retry loop. Fail-closed on ambiguity is the default; fail-open requires an explicit policy exception with a named owner.
- A hash-linked action chain, per material decision: Every material action lands one row in an append-only chain: inputs (hashed), policy version and hash, verifier tool version and result, model version, human-in-the-loop identity if any, output effect. The chain is queryable by an auditor without the engineering team building a one-off pull. Chain integrity is verifiable independently of the platform via published signature (Ed25519 in our own case; some vendors use RFC 3161 timestamping).
- A drift signal that is loud when it should be loud: The measured miss-rate on a certified category is a live metric with an SLO. When measured miss-rate crosses the threshold set at certification time (typically 1%), the category is auto-paused, the certification is marked stale, and the human on-call is paged. The system does not silently keep operating on the assumption that yesterday's certification still holds. Re-certification is a defined procedure, not an ad-hoc email.
A platform that ships those four primitives is doing safety engineering for the AI layer. It borrows scanning and red-teaming from the security tradition when the moment calls for them. But its organizing question is not is anyone attacking this? It is is this system doing what it is supposed to do, and would we know if it were not?
Force the vendor to answer the diagnostic question at the top of this piece with a live demo, not a slide. If the demo shows attacks blocked, ask them to walk one integrity failure caught in the absence of any attack, with the specific policy invariant that was violated, the verifier result that fired, and the row in the audit trail that recorded it. If they cannot, they built a security platform and marketed it as governance. That is a fine platform to buy. It is not the platform that covers the other half of the exposure, and you should know which one you are buying before the PO goes out.
Why we take the “do no harm” stance
Security failures make headlines. Integrity failures make settlements, audit findings, and re-issued financial statements. The customer we serve most often can absorb a red-team report but cannot absorb a quiet six-month drift into a wrong answer that eventually reaches a regulator.
Our operating conviction is that the second class of failure is the one an AI governance platform must be built to catch first. The attack posture is a second line item on that platform, added to a primary loop that watches whether the system is doing what it is supposed to do. That primary loop is the one we build. That is why our platform reads more like a safety-engineered flight deck than a red-team dashboard, and that is deliberate.