A consistent pattern emerged: organizations can increasingly prove what AI systems did, but cannot reliably prove who owned decisions at runtime, or whether meaningful human judgment was exercised.
In practice, “human-in-the-loop” often degrades into habitual approval. Review becomes ceremonial, and AI systems transition into de facto automation without explicit intent or governance.
This failure rarely originates in model error. It arises from gradual behavioral drift and organizational dynamics that existing AI governance frameworks are not designed to observe or manage.
The result is an audit and accountability gap: when incidents occur, decision rationale cannot be reconstructed without re-running systems or relying on interviews.
We wrote a short research paper documenting these failure modes and the structural gap between governing AI systems and governing AI usage.
Genuinely interested in critique from people who have seen similar dynamics in production systems, audits, or post-incident reviews.