AI Academy

Every few months, I run a simple mental exercise: What happens when the AI we govern becomes capable of governing itself?

We are entering a world where Large Language Models and Agentic AI systems can draft policies, implement workflows, generate compliance documentation, audit their own behaviour, and even rewrite the rules when the environment changes.

This raises a question no regulator, no bank, no enterprise has fully answered:

How do you govern an AI that can write its own governance?

Let’s break it down.

1️⃣ Governance Today: Humans Write the Rules. AI Follows.

Traditional governance models assume:

• Humans define the rules
• AI systems operate within them
• Risk teams verify compliance
• Regulators audit after the fact

This structure depends on one assumption: Human oversight is always upstream of AI behaviour.

But agentic system, capable of reflection, reasoning, planning, and self-improvemenT, challenge this assumption entirely.

2️⃣ Governance Tomorrow: AI Writes the Rules. Humans Validate.

Consider what AI systems can already do:

• Draft policy frameworks
• Suggest guardrails
• Create controls
• Run risk assessments
• Simulate policy outcomes
• Generate compliance reports
• Monitor themselves for drift or violations

This leads to a provocative future scenario:

AI generates a governance policy faster, more accurately, and more contextually than a human.

If that happens:

• Does the AI become the policy author?
• Do humans become approvers instead of designers?
• What does “governance” look like in a closed loop where the system manages its own rules?

3️⃣ The Real Risk: Recursive Governance Loops

The moment AI can rewrite guardrails dynamically, you get a recursive loop:

1. AI identifies a constraint
2. AI rewrites the rule
3. The new rule modifies the AI
4. The AI behaves differently
5. AI then re-evaluates the rule again

This loop is fast. Faster than any governance committee or board can react.

How do you govern a loop you cannot keep up with?

4️⃣ The Answer: You Don’t Control the AI. You Control the “Meta-Rules.”

We must shift from governing outputs to governing the rule-generation process itself.

This is Meta-Governance, the governance of the governance engine.

The core principle becomes:

“AI can propose rules, but only humans can authorize rule changes.”

This is how you keep the human in the loop at the policy layer, not at the task layer.

5️⃣ The Meta-Governance Framework (A Starting Point)

A future-ready governance model might look like this:

1. Policy Sandbox for AI

AI systems can generate draft policies but cannot apply them until validated.

2. Human Approval Layer

A policy is only “live” after:

• Human review
• Risk scoring
• Legal assessment
• Ethical validation

3. Immutable Control Ledger

Policy changes logged on:

• Blockchain
• Verifiable Registry
• Cryptographic audit trail

No silent modifications. Ever.

4. Agent Permissions

AI cannot modify:

• Core principles
• Regulatory obligations
• Constitutional constraints
• Safety rules

This prevents “self-optimisation loops” that degrade safeguards.

5. Kill-Switch Logic

If the AI attempts unauthorized meta-rule changes, a global control automatically deactivates the policy engine.

6️⃣ The Paradox: AI May Become Better at Governance Than Humans

Here’s the uncomfortable truth:

AI may eventually become more consistent, less biased, and more rigorous in applying governance controls than humans.

When that happens, do we fight it? Or do we redefine the governance hierarchy?

This is the philosophical challenge every CAIO must prepare for.

7️⃣ My View: Humans Should Never Be Out of the Loop, But the Loop Will Evolve

Governance does not disappear. It moves upstream.

We don’t govern tasks. We govern rule-generation. We govern the intent of the system. We govern the boundaries of autonomy.

The future isn’t “AI governing itself.” The future is “AI proposing governance, humans deciding.”

That is the balance between innovation and safety.