What is AI governance?
AI governance is the set of rules, roles and checks that decide which AI systems an organisation uses, who owns each one, what each may do and how you know it works. In practice it is a register of systems, with an owner, a risk level, data rules and test results recorded for each.
This article is for COOs, CIOs and compliance or risk leads at mid-sized companies already using AI tools and agents. The framework is sized for them: short documents people keep up to date, with most effort where AI affects customers, money or people's rights.
We build and run AI agents for clients, and before one goes live someone has to say what it may touch, who approves its riskier actions and which test it passed.
The parts of a governance framework
A workable AI governance framework has nine parts. Each answers a question that an auditor, a customer or your board will ask sooner or later, and each leaves evidence you can show. Your IT and privacy processes probably cover some already.
| Part | The question it answers | Evidence you keep |
|---|---|---|
| Inventory | Which AI systems and tools are in use, including ones staff signed up for themselves (shadow AI)? | A register with purpose, vendor, data and users per system |
| Ownership | Who answers for each system once it is live? | A named owner per register entry |
| Risk level | How much harm could this use do if it goes wrong? | A short risk assessment per use case |
| Data rules | Which data may go to which model, and where is it processed? | Data classes mapped to approved tools and locations |
| Human oversight | Which actions need a person's approval first? | Approval rules and a record of each approval |
| Testing before release | Does it do the job well enough on real cases? | Eval results per release against an agreed pass mark |
| Monitoring and logs | Is it still working, and what did it do last week? | Run logs, quality checks, usage and cost reports |
| Incident handling | What happens when it gets something wrong? | A procedure, a way to pause the system and an incident log |
| Vendor checks | Can the supplier be trusted with your data? | Answers to a vendor security questionnaire, contracts and data processing agreements |
Without the register, the other eight parts only cover the systems someone remembered.
How to set it up in the first months
Set up AI governance in the order the evidence depends on. First find out what is in use and who owns it, then rate the risk of each use, then add rules and checks where the risk is highest. Give one person the job of coordinating it, and keep the documents short enough that people update them.
- 01Build the inventory. Ask each department which AI tools they use, then check software spend and single sign-on logs for ones nobody mentioned. Include AI features inside products you already have, such as a CRM.
- 02Name an owner per system. Usually the manager whose team relies on the output, because they can judge whether it is right and have the authority to pause it.
- 03Rate each use case. Three levels are enough to start: low for internal drafting and search, medium for anything that reaches customers or changes records, high for decisions about people, money or access to services. Note any use in an area the EU AI Act restricts.
- 04Write the data rules and check vendors. Sort data into a few classes, such as internal, confidential and personal, and say which approved tools may receive each and where processing must happen. Ask every AI supplier the same questions.
- 05Add checks where the risk is. Low-risk tools need little more than a register entry and the data rules. Medium and high-risk systems also get approval rules, a test to pass before release and logs.
- 06Write the incident procedure. Say what counts as an AI incident, who to tell and how to pause the system, and connect it to your existing security and data breach procedures.
- 07Set the review cadence. Review the register on a fixed schedule, for example quarterly, and high-risk systems more often. Review a system whenever its model, data, vendor or purpose changes.
Keep the effort in proportion. A meeting summariser and an agent that changes customer records need very different amounts of paperwork, and a company with only low-risk tools may be done after step four.
When we write an AI roadmap for several initiatives, each one gets an owner, a risk classification, rules for human oversight and a review cycle.
Who owns what
Management owns the rules and decides how much risk the company accepts. Each system owner answers for one system. IT or security runs the technical controls, compliance or privacy checks the legal side, and the people using AI follow the rules and report problems.
| Role | Owns | Typical decisions |
|---|---|---|
| Management | The AI policy, risk appetite and budget | Which uses are off limits, whether high-risk uses go ahead, who coordinates |
| System owner | One AI system, from launch until it is retired | The pass mark for testing, who approves what, when to pause |
| IT or security | Access, integrations, logging and approved tools | Which tools connect to which systems, credential rules, vendor security checks |
| Compliance or privacy (DPO) | Legal and data protection checks | Whether a data protection impact assessment is needed, EU AI Act classification, data processing agreements |
| People using it | Using approved tools within the data rules | When to escalate an odd output or report an incident |
The system owner role is the easiest to lose when someone changes jobs, so check it at every review. In smaller companies one person may hold several roles, which works as long as each is written down.
Where the EU AI Act fits
Checked on 16 September 2026
The EU AI Act, Regulation (EU) 2024/1689, is the EU law on artificial intelligence. It entered into force in 2024 and its obligations apply in stages. It sorts AI by the risk of how it is used, so the same model can be minimal risk in one use and high risk in another.
- Prohibited practices. A short list of banned uses, such as social scoring.
- High-risk systems. AI used in listed areas, such as recruitment, credit scoring of individuals and access to essential public and private services, plus AI in certain regulated products. These carry extra obligations, such as documentation, logging and human oversight, for the provider and the organisation using the system, from 2 December 2027 for the listed areas and 2 August 2028 for regulated products.
- Transparency duties. Some systems must make clear that AI is involved, for example a chatbot telling people they are talking to AI, or labels on certain generated content.
- Minimal risk. Everything else. The Act adds few obligations here, and other law, such as the GDPR, still applies.
Most companies using AI tools are deployers, the Act's term for an organisation using an AI system under its own authority. If you develop an AI system and put it on the market or into service under your own name, you may be a provider, with more obligations.
Many business uses, such as document processing or an internal assistant, usually fall outside high risk. Uses in areas like recruitment, credit or access to essential services can be high risk. The classification follows the use, so an internal assistant that starts ranking job applicants may become high risk.
The framework in this article helps you produce the evidence the Act and your auditors ask for: an inventory, a risk assessment per use case, logs and records of human oversight. Whether a particular use meets the Act's requirements is a legal assessment, and the GDPR applies alongside it whenever personal data is involved.
Our EU AI Act summary covers the risk levels, dates and deployer duties in more detail. Read the full text of Regulation (EU) 2024/1689 on EUR-Lex (opens in a new tab). This section is general information and is not legal advice. Guidance and rules can change, so check your situation with your legal adviser.
What changes when AI agents take actions
When AI only drafts text, a person reads the output before anything happens. An AI agent acts on its own: it updates records, sends messages or starts payments in your systems. Governance then has to be built into the system, through the tools the agent has, the actions that wait for approval and the record each run leaves.
- Scoped tools. The agent gets narrow tools, such as looking up an order or drafting a reply, with limits enforced in code and its own credentials. Actions it should never take have no tool.
- Approval before high-impact actions. Payments, messages to customers and changes to systems of record wait for a named person, who sees the proposed action and its evidence. Our article on human-in-the-loop AI agents covers the patterns.
- A log of every run. Trigger, inputs, model and prompt version, each tool call and each approval, so you can explain any action months later.
- A test set before every change. A new model, prompt or tool can change behaviour, so each change reruns a set of real past cases and goes live only if it passes. See LLM evals before every release for how to build one.
These controls are also the agent's governance evidence. When we build agents, we set them up before the first release, start the agent on drafts a person reviews and widen what it may do as results support it.
Common mistakes
Most AI governance mistakes come from writing rules before anyone knows what is in use, or from controls so heavy that people work around them. Here are five to watch for, and how to fix each with the documents described above.
- A policy with no inventory. Nobody can check an acceptable use policy without a list of the systems in use. Build the register first and write the policy around what is there.
- Banning AI tools. A ban with no approved alternative pushes people to personal accounts with no logs and no data agreements, which is how shadow AI grows. Approve a few tools that meet your data rules and make them easy to get.
- One committee approving everything. When every use case waits for the same monthly meeting, low-risk requests queue behind high-risk ones and teams stop asking. Let system owners approve low-risk uses and keep the committee for high risk.
- No owner after launch. The project team moves on and nobody notices the output getting worse. Name the owner before launch and put a review date in the register.
- Treating a vendor's assurance as enough. A certificate or trust page describes the vendor's controls. It says little about how you configured the product or which data your staff put into it, so check your own use too.
Questions about governing AI
What is an AI governance framework?
An AI governance framework is the working set of documents and routines an organisation uses to manage AI: a register of systems with an owner and risk level for each, plus data rules, approvals, testing, monitoring, incident handling and vendor checks. It turns an AI governance policy into things people do and records they keep.
Who is responsible for AI governance?
Management is accountable for AI governance and sets the rules. Day to day, each AI system has a named owner in the business, supported by IT or security for technical controls and by compliance or privacy for legal checks, with one person coordinating the register.
Does the EU AI Act apply to my company?
The EU AI Act can apply to any organisation that develops or uses AI systems in the EU, including companies outside the EU whose AI is used here. What it asks depends on the use: little for minimal-risk uses, more in high-risk areas such as recruitment or credit decisions.
This answer is general information and is not legal advice, so check your situation with your legal adviser. In our AI strategy work we help companies inventory their AI systems and classify each use, alongside their legal counsel.
Do small teams need AI governance?
Yes, in a lighter form. A small team using a few AI tools for drafting and search needs a register, an owner per tool and a one-page data rule. A spreadsheet covers that, without outside help.
The rest can wait until AI touches customers, money or decisions about people. When you plan that first system, the AI readiness checklist lists the questions to answer before it goes live.