AI Agent Governance: Emerging Challenges and Frameworks
The Era of AI Agent Governance
We are entering a new and consequential phase in the evolution of artificial intelligence, one that demands we raise our standards of governance significantly. If the challenge of the last few months has been governing AI tools, the challenge of the next few months will be governing AI agents. The organisations that fail to appreciate it risk being caught badly off guard.
The Black Box Problem, Amplified
AI products have long presented governance challenges, not least because they operate, to a significant degree, as black box systems. Decisions and outputs are generated in ways that are difficult, sometimes impossible, to interrogate with confidence. That challenge is now being compounded by a new development: the growing ability of lawyers, IT professionals, and knowledge professionals to build their own agents, automated systems capable of taking actions, making decisions, and producing outputs, often with limited human oversight.
This is, in one sense, progress. The democratisation of agent development unlocks genuine efficiencies and opens up new possibilities for professional services. But it also demands that we pause and ask some uncomfortable questions. Who should be permitted to build an agent? What data should that agent be allowed to access? Should there be a meaningful distinction between agents producing deterministic outputs and those operating under more fluid, supervised conditions? What level of control and oversight is appropriate for which category of agent?
These are not abstract questions. In the not-so-distant past, a professional support lawyer or knowledge professional would take responsibility for ensuring that a precedent or standard document was fit for purpose. There was a clear line of accountability. In the world of AI agents, that line has become blurred. Should a trainee solicitor be able to develop an agent that reviews or generates legal work? Should a non-legally qualified technologist? The answer may well vary depending on the task, but the point is that firms and in-house teams need to have a clear answer, and at present, many do not.
The Data Challenge
Alongside the governance of agent development sits a growing data challenge. Data hygiene has always mattered in the context of AI; in the context of AI agents, it is becoming acute. As agents increasingly interact with multiple data sources, the questions multiply. Is the data reliable? Is it appropriate for the agent to access it? Is it sensitive? Are we respecting applicable data protection, confidentiality, and ethical obligations? How will MCP calls be controlled? The list is long, and the consequences of getting it wrong are serious.
The Ongoing Governance Problem
There is a further dimension that deserves close attention: the ongoing governance of AI systems once they are in place. Many legal technology platforms are not building their own AI from the ground up, they are procuring it from large AI technology companies an applying an interpretive context layer and interface. This creates a governance challenge on two fronts. First, law firms and in-house legal teams will frequently have no visibility on what their supplier is doing in terms of updating or altering the underlying AI model or context layer. Second, the suppliers themselves may have limited visibility on what the major AI technology companies are doing at the foundational LLM level. Firms and in-house teams will be on the receiving end of changes they did not anticipate and may not immediately detect as any changes can alter the way agents work.
Managing this requires deliberate effort. One important mechanism is the use of controlled data sets, run at periodic intervals, to test whether AI outputs remain consistent and reliable over time. This is a sensible practice for AI generally, but it is especially important for agents, where the stakes of an undetected change in behavior can be high. Different categories of agents will also require different controls and testing regimes, calibrated to their function and the significance of their outputs.
The Consumption Question
One further issue needs further thought: cost and consumption. Not every headline about AI economics is reliable, but there is a general trend towards consumption-based pricing in the more complex AI models. Every agent that is developed needs to account for this. Firms and in-house teams should ensure that appropriate cost control mechanisms are built into agent development from the outset, and that the rate of consumption and associated costs are kept under constant review. Also, what approval and limits need to be built into the system? Can it operate on a fixed basis with additional capability accessible by use of a pre-approved “power” button with cost allocation?
Conclusion
The developments we are seeing in the field of AI agents are genuinely impressive, and it would be both futile and short-sighted to attempt to resist them. These technologies will continue to advance, and the efficiency benefits they bring will be substantial. However, we cannot afford to rush in carelessly.
What is needed now is clear governance: structured, agile, and kept under regular review. That means establishing firm-wide policies that set out clearly who can develop agents, under what conditions, and subject to what controls. It means identifying agent owners, implementing application and approval processes, and maintaining agent logs. These are disciplines that software development has long understood, but the complexity and stakes in the context of AI agents make them more important than ever and other groups of people than IT professionals will be undertaking this work.
The core message is straightforward: every law firm and in-house team needs to get ahead of this now. The worst outcome is to allow a proliferation of ungoverned agents that creates a problem that is difficult and costly to unwind. AI governance committees need to think this through carefully and ensure that the right foundations are in place, not just for managing risk today, but for enabling responsible innovation tomorrow.
Hyperscale Group are a Technology, Digital and Operational Advisory and Implementation Business with over 25 years plus deep market experience. We work for In-House teams and Professional Services Firms all around the world and support them developing and implementing their strategies. We help make the right things happen.
Our Experience: Click here - Our Services: Click here - Client Testimonials: Click here
How We Work: Click here
For more information on how we are supporting clients in this area, please contact dereksouthall@hyperscalegroup.com
