Governing an Agent That Acts

Mechanism one — stripping the toolset

The bluntest and most reliable mechanism is not showing the agent the capability at all.

At safe autonomy, tools classified as risky are removed from the toolset before the model is called. The same pattern is used for every opt-in feature in the product: when outcome verification is off, the outcome tools are not offered; when task capture is off, the triage tools are not offered. A workspace that never enabled a feature sees an agent that is byte-identical to the one that existed before the feature shipped.

This matters because it is the only mechanism that does not depend on the model's cooperation. Every other gate in this lesson is a check that runs when the agent reaches for something. Stripping means there is nothing to reach for. When you are deciding how to constrain an agent, prefer taking the capability away over telling it no — instructions are negotiable in a way that an absent tool is not.

The corollary is a real cost: a stripped agent cannot tell you it wanted the tool. If your agent seems oddly unhelpful in one domain, check whether a scope or a level has quietly removed the capability rather than assuming the model is being dim.

Mechanism two — the risk taxonomy

Governance needs a definition of risky, and the definition has to be maintained rather than inferred.

Kavanah classifies a tool as risky if it sends something externally, moves money, deploys, changes membership, writes to a connected third-party tool, or spends AI usage at scale. Some families are caught by pattern — anything that sends, anything that deletes. Everything else is caught by an explicit list, and the list is the part that rots.

The design rule worth internalising: a tool absent from the risky list is not stripped in safe mode and not parked for approval in full mode. Silence is permission. This is why the product treats adding a write capability as a two-registration change rather than one, and why the invariant is enforced by a test rather than by intention. Governance systems fail at the seams — the new thing nobody remembered to classify — far more often than they fail at the mechanism.

As a manager you do not maintain that list. But you should know it exists, because it explains the otherwise-mysterious case where the agent did something risky-feeling without asking: the action was not in a category, so no policy applied to it.

Mechanism three — graduated trust via action policies

The naive version of full autonomy parks every risky action for approval. It is safe and it does not work, for a reason worth stating plainly: approval load scales linearly with usefulness. The more the agent does, the more you approve, and the ceiling on the agent's value becomes your attention span. Teams respond by rubber-stamping, which is worse than no gate at all — it produces the paperwork of oversight with none of the substance.

Action policies replace that with graduated trust. Each risky tool maps to a category — send messages and email, move money, delete and disconnect, membership and access, deploy and run external workflows, write to connected tools, spend AI usage — and you set one policy per category rather than one decision per action. The four policies are allow, allow up to a daily budget, park for approval, and block.

The budget option is the interesting one and the most under-used. It fits any category where the risk is cumulative rather than per-action: one AI-spend action is fine, forty in an afternoon is a bill. A budget lets the agent work without asking while capping the blast radius by volume, which is the right shape for most of the categories most of the time.

No policy row means park. This is deliberate: a category you have never thought about behaves exactly like the old approve-everything default, so shipping a new category cannot silently widen anyone's exposure.

Mechanism four — approvals, and why the queue must stay small

A parked action waits for a human. That is the easy part. The hard part is that an approvals queue is a decision surface, and decision surfaces sit on top of a failure mode that human-factors research has spent forty years documenting.

The finding is sharper than the folk version, and the precision matters. Automation bias — accepting an automated recommendation without adequate verification — produces both omission errors (missing something because the automation did not flag it) and commission errors (doing something wrong because the automation said so). It occurs in expert operators, not just novices. And, in the authors' words, it cannot be prevented by training or instructions (Parasuraman and Manzey 2010). Related but distinct, automation complacency — under-monitoring a system you have come to trust — shows up specifically under multiple-task load, when the thing you are supposed to be watching is competing for attention with the work you are actually doing. It is not a claim that people always under-monitor; it is a claim about what happens when monitoring is one job among several. Which is, of course, exactly the manager's situation.

Read those two findings together and the design conclusion is forced: you cannot fix an approvals queue by asking people to be more careful. Telling reviewers to read properly is the intervention the evidence specifically says does not work. The mitigation has to be structural.

So the structural moves, all of which are just action policies used well: keep the queue small enough that each item still gets read; move anything you would rubber-stamp to allow or allow-with-a-budget, so the queue contains only genuine decisions; block outright anything you would always reject, so it never becomes a queue item. A short queue that gets read is worth more than a comprehensive one that gets skimmed.

The canonical thing a skim misses is the item that is almost right, but not quite. In the 2025 Stack Overflow survey that was developers' single largest frustration with AI output, named by about two-thirds of respondents (Stack Overflow 2025). An obviously-wrong action is caught by anyone glancing. An almost-right one is caught only by someone reading — which is a budget, not a virtue.

This is not just good practice; where it is regulated, it is close to the letter of the law. The EU AI Act's human-oversight article requires that people overseeing a high-risk system remain aware of the tendency to over-rely on its output — it names automation bias explicitly — and that they be able to disregard, override, or reverse it, and to stop the system in a safe state. Kavanah is not a high-risk system under that regulation, but Article 14 is a good checklist regardless: can your people see it, override it, and stop it?

Mechanism five — the ledger, and why it is not optional

Every action the agent takes is recorded: what it did, which tool, under whose authority, whether a policy allowed it or a human approved it. The ledger keeps recording whether or not anyone has turned on the UI that displays it — dropping rows because nobody is currently looking would defeat the point of an audit trail, which is that you consult it after something has gone wrong rather than before.

The ledger answers the question governance exists to answer: what did it do, and who said it could. That question is asked in exactly two situations, and both are bad ones to be unprepared for — when something went wrong, and when someone external asks. Neither is a moment to discover you were not recording.

There is a useful philosophical frame for why recording is not merely paperwork. One well-developed account of meaningful human control over autonomous systems sets two conditions: tracking, meaning the system is demonstrably responsive to the human reasons that ought to govern it, and tracing, meaning its actions can be traced back to a human who understood what they were authorizing (Santoni de Sio and van den Hoven 2018). Your charter and your action policies are the tracking condition — they are where the reasons live. The ledger is the tracing condition. An agentic workspace with policies but no ledger has opinions about what should happen and no way to establish what did; one with a ledger but no policies has a perfect record of decisions nobody made deliberately. You need both, and neither substitutes for the other.

Alongside it sits the Negation ledger from lesson 5.4, which records the work the charter refused. The two are complements: the action ledger is what the agent did, the Negation ledger is what it was stopped from doing. A governance review reads both, because an agent with an empty refusal ledger is either perfectly aligned or completely unconstrained, and the action ledger alone cannot tell you which.

One honest caveat about all five mechanisms. The KVN gate — the one that checks work against your Negation — deliberately fails OPEN: if the model call behind it errors or times out, the write proceeds. That is the opposite of every other gate here, and it is a considered trade: the gate sits in front of ordinary task creation, and hard-failing would take the product down with an upstream provider. It is a guard rail, not a security boundary. Knowing which of your gates is which is part of the job.