Human-in-the-loop AI agents: design patterns for production

An AI agent is a system that uses a language model to decide which actions to take, calls tools to take them and works towards a goal over several steps. That makes agents useful for operations work such as handling shipment exceptions, triaging invoices or preparing customer replies. It also means the system acts, and actions have consequences that generated text alone does not.

This post is for operations leaders and engineers designing agents that touch real systems. Human-in-the-loop design means deciding in advance, and in code, which decisions an agent makes alone, which it proposes for a person to approve and which it never makes. Done well, it lets you automate most of the volume while people handle the cases that need judgement.

An approval checkpoint pauses the agent before a consequential action, shows a person what it intends to do and why, and continues only once approved. The key design choices:

• Where to pause. Put checkpoints before actions that are hard to reverse or that leave the organisation: payments, refunds, emails to customers and changes to systems of record.
• What to show. The proposed action, the inputs it is based on, the reasoning in a sentence or two and the source for any claim, such as the policy clause or order record.
• How to respond. Approve, edit then approve, reject with a reason, or take over. Rejections with reasons are valuable eval data.
• What happens while waiting. The agent's state is persisted so it can resume hours later, and approvals expire if they wait too long.

The approval screen decides whether the checkpoint works. If a reviewer needs five minutes to reconstruct the context, they will start approving without reading. With the reason and evidence on screen, most approvals take seconds.

A checkpoint on every action defeats the point of automation. Routing by risk and confidence keeps people on the cases that need them. We classify each tool by risk, then decide per action whether it runs, waits for approval or is refused:

from dataclasses import dataclass
from enum import Enum

class Risk(Enum):
    READ = "read"
    REVERSIBLE = "reversible"
    IRREVERSIBLE = "irreversible"

TOOL_RISK = {"lookup_order": Risk.READ, "draft_reply": Risk.REVERSIBLE, "issue_refund": Risk.IRREVERSIBLE}
AUTO_THRESHOLD = 0.85

@dataclass
class ProposedAction:
    tool: str
    args: dict
    confidence: float  # built from checkable signals, see below
    reason: str

def route(action: ProposedAction) -> str:
    risk = TOOL_RISK.get(action.tool)
    if risk is None:
        return "reject"  # unknown tools never run
    if risk is Risk.READ:
        return "execute"
    if risk is Risk.REVERSIBLE and action.confidence >= AUTO_THRESHOLD:
        return "execute"
    return "queue_for_approval"

A caution about confidence: a model's self-reported confidence is poorly calibrated, so do not use it on its own. Build the score from signals you can check, such as whether extracted fields validated, whether the retrieved policy clearly covers the case, whether two independent passes agree and how similar cases performed in your eval set. Then tune the threshold against real outcomes, starting conservatively and lowering it as the data supports.

An agent can only do what its tools allow, which makes tool design the most effective safety control you have.

Rules we apply to every agent:

• Give the agent narrow tools, such as issue_refund(order_id, amount), and avoid general ones like run_sql.
• Enforce limits in the tool code: maximum amounts, allowed recipients and rate limits. The prompt guides the agent; the tool enforces the rules.
• Run tools with credentials scoped to the agent, never with an administrator's.
• Treat text the agent reads from emails, documents and web pages as untrusted, since it may contain instructions aimed at the agent.

When an agent makes a decision, someone will eventually ask why. An audit trail should let you answer that for any action, months later. For each run, record:

• The trigger and the input data, or a reference to it.
• Each model call with its prompt version, model id and output.
• Each tool call with its arguments, result and timestamp.
• The routing decision, the confidence signals behind it and the threshold in force at the time.
• Who approved, edited or rejected the action, and their stated reason.

Store this in a structured, queryable form. Beyond compliance, the trail is how you debug failures and how you find new cases for your eval set.

Agents will meet inputs they cannot handle and systems that are down. Design the failure paths explicitly:

1. 01Escalate with context. When the agent cannot decide, it hands the case to a person with what it has found so far, instead of retrying indefinitely.
2. 02Cap the loop. Limit steps, tool calls and spend per run, and escalate when a limit is reached.
3. 03Degrade gracefully. If a dependency is unavailable, queue the work or fall back to the manual process, and alert the team.
4. 04Provide a stop switch. Operations staff should be able to pause an agent, or a single tool, without a deployment.

• Every tool is classified by risk, and out-of-scope actions have no tool.
• Limits are enforced in tool code, with scoped credentials.
• Irreversible and external actions require approval, with evidence on the approval screen.
• Confidence is built from checkable signals and tuned against outcomes.
• Every run leaves a complete, queryable audit trail.
• Loops are capped, failures escalate with context and there is a stop switch.

We design agents this way from the first version, then widen automation as the audit trail and evals show it is safe. Our invoice approval agent and freight exceptions case show these patterns in practice, and how we work covers how we take agents into production.