Guardrails, permissions, and human-in-the-loop

# Guardrails, permissions, and human-in-the-loop

In 2024, a car dealership chatbot was talked into agreeing to sell a truck for one dollar. It was "legally binding," the customer joked, screenshotting the reply. The bot had no brakes. It could say anything, and (in a more dangerous version) it could have *done* anything.

Now imagine that same agent connected to your email, your calendar, and your company credit card. An agent that can act in the real world needs limits it cannot cross. This lesson is about those limits: what to allow, when to ask a human, and how to cap the damage.

Why agents need brakes

A chatbot only talks. An agent (an AI that can take actions using tools, like sending an email or calling an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →) actually does things. That is the whole point, and also the whole risk.

Three failure modes show up constantly:

• Wrong action. The agent refunds $5,000 instead of $50.
• Right action, wrong target. It emails the draft to the entire company instead of one person.
• Runaway loops. It retries a failing task 400 times, each retry costing money.

Guardrails are how you keep the usefulness while removing the catastrophe. There are three main tools: permissions (what the agent *may* do), approvals (when a human must say yes first), and limits (hard ceilings it cannot exceed).

Permissions: the tool allow-list

The first guardrail is deciding which tools the agent can even touch. This is a tool allow-list: an explicit list of the actions available to it. Anything not on the list simply does not exist for the agent.

Think of it like giving a new intern a keycard. You do not hand them access to every room in the building. You give them the three doors they need.

A support agent might get:

• search_knowledge_base (read-only, safe)
• create_draft_reply (produces text, sends nothing)
• escalate_to_human (hands off)

Notice what is *not* there: no send_email, no issue_refund, no delete_account. The agent can prepare work, but a person or a stricter system handles anything irreversible.

A useful habit: sort every tool into read (safe, look things up), write (changes something, riskier), and irreversible (spends money, sends messages, deletes data). Read tools can usually run freely. Write and irreversible tools deserve extra scrutiny.

Scope the tool, not just the name

Giving an agent a send_email tool is not enough control. *Scope* it. For example, restrict the tool so it can only email addresses inside your own company domain, or only reply to threads the user started. The tool itself enforces the rule, so the agent cannot break it no matter what it "decides."

Human-in-the-loop: the approval step

For risky actions, the safest brake is a human. Human-in-the-loop (often shortened to HITL) means the agent pauses before a sensitive action and waits for a person to approve or reject it.

The pattern is simple:

1. Agent decides it wants to take an action (send this refund).

2. Instead of doing it, the agent *proposes* it and pauses.

3. A human sees the proposal and clicks approve or reject.

4. Only after approval does the action run.

You do not need approval on everything. That would make the agent as slow as doing the work yourself. Approve only the actions that are irreversible or expensive. Let the safe stuff flow.

Here is a vendor-neutral sketch of what an approval gate looks like in a tool-call loop. The logic is the same whether you use the OpenAI Agents SDK, the Claude Agent SDK, or Google's ADK:

# Tools that always require a human to approve before running
NEEDS_APPROVAL = {"send_email", "issue_refund", "delete_record"}

def run_tool(name, args):
    if name in NEEDS_APPROVAL:
        # Pause and ask a person
        if not human_approves(name, args):
            return {"status": "rejected", "reason": "Human declined."}
    return execute(name, args)

# The agent loop
while not done:
    action = agent.next_action()          # model proposes a tool call
    result = run_tool(action.name, action.args)
    agent.observe(result)                 # feed the outcome back in

The key idea: the model *proposes*, your code *decides* whether to run it. The agent never has direct hands on the dangerous button.

Make approvals easy to review

An approval request that just says "Approve action?" is useless. The human cannot judge it. Show the full detail:

> Agent wants to send an email

> To: jane@acme.com

> Subject: Your refund of $50.00

> Body: Hi Jane, we have processed your refund...

> [Approve] [Reject] [Edit]

Give the reviewer enough context to catch a mistake in two seconds. An "Edit" option is a bonus: the human fixes the amount from $500 to $50 and then approves.

Limits: caps the agent cannot cross

Approvals rely on a human paying attention. Limits work even when nobody is watching. They are hard ceilings enforced by your code, not by the model's judgment.

Common limits:

• Spending cap. The agent may spend at most $100 per day. Attempt 101 fails automatically.
• Rate limit. No more than 20 emails per hour, to stop a runaway loop from spamming.
• Step budget. Stop after 15 actions on a single task, so it cannot loop forever.
• Data boundary. It can read the support database but never the payroll database.

A spending cap in practice is just a running total your code checks before each spend:

DAILY_CAP = 100.00

def issue_refund(amount):
    if spent_today() + amount > DAILY_CAP:
        return {"status": "blocked", "reason": "Daily cap reached."}
    charge(amount)
    return {"status": "ok"}

Because this runs in your code, the agent cannot argue its way past it. No clever prompt makes the number go higher. That is what makes limits stronger than instructions.

Instructions are not guardrails

Writing "never spend more than $100" in the system prompt is a *suggestion*, not a guarantee. Models can be confused, jailbroken, or simply mistaken. Real guardrails live in the surrounding code and tools, where the model has no vote. Use the prompt to guide behavior, but enforce the hard rules outside the model.

For a solid, free overview of how these controls fit into a larger safety picture, the NIST AI Risk Management Framework is readable and vendor-neutral.

Putting it together: a refund agent

Let us design a realistic agent and layer all three guardrails.

Job: handle customer refund requests in a support inbox.

Permissions (allow-list):

• look_up_order (read, safe)
• check_refund_policy (read, safe)
• draft_refund (write, prepares but does not execute)
• issue_refund (irreversible)

Approvals:

• Refunds under $50 that match policy: auto-approved, the agent runs them.
• Refunds $50 and over, or anything outside policy: paused for a human.

Limits:

• Daily cap of $2,000 total across all refunds.
• Max 30 refund actions per hour.
• Stop and escalate after 10 steps on one case.

Now trace a request. A customer wants $200 back for a late order. The agent looks up the order (safe), checks policy (safe), and drafts the refund. Because $200 crosses the $50 threshold, it pauses. A support rep sees the full proposal, confirms the order was genuinely late, and clicks approve. The refund runs, and it counts against the $2,000 daily cap.

If a bug caused the agent to try 50 refunds in a minute, the rate limit stops it at 30. If a prompt injection (a hidden instruction planted in a customer message trying to hijack the agent) told it to refund $10,000, the daily cap blocks it and the approval step catches it. The layers back each other up.

That layering is the real lesson. No single guardrail is enough. Permissions shrink what is possible, approvals catch judgment errors, and limits cap the worst case. Together they let an agent run mostly on its own while making a disaster nearly impossible.

Tuning the brakes over time

Start strict, then loosen. Launch with approval on almost everything. Watch the proposals for a week. The actions humans approve every single time without changes are candidates to auto-approve. The ones humans frequently edit or reject stay gated.

This gives you data instead of guesses. You learn *where* the agent is trustworthy and let it run freely there, while keeping a human on the genuinely risky calls.

For provider-specific ways to wire these controls (native approval hooks, tool permission settings, and budget features), see the deep-dive blocks on the OpenAI Agents SDK, the Claude Agent SDK, and Google's ADK. The concepts here transfer directly; only the syntax changes.

Key Takeaways

• Use a tool allow-list. The agent can only use tools you explicitly grant. Sort every tool into read, write, and irreversible, and scope the risky ones tightly.
• Put a human in the loop for irreversible or expensive actions. The model proposes, your code decides whether to run it. Show reviewers full detail so approval takes seconds.
• Enforce hard limits in code, not in the prompt. Spending caps, rate limits, and step budgets stop runaway loops and jailbreaks even when nobody is watching.
• Layer the guardrails. Permissions, approvals, and limits each catch what the others miss. No single brake is enough.
• Start strict and loosen with data. Gate everything at launch, then auto-approve the actions humans always wave through.