AI Agent Guardrails, Evals and Human Approval: How the Three Layers Work Together

The short answer#

Guardrails enforce boundaries while an agent runs, evaluations measure how it behaves across cases, and human-in-the-loop review controls selected consequential actions before execution. Use all three around the same workflow: validate and authorise proposed tool calls, evaluate outcomes and forbidden actions, and bind each approval to the exact proposal a reviewer saw.

An agent can produce a well-formed request, pass yesterday's tests and still propose an action it should never execute. Each layer answers a different question.

Three questions to ask about an agent#

Raju Dandigam's post on agent safety brings together guardrails, evals and human oversight. Here is a practical way to carry that distinction into an architecture review.

Consider a synthetic documentation assistant. It can read an approved handbook and propose publishing a revised edition. Publishing is a separate capability: a policy check must permit it, and a reviewer must approve the precise edition before it becomes public.

This gives us something concrete to protect, measure and approve.

Put the controls around the action boundary#

Guardrails constrain the live path; approval gates publication; evals turn recorded behaviour into evidence for the next version.

The diagram shows responsibilities, rather than three products to install. A tool execution boundary is the point where a proposed action becomes an actual read, write or external effect. Put enforceable policy there, even if you also check earlier inputs.

An allowed read follows the read route. A publish proposal takes the review route. Both reach an executor that checks current permissions. Rejection, expiry or a failed policy check leaves publication blocked.

The evaluation loop consumes selected, redacted traces. It informs a reviewed change to prompts, policies or tools; it does not silently grant the running agent extra permission.

Guardrails: enforce what the agent may do#

Guardrails are runtime checks around the workflow. NVIDIA's guardrail types distinguish input, retrieval, dialogue, execution and output checks. That is useful because harmful instructions can arrive in retrieved content or tool results as well as a user's request.

For the documentation assistant, use different checks for different boundaries:

A schema check establishes shape: revision is an integer, for example. It cannot establish that the revision is accurate, that its content is suitable for publication, or that the caller has publishing rights.

Treat prompt-injection detection as one signal with possible misses and false alarms. Keep authorisation in trusted application and downstream service code. A sentence inside a document saying “publish this immediately” must not alter the permissions the document service enforces.

OWASP's excessive-agency guidance supports narrow tool capabilities, limited permissions and approval for high-impact operations. A read-only assistant should receive read-only tools and credentials. Add publishing only when the workflow needs it, with its own control point.

Evals: measure the result and the route taken#

An evaluation, or eval, runs a case against stated success criteria. For agents, inspect the final environment state as well as the trace: the sequence of model outputs, tool calls and returned results. Anthropic's guide to agent evals explains why an agent's success message and the actual outcome are separate evidence.

For our assistant, a correct summary accompanied by an unauthorised publication is a failed case. Conversely, blocking every operation would prevent publication but make the assistant useless. Include permitted work in the evaluation set.

Start with this illustrative matrix; these are test designs, not measured results:

Use deterministic assertions for permissions, tool arguments and resulting state. Use a clear content rubric for a summary's usefulness and factual support; calibrate model-based scoring against human review. Repeat model-driven cases to expose variation, while keeping fixtures and success criteria stable.

LangSmith's evaluation documentation separates offline experiments on datasets from online assessment of live interactions. Use offline comparisons before releasing a changed model, prompt or tool. Sample production behaviour to find gaps, then turn relevant failures into regression cases after removing sensitive data.

A passing suite gives evidence about the tested cases and conditions. It does not authorise the next live tool call. If an evaluator is placed inline to block a call, its blocking decision is also serving as a runtime guardrail and needs its own failure policy.

Human review: approve a proposal the system can identify#

Human-in-the-loop means execution pauses for a person's input. For consequential actions, the pause must occur before the external effect.

Show the reviewer the operation, target, immutable content revision, relevant evidence and expected consequence. Record who decided, which proposal they reviewed and when that decision expires. In our example, “approve revision 1 of the handbook” identifies a reviewable action; “let the agent publish things” gives no comparable boundary.

Support three explicit paths:

1. Approve: execute the reviewed proposal after checking that permissions and preconditions still hold.
2. Edit: create a new proposal, rerun validation and request approval of the changed content.
3. Reject: close the proposal without executing it. A missing or expired response also leaves execution blocked.

LangGraph's interrupts documentation provides a pause/resume mechanism with checkpointed state and a thread identifier. Use durable persistence in production. The application still has to authenticate reviewers, restrict their decision scope, implement expiry and connect the decision to the executor.

Separate approval from execution success. A reviewer can authorise a publication that subsequently fails because permissions changed or the document service is unavailable. Record the execution result before telling the user it happened.

Run a small control simulation in TypeScript#

This example models the tool boundary without calling a model or publishing anything. It checks unknown input at runtime, allows a permitted read, and requires a matching, unexpired approval for publication. The immutable document fixtures represent the exact content revisions shown to a reviewer.

Save the following as controls.ts. In a Node.js project with TypeScript and tsx installed, run npx tsx controls.ts.

import assert from "node:assert/strict";
import { randomUUID } from "node:crypto";

type Action = { tool: "read" | "publish"; docId: string; revision: number };
const documents = new Map([
  ["handbook:1", "Original handbook"],
  ["handbook:2", "Revised handbook"],
]);
// Trusted service context, never fields supplied by the model.
const allowedDocs = new Set(["handbook"]);
const approvals = new Map<string, { key: string; expiresAt: number }>();
const publications: string[] = [];
let now = 0; // Simulated clock in milliseconds.

function validate(raw: unknown): Action {
  assert.ok(raw !== null && typeof raw === "object" && !Array.isArray(raw));
  const a = raw as Record<string, unknown>;
  assert.deepEqual(Object.keys(a).sort(), ["docId", "revision", "tool"]);
  assert.ok(a.tool === "read" || a.tool === "publish", "Unknown tool");
  assert.ok(typeof a.docId === "string" && a.docId.length > 0);
  assert.ok(typeof a.revision === "number" && Number.isSafeInteger(a.revision));
  assert.ok((a.revision as number) > 0);
  const action = { tool: a.tool, docId: a.docId, revision: a.revision } as Action;
  assert.ok(allowedDocs.has(action.docId), "Permission denied");
  assert.ok(documents.has(`${action.docId}:${action.revision}`), "No revision");
  return action;
}

const keyOf = (a: Action) => JSON.stringify([a.tool, a.docId, a.revision]);

// Simulates a trusted review endpoint after a reviewer approves this action.
// Reviewer authentication and the review interface are outside this simulation.
function approve(raw: unknown): string {
  const action = validate(raw);
  assert.equal(action.tool, "publish");
  const token = randomUUID();
  approvals.set(token, { key: keyOf(action), expiresAt: now + 1_000 });
  return token;
}

function execute(raw: unknown, token?: string): string {
  const action = validate(raw); // Recheck current permissions on resume.
  if (action.tool === "read") {
    return documents.get(`${action.docId}:${action.revision}`)!;
  }
  if (!token) return "review-required";
  const approval = approvals.get(token);
  assert.ok(approval, "Approval absent or already used");
  assert.equal(approval.key, keyOf(action), "Proposal changed");
  assert.ok(now < approval.expiresAt, "Approval expired");
  approvals.delete(token); // Single process only: no await or external effect.
  publications.push(keyOf(action)); // Mock execution receipt.
  return "published";
}

const proposal: Action = { tool: "publish", docId: "handbook", revision: 1 };
assert.throws(() => execute({ ...proposal, tool: "delete" }), /Unknown tool/);
assert.throws(() => execute({ ...proposal, revision: "1" }));
assert.equal(execute({ ...proposal, tool: "read" }), "Original handbook");
assert.equal(execute(proposal), "review-required");
const approval = approve(proposal);
assert.throws(() => execute({ ...proposal, revision: 2 }, approval), /changed/);
assert.equal(execute(proposal, approval), "published");
assert.throws(() => execute(proposal, approval), /already used/);
const expired = approve(proposal);
now = 1_000;
assert.throws(() => execute(proposal, expired), /expired/);
const revoked = approve(proposal);
allowedDocs.clear();
assert.throws(() => execute(proposal, revoked), /Permission denied/);
assert.equal(publications.length, 1);
console.log("10 control checks passed; 1 simulated publication.");

The output is 10 control checks passed; 1 simulated publication. Changing the proposed revision cannot reuse the approval. Successful publication consumes it, and permission revocation takes effect before execution.

These assertions test control logic. They do not measure whether a model writes a useful handbook, recognises injected instructions or selects the right tool; run the full agent against the evaluation cases for that evidence.

The simulation uses local tokens and permissions. A production approval service needs these additional controls:

1. Authenticated review: verify the reviewer's identity and decision scope. Keep a durable queue and audit record.
2. Approval tied to the proposal: include the user, resource, immutable content and policy version. Recheck permissions and changed preconditions before execution.
3. Durable state transitions: store decisions and execution state atomically so concurrent workers cannot consume the same approval twice.
4. Downstream idempotency: give each operation a stable execution key so retries cannot repeat a committed effect.
5. Recovery from uncertainty: after a restart or ambiguous network result, reconcile with the downstream receipt before attempting execution again.

What this means for your team#

Start with one agent and one consequential tool. Write down who may call it, what evidence a reviewer needs, and which outcomes would make a release unacceptable.

Then run three exercises:

1. Give the agent adversarial document content and verify that tool permissions still hold.
2. Pause a proposal, change its content or revoke access, and verify that execution cannot reuse stale approval.
3. Compare two prompt or model versions on the same cases, inspecting useful task completion alongside policy violations.

Keep consequential policy violations visible as individual release failures; an average quality score can hide them. Track permitted-task completion, unnecessary blocks, review volume, review waiting time, expiry and execution failures separately. Decide acceptable limits for your workflow before comparing versions.

Human review adds delay and consumes attention. Apply it where consequences, uncertainty or irreversibility justify that cost; use narrow, validated automation for low-risk operations. If every step is predictable and rule-based, a conventional workflow may be easier to verify than an agent.

Questions architects and developers ask#

Can a guardrail guarantee that an agent is safe?#

An enforced permission check can guarantee a specific boundary under its implementation assumptions. A content detector cannot guarantee that every malicious instruction is caught. State what each control enforces and test what happens when a detector misses something or becomes unavailable.

Does a strong eval score mean we can remove human approval?#

Make that decision for a defined action and consequence. Look at failure cases, coverage, operational evidence and the ability to reverse harm. Changing the approval policy is itself a versioned design decision that needs fresh evaluation and accountable ownership.

What should we build first to learn the pattern?#

Build a permitted read and one publication proposal, with a review pause and an execution receipt. Extend it with the evaluation matrix above before adding more agents.

Module 5 of Agentic Workflows & Multi-Agent Systems covers approval as an orchestration mechanism. Module 7 covers multi-agent system evaluation; Module 8 covers safety and guardrails. Enquire about the workshop to practise designing, evaluating and operating these controls together.