AI Agent Safety Governance: An Engineer's Playbook

This is the playbook I wish I had when I started building production AI agents. It covers the threat model, the runtime policy, the defense against prompt injection, the audit requirements, and the tooling matrix. It is the engineer’s version of the Substrate Pattern and Defence in Depth frameworks.

The threat model

Five threats, in order of likelihood:

1. Prompt injection: the attacker injects instructions into the agent’s context (via a tool output, a user message, or a retrieved document) that cause the agent to take an unintended action. This is the #1 threat. Every production agent system must defend against it.

2. Scope violation: the agent calls a tool or accesses memory outside its intended scope. Less likely than prompt injection but more dangerous — it can lead to data breaches.

3. Cost runaway: the agent loops, calling the LLM API repeatedly until the budget is exhausted. Common in early-stage deployments.

4. Hallucinated action: the agent invents a tool call that doesn’t exist, or calls an existing tool with wrong parameters. The runtime should catch this — but if it doesn’t, the action fails silently.

5. Identity confusion: the agent acts on behalf of the wrong user. Rare but catastrophic — a cross-tenant data breach.

The defense: Substrate Pattern + Defence in Depth

The Substrate Pattern provides the architecture: four independent layers (memory, tool, action, identity) that each gate the agent’s behavior. Defence in Depth provides the principle: layer the defenses so no single failure compromises the system.

The implementation:

1. L1 — Application logic: treat as untrusted. Validate all inputs. Never pass user input directly to the model without sanitisation.
2. L2 — Tool execution: sandbox every tool call. Rate limit. Scope check. Audit log.
3. L3 — Action policy (Substrate): pre-action gate. Validate against the agent’s scope. Pre-conditions, post-conditions, identity checks.
4. L4 — Circuit breaker: monitor for anomalies. Rate spikes, scope violations, identity mismatches. Trip the breaker. Stop the agent.
5. L5 — Kill switch: manual or automated. Stops the agent, rolls back state, pauses the queue. Reachable in under 30 seconds.

Prompt injection defense

Prompt injection is the #1 threat. The defense:

1. Never trust tool output: tool output is untrusted input. Sanitise before it enters the agent’s context. Strip any instructions from tool output.
2. Separate system and user context: the system prompt and the user prompt should be in separate channels. The model should not be able to overwrite system instructions via user input.
3. Validate actions, not prompts: the prompt can say anything. The action policy (L3) is what matters. If the agent tries to call a tool outside its scope, the action gate blocks it regardless of what the prompt says.
4. Audit every action: log every tool call, every memory access, every action. If something goes wrong, you can trace it.

The tooling matrix

Layer	Tool	What it does
L1	Input validation	Zod, Pydantic, serde — validate all inputs
L2	Tool sandboxing	Custom runtime — scope, rate limit, audit
L3	Action policy	Substrate Pattern runtime — pre/post conditions
L4	Circuit breaker	Custom monitoring — anomaly detection, auto-trip
L5	Kill switch	Manual + automated — stop, rollback, pause
Audit	Tracing	tracing (Rust), Langfuse, LangSmith — structured logs
Red-team	Adversarial testing	Prompt injection tests, scope violation tests

The audit requirements

For regulated industries (financial services, healthcare), the audit requirements are:

1. Who: every action is attributed to a specific user identity. Chain of custody from request to action.
2. What: every tool call, every memory access, every action is logged with parameters and results.
3. When: timestamps on every event. Ordered, immutable.
4. Why: the model’s reasoning (if available) is logged alongside the action.
5. Reversibility: for T2 (act under supervision), every action is reversible within a defined window.

How to engage

The AI Agent Infrastructure consulting engagement is designed for teams that need to implement this playbook. Architecture review: USD 25K. Full implementation: USD 50K-200K.

— Dipankar Sarkar, Founder of Neul Labs