An assistant that answers questions is a content problem. An assistant that can issue a refund, reschedule a job or update a record is a security problem, and it should be built by whoever would build any other feature that writes to your database.
The framing that makes this tractable: the model supplies judgement about intent, and your code supplies the actions, the permissions and the limits. Every control belongs on the code side.
A prompt is not a control
You can tell a model not to discuss competitors, not to promise a refund, and not to take an action above a threshold. It will usually comply. Usually is not a security property.
Retrieved content can contain instructions. A customer message can contain instructions. A document in your own index, written by someone years ago, can contain a sentence that reads like a directive. Any of these can move a model away from what the system prompt asked for.
The defence is not a better prompt. It is that the instruction, even if followed, cannot produce an effect, because the tool refuses the arguments.
Typed tools with validated arguments
Every action the agent can take is a function with a schema. The model returns a call, your code validates the arguments against the schema, and then checks them against the permissions of the requesting user before executing anything.
That last step is the one that gets skipped. A tool that updates a booking should not accept any booking identifier. It should accept one the current user is entitled to modify, and the check runs in your authorisation layer rather than being inferred from the fact that the model asked.
Treat the model as an untrusted client. It is a very capable one, and it is still a client.
Read broadly, write narrowly
Read tools still need least privilege. Restrict them to the records and fields needed for the task. A broad read can disclose confidential data or supply it to a later tool, even when no record changes.
Write tools should be as narrow as the action requires. Not update_record with an arbitrary field map, but reschedule_appointment with an appointment identifier and a slot. Narrow tools are easier to validate, easier to authorise and easier to reason about when reviewing a transcript.
Every write should be idempotent, keyed on the business operation. An agent that retries after a timeout must not create a second task or send a second message.
Propose rather than perform
Where an action has a cost, the right pattern is usually a proposal.
The agent gathers the context, determines the action and prepares it. A person sees the prepared action with the reasoning and releases it. The work of assembling is automated, which is the expensive part, and the authority stays with a person.
Preparing the context can remove repetitive work while preserving a person’s decision. Measure preparation and review effort on the actual task, including cases where the proposed action is wrong. Do not assume that reviewing an AI proposal is always quick or easy.
Hard limits in code
Set them explicitly and enforce them outside the model.
Spend caps per user and per feature, so a loop cannot consume a month of budget. Rate limits per user, so a single conversation cannot generate a thousand calls. Value thresholds above which an action requires approval regardless of context. Record scoping, so tools operate only within the tenant and the entity set the session covers. Time boundaries, so an agent cannot modify records outside a permitted window.
These controls need tests at their boundaries and under concurrent requests. A limit checked independently by several workers may not be a limit at all.
Design the failure
Timeouts, provider errors and low confidence need a defined behaviour that is not a guess.
A short deterministic response and a handoff is almost always the right answer. A queued task is the next best. Silence is bad. An invented fact is worst, because it is indistinguishable from a good answer to the person reading it.
Keep a bounded audit trail and evaluate
Record the actor, tool, affected record, approval reference and outcome. Redact secrets and unnecessary personal information before logging. Set retention and access rules rather than storing complete conversations indefinitely. The OWASP logging guidance distinguishes useful security events from sensitive material that should not enter logs.
Then build a privacy reviewed evaluation set from representative interactions. Score expected outcomes before a prompt or model change ships. An appealing response is not evidence that its proposed action was authorised.
Approval has to identify the exact action
A confirmation button is not enough if the action can change between review and execution. Store the proposed operation with its target, material arguments, expiry and requesting identity. Present those details to the reviewer. When they approve it, execution should consume that specific proposal rather than asking the model to produce another one.
Suppose an agent proposes moving an appointment to Tuesday morning. The reviewer approves, but another operator has already moved it to Wednesday. Recheck the appointment version and availability before writing. Reject a stale proposal with an explanation; do not quietly override the newer decision. The approval authorised a particular change to a particular state, not permission to keep trying until something fits.
Make the proposal single use. If execution times out, retrieve its status using the same operation identifier. A second click must not create a second appointment or notification. Revoking the reviewer's permissions before execution should also stop the operation.
Test these cases with two concurrent sessions, an expired approval and a changed target record. Include a malicious document asking the agent to use an unrelated tool. The expected result is a denied operation with a useful audit event, not a promise from the model that it will behave.