I keep seeing "AI support agent" sold as the obvious next product. Chat in, account updates out, refunds issued, questions answered. When I designed a support-agent architecture for client work like this, I ended up somewhere less exciting and more honest.
After you make the thing safe enough for production money and legal answers, the model is often reduced to intent classification and slot filling. At that point you have to ask: is this leverage, or a marketing-shaped wrapper around flows that should already be self-serve UI plus deterministic policy?
I'm not saying never use AI for support. I'm saying know when the agent is real leverage versus theatre.
The paradox after you design for safety
The brief was familiar. Case comes in. Agent should change a plan, update account fields, release a refund, or answer a question.
Refunds are the obvious landmine. So the design was strict from day one: the model does not dictate the refund amount. No refund without human-in-the-loop except under very tight, Finance-signed conditions. Once you take amount and open-ended write authority away from the model, what is left for the LLM on that path? Read the ticket. Classify intent as refund. Hand off to policy.
Plan change and account update look similar if you are honest. Anything the agent is allowed to change automatically is usually already possible in the product UI, and if it isn't, that is often a product gap, not an AI gap. Questions are worse in a different way: answers carry legal consequences. So the agent does not write open prose. It fills versioned FAQ templates with bound params. Anything outside the template library escalates to a human.
You can still argue the classifier recognises messy natural language better than a traditional form. That may be true. The open question is how much of that is real operational improvement versus a cool toy marketing can sell.
The loop: classify, then policy, then a write gateway
The flow from that project looks roughly like this.
A case is POSTed into an Orchestrator that holds read-only credentials into the account system. The Orchestrator always looks up the account and binds a snapshot. Then one Bedrock classify call returns strict JSON: refund, plan change, account update, question, or abuse. Policy comes from AppConfig. Writes never go through the Orchestrator. They go through a separate Tool Gateway with write credentials. DynamoDB covers audit, lock, and idempotency. HITL lands in a console queue. HITL is not the same as escalate: one is a controlled human approval step, the other is "we are out of policy, give this to support."
Caps matter: two model calls max, one write max, escalate when unsure. The credential boundary is the real safety net. If the model cannot reach write credentials, a clever prompt cannot invent a refund.
And the critical rule for every tool: the model never supplies the amount, the plan id, the field values, or the reply prose. Those bind from the account snapshot, the plan catalog, or FAQ template slots.
Where each flow collapses
Refunds. Week-one tool: look up account is auto. Issue refund is auto-if-policy only when Finance-signed knobs all pass, for example amount ≤ $50, account age, no prior refund, otherwise HITL. The model never chooses the amount. Once you accept that, a large share of refund tickets are "customer asked for a refund" plus policy. A "Request refund" button plus the same policy engine gets you most of the way there without a chat agent in the middle.
Plan change. Auto-if-policy with an empty allowlist is effectively HITL. If the allowlist later grows, the safe changes are usually the same ones a billing settings page already offers. The agent is not inventing a new capability; it is routing language into an existing mutation.
Account update. Every field HITL in week one. That is the honest default when a wrong email or wrong address creates downstream pain. Again: if a field is safe enough for the agent to write without a human, it is usually safe enough for the customer to edit in the UI.
Questions. Versioned FAQ pack keyed by questionId, template, and slots, not open RAG. Miss → escalate. Reply is auto only for allowlisted templates with every slot bound. That is not a conversational genius. It is a classifier plus a mail-merge with better intent parsing.
Put together, the "agent" is mostly: parse messy input → choose a labelled intent → apply policy → either fill a template, queue HITL, or escalate. The LLM sits in a narrow band of that pipeline.
When an LLM still earns its keep
I'd still use a model in a few places.
Natural language is messy. Customers do not click the right form. They paste screenshots of feelings into a ticket. A good classifier over free text can route that faster than forcing every request through rigid UI, especially across email and chat channels where you do not control the entry point.
Ambiguous tickets benefit from a first-pass label before a human opens them, refund vs billing question vs abuse, as long as the label is cheap to verify and never trusted for the write.
And for high-volume FAQ patterns with stable templates, slot filling from an account snapshot is real leverage: less copy-paste for agents, consistent wording, audit trail on which template shipped.
What I would not trust the model with, in this design: inventing amounts, inventing plan ids, free-form legal answers, or any write path that bypasses the Tool Gateway.
A practical decision framework
If you are a founder or CTO evaluating an "AI support agent," I'd run the design through five checks before you buy the story.
- Would a button plus policy do 80% of this? If yes, build the button. Use the model on the channel where customers refuse the button.
- Who supplies the money and the facts? If the answer is "the model," stop. Bind from systems of record.
- Is HITL a real queue with audit, or a slide? HITL without lock, idempotency, and a clear console is theatre.
- Are answers open text or templates? Open text on regulated or contractual topics is a legal product, not a chat feature.
- What happens when the model is unsure? Escalate must be cheaper and safer than guessing. Cap model calls and writes so the system fails closed.
If after those checks the remaining job is mostly classification and slot fill, name it that way. Ship the classifier. Don't dress it up as an autonomous agent unless the autonomy is real, and safe.
Closing
The worst possible application of LLMs is not "customer support" in the abstract. It is spending a year wrapping deterministic money and legal flows in a chat UI so the demo looks magical, while production quietly strips the model back to intent labels because anything more is unsafe.
Design the safe version first. See how much of the LLM survives. Keep what earns its keep. Cut the rest without guilt.
That is the difference between product value and marketing theatre. I'd rather know which one I am shipping.