Over two years of building AI support agents we've seen both 400% six-month ROI cases and failures that cost a client 8% of their customer base in churn. The difference always comes down to one thing: whether the scenario was chosen correctly. This article breaks down where AI customer support agents reliably beat humans, where they damage the experience, how to measure success honestly and what a pilot really costs.

What the industry data actually says

The headline numbers are impressive. In February 2024 Klarna reported that its AI assistant handled 2.3 million conversations in its first month — two-thirds of all customer service chats — doing the work of about 700 full-time agents, with customer satisfaction on par with humans and resolution time down from 11 minutes to under 2 (Klarna, 2024). Gartner predicts that by 2029 agentic AI will autonomously resolve 80% of common customer service issues (Gartner, 2025).

The other side of the story is just as real. A Gartner survey of 5,728 customers found that 64% would prefer companies didn't use AI in customer service, and the top fear was that it would become harder to reach a person (Gartner, 2024). And in 2025 Klarna itself started hiring human agents again so customers could always talk to a person (Bloomberg, 2025).

Both things are true at once: AI handles routine work very well, and customers punish companies that use it to hide humans. Good design lives between those two facts.

Where AI agents reliably outperform humans

  • FAQs and common operations — order status, re-shipment, password reset, delivery changes. High volume, clear answers, low emotional stakes.
  • Scenarios with a clean API — ticket creation, bookings, access grants, subscription changes. If the action is one API call with clear validation, the agent can execute it end to end.
  • Night support and markets outside your primary time zone. An agent that answers at 3 a.m. in the customer's language beats a promise to reply "within one business day".
  • First-line triage — classifying intent, collecting order numbers and screenshots, and routing to the right team with a summary, so the human starts with context instead of "how can I help?".
  • Agent assist. Even where AI should not talk to customers, it can draft replies and surface knowledge for human agents. A study of 5,179 support agents found that access to an AI assistant raised productivity by 14% on average and by 34% for novice workers (Brynjolfsson, Li & Raymond, NBER, 2023).

The common thread: the agent needs accurate knowledge and permission to act. Knowledge comes from retrieval over your documentation, which we cover in detail in RAG for business: how to build an AI agent that actually knows your docs.

Where AI breaks the experience

  1. Conflict conversations — refunds, complaints, legal questions. An upset customer wants to be heard and wants someone with authority to make an exception. A polite bot quoting policy makes it worse.
  2. Cross-domain tasks where the answer depends on three or more systems without a unified API. Every missing integration becomes a place where the agent guesses.
  3. Markets where customers expect a personal touch, such as B2B enterprise accounts with €100k+ contracts and a named account manager.
  4. Cases where one mistake costs more than the agent saves in a year — medical, financial or safety-critical advice, or anything that creates a binding commitment.

That last point is not theoretical. In Moffatt v. Air Canada (2024 BCCRT 149), a Canadian tribunal held the airline liable after its website chatbot invented a refund rule for bereavement fares, rejecting the argument that the chatbot was a separate entity responsible for its own words (American Bar Association summary). What your agent says, your company says.

The best AI support agents are the ones that know when to hand off to a human. We always design escalation logic in sprint one.

Designing escalation that customers don't hate

Escalation is a product feature, not a fallback. Our default rules:

  • Always offer a human. A visible "talk to a person" option at every step. Hiding it raises containment on the dashboard and lowers trust everywhere else.
  • Escalate on signals, not only on request: negative sentiment, repeated rephrasing, words like "cancel", "lawyer" or "complaint", high-value customers and any action above a set amount.
  • Hand over with context. The human receives the conversation summary, detected intent, customer data already fetched and what the agent tried, so the customer never repeats themselves.
  • Confirm before side effects. Refunds, cancellations and account changes require explicit confirmation, and money-moving actions above a threshold require a human.
  • Disclose that it's AI. Customers should know they are talking to a machine. In the EU this becomes a legal obligation: Article 50 of the AI Act requires providers to inform people that they are interacting with an AI system, with transparency obligations applying from August 2026 (EU AI Act, Article 50; Regulation (EU) 2024/1689).

A simple routing policy in code looks like this:

ESCALATE_INTENTS = {"refund_dispute", "complaint", "legal", "cancel_contract"}

def should_escalate(turn, customer) -> bool:
    return (
        turn.intent in ESCALATE_INTENTS
        or turn.sentiment < -0.5
        or turn.rephrase_count >= 2          # the customer is going in circles
        or customer.annual_value > 100_000
        or turn.user_asked_for_human
    )

Anthropic's engineering guide on building effective agents makes the same point from the model side: start with simple, predictable workflows and give the model autonomy only where you can verify the results.

How to measure success

  • Containment rate — the share of conversations the agent closes without handoff. Measure it together with recontact rate within 7 days; a "contained" conversation where the customer comes back tomorrow is a failure.
  • CSAT separately for AI and human conversations. Compare them, don't blend them, and segment by intent: the agent may be excellent at order status and poor at billing.
  • First response time (FRT) and time to resolution (TTR). AI wins FRT by design; TTR shows whether it actually solves problems.
  • Escalation quality — how often humans have to ask the customer for information the agent already had.
  • Cost per resolved case — the metric your CFO cares about, including model costs, platform fees and the human time spent on escalations.
Metric Healthy pilot after 3 months Red flag
Containment rate 40–60% for chosen intents above 80% with falling CSAT
7-day recontact rate below 10% above 20%
AI vs human CSAT within 5 points gap above 10 points
Escalations with full context above 90% customers repeat their issue

A phased rollout that works

  1. Agent assist first. The AI drafts replies, humans send them. You collect quality data without customer risk.
  2. Automate three to five high-volume intents with clean APIs and clear policies.
  3. Expand intent by intent, each with its own golden set of real conversations and CSAT tracking.
  4. Review weekly. Read a sample of conversations, especially escalated and low-rated ones, and fix knowledge gaps at the source.

What it costs to launch

A pilot connected to one system (for example, Zendesk plus a knowledge base) takes 6–10 weeks and €25,000–€45,000. ROI typically lands in 5–9 months with a containment rate above 50%. If your traffic is below 3,000 requests per month, the economics don't work — don't start; invest in a better help center and agent-assist tools instead.

The budget usually splits into three parts: integration and tooling (30–40%), knowledge base cleanup and evaluation set (25–35%), and conversation design, escalation logic and testing (the rest). Teams that skip the knowledge cleanup spend the savings on escalations.

The metrics above only work with a solid evaluation process; we describe ours in How to evaluate LLM applications. Support agents read untrusted customer emails, so the safeguards in AI agent security apply directly, and if you serve EU customers, check the disclosure duties in EU AI Act for software companies.

For the architecture behind support agents, see workflows vs agents and designing tools for LLM agents; customer data in prompts is covered in privacy for LLM apps.

FAQ

Will AI agents replace our support team? No. They take over repetitive intents and first-line triage, so humans focus on complex, emotional and high-value cases. Most teams we work with redeploy people rather than cut them.

What containment rate is realistic? For well-chosen intents, 40–60% in the first months. Overall containment depends on how much of your volume is routine.

Do we have to tell customers they are talking to AI? It is best practice everywhere, and in the EU the AI Act makes disclosure mandatory for systems that interact with people.

Which channel should we start with? Chat or messaging. Email tolerates slower, more careful answers and is a good second step; voice is the hardest and should come last.

Sources

  1. Klarna (2024). Klarna AI assistant handles two-thirds of customer service chats in its first month.
  2. Bloomberg (2025). Klarna turns from AI to real person customer service.
  3. Gartner (2024). Survey finds 64% of customers would prefer that companies didn't use AI for customer service.
  4. Gartner (2025). Agentic AI will autonomously resolve 80% of common customer service issues by 2029.
  5. Brynjolfsson, Li & Raymond (2023). Generative AI at Work. NBER Working Paper 31161.
  6. American Bar Association (2024). BC tribunal confirms companies remain liable for information provided by AI chatbot (Moffatt v. Air Canada, 2024 BCCRT 149).
  7. European Union. Regulation (EU) 2024/1689 (AI Act), Article 50.
  8. Anthropic (2024). Building effective agents.