The Demo That Convinced Me
I saw the demo in January. A colleague showed me an AI agent that could read a Slack message, look up a customer in our CRM, pull their billing history, draft a response, and post it back — all from a single prompt. No code. No API endpoints. Just a description of what it should do and a handful of tool permissions. The whole thing took about forty seconds.
I walked back to my desk and started sketching an agent for our own internal support queue. We get maybe fifty tickets a day, mostly password resets and billing questions with clear answers. If an agent could handle even half of those, that's hours of human time freed up every week. The demo made it look trivial. I figured I'd have a working prototype in two weeks.
It took two months. And the thing I shipped barely resembles what I started with.
What I Actually Built
The agent was supposed to do three things: read incoming support tickets from our helpdesk, look up customer information from our database, and draft a response that a human would review before sending. Simple scope. Defined inputs, defined outputs, clear success criteria.
The first version took about a week. I used LangChain with GPT-4, wired up a tool that queries our PostgreSQL database, added a tool that reads from our knowledge base (a Notion workspace with about forty articles), and set up a basic loop: receive ticket, classify urgency, look up customer, find relevant knowledge article, draft response, present to human for approval.
It worked perfectly on the three test tickets I ran it against. Then I pointed it at the real queue.
The first failure was分类. I'd given it four urgency categories: low, medium, high, critical. It classified a billing question about a double charge as "critical" because the word "charged" triggered its urgency heuristic. Then it escalated the ticket, sent an internal alert to three managers, and drafted a response that started with "We understand this is an urgent situation." It was a $12 double charge on a $29/month subscription. Not critical.
The second failure was worse. The agent tried to look up a customer whose email address had a typo in the ticket. Instead of returning "customer not found" and asking for clarification, it guessed — matched to a different customer with a similar email, pulled their billing history, and drafted a response addressing them by the wrong name. The human reviewer caught it, but only because they were paying attention. If the agent had sent that automatically, we'd have leaked one customer's billing details to another.
That's when I learned the first rule of building agents: the failure modes are not the same as the failure modes of regular software. Regular software crashes or returns an error. An agent makes a confident, plausible, completely wrong decision and moves on to the next step as if nothing happened.
The Error Compounding Problem
Here's what I didn't understand before I started: agents don't just fail. They fail forward.
A regular API call that returns bad data triggers an error handler. An agent that receives bad data incorporates it into its reasoning, draws a conclusion, and takes an action based on that conclusion. The error doesn't stop the process — it becomes part of the process. And every subsequent step builds on the mistake.
I saw this play out in a customer support scenario. A ticket came in from someone asking about our enterprise pricing. The agent looked up their account, found they were on a free trial, and — instead of routing them to the sales team — tried to answer the pricing question itself. It pulled information from our public pricing page, which lists individual plan prices, and presented them as enterprise pricing. The customer got a response that was confident, well-formatted, and completely wrong. The human reviewer approved it because it looked right.
We caught it only because the customer replied asking why the enterprise price was the same as the individual price. That reply went back into the queue, got classified as a new ticket, and the agent tried to answer it again — this time pulling from a different knowledge article and contradicting its first response. The customer was now confused and annoyed, and we'd lost twenty minutes of a human agent's time that we wouldn't have spent if we'd just answered the ticket manually in the first place.
The economics flipped. The agent was supposed to save time. Instead it created more work than it prevented, and the only reason we caught the problem was human diligence — the exact thing the agent was supposed to reduce.
What Actually Works
I don't want to make it sound like the whole thing was a disaster. It wasn't. After two months of iteration, the agent does three things well.
Ticket classification and routing. Not responding to tickets — just reading them and deciding who should handle them. The agent reads an incoming ticket, checks the customer's plan and history, and assigns a category and priority. It's right about 85% of the time. That's not good enough for autonomous action, but it's good enough to save our triage team fifteen minutes a day. We added a human-in-the-loop step: the agent proposes a classification, a human confirms or overrides. That single change eliminated the compounding errors because the human catches the wrong classifications before they cascade.
Drafting responses for simple, well-documented issues. Password resets, billing cycle questions, feature requests that map directly to knowledge base articles. For these, the agent drafts a response that a human reviews. The draft is usually 70-80% right. The human edits the rest. Net time savings: maybe three minutes per ticket. Not revolutionary, but it adds up.
Summarizing long threads. When a ticket has fifteen back-and-forth messages and gets escalated to a senior agent, the new agent has to read the whole history. Our agent now generates a two-paragraph summary of the thread: what the customer asked, what was tried, what the current status is. Senior agents say this is the most useful feature. It saves them five minutes per escalation, and the summaries are consistently accurate.
What all three of these have in common: they're narrow, they're supervised, and they don't take irreversible actions. The agent proposes; a human disposes. That's the pattern that works. The pattern that doesn't work — agent acts autonomously, human reviews after — is the one that produced every incident I described above.
The Cost Reality
I tracked every dollar. Here's what three months of building and running an AI agent cost.
API costs during development: $340. Most of that was GPT-4 calls during testing and iteration. Each test run processed ten to twenty tickets and cost roughly $0.50-$1.00. I ran about two hundred test cycles. The costs crept up slowly enough that I didn't notice until I checked the billing dashboard.
API costs in production: $85/month. The agent processes about fifty tickets a day, but it only drafts responses for the simple ones — maybe twenty per day. Each draft costs about $0.06-$0.08. Add the classification and summarization tasks, and it lands around $85/month.
My time: Roughly 160 hours over two months. At my billable rate, that's about $12,000 in development time. The agent saves maybe fifteen minutes a day across the support team. At the team's average hourly cost, that's about $375/month in savings.
Do the math. $85/month in API costs, $375/month in savings, netting $290/month. Against $12,000 in development time, the payback period is over three years. And that's assuming the agent keeps running reliably, which — given how often our knowledge base updates and how often the model's behavior changes — is a generous assumption.
Was it worth it? For the learning, absolutely. For the business case, not yet. The agent needs to handle more ticket types, reduce its error rate from 15% to under 5%, and run for six more months before the numbers start making sense. I'm not confident all three will happen.
What I'd Do Differently
If I were starting over, I'd build the classifier first and nothing else. Skip the response drafting, skip the summarization, skip the full pipeline. Just build a system that reads tickets and assigns categories. Ship that. Measure whether the classification accuracy is high enough to be useful. Get the team comfortable with an AI tool in their workflow.
Then, and only then, add the next capability. Maybe response drafting for the most common ticket type — password resets. Measure that in isolation. Does it actually save time? Do humans trust the drafts? Does the error rate justify the automation?
I did it backwards. I built the full pipeline first and tried to ship it all at once. That meant I was debugging classification, response drafting, knowledge base retrieval, and human review workflows simultaneously. Each failure interacted with every other failure. The debugging was ten times harder than it needed to be.
The other thing I'd change: I'd set up logging from day one. For the first two weeks, I had no visibility into what the agent was doing or why. When a response looked wrong, I had to trace through the prompt, the tool calls, and the model's reasoning manually. By the time I added proper logging — every tool call, every model decision, every input and output — I'd already wasted hours on debugging that would have taken minutes with the right observability.
The Honest Picture
AI agents are real. They work. The technology is genuinely impressive — a system that can reason over multiple steps, use tools, and produce useful output from a natural language description is something that would have seemed like science fiction five years ago.
But the gap between a demo and a production system is enormous. The demo shows the happy path. Production is all the unhappy paths — the edge cases, the ambiguous inputs, the failure modes that compound instead of stopping. Closing that gap takes months of iteration, careful guardrails, and a willingness to accept that the agent will never be as reliable as a well-designed traditional software system. It will be more flexible, more adaptable, and more capable of handling novel situations. It will also be more unpredictable, harder to debug, and more likely to fail in surprising ways.
If you're thinking about building an agent for your team or your business, start smaller than you think you should. Pick one narrow task, keep a human in the loop, and measure everything. The organizations that succeed with agents are the ones that treat them as tools that assist humans, not replacements that automate humans. The ones that fail are the ones that ship the demo and hope for the best.
I shipped the demo. It broke. I rebuilt it as a tool that assists. Now it works. That's the story nobody puts in the conference talk.