Why your AI agent works in the demo and fails in production

A demo is a sample of one

In a demo, someone picks a task, runs the agent once and it works. The audience sees one run of one task, chosen by a person who already knows it will go well. Production is the opposite. Every customer brings their own task, the same kinds of task come back all day, and nobody picks the easy ones.

The clearest measurement I know of is Sierra's tau-bench, published in June 2024. It puts an agent in a simulated shop with a simulated customer and a set of tools, gives it ordinary jobs such as changing or returning an order, and then checks the database to see whether the right thing happened. Its new measure, pass^k, is the chance that all k tries at the same task come out right. A gpt-4o agent using function calling got about 61% of the retail tasks right on one try. Asked to get a task right on all eight tries, it managed under 25%.

Slide the figure from 1 run to 8. The 20 tasks are made up, but they are shaped like the benchmark: a few the agent nearly always gets right, a few it rarely does and many in between. On the first run 13 of the 20 look fine. Ask for all eight runs to be right and 5 are left. If every run were an independent 61% coin flip, eight in a row would happen about 2% of the time (0.61 to the power of 8 is 0.019). The real figure is higher because some tasks are easy, and the average success rate hides which ones.

Surveys show the same gap in business terms. Camunda's research with Coleman Parkes among 1,150 senior IT leaders (September to October 2025) reported that 71% say they use AI agents, while only 11% of agentic use cases reached production in the last year. It is a vendor-commissioned survey of what people say about themselves, so read it as a hint.

Every step has to work

An agent that issues a refund does not make one decision. It reads the message, finds the order, checks the policy, calls a tool, reads what came back and writes a reply. The run is only right if every step was right. Anthropic's guide to building agents, published in December 2024, says the autonomous nature of agents "means higher costs, and the potential for compounding errors."

The arithmetic is multiplication. If each step is right 95% of the time, ten steps in a row are all right 0.95 to the power of 10 of the time, which is 0.60. Of 100 runs, about 60 get through.

Set the steps to 20 and leave each step at 95%: 36 of 100 runs finish (0.95 to the power of 20 is 0.36). Raise each step to 99% and ten steps leave 90 of 100, twenty leave 82. The model is simple on purpose. Real steps are not independent, and an agent can sometimes notice its own mistake, which is why checks between steps help.

A retry can do the job twice

Tools fail in an awkward way. Sometimes the call worked and only the reply was lost. The agent sees a timeout and cannot tell the two cases apart, so it tries again. For a read that is harmless. For a refund, a booking or an email, the action runs twice.

Switch the figure between the two policies. With a plain retry the ledger ends with two refunds and the customer is paid $80 instead of $40. With an idempotency key, a unique code the caller sends with the request, the API recognises the repeat and returns the saved reply. Stripe's API documentation describes this as a way to retry "without accidentally performing the same operation twice." The agent loop is another caller that retries, so it needs the same protection. Make the key once per intended action, from the order and the reason for example, and not once per attempt, or every retry will look like a new request.

The agent can do more than the job needs

The agent will be wrong sometimes. It misreads an order, gets a vague message, or treats a line in a pasted email as an instruction. What matters then is what it is allowed to do. OWASP's list of risks for language-model applications calls the problem excessive agency: damaging actions taken in response to unexpected, ambiguous or manipulated model output, whatever caused the model to misbehave. It traces it to excessive functionality, excessive permissions and excessive autonomy.

With broad access, all five proposals go through, including deleting an account and exporting every customer email. Switch to narrow tools and the agent has a refund tool and nothing else. At the default limit of $50 the $20 and $45 refunds go through, the $380 refund waits for a person, and the two dangerous actions have no tool to call. OWASP's advice says the same in other words: keep the tools and their permissions to the minimum, ask a person to approve high-impact actions, and check authorization in the system being called, since the model should not decide what it may do.

A long run costs more than it looks

A model has no memory between calls. Anthropic's documentation describes its Messages API as supporting "stateless multi-turn conversations": you send the earlier turns with every request. An agent loop does that on every step, so the model re-reads the instructions, the tool descriptions and everything that happened so far, each time.

Say the fixed prompt is 3,000 tokens and each step adds 1,000, a tool result and the model's note. Step 1 sends 3,000 tokens and step 10 sends 12,000. Ten steps send 75,000 in total (10 times 3,000 plus 1,000 times 45) and twenty steps send 250,000 (20 times 3,000 plus 1,000 times 190).

Drag the steps and the tokens added per step. Twice the steps cost 3.3 times the input here (250,000 against 75,000), because every extra step also carries all the earlier ones. Prompt caching lowers the price of the repeated part, so check what your provider charges for it, but it does not shorten what the model has to read. A limit on steps is the cheapest protection, and Anthropic's guide says it is common to include stopping conditions such as a maximum number of iterations.

Patterns that hold up

Start by asking whether the steps are always the same. If they are, a plain script or a fixed workflow has none of these problems, and Anthropic's own advice is to find the simplest solution possible and add complexity only when needed. If the input really is messy, these habits help.

Run every important case many times

Keep a set of real requests with the outcome you expect, and run each one at least eight times before a release. Score the end state, as tau-bench does, by checking the records themselves, because the wording of a reply proves little. A case that passes five times out of eight is a finding, whatever the average says. Anthropic recommends extensive testing in sandboxed environments along with appropriate guardrails.

Give every action a key and every loop a limit

async function runAgent(task, { maxSteps = 8 } = {}) {
  const messages = [task.message];
  for (let step = 0; step < maxSteps; step++) {
    const move = await model.next(messages);
    if (move.type === "reply") return move.text;

    // one key per intended action, not per attempt
    const key = [task.id, move.tool, hash(move.args)].join(":");
    const result = await callTool(move, { idempotencyKey: key });
    messages.push(move, result);
  }
  return handToPerson(task); // out of steps: stop and hand over
}

Hand over narrow tools

Give the agent a tool for one job, such as issue_refund(order_id, amount), and no general database login or shell. Put the limit inside the tool and not only in the prompt, so a confused or manipulated model cannot talk its way past it. Anything above the limit goes into a queue for a person.

Keep a record of every step

Store each prompt, tool call and result under a run id. When a customer says the agent did something odd, you need to replay that run, and nobody can do that from memory.

How this looked in a real product

Oh Crap! Chat is a paid AI coach we built with Prologue Partnerships. It answers questions from one author's material, so it has fewer ways to go wrong than an agent that acts on records, but the same limits are in it. The API has one time limit for the first words and another for the whole answer, and it retries with a growing delay when the model is busy. Access is checked before any model call, so an expired pass never spends tokens. The model only sees passages pulled from the author's material, which keeps answers inside the book. A dashboard shows AI usage and an estimated daily cost next to revenue. The case study has the full picture.

Questions to answer before launch

1. If you run your ten most common requests eight times each, how many come out right every time?
2. How many steps can one run take, and what happens when it reaches the limit?
3. If a tool call times out, can the same action run twice?
4. What is the worst thing the agent can do with the access it has today, and who approves it?
5. Is the limit on each action inside the tool, or only written in the prompt?
6. Do you know what a 20-step run costs, and can you replay yesterday's odd run from your logs?