AI Agents Aren't as Reliable as You Think. Here's the Number That Proves It.
By Warren Schuitema, Founder | Matchless Marketing | The AI Dad
Your AI agent completed the task. You watched it work. You got the output you needed.
Now ask yourself: would it do that again, exactly the same way, if you ran it right now?
IBM Research dropped a blog post on Hugging Face yesterday that answers this question with data, and the number should matter to anyone running AI agents inside a real business. I'm going to translate it out of research language and into what it actually means for you.
The Stat That Changes How You Should Think About AI Agents
IBM Research tested a ReAct agent running on GPT-4.1 against the AppWorld benchmark, which simulates real multi-step tasks across applications like email, calendar, and maps. The agent succeeded 77.4% of the time on average across five runs, yet only 53.0% of tasks passed all five runs, a 24.4-point consistency gap.
Read that again. The agent's average looked fine. But when IBM asked "does it succeed every single time?", the answer was no, nearly half the time.
In production, that's a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, like reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper.
For you, as a small business owner, that means the automation you trusted to send a follow-up email, pull a report, or update your CRM might work Monday and silently fail Tuesday. No error message. No alert. Just a missed task.
Why This Happens (And It's Not What You Think)
You might assume this is a temperature problem. Set the model to temperature zero, make it deterministic, problem solved.
The gap actually comes from flat probability distributions at decision points, where the model is nearly torn between choices, making outcomes sensitive to small platform-level noise even at temperature zero.
That's the part that stops most people cold. The agent isn't randomly wandering off. It's hitting a fork in the road where both options look almost identical to it, and tiny, invisible system-level differences tip the outcome one way or the other. You can't configure your way out of it. The instability is baked into those decision moments.
Most agents can reread yesterday's transcripts, but they struggle to learn the underlying principles of a domain. That gap shows up as repeated mistakes, brittle behavior when inputs shift, and poor transfer of lessons to new situations.
Think of it this way: your agent has read every playbook, but it starts fresh every morning. It doesn't remember that the last time it hit step four of this workflow, option B was the right call. It evaluates from scratch. Again.
What IBM Built to Fix It
IBM Research introduced consistency guidelines built on a new diagnostic called the Consistency Analyzer, part of the open-source ALTK-Evolve toolkit, to address this hidden reliability problem in LLM agents.
Here's how it works without the academic language.
The Consistency Analyzer resamples decision points in a single recorded trajectory, with no ground truth required and no need to re-run the task, to flag flip-prone steps. Those steps are then turned into reusable guidelines injected at inference time.
So instead of the agent starting blind, it gets a note that says: "Hey, at step four of this type of task, you've historically wavered. Here's what works." The guidelines aren't hand-written. They're distilled automatically from the agent's own past runs.
Incorporating these guidelines into the ALTK-Evolve framework halved the consistency gap on the AppWorld benchmark, raising Pass^5 from 53.0% to 69.0% using a GPT-4.1 ReAct agent.
On AppWorld, adding just-in-time guidance from the memory system increased Scenario Goal Completion by +8.9 points overall, with the largest gains on the hardest tasks, +14.2 points.
The harder the task, the more this matters. Which tracks with how business automation actually works, because the workflows you care most about are never the simple ones.
What This Means for Your Business Right Now
IBM's research is still narrow. The evidence covers AppWorld only, ReAct agents only, GPT-4.1 only, five runs per task. It doesn't yet show whether the Consistency Analyzer transfers to other benchmarks, other scaffolds, or models tuned for determinism.
But the concept it proves is real, and you don't need ALTK-Evolve installed to act on it today.
Here's your practical takeaway: your AI agents have consistency gaps you haven't measured yet. The fact that a workflow succeeded doesn't mean it'll succeed repeatably. You need to know the difference between your agent's average and its all-five rate.
Step 1: Pick one agent workflow you're trusting with real outcomes. Could be a n8n automation that handles lead follow-up, a ChatGPT assistant that processes intake forms, or a Claude workflow that drafts client-facing responses. One workflow. That's it.
Step 2: Run it five times on the same input. Manually trigger it with the same trigger data, back to back. Don't change anything. Watch what happens across all five runs. Note the differences.
Step 3: Find the flip-prone step. Where did the output vary? That's your consistency gap. It's rarely a random failure. It's usually the same decision point showing up differently.
Step 4: Write a guideline for that step. Add a plain-language note to your system prompt or workflow instructions: "When you reach X decision, do Y, not Z, because the last time you chose Z it caused this problem." Specific beats vague here. "Prioritize the most recent contact record when duplicates exist" beats "handle duplicates carefully."
Step 5: Run it five more times. See if the gap closed.
That's it. You don't need a research team or ALTK-Evolve to apply the core principle. You identify the wobbly decision point, write a guideline that addresses it, inject it back into the agent's context. Rinse and repeat.
This is how you build an agent that gets better at your business over time, instead of one that stays at the same coin-flip reliability forever.
The Bigger Point
IBM's research is telling us something the AI marketing world doesn't want to say out loud: average success rates on benchmarks are not the same as reliability.
Most benchmarks hide this variability behind an average. The headline number looks good. The repeated-run number is where the real story lives.
If you're building any kind of AI-assisted workflow inside your business, you're not just evaluating whether the agent can do the task. You're evaluating whether it does it consistently enough to trust with your customers, your revenue, and your reputation.
Measure the all-five rate. Find the flip-prone steps. Write guidelines that close the gap. That's how agents go from impressive demos to actual business tools.
Warren Schuitema is the founder of Matchless Marketing and the creator of The AI Dad, a brand and platform helping small business owners and solopreneurs implement AI tools without hype, overwhelm, or a developer on retainer. He builds, tests, and documents real AI systems live so his audience can follow what actually works inside a running business. In this article, that experience applies directly: the consistency gap IBM Research identified is the same reliability question Warren asks about every agent running inside Matchless Marketing's own operations.