The Verification Bottleneck
An agent files an insurance claim. The portal accepts it, the screen confirms, and a dashboard somewhere turns the task green.
Two days later an adjuster calls the office to verify a policy number before the claim can move, and the phone rings in an empty room.
So the claim stalls until a human notices the payment that never arrived, which can take a quarter, if anyone notices at all.
The agent did everything right. It read the screen, filled every field and clicked the right buttons, and it still had no way to learn that correct and finished are different things.
The industry is still debating whether models can operate a desktop at all, and the answer is yes. Companies pay for finished work, though, delivered without someone standing behind the machine checking every line, and the gap between clicking and finishing is where this year’s deployments either hold up or fall apart.
together with Plaid
An agent that can file a legitimate claim can file a fake one just as fast. Most identity checks still assume a human at the keyboard.
Plaid’s latest white paper traces how fraud evolved across each internet wave, and why this one needs a new approach.
Inside:
▫️ Why point-in-time verification is breaking down
▫️ How AI-driven fraud is scaling faster than traditional controls
▫️ Why financial behavior offers stronger, harder-to-fake context
▫️ How broader, cross-platform visibility reveals patterns others miss
Table of Contents
1. What The Current Benchmarks Are
2. A Benchmark Score and a Business Process Are Different Units
3. The Best Architecture in Production Is Fifteen Years Old
4. The Number That Settles This
5. The Attack Surface Is the Product
6. Why the Back Offices Are Still Hiring
1. What The Current Benchmarks Are
Claude Fable 5 ranked top of OSWorld-Verified with 85%, sourced to llm-stats in June 2026. Human testers score around 72% on the same task set.
A year earlier the best model managed just 42%.
By the 29th of August, BenchLM’s tracking of the same benchmark had Qwen3.8 Max in front at 86.1%, with Claude Mythos 5 and Claude Fable 5 tied behind at 85.
Eleven weeks. New leader, different lab, different continent.
Normalised against the human baseline of 72.36 and the arc still deserves respect. Last year’s best model sat at 58% of human performance and this year’s is at 119%. Something real crossed over in early 2026.
What a score of 85 contains
OSWorld-Verified is a set of 369 desktop tasks, and a model passes one by actually finishing it rather than describing how.
Eight of the tasks can be left out under the official rules. Some results get re-run by the people who maintain the leaderboard and others are simply reported by the labs, using different time limits, different operating systems and different tools.
There is also a newer version, OSWorld 2.0, whose scores are not comparable with the older one.
So an 85 belongs to a model, plus the software wrapped around it, plus how many steps it was allowed to take. Two companies both claiming 85 may not be running the same thing at all.
Andreessen Horowitz published a piece in August, where they interviewed the people actually running these systems, and one of them turned out to be running millions of automated tasks a month without knowing which model does the work. He had never needed to find out. His supplier swaps models underneath him the way a cloud provider swaps servers.
That detail says more about where computer-use agents have got to than any leaderboard does. When the heaviest users stop checking which model they are on, the model has stopped being the thing that decides whether the work gets done.
2. Per Task and Per Step Are Different Maths
What a benchmark measures and what a company actually buys are two completely different things. Here’s the explanation.
Task success and step success
A benchmark scores whole tasks, while a real office job is a chain of steps, and every step has to work for the job to count as done.
So if a job has 20 steps and each step works 95% of the time, the job works 36% of the time. That is 0.95 multiplied by itself twenty times. Here is what the same sum looks like across a few reliability levels.
This is simplified. A well-built system retries and recovers, so real numbers land better than the table. Read it as the shape of the problem rather than a prediction.
Why a better model buys less than it looks
Agents hold up on short jobs that follow a fixed script and come apart on long ones. That is exactly the pattern their interviews describe, without ever naming the reason underneath it.
Going from 95 to 98 per step nearly doubles end-to-end success at 20 steps, and still leaves you failing two thirds of the time at 50.
Model improvements arrive in a straight line, but failure compounds. That is an unfair fight, and it is why the last few points on a leaderboard matter far less at work than they do in a launch, and why the people running this at volume have all landed on the same answer.
3. Why the Winning Architecture Is Fifteen Years Old
The design pattern serious teams advocate for is being seen as an emerging AI-native idea. But that is not the case. More like far from it.
What run-caching does
The a16z piece describes the pattern well. The agent runs a workflow once, the system caches it as deterministic code, every subsequent run executes as cheap code, and the model is only called back when something breaks so it can diagnose, fix and re-cache.
They frame this as cost optimisation.
But it’s more like reliability architecture, and what it does is collapse N stochastic steps into one checkable event. You basically stop rolling the dice 50 times and start rolling it once, at the moment the portal changes.
What the scraper case shows
Their flagship case, reported first-hand, is a CPG data platform running 15 to 20 million automated portal interactions a month, which halved the engineering team assigned to scraper maintenance. Excellent outcome, real money.
However, this is actually using hand-coded scrapers to do the work. The agent is the repair crew that fixes them when a retailer changes the page.
That is not an agent using a computer. That is an agent fixing the thing that uses the computer.
The conclusion is that software that clicks through screens on a fixed script has existed for 15 years, under the name Robotic Process Automation, and it always broke the moment a website changed.
If the honest story right now is that AI models are the fix for that 15-year-old flaw, that is a genuinely valuable business worth building. It is also a much smaller claim than the one printed on the tin.
The same tension runs through the speed fix they point to. Reading a page’s underlying structure instead of looking at a picture of it is faster, and it ties you straight back to how that particular application is built. Screenshots removed that dependency.
This puts it back. Speed bought with fragility is not free.
4. The Number That Settles This
Here’s the metric that does not exist, and its absence is the most interesting thing about this whole conversation.
The a16z cost comparison puts an agent at $6-8 an hour, against roughly 10 for offshore BPO and 30 to 45 for a US back-office worker, the last built up from a BLS median of 20.59 with benefits at about 30% of total comp. Fine as far as it goes.
But an hour of agent time is not the thing a buyer purchases. A completed, correct task is.
Let’s assume $7 per hour, 9 minutes per agentic task, gives you about $1.05 an attempt. At 85% first-pass success with independent retries you’re looking at roughly 1.18 attempts, so $1.24. Now price the 15% that need a person. 10 minutes of a US operator at 35 fully loaded is $5.83, and 0.15 of that adds 87 cents. You land near $2.10 per completed task.
Double the headline. And that’s the optimistic version, because it assumes every failure gets caught.
Nobody is claiming $2.10 is the real number. The real number doesn’t exist yet and neither does anybody quoting $6-8 an hour. However, cost per verified completed task, including escalation, is the only figure that decides whether any of this works, and I cannot find a single vendor publishing it.
Same for the second missing metric. Nobody reports a silent error rate.
The a16z piece names the failure mode precisely and then walks past it. An agent transcribing payment terms into an ERP reads net 60 as net 30.
The record looks plausible as it passes every visual check. It surfaces weeks later when an invoice goes out wrong. An agent files an insurance claim, the screen confirms receipt, and two days later an adjuster phones about a policy number the agent will never know was asked for.
A loud failure costs you a retry. A silent one costs you the invoice, the relationship and the audit. Those are not the same 15%, and no benchmark on earth distinguishes them.
5. The Attack Surface Is the Product
What prompt injection is
An agent reads the instructions from its owner and the text on the webpage as the same kind of thing: words on a screen. It has no reliable way of telling which words came from the person paying for it. So anyone who can get text in front of the agent can give it orders.
Picture a temp who does whatever any note left on the desk says, because nobody explained which notes come from the boss. Hide a note on a page the agent is going to read, and the note becomes an instruction.
Three conditions, all of them shipped on purpose
The recurring finding across serious prompt injection incidents is one configuration where an agent has access to private data, exposure to untrusted content, and the ability to act externally.
The use case is that an agent holding live enterprise credentials, logging into government and insurance portals it does not control, taking actions on the other side of them.
All three conditions. By design. As the product.
In May 2026 the Five Eyes intelligence agencies published joint guidance naming prompt injection as a main route for manipulating these systems, and told organisations to assume their agents will sometimes behave in ways nobody planned for. Attackers who adjust their approach get past almost every published defence.
Screenshot-driven agents add a channel that text-only systems simply do not have. Brave’s security team demonstrated it against Perplexity Comet using white text on white backgrounds and instructions buried in HTML comments, which got the agent fetching one-time passwords out of email when the user had only asked it to summarise a page.
The same blind spot, with far worse consequences
The conditions under which a silent error goes undetected are precisely the conditions under which a successful injection goes undetected. Same gap in the process, wildly different bill at the end.
Two further questions sit alongside it.
The first is who pays when an agent files the wrong thing with a regulator? Is it the supplier, the buyer, or the lab that built the model?
Nobody has settled that, and it stalls deals in legal review long after the trial run went well.
The second is permission from the other side. The main use case here is automating someone else’s website, and plenty of them forbid automated access in their terms and run software to spot it. Some will push back.
together with Plaid
The agents that click through portals can also spoof the signals identity checks rely on.
Plaid’s white paper explains why point-in-time verification is breaking down, and what holds up better.
6. What the Hiring Data Shows
If agents at $7 an hour genuinely substituted for offshore labour at $10, the offshore numbers would be falling. They are not, and that disagreement is the most useful data in this entire debate.
What the employment numbers show
None of what follows appears in their piece.
Global contact-centre employment is forecast to grow from 15.3 million in 2025 to 16.8 million by 2029, even while automation suppresses an estimated 1.9 million individual roles across the same window.
The Philippines produced $40 billion in BPO export revenue with 1.9 million workers in 2025 and is tracking toward $42.3 billion and 1.96 million in 2026.
Sam Altman said in May 2026 that he was delighted to have been wrong about customer support roles disappearing, reversing his own position from ten months earlier. Against him, Vinod Khosla ahead of the India AI Impact Summit: “IT and BPO services will disappear, almost certainly within the next five years.”
One of those readings will look foolish in three years. I do not know which, and I would be suspicious of anybody who claims they do.
What the industry is repositioning around
The firms being valued upward are the ones building the checking layer. That includes supervision, quality control, and handling the cases that go wrong.
That surviving human function is the verification layer, sitting in plain sight in the labour data, which is their own argument turning up in the employment statistics.
Which brings the moat claim into focus. Context is the moat. That means the written procedures, the unwritten know-how, who to escalate to.
Against a model provider that defends beautifully. Against another startup it defends less well, and their own examples suggest why.
If the knowledge lives in one recorded video of somebody doing the job once, that video is cheap to make and any competitor can use it. Procedures and logins belong to the customer, not to whoever holds them this quarter.
What they are describing is a switching cost. It makes leaving annoying rather than impossible, and it gets weaker as setting up a new supplier gets cheaper, which is exactly what they forecast will happen.
The more durable version is the one their buyers gave them directly. Being the vendor an enterprise is permitted to run in production. Security review, audit trail, liability terms, procurement sign-off. Slow to earn, slow to lose, and impossible to show off on a stage.
So, can agents use a computer. Yes, and that question closed months ago without much ceremony at all.
The one that replaced it is whether you can tell when they got it wrong, cheaply enough and fast enough for it to matter. Where you can, the economics already beat every human alternative and the deployments are real and growing. Where you cannot, a smarter model changes absolutely nothing, because the model was never the thing that failed.
Doing the work stopped being the hard part somewhere around February.
Vouching for it never was easy.
"Correct and finished are different" is the right frame, and it cuts the other way too: an agent that can't tell when it's finished also can't tell when it shouldn't have started. For everyday users that argues for putting the check before irreversible steps (send, submit, pay), not only after the task, since a review after the fact can't unsend anything.
Automation benchmarks often ignore the friction of trust. Scaling human output through agents creates a paradox where efficiency gains are offset by the growing surface area for social engineering. As back office operations grow, the target moves from the internal network to the external relationship. When these agents handle sensitive communication, they become the primary vector for extortion. Attackers no longer need to break the firewall if they can manipulate the automated workflows that already have access to your client data. This shift changes the economics of extortion, turning every automated touchpoint into a potential ransom event for your customers.
https://cyrilsimonnet.substack.com/p/the-ransom-note-now-goes-to-your?utm_source=substor&utm_medium=substack&utm_campaign=comment